// Metadata created by nebel
//

[id="mysql-connector-configuration-properties_{context}"]
= MySQL connector configuration properties

The configuration properties listed here are *required* to run the {prodname} MySQL connector. There are also <<advanced-mysql-connector-properties, advanced MySQL connector properties>> whose default value rarely need changed and therefore, specified in the connector configuration.

TIP: The {prodname} MySQL connector supports _pass-through_ configuration when creating the Kafka producer and consumer. See link:http://kafka.apache.org/documentation.html[the Kafka documentation] for more details on _pass-through_ properties.

[cols="3,1,6"]
|===
|Property |Default |Description

|`name`
|
|Unique name for the connector. Attempting to register again with the same name will fail. (This property is required by all Kafka Connect connectors.)

|`connector.class`
|
|The name of the Java class for the connector. Always use a value of `io.debezium{zwsp}.connector.mysql.MySqlConnector` for the MySQL connector.

|`tasks.max`
|`1`
|The maximum number of tasks that should be created for this connector. The MySQL connector always uses a single task and therefore does not use this value, so the default is always acceptable.

|`database.hostname`
|
|IP address or hostname of the MySQL database server.

|`database.port`
|`3306`
|Integer port number of the MySQL database server.

|`database.user`
|
|Name of the MySQL database to use when connecting to the MySQL database server.

|`database.password`
|
|Password to use when connecting to the MySQL database server.

|`database.server.name`
|
|Logical name that identifies and provides a namespace for the particular MySQL database server/cluster being monitored. The logical name should be unique across all other connectors, since it is used as a prefix for all Kafka topic names emanating from this connector.
Only alphanumeric characters and underscores should be used.

|`database.server.id`
|_random_
|A numeric ID of this database client, which must be unique across all currently-running database processes in the MySQL cluster. This connector joins the MySQL database cluster as another server (with this unique ID) so it can read the binlog. By default, a random number is generated between 5400 and 6400, though we recommend setting an explicit value.

|`database.history.kafka.topic`
|
|The full name of the Kafka topic where the connector will store the database schema history.

|`database.history{zwsp}.kafka.bootstrap.servers`
|
|A list of host/port pairs that the connector will use for establishing an initial connection to the Kafka cluster. This connection will be used for retrieving database schema history previously stored by the connector, and for writing each DDL statement read from the source database. This should point to the same Kafka cluster used by the Kafka Connect process.

|`database.whitelist`
|_empty string_
|An optional comma-separated list of regular expressions that match database names to be monitored; any database name not included in the whitelist will be excluded from monitoring. By default all databases will be monitored. May not be used with `database.blacklist`.

|`database.blacklist`
|_empty string_
|An optional comma-separated list of regular expressions that match database names to be excluded from monitoring; any database name not included in the blacklist will be monitored. May not be used with `database.whitelist`.

|`table.whitelist`
|_empty string_
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be monitored; any table not included in the whitelist will be excluded from monitoring. Each identifier is of the form _databaseName_._tableName_. By default the connector will monitor every non-system table in each monitored database. May not be used with `table.blacklist`.

|`table.blacklist`
|_empty string_
|An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables to be excluded from monitoring; any table not included in the blacklist will be monitored. Each identifier is of the form _databaseName_._tableName_. May not be used with `table.whitelist`.

|`column.blacklist`
|_empty string_
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event message values. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.

|`column.truncate.to._length_.chars`
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be truncated in the change event message values if the field values are longer than the specified number of characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.

|`column.mask.with._length_.chars`
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of character-based columns whose values should be replaced in the change event message values with a field value consisting of the specified number of asterisk (`*`) characters. Multiple properties with different lengths can be used in a single configuration, although in each the length must be a positive integer or zero. Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.

|`column.propagate.source.type`
|_n/a_
|An optional comma-separated list of regular expressions that match the fully-qualified names of columns whose original type and length should be added as a parameter to the corresponding field schemas in the emitted change messages.
The schema parameters `pass:[_]pass:[_]{prodname}.source.column.type`, `pass:[_]pass:[_]{prodname}.source.column.length` and `pass:[_]{prodname}.source.column.scale` will be used to propagate the original type name and length (for variable-width types), respectively.
Useful to properly size corresponding columns in sink databases.
Fully-qualified names for columns are of the form _databaseName_._tableName_._columnName_, or _databaseName_._schemaName_._tableName_._columnName_.

|`time.precision.mode`
|`adaptive_time{zwsp}_microseconds`
| Time, date, and timestamps can be represented with different kinds of precision, including: `adaptive_time_microseconds` (the default) captures the date, datetime and timestamp values exactly as in the database using either millisecond, microsecond, or nanosecond precision values based on the database column's type, with the exception of TIME type fields, which are always captured as microseconds;
ifndef::cdc-product[]
// Do not include deprecated content in downstream doc
`adaptive` (deprecated) captures the time and timestamp values exactly as in the database using either millisecond, microsecond, or nanosecond precision values based on the database column's type;
endif::cdc-product[]
or `connect` always represents time and timestamp values using Kafka Connect's built-in representations for Time, Date, and Timestamp, which uses millisecond precision regardless of the database columns' precision.

|`decimal.handling.mode`
|`precise`
| Specifies how the connector should handle values for `DECIMAL` and `NUMERIC` columns: `precise` (the default) represents them precisely using `java.math.BigDecimal` values represented in change events in a binary form; or `double` represents them using `double` values, which may result in a loss of precision but will be far easier to use. `string` option encodes values as formatted string which is easy to consume but a semantic information about the real type is lost.

|`bigint.unsigned.handling.mode`
|`long`
| Specifies how BIGINT UNSIGNED columns should be represented in change events, including: `precise` uses `java.math.BigDecimal` to represent values, which are encoded in the change events using a binary representation and Kafka Connect's `org.apache.kafka.connect.data.Decimal` type; `long` (the default) represents values using Java's `long`, which may not offer the precision but will be far easier to use in consumers. `long` is usually the preferable setting. Only when working with values larger than 2^63, the `precise` setting should be used as those values can't be conveyed using `long`.

|`include.schema.changes`
|`true`
|Boolean value that specifies whether the connector should publish changes in the database schema to a Kafka topic with the same name as the database server ID. Each schema change will be recorded using a key that contains the database name and whose value includes the DDL statement(s). This is independent of how the connector internally records database history. The default is `true`.

|`include.query`
|`false`
|Boolean value that specifies whether the connector should include the original SQL query that generated the change event. +
Note: This option requires MySQL be configured with the binlog_rows_query_log_events option set to ON. Query will not be present for events generated from the snapshot process. +
WARNING: Enabling this option may expose tables or fields explicitly blacklisted or masked by including the original SQL statement in the change event. For this reason this option is defaulted to 'false'.

|`event.processing{zwsp}.failure.handling.mode`
|`fail`
| Specifies how the connector should react to exceptions during deserialization of binlog events.
`fail` will propagate the exception (indicating the problematic event and its binlog offset), causing the connector to stop. +
`warn` will cause the problematic event to be skipped and the problematic event and its binlog offset to be logged. +
`skip` will cause problematic event will be skipped.

|`inconsistent.schema.handling.mode`
|`fail`
| Specifies how the connector should react to binlog events that relate to tables that are not present in internal schema representation (i.e. internal representation is not consistent with database)
`fail` will throw an exception (indicating the problematic event and its binlog offset), causing the connector to stop. +
`warn` will cause the problematic event to be skipped and the problematic event and its binlog offset to be logged. +
`skip` will cause the problematic event to be skipped.

|`max.queue.size`
|`8192`
|Positive integer value that specifies the maximum size of the blocking queue into which change events read from the database log are placed before they are written to Kafka. This queue can provide backpressure to the binlog reader when, for example, writes to Kafka are slower or if Kafka is not available. Events that appear in the queue are not included in the offsets periodically recorded by this connector. Defaults to 8192, and should always be larger than the maximum batch size specified in the `max.batch.size` property.

|`max.batch.size`
|`2048`
|Positive integer value that specifies the maximum size of each batch of events that should be processed during each iteration of this connector. Defaults to 2048.

|`poll.interval.ms`
|`1000`
|Positive integer value that specifies the number of milliseconds the connector should wait during each iteration for new change events to appear. Defaults to 1000 milliseconds, or 1 second.

|`connect.timeout.ms`
|`30000`
|A positive integer value that specifies the maximum time in milliseconds this connector should wait after trying to connect to the MySQL database server before timing out. Defaults to 30 seconds.

|`gtid.source.includes`
|
|A comma-separated list of regular expressions that match source UUIDs in the GTID set used to find the binlog position in the MySQL server. Only the GTID ranges that have sources matching one of these include patterns will be used. May not be used with `gtid.source.excludes`.

|`gtid.source.excludes`
|
|A comma-separated list of regular expressions that match source UUIDs in the GTID set used to find the binlog position in the MySQL server. Only the GTID ranges that have sources matching none of these exclude patterns will be used. May not be used with `gtid.source.includes`.

ifndef::cdc-product[]
// Do not include deprecated content in downstream doc
|`gtid.new.channel.position` +
_deprecated and scheduled for removal_
|`earliest`
| When set to `latest`, when the connector sees a new GTID channel, it will start consuming from the last executed transaction in that GTID channel. If set to `earliest` (default), the connector starts reading that channel from the first available (not purged) GTID position. `earliest` is useful when you have a active-passive MySQL setup where {prodname} is connected to master, in this case during failover the slave with new UUID (and GTID channel) starts receiving writes before {prodname} is connected. These writes would be lost when using `latest`.
endif::cdc-product[]

|`tombstones.on.delete`
|`true`
| Controls whether a tombstone event should be generated after a delete event. +
When `true` the delete operations are represented by a delete event and a subsequent tombstone event. When `false` only a delete event is sent. +
Emitting the tombstone event (the default behavior) allows Kafka to completely delete all events pertaining to the given key once the source record got deleted.

|`message.key.columns`
|_empty string_
| A semi-colon list of regular expressions that match fully-qualified tables and columns to map a primary key. +
Each item (regular expression) must match the `<fully-qualified table>:<a comma-separated list of columns>` representing the custom key. +
Fully-qualified tables could be defined as `DB_NAME.TABLE_NAME` or `SCHEMA_NAME.TABLE_NAME`, depending on the specific connector.

|===

== Advanced MySQL connector properties
[[advanced-mysql-connector-properties]]

[cols="3,1,6"]
|===
|Property |Default |Description

|`connect.keep.alive`
|`true`
|A boolean value that specifies whether a separate thread should be used to ensure the connection to the MySQL server/cluster is kept alive.

|`table.ignore.builtin`
|`true`
|Boolean value that specifies whether built-in system tables should be ignored. This applies regardless of the table whitelist or blacklists. By default system tables are excluded from monitoring, and no events are generated when changes are made to any of the system tables.

|`database.history.kafka.recovery.poll.interval.ms`
|`100`
|An integer value that specifies the maximum number of milliseconds the connector should wait during startup/recovery while polling for persisted data. The default is 100ms.

|`database.history.kafka.recovery.attempts`
|`4`
|The maximum number of times that the connector should attempt to read persisted history data before the connector recovery fails with an error. The maximum amount of time to wait after receiving no data is `recovery.attempts` x `recovery.poll.interval.ms`.

|`database.history.skip.unparseable.ddl`
|`false`
|Boolean value that specifies if connector should ignore malformed or unknown database statements or stop processing and let operator to fix the issue.
The safe default is `false`.
Skipping should be used only with care as it can lead to data loss or mangling when binlog is processed.

|`database.history.store.only.monitored.tables.ddl`
|`false`
|Boolean value that specifies if connector should should record all DDL statements or (when `true`) only those that are relevant to tables that are monitored by {prodname} (via filter configuration).
The safe default is `false`.
This feature should be used only with care as the missing data might be necessary when the filters are changed.

|`database.ssl.mode`
|`disabled`
|Specifies whether to use an encrypted connection.  The default is `disabled`, and specifies to use an unencrypted connection.

The `preferred` option establishes an encrypted connection if the server supports secure connections but falls back to an unencrypted connection otherwise.

The `required` option establishes an encrypted connection but will fail if one cannot be made for any reason.

The `verify_ca` option behaves like `required` but additionally it verifies the server TLS certificate against the configured Certificate Authority (CA) certificates and will fail if it doesn't match any valid CA certificates.

The `verify_identity` option behaves like `verify_ca` but additionally verifies that the server certificate matches the host of the remote connection.

|`binlog.buffer.size`
|0
|The size of a look-ahead buffer used by the binlog reader. +
Under specific conditions it is possible that MySQL binlog contains uncommitted data finished by a `ROLLBACK` statement.
Typical examples are using savepoints or mixing temporary and regular table changes in a single transaction. +
When a beginning of a transaction is detected then {prodname} tries to roll forward the binlog position and find either `COMMIT` or `ROLLBACK` so it can decide whether the changes from  the transaction will be streamed or not.
The size of the buffer defines the maximum number of changes in the transaction that {prodname} can buffer while searching for transaction boundaries.
If the size of transaction is larger than the buffer then {prodname} needs to rewind and re-read the events that has not fit into the buffer while streaming. Value `0` disables buffering. +
Disabled by default. +
_Note:_ This feature should be considered an incubating one. We need a feedback from customers but it is expected that it is not completely polished.

|`snapshot.mode`
|`initial`
|Specifies the criteria for running a snapshot upon startup of the connector. The default is `initial`, and specifies the connector can run a snapshot only when no offsets have been recorded for the logical server name. The `when_needed` option specifies that the connector run a snapshot upon startup whenever it deems it necessary (when no offsets are available, or when a previously recorded offset specifies a binlog location or GTID that is not available in the server). The `never` option specifies that the connect should never use snapshots and that upon first startup with a logical server name the connector should read from the beginning of the binlog; this should be used with care, as it is only valid when the binlog is guaranteed to contain the entire history of the database. If you don't need the topics to contain a consistent snapshot of the data but only need them to have the changes since the connector was started, you can use the `schema_only` option, where the connector only snapshots the schemas (not the data).

`schema_only_recovery` is a recovery option for an existing connector to recover a corrupted or lost database history topic, or to periodically "clean up" a database history topic (which requires infinite retention) that may be growing unexpectedly.

|`snapshot.locking.mode`
|`minimal`
|Controls if and how long the connector holds onto the global MySQL read lock (preventing any updates to the database) while it is performing a snapshot.  There are three possible values `minimal`, `extended`, and `none`. +

`minimal` The connector holds the global read lock for just the initial portion of the snapshot while the connector reads the database schemas and other metadata. The remaining work in a snapshot involves selecting all rows from each table, and this can be done in a consistent fashion using the REPEATABLE READ transaction even when the global read lock is no longer held and while other MySQL clients are updating the database. +

`extended` In some cases where clients are submitting operations that MySQL excludes from REPEATABLE READ semantics, it may be desirable to block all writes for the entire duration of the snapshot. For these such cases, use this option. +

`none` Will prevent the connector from acquiring any table locks during the snapshot process. This value can be used with all snapshot modes but it is safe to use if and _only_ if no schema changes are happening while the snapshot is taken. Note that for tables defined with MyISAM engine, the tables would still be locked despite this property being set as MyISAM acquires a table lock. This behaviour is unlike InnoDB engine which acquires row level locks.

|`snapshot.select.statement.overrides`
|
|Controls which rows from tables will be included in snapshot. +
This property contains a comma-separated list of fully-qualified tables _(DB_NAME.TABLE_NAME)_. Select statements for the individual tables are specified in further configuration properties, one for each table, identified by the id `snapshot.select.statement.overrides.[DB_NAME].[TABLE_NAME]`. The value of those properties is the SELECT statement to use when retrieving data from the specific table during snapshotting. _A possible use case for large append-only tables is setting a specific point where to start (resume) snapshotting, in case a previous snapshotting was interrupted._ +
*Note*: This setting has impact on snapshots only. Events captured from binlog are not affected by it at all.

|`min.row.count.to.stream.results`
|`1000`
|During a snapshot operation, the connector will query each included table to produce a read event for all rows in that table. This parameter determines whether the MySQL connection will pull all results for a table into memory (which is fast but requires large amounts of memory), or whether the results will instead be streamed (can be slower, but will work for very large tables). The value specifies the minimum number of rows a table must contain before the connector will stream results, and defaults to 1,000. Set this parameter to '0' to skip all table size checks and always stream all results during a snapshot.

|`heartbeat.interval.ms`
|`0`
|Controls how frequently the heartbeat messages are sent. +
This property contains an interval in milli-seconds that defines how frequently the connector sends heartbeat messages into a heartbeat topic.
Set this parameter to `0` to not send heartbeat messages at all. +
Disabled by default.

|`heartbeat.topics.prefix`
|`__debezium-heartbeat`
|Controls the naming of the topic to which heartbeat messages are sent. +
The topic is named according to the pattern `<heartbeat.topics.prefix>.<server.name>`.

|`database.initial.statements`
|
|A semicolon separated list of SQL statements to be executed when a JDBC connection (not the transaction log reading connection) to the database is established.
Use doubled semicolon (';;') to use a semicolon as a character and not as a delimiter. +
_Note: The connector may establish JDBC connections at its own discretion, so this should typically be used for configuration of session parameters only, but not for executing DML statements._

|`snapshot.delay.ms`
|
|An interval in milli-seconds that the connector should wait before taking a snapshot after starting up; +
Can be used to avoid snapshot interruptions when starting multiple connectors in a cluster, which may cause re-balancing of connectors.

|`snapshot.fetch.size`
|
|Specifies the maximum number of rows that should be read in one go from each table while taking a snapshot.
The connector will read the table contents in multiple batches of this size.

|`enable.time.adjuster`
|
|MySQL allows user to insert year value as either 2-digit or 4-digit.
In case of two digits the value is automatically mapped to 1970 - 2069 range.
This is usually done by database. +
Set to `true` (the default) when {prodname} should do the conversion. +
Set to `false` when conversion is fully delegated to the database.

|`source.struct.version`
|v2
|Schema version for the `source` block in {prodname} events; {prodname} 0.10 introduced a few breaking +
changes to the structure of the `source` block in order to unify the exposed structure across
all the connectors. +
By setting this option to `v1` the structure used in earlier versions can be produced.
Note that this setting is not recommended and is planned for removal in a future {prodname} version.

|`sanitize.field.names`
|`true` when connector configuration explicitly specifies the `key.converter` or `value.converter` parameters to use Avro, otherwise defaults to `false`.
|Whether field names will be sanitized to adhere to Avro naming requirements.

|===
