MySQL integrated CDC pipelines reach general availability in Databricks

The integrated CDC pipeline for MySQL is now generally available (GA). It consolidates binlog extraction and change application into a single pipeline update, removing the need for a dedicated ingestion gateway.

Use cases

Imagine you need to replicate MySQL relational tables into Databricks on scheduled runs without paying for always-on compute or managing intermediate ingestion infrastructure. With integrated CDC pipelines, you can:

• Run extraction and merge stages within a single pipeline execution.
• Replicate individual tables or entire schemas through a Unity Catalog connection.
• Perform automatic full refreshes, which is not supported in legacy gateway-based setups.

How to use

You can configure the pipeline using Databricks Asset Bundles, Python SDK, Databricks CLI, or REST API. Here is an example using a bundle YAML file:

resources:
  pipelines:
    meu_pipeline_cdc:
      name: "cdc-mysql-lakeflow"
      catalog: "meu_catalogo"
      schema: "bronze"
      ingestion_definition:
        connection_name: "conexao_mysql_uc"
        connector_type: CDC
        objects:
          - table:
              source_schema: "meu_banco"
              source_table: "clientes"
              destination_catalog: "meu_catalogo"
              destination_schema: "bronze"
              destination_table: "clientes"
              table_configuration:
                scd_type: "SCD_TYPE_1"

What happens in this code:

1. Defines a managed pipeline referencing a MySQL connection in Unity Catalog (conexao_mysql_uc).
2. Explicitly configures connector_type as CDC for the source table clientes.
3. Applies extracted changes to the destination streaming table in meu_catalogo.bronze using SCD_TYPE_1.

Limitations and requirements

• Status: Generally Available (GA).
• Compute: Requires classic compute (recommended minimum of 8 cores for extraction). Serverless compute is not supported. The compute plane must have network connectivity to MySQL (via VPC peering, public endpoints, Direct Connect, or VPN).
• Execution mode: Runs in triggered mode by default. Continuous execution is in Beta. An execution interval of 60 minutes or longer is recommended.
• Permissions: Requires USE CONNECTION (or CREATE CONNECTION) in Unity Catalog, USE CATALOG on destination catalog, and USE SCHEMA, CREATE TABLE, and CREATE VOLUME on the destination or staging schema.

Using it in a pipeline

This pipeline lands CDC changes into Bronze streaming tables inside Unity Catalog with exactly-once semantics, preparing clean records for downstream Silver and Gold medallion processing.