Parameters for Lakeflow pipelines are now generally available

Use cases

Imagine you need to run the exact same data transformation logic in both development and production environments, changing only the target catalog or source path. Pipeline parameters prevent you from having to manually edit source code for each environment change.

• Reuse the same SQL code across different catalogs, schemas, or tables.
• Define default values in pipeline settings or override them when starting an update.
• Reference parameters directly in SQL source code using named parameter syntax.

How to use

You can define parameters in JSON format within your pipeline settings and reference them in your SQL code.

{
  "name": "Sales pipeline",
  "parameters": {
    "source_catalog": "dev_catalog",
    "source_schema": "sales",
    "start_date": "2026-01-01"
  }
}

What happens in this code:

1. The JSON object defines the pipeline name and the parameters map.
2. The keys source_catalog, source_schema, and start_date receive default values.
3. These values can be injected into SQL queries during pipeline execution.

To use the parameter in SQL source code, prefix the name with a colon:

CREATE OR REFRESH MATERIALIZED VIEW transaction_summary AS
SELECT account_id,
  COUNT(txn_id) AS txn_count,
  SUM(txn_amount) AS account_revenue
FROM :source_catalog.sales.transactions
WHERE txn_date >= :start_date
GROUP BY account_id

What happens in this code:

1. Databricks binds the parameter value to the :source_catalog placeholder.
2. The date filter uses :start_date to limit the scope of processed data.
3. The materialized view is created or refreshed using the dynamic runtime values.

Limitations and requirements

• Generally Available (GA) feature.
• Named parameter syntax is supported in SQL source code only.
• Pipelines run a single update at a time; Databricks caps concurrency to 1 in specific job tasks or loops to prevent update overlaps.
• Date filters bounded on both start and end can invalidate incremental processing and trigger a full refresh.

Using it in a pipeline

Use parameters in Bronze or Silver layers to dynamically switch between test and production data sources without altering the underlying code base.