Deleting a Unity Catalog pipeline now retains tables by default

When you delete a Unity Catalog pipeline that uses the default publishing mode, the UI and the delete API keep associated tables, views, and materialized views intact. The retained tables remain queryable, but they cannot be updated until moved into an active pipeline. To remove both the pipeline and its tables together, you must configure the cascade parameter in the request.

Use cases

Imagine you need to remove an obsolete ingestion pipeline, but you want to ensure that processed data and downstream tables are not accidentally dropped.

• Prevents accidental data loss when deleting a pipeline.
• Allows querying orphaned tables until they are reallocated.
• Provides explicit API control for complete removal.

How to use

To delete the pipeline and retain the associated tables, use the HTTP DELETE method on the pipelines API.

curl -X DELETE "https://<databricks-instance>/api/2.0/pipelines/<pipeline_id>" \
  -H "Authorization: Bearer <token>"

What happens in this code:

1. The DELETE request is sent to the specific pipeline endpoint.
2. Databricks removes the pipeline.
3. Associated tables and views are retained in Unity Catalog.

To delete the pipeline and remove all associated tables in cascade, add the cascade parameter.

curl -X DELETE "https://<databricks-instance>/api/2.0/pipelines/<pipeline_id>?cascade=true" \
  -H "Authorization: Bearer <token>"

What happens in this code:

1. The DELETE request includes the cascade parameter.
2. Databricks removes both the pipeline and its associated resources.

Limitations and requirements

• Applies only to Unity Catalog pipelines using the default publishing mode.
• Does not affect Hive metastore pipelines.
• Retained tables cannot be updated until they are moved into an active pipeline.