Data Classification API and Terraform resource reach GA

The Data Classification API and its corresponding Databricks Terraform resource are now generally available (GA). This update enables teams to manage sensitive data scanning across Unity Catalog catalogs programmatically.

Use cases

Imagine you need to enforce data governance standards across dozens of Unity Catalog catalogs using CI/CD pipelines or Infrastructure as Code (IaC), avoiding manual UI configuration.

With this API, you can:

• Enable data classification on a catalog by creating its configuration resource.
• Scope scans by explicitly including or excluding specific schemas.
• Perform partial updates on existing configurations using update_mask via PATCH.
• Disable data classification by deleting the catalog's configuration via DELETE.

How to use

The API adheres to an "At Most One Resource" pattern per catalog via /api/data-classification/v1/{parent=catalogs/*}/config.

curl -X POST https://<databricks-instance>/api/data-classification/v1/catalogs/meu_catalogo/config \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "included_schemas": {
      "names": ["bronze", "silver"]
    }
  }'

What happens in this code:

1. Sends a POST request to create the configuration for meu_catalogo.
2. Specifies included_schemas to limit scans exclusively to bronze and silver.
3. Creates the resource and immediately activates data scanning for the selected schemas.

To update configuration rules without deleting the resource:

curl -X PATCH "https://<databricks-instance>/api/data-classification/v1/catalogs/meu_catalogo/config?update_mask=excluded_schemas" \
  -H "Authorization: Bearer <your-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "excluded_schemas": {
      "names": ["temp_staging"]
    }
  }'

What happens in this code:

1. Invokes the PATCH method on the catalog configuration path.
2. Supplies update_mask=excluded_schemas as a query parameter to indicate which attribute is being modified.
3. Replaces the scan scope to exclude temp_staging, scanning all other schemas in the catalog.

What it returns

The endpoints return the CatalogConfig object:

Field	Description
name	Immutable resource name formatted as catalogs/{catalog_name}/config.
included_schemas	Schemas included in scans. Mutually exclusive with excluded_schemas.
excluded_schemas	Schemas excluded from scans. Mutually exclusive with included_schemas.
auto_tag_configs	List of automated tagging rules (system tags or governance tags) and their status.

Limitations and requirements

• Maturity: The core configuration API and Terraform resource are GA. The auto-tagging configuration block (auto_tag_configs) remains in Public Preview.
• Permissions: API calls require the dataclassification scope.
• Scope constraints: included_schemas and excluded_schemas are mutually exclusive. If neither is specified, all schemas in the catalog will be scanned.
• Cardinality: At most one configuration resource can exist per catalog. Attempting to POST when a configuration already exists will fail.