Write to Unity Catalog managed Delta tables from external Spark clients

Support for creating and writing to Unity Catalog managed Delta tables from external Apache Spark clients is now generally available (GA). Access is provided through the Unity REST API on tables configured with catalog commits.

Use cases

Imagine having your own Apache Spark processing infrastructure running outside of Databricks (such as on Kubernetes or EMR) and needing to load data directly into corporate managed lakehouse tables without demoting them to external tables.

With this capability, you can:

• Append and insert data into managed Delta tables directly from external Spark jobs.
• Create new managed tables in Unity Catalog from external systems.
• Keep governance and access control centralized in Unity Catalog while processing data across external compute.

How to use

To allow external writes, the managed Delta table must have catalog commits enabled (delta.feature.catalogManaged).

Create the managed table using SQL in Databricks Runtime:

CREATE TABLE meu_catalogo.meu_schema.minha_tabela (id INT, desc STRING)
USING delta
TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported');

What happens in this code:

1. A managed Delta table is created in the designated catalog and schema.
2. The delta.feature.catalogManaged table property is set to supported, delegating commit coordination to Unity Catalog.

In your external Apache Spark cluster, configure your session with the necessary packages and endpoints (using OAuth M2M authentication):

# SparkSession configuration for external Spark
spark_conf = {
    "spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension",
    "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog",
    "spark.sql.catalog.meu_catalogo": "io.unitycatalog.spark.UCSingleCatalog",
    "spark.sql.catalog.meu_catalogo.uri": "https://meu-workspace.cloud.databricks.com",
    "spark.sql.catalog.meu_catalogo.auth.type": "oauth",
    "spark.sql.catalog.meu_catalogo.auth.oauth.uri": "https://meu-workspace.cloud.databricks.com/oidc/v1/token",
    "spark.sql.catalog.meu_catalogo.auth.oauth.clientId": "<oauth-client-id>",
    "spark.sql.catalog.meu_catalogo.auth.oauth.clientSecret": "<oauth-client-secret>",
    "spark.sql.defaultCatalog": "meu_catalogo",
    "spark.hadoop.fs.s3.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem",
    "spark.jars.packages": "io.delta:delta-spark_4.1_2.13:4.3.0,io.unitycatalog:unitycatalog-spark_4.1_2.13:0.5.0,org.apache.hadoop:hadoop-aws:3.4.2"
}

What happens in this code:

1. Enables Delta Lake SQL extensions and points the catalog integration to Unity Catalog.
2. Sets the REST API endpoint to your Databricks workspace URL.
3. Configures OAuth machine-to-machine (M2M) authentication, automatically handling token renewal for long-running batch jobs.

Limitations and requirements

• Status: Generally available (GA) for Apache Spark. Other engines (DuckDB, Flink, Starburst, StreamNative) remain in Beta.
• Component requirements: Apache Spark 4.0 or 4.1, Delta Spark 4.3.0+, and Unity Catalog Spark client 0.5.0+.
• Databricks Runtime: DBR 16.4+ to read, write, or create tables with catalog commits; DBR 18.0+ to toggle catalog commits on existing tables.
• Permissions: External data access must be enabled on the metastore. The principal requires EXTERNAL USE SCHEMA on the schema, SELECT and MODIFY on the table (or CREATE on the schema for new tables).
• Restrictions: External clients cannot perform maintenance tasks (OPTIMIZE, VACUUM, ANALYZE), alter internal table properties (delta.* or databricks.*), rename columns, add constraints, or write to tables with IcebergCompatV3.
• Concurrency: Modifying the same Delta table on S3 from different writer engines simultaneously is discouraged to prevent data loss or corruption.

Using it in a pipeline

In hybrid architectures, an external Spark cluster can handle initial extraction and write directly to Bronze or Silver managed tables governed by Unity Catalog, leaving Databricks SQL or native jobs to produce final Gold tables.