External engines can now query Unity Catalog views via REST API

External compute engines can now query Unity Catalog views over Delta Lake tables using the Unity REST API, which is now generally available (GA). Accessing Iceberg-compatible views through the Iceberg REST catalog remains in Beta.

Use cases

Imagine your organization runs workloads on external Apache Spark clusters or third-party engines, but you want to maintain business logic and governance centralized in Unity Catalog views without exporting data or duplicating definitions.

With this feature, you can:

• Query Unity Catalog views directly from external engines.
• Run the view's SQL inside the external compute engine while reading the underlying Delta tables.
• Centralize data governance in Unity Catalog while executing compute across external platforms.

How to use

To query a view over Delta tables from an external Apache Spark session on AWS, configure these Spark session properties:

spark.jars.packages io.unitycatalog:unitycatalog-spark_4.0_2.13:0.6.0,io.delta:delta-spark_4.0_2.13:4.3.0,org.apache.hadoop:hadoop-aws:3.4.1
spark.sql.extensions io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog
spark.sql.catalog.meu_catalogo io.unitycatalog.spark.UCSingleCatalog
spark.sql.catalog.meu_catalogo.uri https://meu-workspace.cloud.databricks.com
spark.sql.catalog.meu_catalogo.token dapi_seu_token
spark.sql.defaultCatalog meu_catalogo
spark.hadoop.fs.s3.impl org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem

What happens in this code:

1. Loads the required packages for the Unity Catalog connector, Delta Lake, and AWS S3 Hadoop filesystem.
2. Defines a Spark catalog mapped to your Databricks workspace URL using an access token.
3. Sets the default catalog and configures the S3 file system provider.

Once configured, query the view using standard SQL in Spark:

SELECT * FROM meu_catalogo.meu_schema.minha_view;

What happens in this code:

1. The external engine fetches the view definition from the Unity REST API.
2. The external engine executes the query by reading the underlying Delta table files directly.

Limitations and requirements

• Maturity: GA for Delta views via Unity REST API; Beta for views accessed via Iceberg REST catalog.
• Metastore setting: External data access must be enabled on the Unity Catalog metastore.
• Privileges: The identity querying the view must have USE CATALOG and USE SCHEMA on parent hierarchies, SELECT on the view and every referenced table/view, and EXTERNAL USE SCHEMA on schemas containing referenced tables.
• Authentication: Requires a Databricks personal access token (PAT) or OAuth token.
• Minimum versions: Apache Spark 4.0, Unity Catalog Spark connector 0.6.0, and Delta Lake 4.3.0.
• Unsupported operations: Creating or altering views from external engines is not supported. The external engine must support Spark SQL dialect used in the view definition.