Job triggers on updates for OpenSharing and system tables

Lakeflow Jobs table update triggers now support monitoring shared data through OpenSharing and system tables. This allows you to automate job runs as soon as new data arrives in external shares or platform system logs.

Use cases

Imagine you need to run a transformation job as soon as new billing data lands in system tables or when a data provider updates a shared table.

• Kick off pipelines when system tables receive fresh audit or billing data.
• Automate downstream processing based on updates to OpenSharing objects.
• Control execution flow using minimum time and wait-after-change parameters.

How to use

Configure triggers directly in the Databricks UI under job schedules or via the Jobs API.

1. In the left navigation of your workspace, click Jobs & Pipelines.
2. Select your job and go to Schedules & Triggers.
3. Click Add trigger and choose Table update.
4. Add your system tables or OpenSharing objects and save.

Limitations and requirements

• The principal configured in the job's Run as field must have SELECT permissions on every monitored object.
• A maximum of 1,000 table update triggers on OpenSharing objects or system tables can be created per workspace.
• Cross-regulatory-domain OpenSharing is not supported for table update triggers.

Using it in a pipeline

Use this feature to trigger Silver or Gold layer processing immediately when system data lands or shared updates arrive, removing the need for continuous clusters.