Saving an unchanged job no longer adds rows to system tables

Saving a job in the UI, via the Jobs API, or using Declarative Automation Bundles without changing any settings no longer generates new rows in the system.lakeflow.jobs or system.lakeflow.job_tasks tables. The change_time field remains at the timestamp of the last actual modification.

Use cases

Imagine you open an existing job just to inspect parameters in the UI and click save without changing anything. Previously, this generated a new row in the Lakeflow system tables, cluttering the change history (SCD2). Now, the history stays clean and reflects only actual modifications.

How to use

You can query the current state of jobs in the system tables by filtering for the latest modification:

-- Get the current state of every job, excluding deleted jobs
WITH latest AS (
  SELECT
    *,
    ROW_NUMBER() OVER(PARTITION BY workspace_id, job_id ORDER BY change_time DESC) AS rn
  FROM
    system.lakeflow.jobs
  QUALIFY rn = 1
)
SELECT * FROM latest WHERE delete_time IS NULL;

What happens in this code:

1. The query partitions by workspace_id and job_id to order rows by the latest change timestamp (change_time).
2. The ROW_NUMBER() function identifies the newest version of each job in the SCD2-backed table.
3. The final filter ensures only active jobs are returned by excluding those with a populated delete_time.

Limitations and requirements

• This applies to saves performed via the UI, Jobs API, and Asset Bundles.
• To access Lakeflow system tables, users must be metastore/account admins or have USE and SELECT permissions on the system schemas.