Run Lakeflow pipelines with a group as identity

You can now configure the run-as identity of a Lakeflow pipeline to use an account group instead of relying solely on an individual user or service principal. This simplifies collaborative management of automated workflows, ensuring that pipelines keep running smoothly even if team members leave the organization.

Use cases

Imagine you have a production data pipeline created by an analyst who left the company, and the pipeline stopped refreshing because their personal account was deactivated. With this feature, you can:

• Assign pipeline execution to a corporate account group.
• Keep automated pipelines under the responsibility of an entire team.
• Prevent failures caused by the deactivation of individual accounts.

How to use

You can configure the group identity in the workspace UI, via the REST API, or using Asset Bundles.

{
  "name": "sales-etl",
  "run_as": {
    "group_name": "Test Group"
  }
}

What happens in this code:

1. Sets the pipeline name to sales-etl.
2. Configures the run_as block to target the specified account group for execution.

resources:
  pipelines:
    sales_etl:
      name: sales-etl
      run_as:
        group_name: Test Group

What happens in this code:

1. Declares a pipeline resource named sales_etl.
2. Assigns Test Group to the run_as mapping property in the bundle configuration file.

Limitations and requirements

• The group must be an account group assigned to the workspace (workspace-local groups are not supported).
• Built-in broad groups like admins, users, and account users cannot be used.
• The user making the change must be a member of the group or have Assume permission on the group.
• The group must be granted all necessary permissions to access source code, catalogs, and Unity Catalog schemas used by the pipeline.