Managed Azure DevOps Ingestion with Lakeflow Connect

Databricks has added a managed Azure DevOps connector to Lakeflow Connect. This feature loads project, code, issue tracking, and CI/CD data directly into tables governed by Unity Catalog.

Use cases

Imagine you need to track engineering metrics (such as DORA metrics), team productivity, or release auditing. Instead of writing custom scripts against Azure DevOps REST APIs, the managed connector handles extraction and loading automatically.

It allows you to ingest:

• Projects, git repositories, and commit history (with incremental ingestion support).
• Work items, iterations, and area paths.
• Build definitions, test runs, and releases.
• Identities, teams, user groups, and service principals.

How to use

Setup is performed in three main steps using the UI, APIs, or Databricks Asset Bundles (DABs):

1. Configure Azure DevOps: Set up a Microsoft Entra service principal with the necessary access permissions across your Azure DevOps organization.
2. Create a Unity Catalog connection: In Catalog Explorer, create a new Azure DevOps connection storing the M2M (Machine-to-Machine) credentials for the service principal.
3. Create an ingestion pipeline: Build an ingestion pipeline using the connection, selecting target Unity Catalog schemas along with the tables and columns to sync.

The connector integrates with Databricks Workflows for orchestration and handles automated schema evolution for added or deleted columns.

Limitations and requirements

• Maturity: Beta. A workspace admin must enable Lakeflow Connect for Azure DevOps from the Previews page.
• Authentication: Supports OAuth M2M (Microsoft Entra service principal) only. User credentials (OAuth U2M) and basic authentication are not supported.
• Filters and schemas: Supports column selection/deselection via API, but row filtering via API is not supported. Data type changes and column renames require a full refresh.
• SCD Type 2: Not supported on tables containing VARIANT columns (such as commits, work_items, builds, pull_requests, and git_repositories).

Using it in a pipeline

Raw Azure DevOps data lands in bronze tables managed under Unity Catalog. Downstream silver and gold pipelines in SQL or Python can then process these records to generate dashboards tracking pull request cycle times, deployment frequency, or issue traceability.