Managed Cribl connector in Lakeflow Connect for metadata and log ingestion

Databricks released the managed Cribl connector in Lakeflow Connect in Beta. This feature enables teams to ingest Cribl.Cloud configurations, worker health telemetry, and Leader audit logs directly into the lakehouse.

Use cases

Imagine you use Cribl.Cloud to collect and route observability telemetry and need to audit access or monitor worker health alongside your analytical data. With this connector, you can:

• Ingest Worker Group configurations and organization metadata.
• Monitor operational status and health metrics across workers.
• Incrementally ingest Leader access and audit logs.

Limitations and requirements

• Stage: Beta. A workspace admin must enable Lakeflow Connect for Cribl from the Previews page.
• Authentication: Supports only OAuth machine-to-machine (M2M) using Client ID and Client Secret from Cribl.Cloud API credentials. OAuth U2M and basic authentication (username/password) are not supported.
• Pipeline authoring: Supported through the Databricks UI, API, and Declarative Automation Bundles (DABs), orchestrated by Databricks Workflows.
• Filtering and columns: Supports API-based column selection and deselection; API-based row filtering is not supported.
• SCD Type 2: Supported for standard tables, but not available on tables containing VARIANT columns (workers, pipelines, sources, destinations, parsers, breakers, collectors, packs, inputs_status, outputs_status, functions, api_credentials, audit_log, and access_log).
• Schema evolution: New and deleted columns are handled automatically. Data type changes and column renames are not supported and require a full refresh.
• Setup flow: Requires configuring Cribl.Cloud for authentication, creating a connection in Unity Catalog's Catalog Explorer, and setting up an ingestion pipeline tied to that connection.

Using it in a pipeline

Metadata and event logs land in the Bronze layer as raw operational data. In the Silver layer, you can combine Cribl audit logs with Unity Catalog system tables to audit end-to-end data pipeline operations and alert on failing workers.