Create managed tables and multiple append flows in Lakeflow pipelines

Use cases

Imagine you need to consolidate data from two different streaming sources into a single table within Lakeflow. With this feature, you can create managed tables and target multiple append flows to the same destination. It also allows you to:

• Consolidate multiple append flows into a single managed table.
• Migrate existing Structured Streaming workloads without reprocessing the source using checkpoints.

How to use

from pyspark import pipelines as dp

lp.create_table(
    name="my_managed_table",
    comment="Combined table from two sources"
)

@dp.append_flow(target="my_managed_table")
def stream_a():
    return spark.readStream.table("source_a")

@dp.append_flow(target="my_managed_table")
def stream_b():
    return spark.readStream.table("source_b")

What happens in this code:

1. The dp.create_table function defines and creates a new managed table within the pipeline.
2. The @dp.append_flow decorator points the stream_a function to write streaming data into the combined table.
3. The second @dp.append_flow decorator adds stream_b writing to the same target table.

Limitations and requirements

• This feature is in Beta.
• Managed tables do not support CDC change flows.
• Only append flows (append_flow) are supported; replace flows are not allowed.
• Requires pipelines configured with Unity Catalog.
• You cannot reuse the name of an existing streaming table for a new managed table.

Using it in a pipeline

Use this approach in your Silver or Gold layers to unify real-time ingestion streams coming from different bronze sources into a single consolidated table.