Resize stateful streaming partitions without losing checkpoints in Databricks

On-demand state repartitioning allows you to adjust the number of partitions in stateful streaming queries without losing the checkpoint history. Previously, changing this value required restarting the query from scratch with a new checkpoint directory.

Use cases

Imagine you have a streaming query aggregating IoT data that needs to scale up or down as data volume fluctuates.

• Tune streaming query performance without rebuilding the checkpoint.
• Scale resources to match workload changes.

How to use

Use the spark.sql.streaming.stateStore.partitions configuration to set the new partition count before restarting the query.

# Stop the running query
query.stop()

# Set the new partition count
spark.conf.set("spark.sql.streaming.stateStore.partitions", "100")

# Restart the query with the same options
query = (
    df.withWatermark("event_time", "10 minutes")
    .groupBy(window("event_time", "5 minutes"), "id")
    .count()
    .writeStream.format("delta")
    .option("checkpointLocation", "/checkpoint/path")
    .outputMode("append")
    .start()
)

What happens in this code:

1. The active query is stopped using the .stop() method.
2. The Spark configuration sets the new state partition count.
3. The query restarts using the same checkpoint path, automatically applying the repartitioning operation.

Limitations and requirements

• Requires Databricks Runtime 18 LTS or above.
• The query must use the RocksDB state store provider.
• Feature is generally available (GA).