DataFrame caching arrives on Databricks serverless compute

Serverless compute on Databricks now supports DataFrame caching using df.cache() and df.persist(). When called, Databricks computes the query right away and materializes the result as a session snapshot, preventing repeated executions during subsequent operations.

Use cases

Imagine running an expensive join between orders and customers to produce multiple aggregations in the same notebook, or processing a streaming foreachBatch that outputs to multiple sinks. Without caching, Databricks re-executes the entire join for each action. With this feature, you can:

• Reuse the output of heavy joins, filters, or aggregations multiple times across the same session.
• Compute a streaming micro-batch once and dispatch it to multiple destination tables.
• Train and evaluate multiple machine learning models over identical preprocessed feature sets.
• Maintain an immutable, stable snapshot across multi-step exploratory workflows even if source tables receive updates.

How to use

Call .cache() directly on the DataFrame. Subsequence operations (filter, groupBy, select) derived from that reference will read straight from the cached snapshot.

from pyspark.sql import functions as F

# DataFrame resulting from an expensive join
orders_detailed = (
    spark.table("meu_catalogo.vendas.pedidos")
    .filter(F.col("order_date") == "2026-10-15")
    .join(spark.table("meu_catalogo.vendas.clientes"), "customer_id")
)

# Immediately materializes the snapshot for the session
orders_detailed.cache()

# Both queries read the snapshot rather than recomputing the join
total_revenue = orders_detailed.agg(F.sum("amount")).first()[0]

revenue_by_region = orders_detailed.groupBy("order_date", "region").agg(F.sum("amount").alias("revenue"))
revenue_by_region.write.mode("append").saveAsTable("meu_catalogo.relatorios.receita_regiao")

# Release the storage as soon as you are done
orders_detailed.unpersist()

What happens in this code:

1. The join between orders and customers is defined and materialized immediately upon calling .cache().
2. Both the scalar revenue computation and the regional breakdown query read directly from the cached snapshot without touching source tables.
3. Calling .unpersist() reclaims cloud storage; after this call, orders_detailed and any derived DataFrames are invalidated.

Limitations and requirements

• Maturity: Public Preview, enabled by default in most workspaces (workspace admins can toggle it via Previews settings). Unavailable in workspaces with compliance security profiles enabled.
• Environment: Serverless notebooks and jobs require environment version 5 or higher.
• Client versions: Requires Databricks Connect for Python 18.1.8+ or Databricks Connect for Scala 18.1.5+ (older clients throw CACHE_REQUIRES_NEWER_CLIENT_ON_SERVERLESS_GC).
• Derivation semantics: Only DataFrames explicitly derived from the cached variable reuse the snapshot. Writing an identical query in another DataFrame reads from the underlying tables.
• Lifecycle & scope: Cached snapshots last until session termination or at most 7 days, fully isolated to the active session. The persist() parameter accepts any storage level, but serverless enforces DISK_ONLY.
• Unpersist destruction: Executing .unpersist() releases the snapshot and renders that DataFrame and its derived children unusable (raising CACHE_DATAFRAME_ACCESS_AFTER_UNCACHE).