Session restore for serverless jobs in Databricks

Use cases

Imagine you ran a long serverless job that failed after forty minutes or you need to inspect results without rerunning the entire workflow. With this feature, you can open a snapshot of the previous job state directly inside a new interactive notebook.

• Debug job failures without full re-execution.
• Explore Python variables and Spark session state from past runs.
• Create isolated notebook copies based on run states.

How to use

To restore state from a job run:

1. View the job run details and open the notebook view of the run.
2. Click Clone into new notebook or Debug in new notebook on a failed run.
3. Wait for Databricks to create a new notebook in your home folder with the restored Python variables and Spark session.

What happens in this process:

• Databricks creates a new notebook named Job Run of <notebook name> at <timestamp> (Run <run ID>) in your home folder.
• The environment rebuilds variables and the session from the automatically captured snapshot.
• You edit the restored copy without modifying the original notebook or job.

Limitations and requirements

• This feature is in Public Preview.
• Only the run-as user can restore a run's state, and only for notebook tasks.
• Applies to runs using serverless compute with environment version 5 or higher.
• State is captured automatically when a run fails, runs 30 minutes or longer (execution time excluding compute wait time), is cancelled, or times out.
• Snapshots are retained for 7 days.
• The restored notebook must stay attached to serverless compute.