Base environments now supported on classic compute in Databricks

Base environments on classic compute allow you to manage Python dependencies at the workload level instead of installing libraries directly on the cluster. This isolates settings, prevents breakages when the cluster changes, and speeds up startup because Databricks caches the environments.

Use cases

Imagine you need to run notebooks with different libraries without affecting the entire cluster or dealing with network package resolution issues during startup.

• Isolate dependencies per notebook or job task.
• Use standard, ML, or custom environments.
• Avoid reinstalling packages every time a session resumes.

How to use

To create compute with the appropriate dependency mode, configure the advanced settings when provisioning the resource.

// Typical compute configuration via UI:
Runtime: Databricks Runtime 19 (Beta) or above
Access mode: Standard access mode
Dependency mode: Manual (Environments) or Auto

What happens in this configuration:

1. Selects Databricks Runtime 19 or above.
2. Sets the dependency mode to Environments.
3. Traditional cluster-level libraries are disabled in favor of base environments.

Limitations and requirements

• Feature is in Beta.
• Requires Databricks Runtime 19 or above.
• Compatible only with Standard access mode on All-purpose compute or classic jobs compute.
• Enabling Environments dependency mode disables Docker, Spark environment variables, init scripts, and cluster libraries.
• Scala code (%scala) and Spark JAR tasks are not supported.

Using it in a pipeline

Use base environments in data engineering and machine learning tasks within notebooks or jobs to ensure specific Python library packages load reliably and in an isolated manner.