Configure Serverless Job Environments Using pyproject.toml

Databricks introduced support for standard pyproject.toml files to define execution environments for serverless job tasks. This feature standardizes dependency management across serverless notebook and Python script tasks.

Para que serve

Imagine you maintain a structured Python data project and want to run serverless tasks without having to scatter %pip install commands across notebooks or manually handle wheel installations. With this capability, you can:

• Standardize job dependencies using Python's modern packaging specification (pyproject.toml).
• Set up the task environment through the Jobs UI, Jobs API, or Databricks Asset Bundles.
• Isolate script and notebook dependencies across project-scoped serverless executions.

Como usar

Define your dependencies within your repository's configuration file:

[project]
name = "meu_projeto_dados"
version = "0.1.0"
dependencies = [
    "pandas>=2.0.0",
    "polars>=1.0.0",
    "pydantic>=2.0.0"
]

O que acontece nesse código:

1. The file declares metadata and required package versions for your project.
2. The serverless compute layer parses the file when starting the task, preparing dependencies before script execution.

You can then select or point to this project environment from the Jobs UI, pass it through the Jobs API, or reference it in an Asset Bundle.

Limitações e requisitos

• Currently in Beta.
• Requires workspace administrators to turn on "Project-Scoped Shared Compute" from the workspace Previews page.
• Limited to notebook and Python script tasks running on serverless compute.

Como aplicar num pipeline

Within a Databricks Asset Bundle pipeline, you keep a single pyproject.toml file in source control. Both bronze ingest scripts and silver transformation notebooks reference this definition, ensuring consistent package versions throughout your entire execution workflow.