Case study · Data platform · DevOps
Building a Databricks platform from scratch
Rolling out and structuring a Databricks data platform, separating development (DEV) and production (PRD) environments and building the infrastructure, governance and CI/CD processes needed to support the lifecycle of data solutions.
- Databricks
- Azure
- Unity Catalog
- Terraform
- Azure DevOps
- Microsoft Entra ID
- CI/CD
Infrastructure architecture
The platform was built on Azure, with logical separation between development and production. The main infrastructure activities included:
- Creating and configuring the DEV and PRD workspaces.
- Provisioning storage resources for each environment.
- Standardizing naming and applying environment, owner and cost center tags.
- Organizing Microsoft Entra ID (Azure AD) access groups and assigning them to the workspaces.
A key architectural decision was to keep resources in the same Resource Group, using logical separation, naming conventions and access controls to tell DEV and PRD apart.
Data governance with Unity Catalog
The project also covered data governance, centralizing access control and organizing analytical assets by environment and layer. The architecture included:
- Separate catalogs and schemas per environment and layer, following the Bronze, Silver and Gold approach.
- Organizing data by business domain.
- Using Volumes to store files and staging data.
- Defining permissions for catalogs, schemas, tables and other objects.
- Splitting administrative responsibilities between the workspace and Unity Catalog.
CI/CD with Azure DevOps and Git
A core part of the project was building a development and deployment flow for Databricks integrated with Azure DevOps.
Azure DevOps Repos
Versioning, branches and change control.
Azure DevOps Pipelines
Automating validation and deployment.
Terraform
Infrastructure and Databricks resources.
Databricks
Jobs, queries and pipelines.
Activities included:
- Integrating the repositories with the Databricks workspaces.
- Structuring the CI/CD pipelines in Azure DevOps.
- Organizing variables and variable groups for global settings and the Terraform backend.
- Handling Git permissions, including branch creation permissions.
Infrastructure as code with Terraform
An Infrastructure as Code (IaC) approach was set up to automate creating and maintaining platform resources. The project used Terraform with the AzureRM and Databricks providers, covering:
- Configuring the remote backend that stores Terraform state.
- Separating variables and settings per environment.
- Automating management of Databricks resources and jobs.
- Defining tasks, job clusters, permissions, tags and dependencies.
- Generating Terraform files automatically from YAML configuration.
Main technical challenges
| Challenge | Work done |
|---|---|
| DEV/PRD separation | Organizing workspaces, storage, tags and access |
| Governance | Designing the Unity Catalog strategy and permissions |
| CI/CD | Configuring pipelines, variables and a self-hosted agent |
| Terraform | Remote backend, providers, versions and resource state |
| Job automation | Code generation from YAML and Jinja templates |
| Version control | Git integration and permission fixes |
| Safe deployment | Handling existing resources and preventing duplicates |