Case study · Data engineering · Healthcare
FHIR API ingestion with Databricks
A data engineering pipeline integrating a healthcare API based on the HL7 FHIR standard into Databricks, using the Google Cloud Healthcare API as the data source and OAuth 2.0 for authentication.
The goal was to automate clinical data ingestion, turn FHIR resources into tabular structures and prepare them for queries, analytics and consumption by other layers of the data platform.
- Databricks
- PySpark
- HL7 FHIR
- OAuth 2.0
- Delta Lake
- Unity Catalog
- Google Cloud
Architecture and data flow
OAuth 2.0 authentication
Token issued by Amil's authentication API, with credentials stored in Secrets.
Google Cloud Healthcare API
Querying FHIR resources, paging and retrieving data with a Bearer Token.
Bronze — raw data
Persisting the original JSON, preserving the structure received from the API.
Silver — FHIR transformation
Exploding resources, handling nested structures and creating one Delta table per entity.
Structured data for consumption
Data organized for SQL queries, integration and downstream analytics.
Main technologies used
Databricks e PySpark
Processing, transforming and structuring the data.
Google Cloud Healthcare API
Source of the clinical data in the FHIR standard.
OAuth 2.0 Client Credentials
Authentication and access token retrieval through Amil's API.
Python e REST API
Running HTTP requests, managing headers and handling responses.
Apache Spark e Delta Lake
Reading JSON, building DataFrames and persisting to Delta tables.
Unity Catalog e Volumes
Organizing and storing the data on the Databricks platform.
Main FHIR resources handled
The pipeline had to handle different resource types, each with its own structure and attributes.
| Resource | Purpose |
|---|---|
| Patient | Patient data |
| Consent | Consent records |
| Observation | Clinical observations and results |
| DiagnosticReport | Diagnostic test reports |
| Encounter | Encounters |
| Coverage | Coverage information |
| Practitioner | Healthcare practitioners |
| Specimen | Biological specimens |
One of the main challenges was turning a heterogeneous Bundle response, which groups multiple FHIR resources, into tabular structures suited to Spark.
Technical challenges
Authentication and integration
Obtaining and correctly using the Bearer Token in API calls.
Pagination
Processing every page returned by the API through the FHIR Bundle's next links.
Incremental ingestion
Using the _lastUpdated parameter to fetch records updated since a watermark.
Complex JSON
Handling nested structures, optional objects and arrays without assuming every resource shares the same schema.
Data modeling
Splitting resources into dedicated tables and extracting the relevant attributes.
Storage and performance
Organizing data into Delta files and tables, evaluating strategies such as Liquid Clustering, statistics and query optimization.