Set maintenance windows for continuous Lakeflow pipelines

Continuous pipelines run without stopping, requiring restarts to receive runtime or serverless infrastructure updates. With maintenance windows, you define the day of the week, start hour, and time zone so these pauses occur only at planned times.

Use cases

Imagine you manage a streaming pipeline that cannot experience surprise downtime during peak business hours. This configuration ensures restarts for security patches or new versions happen only overnight.

• Pick the exact day of the week and hour for maintenance.
• Prevent unexpected performance drops caused by infrastructure updates.
• Set up notifications for maintenance start and completion.

How to use

You can configure the maintenance window through the Databricks UI or via the Jobs REST API when managing a continuous job.

{
  "continuous": {
    "pause_status": "UNPAUSED",
    "maintenance_window": {
      "start_hour": 2,
      "day_of_week": "TUESDAY",
      "timezone_id": "America/New_York"
    }
  }
}

What happens in this code:

1. The continuous parameter specifies that the job runs in continuous mode.
2. The maintenance_window block groups the update rules.
3. The start_hour field sets the starting hour in 24-hour format.
4. The day_of_week and timezone_id fields specify the evaluated day and time zone.

Limitations and requirements

• Maintenance windows apply only to continuous jobs and pipelines and are not available for scheduled triggers.
• All three fields (start_hour, day_of_week, and timezone_id) are required when setting a window.
• Serverless clusters have a maximum lifetime; if that limit is reached before your next window, Databricks restarts the pipeline to enforce infrastructure recycling.