Reliable Databricks pipelines: retries, zones, instance pools and Delta Live Tables recovery

Databricks itself is a managed control plane; your pipelines' availability depends on how compute and jobs are configured.

#Compute

• Job clusters per job (isolation; a broken all-purpose cluster doesn't stop all jobs).
• On-demand driver; spot/low-priority workers with fallback to on-demand (first_on_demand: 1, availability: SPOT_WITH_FALLBACK on AWS, SPOT_WITH_FALLBACK_AZURE on Azure).
• AWS: zone_id: auto (auto-AZ) so clusters launch in an AZ with capacity.
• Instance pools with a few idle instances for critical jobs (faster start, less exposure to capacity shortages).
• Autoscaling with sensible min/max.

"aws_attributes": { "first_on_demand": 1, "availability": "SPOT_WITH_FALLBACK", "zone_id": "auto" }

#Jobs

• Task-level retries (max_retries: 2, min_retry_interval_millis: 60000, retry_on_timeout: true where safe).
• Timeouts per task so a hung task doesn't block the schedule.
• Task dependencies with repair run used after failures (rerun only failed tasks and their dependents).
• Notifications/webhooks on failure and on duration above threshold.
• Concurrency: max_concurrent_runs: 1 for jobs that must not overlap.

#Idempotent writes

• MERGE on keys, or replaceWhere partition overwrite, so retries don't duplicate data.
• Avoid "append then hope"; if appending, use a batch ID column and dedupe downstream.

(df.write.format("delta").mode("overwrite")
   .option("replaceWhere", "load_date = '2026-10-22'")
   .saveAsTable("prod.silver_sales.orders"))

#Streaming

• Checkpoints on durable storage, one per query, never shared or deleted casually.
• Structured Streaming jobs set to retry indefinitely (continuous jobs) with alerts on repeated restarts.
• DLT/Lakeflow pipelines in production mode (automatic retries, cluster restart on failure).

#Dependencies outside Databricks

• Source systems: retries with backoff on JDBC/API reads.
• Secrets in secret scopes / Unity Catalog connections, not in notebooks.

#Monitoring

• System tables system.lakeflow.job_run_timeline / job run history dashboards: failure rate, duration trend.
• Alert on SLA misses (data not fresh), not only job failures.

Tip

Most "Databricks outages" customers report are capacity errors (CLOUD_PROVIDER_LAUNCH_FAILURE, insufficient instances in AZ). Fallback to on-demand, auto-AZ, multiple instance types and pools solve most of them.