Databricks production go-live readiness checklist for pipelines and workloads

Use this per workload (a pipeline, a data product or a dashboard set). Every "no" needs an owner and a date, or an accepted risk.

#Code and deployment

• Code in Git, reviewed through a pull request; no notebooks edited directly in prod.
• Deployed with Asset Bundles (or Terraform) to prod by CI using a service principal.
• Parameters (catalog, schema, paths) come from bundle variables, not hard-coded.
• Unit tests and an integration run on test data passed.
• No dbfs:/mnt, account keys, or personal paths in code.

#Compute

• Job compute (or serverless), not an all-purpose cluster.
• Cluster policy applied; LTS runtime pinned; node types right-sized from a test run.
• Spot with fallback for workers where interruption is acceptable.
• Photon decision made from a benchmark (cost vs runtime).

#Data quality

• Expectations or checks on critical columns (nulls, ranges, uniqueness, referential integrity).
• Row count and freshness checks; behaviour defined for bad data (fail, quarantine or warn).
• Schema evolution handled deliberately (mergeSchema only where intended).

#Security and access

• Job run_as = service principal with least privilege on the catalogs/schemas it needs.
• Output tables owned by a group; consumer access granted to groups.
• Sensitive columns classified (tags) and masked or filtered where required.
• Secrets from a secret scope / vault; none in code or job parameters.

#Reliability

• Task retries and timeouts set; idempotent writes (MERGE, replaceWhere) so reruns are safe.
• Streaming: checkpoint location on durable storage, restart tested.
• Dependencies (upstream tables, APIs) known; behaviour when they are late is defined.
• Backfill procedure documented and tested.
• Recovery: time travel retention adequate; critical tables covered by backups/DR design.

#Monitoring and alerting

• Failure notifications to the team channel / on-call.
• Duration threshold warnings for SLA-critical jobs.
• Freshness alert on the output tables (SQL alert).
• Job and tables appear on the ops dashboard.

#Performance and cost

• Run time and DBUs from a production-sized test run recorded as a baseline.
• Output tables use liquid clustering / predictive optimization.
• Tags (team, cost_center, product) on compute; expected monthly cost agreed with the owner.

#Documentation and handover

• Runbook: what the job does, schedule, dependencies, common failures and fixes, who to call.
• Data product documented in Unity Catalog (table and column comments, owner, tags).
• Support model agreed (who is on call, hours, escalation).
• Operations team walked through it; sign-off recorded.

#Go-live

• First production run monitored live by the build team.
• Hypercare period (for example 2 weeks) with daily checks, then handover.

Tip

Keep this checklist as a pull request template for production deployments. Reviewers tick the boxes, and the evidence lives with the code.