Home / Data Platform / Governance

Data PlatformHow-toIntermediate

CI/CD foundation for data platforms: environments, Git, tests and promotion

Separate dev, test and prod, put pipelines, notebooks, SQL models and infrastructure in Git, run automated tests on pull requests and promote releases with parameterised configs.

Deepak Bhardwaj2 min read

Applies toAzure Data Factory, Databricks Asset Bundles, dbt, Airflow, Fabric; GitHub Actions or Azure DevOps

1. What belongs in Git

Asset How
Infrastructure (storage, workspaces, networking) Terraform/Bicep
ADF / Synapse pipelines Git-integrated factory + ARM templates from the publish branch (or the npm build package)
Databricks jobs, notebooks, libraries Databricks Asset Bundles (databricks.yml)
SQL models and tests dbt project
Airflow DAGs Repo synced to the DAGs folder
Grants and catalogs Terraform or SQL migration scripts

Nothing is created by hand in test or prod.

2. Environments

  • dev: developers build and test with sample or masked data.
  • test/UAT: same configuration as prod, production-like volumes, automated tests.
  • prod: deployed only by pipelines using a service principal.

Configuration differs only by parameters (connection names, storage paths, cluster sizes).

3. Pull request checks

  • Lint (SQLFluff, ruff), unit tests for Python transformations (pytest with small DataFrames).
  • dbt build --select state:modified+ --defer --state prod-artifacts/ against a CI schema.
  • Bundle validation: databricks bundle validate -t test.
  • ADF: validate the factory (npm run build validate with the ADF utilities package).

4. Release pipeline

# GitHub Actions excerpt
jobs:
  deploy-test:
    environment: test
    steps:
      - uses: actions/checkout@v4
      - run: databricks bundle deploy -t test
      - run: dbt build --target test
  deploy-prod:
    needs: deploy-test
    environment: prod          # requires approval
    steps:
      - uses: actions/checkout@v4
      - run: databricks bundle deploy -t prod

Use OIDC federation to cloud identities, not stored secrets.

5. Data-aware deployments

  • Schema changes: backward compatible first (add columns), remove later.
  • Backfills as explicit, reviewed jobs, not side effects of a deploy.
  • Feature flags for new pipelines (deploy disabled, enable after checks).

6. Rollback

  • Code: redeploy the previous release tag.
  • Data: Delta/Iceberg time travel (RESTORE TABLE ... TO VERSION AS OF) for tables changed by a bad release.

Why this is also DR

If the whole platform is code plus data, rebuilding in another region is a pipeline run pointing at replicated storage. Teams with manual configuration discover during DR that nobody knows how prod was built.

Related fixes