Home / Data Platform / Data Factory

Data PlatformRunbookIntermediate

Transformation: Azure Data Factory from 'publish in prod' to Git and automated CI/CD

Stop editing pipelines directly in the production factory. Connect the dev factory to Git, build ARM templates automatically with the npm package, deploy to test and prod with parameter overrides, and handle triggers and secrets correctly.

Deepak Bhardwaj2 min read

Applies toAzure Data Factory, Azure DevOps or GitHub Actions, @microsoft/azure-data-factory-utilities, Azure Key Vault

Target workflow

feature branch → PR to main (review) → build (validate + export ARM) → deploy test → approval → deploy prod

Only the dev factory is connected to Git. Test and prod factories are deployed by the pipeline; nobody edits them in the UI.

1. Connect dev to Git

ADF Studio → Manage → Git configuration: repository, collaboration branch main, root folder /adf. Import existing resources from the dev factory into the repo on first setup.

Turn on "Disable publish" in the Git settings once automated builds work, so the UI publish button can't bypass CI.

2. Parameterise environments

Everything that differs per environment must be a parameter or a Key Vault reference:

  • Linked service URLs, server names, Key Vault base URLs.
  • Integration runtime names.
  • Global parameters.

Use a custom arm-template-parameters-definition.json if you need extra properties parameterised.

3. Build: validate and export

package.json in the build folder:

{ "scripts": { "build": "node node_modules/@microsoft/azure-data-factory-utilities/lib/index" },
  "dependencies": { "@microsoft/azure-data-factory-utilities": "^1.0.0" } }

Azure DevOps build steps:

- script: npm install
  workingDirectory: build
- script: npm run build validate $(Build.Repository.LocalPath)/adf /subscriptions/$(subId)/resourceGroups/rg-data-dev/providers/Microsoft.DataFactory/factories/adf-dev
  workingDirectory: build
- script: npm run build export $(Build.Repository.LocalPath)/adf /subscriptions/$(subId)/resourceGroups/rg-data-dev/providers/Microsoft.DataFactory/factories/adf-dev "ArmTemplate"
  workingDirectory: build
- publish: build/ArmTemplate
  artifact: adf-arm

4. Deploy with trigger handling

  1. Run the Microsoft-provided PrePostDeploymentScript (pre-deployment) — it stops triggers.
  2. Deploy ARMTemplateForFactory.json with environment parameters:
az deployment group create -g rg-data-prod --template-file ARMTemplateForFactory.json \
  --parameters @ARMTemplateParametersForFactory.json factoryName=adf-prod \
  ls_adls_properties_typeProperties_url=https://stdataprod.dfs.core.windows.net/ \
  ls_keyvault_properties_typeProperties_baseUrl=https://kv-data-prod.vault.azure.net/
  1. Run the script again (post-deployment) — it deletes removed resources and restarts triggers.

Warning

If the post-deployment step fails, triggers stay stopped and pipelines silently stop running. Fail the release when that step fails, and verify trigger state at the end.

5. Secrets

No secrets in ARM parameters. Linked services reference Key Vault; each environment's factory managed identity has access to its own vault.

6. Hotfixes

Fix in a branch from main, PR, and release through the same pipeline — never edit prod. If an emergency edit happened in prod, copy it back to dev/Git immediately or the next deployment overwrites it.

Checklist

  • Dev factory in Git; publish disabled in UI
  • Environment values parameterised; secrets in Key Vault
  • Build validates and exports ARM on every PR/merge
  • Release with pre/post scripts and trigger verification
  • Test and prod UI edit rights removed

Related fixes