Symptoms
- Many pipelines fail in the same hour with authentication errors:
AuthorizationFailure,AADSTS7000222,Login failed for user,Server failed to authenticate the request(expired SAS),invalid_client. - Test connection on a linked service fails.
1. Find the common linked service
ADFActivityRun
| where TimeGenerated > ago(6h) and Status == "Failed"
| extend ls = tostring(parse_json(Input).linkedServiceName)
| summarize failures = count(), sample = any(ErrorMessage) by PipelineName, ActivityName
| order by failures desc
Needs ADF diagnostic settings to Log Analytics (resource-specific). Without it, use the Monitor tab and filter failed runs. Usually one linked service (a storage account, a SQL login, a service principal) is behind all of them.
2. Identify the credential type
| Linked service auth | Expires because |
|---|---|
| Service principal secret | App registration secret end date |
| SQL authentication | Password policy, rotated password, disabled login |
| SAS URI / account key | SAS expiry, key rotated |
| Key Vault reference | Secret version disabled, vault access removed, vault firewall |
| SHIR-stored credential | Changed domain password for a Windows account |
az datafactory linked-service show -g rg-data --factory-name adf-prod -n ls_adls_raw --query "properties.typeProperties"
3. Rotate via Key Vault
If the linked service references Key Vault, update the secret — ADF reads the latest version at runtime (unless pinned to a version):
az keyvault secret set --vault-name kv-data --name sp-adf-secret --value "<new-secret>"
If the credential is stored inline in the linked service, change it to a Key Vault reference now; it's the same effort as editing it inline and fixes the next rotation.
4. Check Key Vault access itself
az keyvault show -n kv-data --query "{rbac:properties.enableRbacAuthorization, public:properties.publicNetworkAccess}"
az role assignment list --scope $(az keyvault show -n kv-data --query id -o tsv) --assignee <adf-managed-identity-objectId> -o table
ADF's managed identity needs Key Vault Secrets User (RBAC) or a get/list access policy. With a vault firewall, allow trusted services or use a managed private endpoint.
5. Re-run failed pipelines
Rerun from the failed activity (Monitor → pipeline run → Rerun from failed activity) to avoid repeating successful copies. For tumbling window triggers, rerun the failed windows so dependencies stay in order.
az datafactory pipeline create-run -g rg-data --factory-name adf-prod --name pl_daily_load \
--is-recovery true --reference-pipeline-run-id <failed-run-id> --start-from-failure true
Prevent it next time
- Prefer managed identity authentication for ADLS, Azure SQL, Synapse, Key Vault and Databricks — nothing expires.
- All remaining secrets in Key Vault, referenced (not inlined) by linked services.
- Alert 30 days before expiry of any secret ADF uses, and an alert on >N failed activity runs in 15 minutes.