Home / Data Platform / Data Factory
Azure Data Factory: pipelines, integration runtimes, triggers, CI/CD, cost and DR. 11 posts.
Copy activities sit in Queued for minutes on a self-hosted integration runtime and nightly loads miss their window. Read the IR metrics, find whether CPU, memory, concurrency or network is the limit, and fix it with tuning and scale-out.
Mapping data flows run on managed Spark, so failures look like Spark failures. Find the error code in the activity output, then fix the cluster size, joins/broadcast, partitioning or sink settings.
ADF bills for orchestration, data movement, pipeline activity time and data flow compute separately, and each has its own waste pattern. Use the consumption view to find expensive pipelines, cut activity counts in loops, right-size DIUs, set data flow TTL and core counts sensibly, and review managed VNet and self-hosted IR costs.
Find out why scheduled ADF pipelines stopped running with no failures to alert on. Check trigger runtime state after CI/CD deployments, restart triggers, backfill missed tumbling windows and add a 'no runs' alert.
Restore ADF and Synapse pipelines that fail together because a linked service secret, SAS token, service principal secret or SHIR credential expired. Find the shared credential, rotate it through Key Vault and move to managed identity.
Stop editing pipelines directly in the production factory. Connect the dev factory to Git, build ARM templates automatically with the npm package, deploy to test and prod with parameter overrides, and handle triggers and secrets correctly.
Use the copy activity's duration breakdown to find the bottleneck (queue, source read, sink write or the self-hosted IR), then apply the matching tuning setting.
Prepare a secondary Data Factory in another region from Git and CI/CD, cover linked services, secrets, integration runtimes and triggers, and run failover without duplicate pipeline runs.
Install and harden a self-hosted integration runtime for on-premises and private sources, configure networking, credentials, monitoring and updates, and understand its single-node limits.
Add nodes to a self-hosted integration runtime for HA and scale-out, configure node-to-node communication, plan patching and updates, and monitor node health.
Find failed runs across factories with KQL, read activity error codes, fix self-hosted integration runtime problems and rerun from the point of failure.
No posts in this category yet.