What ADF bills for
| Meter | Driven by |
|---|---|
| Orchestration | Number of activity runs (per 1,000), trigger runs, debug runs |
| Data movement | Copy activity DIU-hours (Azure IR) |
| Pipeline activities | Execution hours of activities like Lookup, Web, Stored Procedure on Azure IR; extra for managed VNet |
| External activities | Calls to Databricks, HDInsight, etc. (the external compute bills separately) |
| Data flows | vCore-hours of the Spark cluster (including TTL idle time) |
| Self-hosted IR | Activity run hours (lower rate) + your VMs |
Check current pricing; the shape matters more than exact rates.
1. Find the expensive pipelines
- Monitor → Pipeline runs → Consumption per run shows activity runs, DIU-hours and data flow vCore-hours.
- Cost Management by resource (factory) and meter.
- Log Analytics (
ADFActivityRun) for counts:
ADFActivityRun
| where TimeGenerated > ago(30d)
| summarize runs = count(), hours = sum(todouble(DurationInMs)) / 3600000 by PipelineName, ActivityType
| sort by runs desc
| take 25
2. Fewer activity runs
| Pattern | Better |
|---|---|
| ForEach over 5,000 files with copy per file | One copy activity with wildcard/folder path or file list |
| Lookup + If + Set Variable for every item | Push logic into a stored procedure or one query |
| Polling with Until loops and Wait | Event triggers or storage event-based ingestion |
| Pipelines triggered every minute "just in case" | Event-based or longer intervals |
3. Right-size DIUs
Copy activity uses auto DIUs by default (from a minimum up to a calculated value). For small, frequent copies, set a low fixed DIU count (e.g. 2–4). Many tiny copies at high DIUs waste money because each run bills a minimum. For big copies, test whether more DIUs actually reduce duration (see the copy activity tuning post).
4. Data flows
- TTL on the Azure IR used by data flows keeps the cluster warm for sequential data flows (faster start), but billed while idle. Set TTL only as long as the gap between consecutive data flows in a pipeline, and use one IR for flows that run back-to-back.
- Core count: start small (e.g. 8 cores general purpose) and scale only if the data flow monitoring shows long stage times that more cores would fix.
- Use quick re-use settings where available, and avoid running data flows for trivial transformations that a copy activity or SQL can do.
- Turn off debug sessions when not in use. They bill while active.
5. Integration runtimes
| IR | Cost consideration |
|---|---|
| Azure IR (public) | Default; cheapest for simple cloud-to-cloud |
| Managed VNet IR | Private endpoints to sources; pipeline/activity rates are higher and there may be start-up time. Use where private access is required |
| Self-hosted IR | Lower activity rates; you pay for VMs. Right-size nodes, share across factories where allowed |
Don't route everything through managed VNet IR if only some sources need private access.
6. Other waste
- Failed runs retried automatically many times. Fix the root cause instead of high retry counts.
- Pipelines for retired sources still scheduled.
- Duplicate triggers after publish mistakes (see the triggers Sev1 post).
7. Track monthly
Factory cost by meter, top 10 pipelines by consumption, activity run counts trend. Review with pipeline owners.
Tip
The single biggest saving is often a ForEach loop doing thousands of tiny copies. Replacing it with one wildcard copy can cut orchestration and DIU cost, and the pipeline usually gets faster too.
Checklist
- Consumption per pipeline and meter reviewed
- Loops and polling reduced
- DIUs fixed low for small copies, tested for big ones
- Data flow TTL, core counts and debug sessions controlled
- IR type matched to need; SHIR right-sized
- Retired pipelines and retry storms cleaned up