Home / Azure

AzureHow-toIntermediate

Optimization: cutting Log Analytics ingestion cost with table plans, DCR filters and commitment tiers

Log Analytics often becomes one of the biggest lines on an Azure bill. Find which tables and resources ingest the most, filter noise at the data collection rule, move verbose tables to the Basic or Auxiliary plan, set retention per table, and pick a commitment tier when volume is steady.

Deepak Bhardwaj4 min read

Applies toAzure Monitor Log Analytics workspaces, Azure Monitor Agent with data collection rules (DCRs), diagnostic settings, Microsoft Sentinel workspaces

Symptom

The Log Analytics charge grows every month, nobody knows why, and the reflex is a daily cap. A cap stops ingestion completely when it's hit (see the "monitoring went blind" Sev1 post), so it's a safety net, not a cost strategy.

1. Find where the volume comes from

By table, last 30 days (Quantity is in MB):

Usage
| where TimeGenerated > ago(30d) and IsBillable
| summarize GB = round(sum(Quantity) / 1024, 1) by DataType
| sort by GB desc

By resource, for a noisy table:

AzureDiagnostics
| where TimeGenerated > ago(7d)
| summarize GB = round(sum(_BilledSize) / 1024 / 1024 / 1024, 2) by _ResourceId, Category
| sort by GB desc
| take 20

By computer, for agent data:

find where TimeGenerated > ago(7d) project _BilledSize, Computer, Type
| summarize GB = round(sum(_BilledSize) / 1024 / 1024 / 1024, 2) by Computer, Type
| sort by GB desc | take 20

Usually a few tables (ContainerLogV2, AzureDiagnostics, Syslog, SecurityEvent, AppTraces) and a handful of resources make up most of the volume.

2. Stop collecting what nobody uses

Source Typical fix
Diagnostic settings sending every category Send only the categories you query or alert on
Windows event collection "all events" DCR with XPath for the event IDs you need
Syslog at debug/info for all facilities Raise minimum level per facility
AKS container logs from chatty system namespaces Exclude namespaces in the Container Insights DCR / ConfigMap
App Insights verbose traces Sampling and log level in the app
Performance counters every 10 s 60 s is enough for most dashboards

3. Filter inside the DCR (transformations)

A transformation drops rows or columns before ingestion, so you don't pay for them:

"dataFlows": [{
  "streams": ["Microsoft-Syslog"],
  "destinations": ["la-prod"],
  "transformKql": "source | where SeverityLevel !in ('debug', 'info') | project-away HostIP",
  "outputStream": "Microsoft-Syslog"
}]

For diagnostic-settings data, use a workspace transformation DCR on supported tables. Test the KQL in the workspace first. A typo that filters everything looks like a quiet day.

4. Use the right table plan

Plan Use for Trade-off
Analytics Data you alert on, dashboards, frequent queries Full KQL, highest ingestion price
Basic Verbose troubleshooting logs queried occasionally Lower ingestion price, per-query charge, reduced KQL and alerting support
Auxiliary Audit/compliance data rarely queried Lowest price, slowest/most limited queries, custom tables via DCR

Supported tables and exact capabilities change. Check the table plan docs, then switch:

az monitor log-analytics workspace table update -g rg-mon --workspace-name la-prod \
  -n ContainerLogV2 --plan Basic

5. Retention per table

Keep interactive retention short for high-volume tables and use long-term (archive) retention only where compliance needs it:

az monitor log-analytics workspace table update -g rg-mon --workspace-name la-prod \
  -n AppTraces --retention-time 30 --total-retention-time 365

6. Commitment tier

When daily ingestion is steady, a commitment tier (starting at 100 GB/day) is cheaper per GB than pay-as-you-go. Check average daily volume after the clean-up above, then choose. Tiers have a minimum commitment period, so don't over-commit.

7. Keep it from growing back

  • Budget alert on the workspace resource group.
  • A monthly "top 10 tables / resources" workbook reviewed by the platform team.
  • Azure Policy for diagnostic settings that deploys a curated category set, not "allLogs" everywhere.

Tip

Measure the effect a week after each change with the Usage query. Changing one source at a time shows what actually saved money.

Checklist

  • Top tables, resources and computers identified
  • Unused categories and verbose levels removed
  • DCR transformations tested and applied
  • Basic/Auxiliary plans for verbose tables
  • Per-table retention set
  • Commitment tier reviewed on cleaned-up volume