Home / Azure

AzureTroubleshootingIntermediate

Sev1: monitoring went blind — Log Analytics daily cap reached, data collection stopped

Restore log ingestion when a Log Analytics workspace hits its daily cap and alerts go silent. Confirm the cap, find the table or resource that spiked, raise the cap safely and stop the noisy source.

Deepak Bhardwaj2 min read

Applies toAzure Monitor Log Analytics workspaces, Microsoft Sentinel workspaces, Azure CLI 2.50+

Symptoms

  • Log-based alerts stop firing; dashboards show a flat line from a certain time.
  • Queries return data up to a point in the day, then nothing.
  • Sentinel analytics rules report no data.

1. Confirm the cap was hit

_LogOperation
| where TimeGenerated > ago(2d)
| where Category =~ "Ingestion"
| where Detail has "OverQuota" or Operation has "Data collection"
| project TimeGenerated, Operation, Detail

az monitor log-analytics workspace show -g rg-mon -n law-prod \
  --query "{capGb:workspaceCapping.dailyQuotaGb, status:workspaceCapping.dataIngestionStatus, reset:workspaceCapping.quotaNextResetTime}" -o json

dataIngestionStatus of OverQuota confirms it. Collection resumes at the reset time if you do nothing.

2. Raise the cap to restore visibility

az monitor log-analytics workspace update -g rg-mon -n law-prod --quota 200
# or remove the cap for now
az monitor log-analytics workspace update -g rg-mon -n law-prod --quota -1

Warning

Data that arrived while the workspace was over quota is dropped, not queued. Raising the cap restores collection from now on only.

3. Find what spiked

Usage
| where TimeGenerated > ago(3d)
| where IsBillable == true
| summarize GB = sum(Quantity) / 1000 by DataType, bin(TimeGenerated, 1d)
| order by GB desc

Then find the resource inside the table:

ContainerLogV2
| where TimeGenerated > ago(1d)
| summarize GB = sum(_BilledSize) / 1e9 by _ResourceId, PodNamespace
| top 10 by GB desc

Swap ContainerLogV2 for AzureDiagnostics, Syslog, Perf or whichever table jumped.

4. Stop the noisy source

Source Fix
A pod logging at debug Fix log level; use a ContainerLogV2 namespace filter in the Container insights config map
Diagnostic setting sending all categories Send only the categories you use
Syslog at debug from all VMs Edit the data collection rule to a higher minimum level
Firewall or flow logs Move to the Basic/Auxiliary logs table plan if you only need occasional queries

5. Put the cap back with headroom

Set the cap above normal peaks and add an alert so it never surprises you again:

_LogOperation
| where Category =~ "Ingestion" and Detail has "OverQuota"

Create a log alert on this query and on a daily Usage threshold at about 80% of the cap.

Prevent it next time

  • Use the cap as a cost safety net with alerts, not as a hard budget for security-critical workspaces.
  • Review top tables monthly and set ingestion-time transformations to drop noise.

Related fixes