Home / Azure

AzureTroubleshootingAdvanced

Log Analytics ingestion latency: data arrives late, alerts fire late or never - measure it and build alerts that survive it

Log-based alerts miss incidents because records land minutes after they happened. Measure agent and pipeline latency with ingestion_time() and _TimeReceived, find the slow source, and write alert queries that don't skip late data.

Deepak Bhardwaj4 min read

Applies toAzure Monitor Logs (Log Analytics workspaces), Azure Monitor Agent with data collection rules, log search alert rules, diagnostic settings

Symptoms

  • A log search alert didn't fire, but when you run its query later, the matching records are there.
  • Dashboards show a dip in the last few minutes that fills in later.
  • Some sources (custom logs, diagnostic settings, a busy agent) are much later than others.

Every record has a TimeGenerated (when the event happened, or when it was collected). It becomes queryable some time later. If an alert evaluates "the last 5 minutes by TimeGenerated" and the data arrives 7 minutes late, the alert never sees it.

Step 1: measure the latency

Two built-ins:

  • _TimeReceived: when Azure Monitor received the record.
  • ingestion_time(): when the record was stored and became queryable.
Heartbeat
| where TimeGenerated > ago(8h)
| extend E2E      = ingestion_time() - TimeGenerated,
         Agent    = _TimeReceived - TimeGenerated,
         Pipeline = ingestion_time() - _TimeReceived
| summarize p50_e2e = percentile(E2E, 50), p95_e2e = percentile(E2E, 95),
            p95_agent = percentile(Agent, 95), p95_pipeline = percentile(Pipeline, 95)
    by Computer
| top 20 by p95_e2e desc

Across all tables:

union withsource = TableName *
| where TimeGenerated > ago(3h)
| extend E2E = ingestion_time() - TimeGenerated
| summarize p95 = percentile(E2E, 95), records = count() by TableName
| order by p95 desc

Typical end-to-end latency is in the range of tens of seconds to a few minutes and varies by data type. Use your own p95 values, not a guess, to design alerts.

Step 2: find where the time goes

Large component Likely cause What to check
Agent (generated → received) Agent buffering, VM overloaded, network/proxy issues, clock skew AMA health, VM CPU, proxy, NTP
Pipeline (received → ingested) DCR transformations, large records, service-side delay Transformation complexity, record sizes, Azure status
Only one table late Source-specific batching (diagnostic settings, custom logs, Logs Ingestion API batching) Source docs; client batch size and flush interval
Negative or huge values Wrong TimeGenerated from the source (clock skew, time zone, parsed field) The source's timestamp mapping

Check the agent side quickly:

Heartbeat
| where TimeGenerated > ago(1h)
| summarize last = max(TimeGenerated), lastIngested = max(ingestion_time()) by Computer, Category, Version
| extend lag = lastIngested - last
| order by lag desc

A clock problem shows as TimeGenerated in the future or far in the past for one machine.

Step 3: write alerts that tolerate late data

Filter on ingestion time, not event time

The key trick: select records by when they arrived, and still report by when they happened.

// Runs every 5 minutes, looks at everything that ARRIVED in the last 5 minutes
AppTraces
| where ingestion_time() > ago(5m)
| where SeverityLevel >= 3 and Message has "PaymentFailed"
| summarize failures = count(), first_seen = min(TimeGenerated) by AppRoleName

Nothing is skipped, because every record arrives exactly once in some window. The cost: the alert fires a little later than the event, never earlier.

Keep a broad TimeGenerated filter too, so the query stays efficient:

| where TimeGenerated > ago(1h)
| where ingestion_time() > ago(5m)

Or: overlap the window

If you must use TimeGenerated, make the alert's lookback longer than its frequency plus your p95 latency (for example, run every 5 minutes over the last 15), and deduplicate in the action (stateful alerts, or a correlation key).

Use metric alerts where they exist

Platform metrics (CPU, availability, queue length) arrive faster and more reliably than logs. Alert on metrics, and use logs to investigate.

Step 4: reduce latency at the source

  • Diagnostic settings: send critical resource logs to the workspace directly, not via Event Hubs and a custom forwarder.
  • Azure Monitor Agent: keep agents updated; avoid DCR transformations that parse huge records when a simple filter would do.
  • Logs Ingestion API: flush small batches more often for alert-critical data.
  • Basic and Auxiliary table plans suit cheap, high-volume data, but have query and alerting limitations; keep alert sources on Analytics tables.

Tip

Add a "latency watchdog" alert: if p95(ingestion_time() - TimeGenerated) for Heartbeat exceeds, say, three times your normal value, notify the platform team. When monitoring is late, you want to know before an incident does.

Heartbeat
| where ingestion_time() > ago(15m)
| summarize p95 = percentile(ingestion_time() - TimeGenerated, 95)
| where p95 > 10m

Prevent it next time

  • Measure latency per table once a quarter and record it next to your alert design.
  • Default new log alerts to ingestion_time() windows.
  • Use metric alerts for anything with a platform metric.
  • Watch agent clock skew; it corrupts both latency numbers and timelines.

Related fixes