Azure diagnostic settings - resource logs missing, duplicated or landing in the wrong table

#Symptom

You open Log Analytics to investigate an incident and the logs for a Key Vault, SQL database, App Gateway or storage account aren't there. Or they are there twice, and the ingestion bill doubled. Or the team's saved queries return nothing because the data is going to AzureDiagnostics while the queries read a resource-specific table like AGWAccessLogs.

All three come down to the diagnostic setting on the resource. Platform logs are not collected by default; each resource needs a setting that names the categories and the destination.

#Quick checks

#Step 1: see what the resource is actually sending

RID=$(az keyvault show -n kv-prod-weu --query id -o tsv)

# Settings on the resource
az monitor diagnostic-settings list --resource "$RID" \
  --query "[].{name:name, workspace:workspaceId, mode:logAnalyticsDestinationType, logs:logs[?enabled].[category, categoryGroup]}" -o jsonc

# Categories the resource supports
az monitor diagnostic-settings categories list --resource "$RID" \
  --query "[].{name:name, type:categoryType}" -o table

What to look for:

• Empty list: nothing is configured. Go to step 3.
• workspaceId is a different workspace than the one you are querying. This is common after a workspace consolidation.
• mode is null: data goes to AzureDiagnostics. Dedicated means resource-specific tables.
• Categories listed individually: any category the service added later is not collected. Category groups (allLogs, audit) pick up new categories automatically.

#Step 2: check what arrives in the workspace

// Resource-specific and AzureDiagnostics data for one resource in the last 24 hours
union withsource=TableName *
| where TimeGenerated > ago(24h)
| where _ResourceId has "/vaults/kv-prod-weu"
| summarize rows = count(), last = max(TimeGenerated) by TableName
| order by rows desc

// What is landing in AzureDiagnostics, by provider and category
AzureDiagnostics
| where TimeGenerated > ago(24h)
| summarize rows = count(), last = max(TimeGenerated) by ResourceProvider, Category, Resource
| order by rows desc

// Duplicate check: same record ingested more than once
AzureDiagnostics
| where TimeGenerated > ago(1h) and Resource =~ "KV-PROD-WEU"
| summarize copies = count() by CorrelationId, OperationName, TimeGenerated
| where copies > 1
| take 20

The union * query scans every table, so keep the time range short. Use _ResourceId rather than resource name; names are upper-cased in AzureDiagnostics and _ResourceId works across tables.

Latency check, so you don't chase data that is just late:

AzureDiagnostics
| where TimeGenerated > ago(2h)
| extend delayMin = datetime_diff('minute', ingestion_time(), TimeGenerated)
| summarize avg(delayMin), percentile(delayMin, 95) by ResourceProvider

#Step 3: create or fix the setting

One setting per resource per destination, category groups instead of lists, resource-specific tables where the service supports them:

WS=$(az monitor log-analytics workspace show -g rg-monitor -n law-prod-weu --query id -o tsv)

az monitor diagnostic-settings create \
  --name diag-to-law-prod \
  --resource "$RID" \
  --workspace "$WS" \
  --export-to-resource-specific true \
  --logs '[{"categoryGroup":"allLogs","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

Remove the duplicate or stale setting once the new one is confirmed:

az monitor diagnostic-settings delete --name old-diag-setting --resource "$RID"

Warning

Switching an existing setting from AzureDiagnostics to resource-specific tables changes where new data lands. Saved queries, workbooks, alert rules and SIEM content that read AzureDiagnostics stop matching new rows. Inventory those before you switch, and run both for a short overlap if needed, accepting the double ingestion cost for that window.

Not every service supports resource-specific mode. If --export-to-resource-specific true has no effect and data still lands in AzureDiagnostics, the service doesn't offer dedicated tables yet; check the service's monitoring reference.

#The Activity Log is a subscription-level setting

The Activity Log isn't covered by resource settings:

az monitor diagnostic-settings subscription list -o table

az monitor diagnostic-settings subscription create \
  --name activity-to-law \
  --workspace "$WS" \
  --logs '[{"category":"Administrative","enabled":true},{"category":"Security","enabled":true},{"category":"ServiceHealth","enabled":true},{"category":"Alert","enabled":true},{"category":"Policy","enabled":true}]'

Check the exact parameters with az monitor diagnostic-settings subscription create --help; depending on your CLI version it may also ask for --location.

#Errors you may hit

#Find resources with no diagnostic setting

Resource Graph doesn't index diagnostic settings, so either check Azure Policy compliance (below) or loop through a resource type:

for id in $(az resource list --resource-type Microsoft.KeyVault/vaults --query "[].id" -o tsv); do
  n=$(az monitor diagnostic-settings list --resource "$id" --query "length(@)" -o tsv)
  [ "$n" = "0" ] && echo "NO DIAG: $id"
done

Get-AzResource -ResourceType Microsoft.Sql/servers/databases | ForEach-Object {
  if (-not (Get-AzDiagnosticSetting -ResourceId $_.ResourceId -ErrorAction SilentlyContinue)) {
    "NO DIAG: $($_.ResourceId)"
  }
}

#Prevent it next time

• Assign the built-in Azure Policy initiatives that deploy allLogs (or audit) category group settings to a Log Analytics workspace, with deployIfNotExists and a remediation task for existing resources. Names vary as Microsoft updates them; search the policy definitions for "category group".
• Use category groups, not category lists, so new categories are picked up.
• Standardise one setting name per destination (for example diag-to-law-prod) so duplicates are easy to spot.
• Prefer resource-specific tables for new deployments; they are easier to query and support table plans for cost control.
• Add a daily query that compares expected resources against _ResourceId values seen in the last 24 hours, and alert on gaps.
• When a resource is deleted and recreated with the same name, check the diagnostic setting again; earlier settings can be reapplied to the new resource.