Home / Azure

AzureCheat sheetAdvanced

AKS control plane and audit logs KQL cheat sheet: who did it, who is hammering the API server and why webhooks fail

KQL for the AKSAudit, AKSAuditAdmin and AKSControlPlane tables. Find who deleted or exec'd into what, which client is throttling the API server, webhook and Entra auth failures, and what the logs cost.

Deepak Bhardwaj7 min read

Applies toAzure Kubernetes Service (all supported Kubernetes versions), Azure Monitor Log Analytics, resource-specific diagnostic tables

Before you query: route logs to resource-specific tables

AKS control plane logs are only useful once a diagnostic setting sends them to Log Analytics. Use resource-specific tables (AKSAudit, AKSAuditAdmin, AKSControlPlane) instead of the legacy AzureDiagnostics. You get typed columns, cheaper queries and no 500-column limit.

AKS_ID=$(az aks show -g rg-aks -n aks-prod --query id -o tsv)
WS_ID=$(az monitor log-analytics workspace show -g rg-mon -n law-platform --query id -o tsv)

az monitor diagnostic-settings create \
  --name aks-control-plane \
  --resource $AKS_ID \
  --workspace $WS_ID \
  --export-to-resource-specific true \
  --logs '[
    {"category":"kube-audit-admin","enabled":true},
    {"category":"kube-apiserver","enabled":true},
    {"category":"kube-controller-manager","enabled":true},
    {"category":"cluster-autoscaler","enabled":true},
    {"category":"guard","enabled":true}
  ]'
Category Table Notes
kube-audit AKSAudit Every API call, including get/list/watch. Very high volume
kube-audit-admin AKSAuditAdmin Audit without get and list. Use it by default
kube-apiserver, kube-controller-manager, kube-scheduler, cluster-autoscaler, guard, cloud-controller-manager, CSI controllers AKSControlPlane Component logs. Filter on Category

Tip

Turn on kube-audit-admin permanently. Turn on full kube-audit only when you need read access (who listed secrets, who is polling), and send it to a separate workspace or a cheaper table plan. Full audit on a busy cluster can easily cost more than the cluster's own workload logs.

The queries below use AKSAuditAdmin for changes and AKSAudit where reads matter. Both have the same schema. Useful columns: Verb, ObjectRef (dynamic: resource, namespace, name, subresource), User (dynamic: username, groups), SourceIps, UserAgent, ResponseStatus (dynamic: code), RequestUri, Stage, RequestReceivedTime, StageReceivedTime.

Who changed or deleted it

Who deleted a deployment, service or namespace:

AKSAuditAdmin
| where TimeGenerated > ago(24h)
| where Verb == "delete"
| where ObjectRef.resource in ("deployments", "services", "namespaces", "statefulsets", "persistentvolumeclaims")
| where Stage == "ResponseComplete"
| project TimeGenerated, user = tostring(User.username), resource = tostring(ObjectRef.resource),
          ns = tostring(ObjectRef.namespace), name = tostring(ObjectRef.name),
          code = toint(ResponseStatus.code), ip = tostring(SourceIps[0]), UserAgent
| order by TimeGenerated desc

Every change to one object, in order:

AKSAuditAdmin
| where TimeGenerated > ago(7d)
| where ObjectRef.namespace == "payments" and ObjectRef.name == "payments-api"
| where Stage == "ResponseComplete" and Verb in ("create", "update", "patch", "delete")
| project TimeGenerated, Verb, user = tostring(User.username), UserAgent, code = toint(ResponseStatus.code)
| order by TimeGenerated asc

UserAgent tells a person apart from automation: kubectl/..., argocd-application-controller/..., helm/... or Go-http-client. An Entra user appears as a UPN or object ID, depending on how the cluster is set up for auth.

Who exec'd or attached into pods, or port-forwarded:

AKSAuditAdmin
| where TimeGenerated > ago(7d)
| where ObjectRef.resource == "pods" and ObjectRef.subresource in ("exec", "attach", "portforward")
| where Stage == "ResponseStarted" or Stage == "ResponseComplete"
| summarize sessions = count(), last = max(TimeGenerated)
    by user = tostring(User.username), ns = tostring(ObjectRef.namespace), pod = tostring(ObjectRef.name), sub = tostring(ObjectRef.subresource)
| order by last desc

RBAC changes, the ones that grant power:

AKSAuditAdmin
| where TimeGenerated > ago(30d)
| where ObjectRef.resource in ("clusterrolebindings", "rolebindings", "clusterroles", "roles")
| where Verb in ("create", "update", "patch", "delete") and Stage == "ResponseComplete"
| project TimeGenerated, Verb, kind = tostring(ObjectRef.resource), ns = tostring(ObjectRef.namespace),
          name = tostring(ObjectRef.name), user = tostring(User.username)

Reads of sensitive objects (needs full kube-audit)

AKSAudit
| where TimeGenerated > ago(24h)
| where ObjectRef.resource == "secrets" and Verb in ("get", "list", "watch")
| where Stage == "ResponseComplete"
| summarize reads = count() by user = tostring(User.username), ns = tostring(ObjectRef.namespace), UserAgent
| order by reads desc

Expect the kubelet, controllers and your secret-sync operator to show up. Look for human accounts, or service accounts reading secrets in namespaces they don't own.

Who is hammering the API server

Top callers by request volume. Run it on AKSAudit, because list and watch are the expensive calls:

AKSAudit
| where TimeGenerated > ago(1h)
| where Stage == "ResponseComplete"
| summarize requests = count() by user = tostring(User.username), UserAgent, Verb
| top 20 by requests

Throttled requests. HTTP 429 comes from API Priority and Fairness or from the managed control plane's limits:

AKSAudit
| where TimeGenerated > ago(6h)
| where toint(ResponseStatus.code) == 429
| summarize throttled = count() by bin(TimeGenerated, 5m), user = tostring(User.username), UserAgent
| render timechart

Unbounded lists across the cluster. A list without a namespace means the whole cluster, and on large clusters that's the classic way to overload the control plane:

AKSAudit
| where TimeGenerated > ago(1h)
| where Verb == "list" and isempty(tostring(ObjectRef.namespace))
| summarize lists = count() by resource = tostring(ObjectRef.resource), user = tostring(User.username), UserAgent
| order by lists desc

Approximate server-side latency per request. This is time from received to response complete, and it excludes watches:

AKSAudit
| where TimeGenerated > ago(1h)
| where Stage == "ResponseComplete" and Verb != "watch"
| extend ms = datetime_diff("millisecond", StageReceivedTime, RequestReceivedTime)
| summarize p50 = percentile(ms, 50), p99 = percentile(ms, 99), n = count() by Verb, resource = tostring(ObjectRef.resource)
| where n > 100
| order by p99 desc

Denied and failing calls

403s. These are RBAC problems, or a workload identity bound to the wrong service account:

AKSAuditAdmin
| where TimeGenerated > ago(1h)
| where toint(ResponseStatus.code) == 403
| summarize denied = count() by user = tostring(User.username), Verb, resource = tostring(ObjectRef.resource), ns = tostring(ObjectRef.namespace)
| order by denied desc

Admission webhook failures. A webhook that times out with failurePolicy: Fail blocks every create in its scope:

AKSControlPlane
| where TimeGenerated > ago(6h)
| where Category == "kube-apiserver"
| where Message has "failed calling webhook" or Message has "admission webhook"
| extend webhook = extract(@'webhook "([^"]+)"', 1, Message)
| summarize failures = count(), sample = take_any(Message) by webhook
| order by failures desc

Entra ID / Azure RBAC for Kubernetes authorisation failures from the guard component:

AKSControlPlane
| where TimeGenerated > ago(6h)
| where Category == "guard"
| where Level in ("Error", "Warning") or Message has_any ("denied", "error", "failed")
| project TimeGenerated, Level, Message
| order by TimeGenerated desc

Errors from the API server and controller manager in general:

AKSControlPlane
| where TimeGenerated > ago(1h)
| where Category in ("kube-apiserver", "kube-controller-manager") and Level in ("Error", "Fatal")
| summarize n = count(), sample = take_any(Message) by Category, msg = substring(Message, 0, 120)
| order by n desc

Cluster autoscaler decisions

AKSControlPlane
| where TimeGenerated > ago(6h)
| where Category == "cluster-autoscaler"
| where Message has_any ("Scale-up", "scale up", "ScaleDown", "scale down", "NotTriggerScaleUp", "failed")
| project TimeGenerated, Level, Message
| order by TimeGenerated desc

Message wording changes between autoscaler versions. Search for a few keywords first, then tighten the filter.

What the logs cost

union withsource = tbl AKSAudit, AKSAuditAdmin, AKSControlPlane
| where TimeGenerated > ago(1d)
| summarize GB = round(sum(_BilledSize) / 1e9, 2) by tbl, _ResourceId
| order by GB desc

Which callers create the most audit volume, so you know what to fix or filter:

AKSAudit
| where TimeGenerated > ago(1d)
| summarize GB = round(sum(_BilledSize) / 1e9, 3) by user = tostring(User.username), Verb
| top 15 by GB

Legacy AzureDiagnostics equivalents

If a cluster still writes to AzureDiagnostics, the audit event is a JSON string in log_s:

AzureDiagnostics
| where Category == "kube-audit-admin"
| extend e = parse_json(log_s)
| where e.verb == "delete"
| project TimeGenerated, user = tostring(e.user.username), resource = tostring(e.objectRef.resource), name = tostring(e.objectRef.name)

Move to resource-specific tables. Parsing log_s on every query is slow and expensive.

Keep in mind

  • Audit events can arrive several minutes late. Leave room for that in alert windows.
  • One request can log several stages. Filter on Stage == "ResponseComplete" (or ResponseStarted for exec and watch) so you don't count it twice.
  • Store these queries as saved functions in the workspace so the on-call engineer runs AksWhoDeleted("payments") instead of retyping filters at 3 a.m.

Related fixes