Azure API Management returns 401, 404 or 500: find the failing stage and fix it

A client gets a 401, 404, 500 or 503 from your API Management (APIM) URL. The first job is to find out who produced that status code: the gateway, a policy or the backend.

#Step 1: Gateway or backend?

Compare the code the client received with the code the backend returned.

ApiManagementGatewayLogs
| where TimeGenerated > ago(1h)
| where ResponseCode >= 400 or ResponseCode == 0
| summarize Calls = count(), Sample = take_any(LastErrorMessage)
    by ApiId, OperationId, ResponseCode, BackendResponseCode,
       LastErrorSource, LastErrorSection, LastErrorReason
| order by Calls desc

LastErrorSource names the policy or step that failed, such as validate-jwt or rate-limit.

#If the table is empty

The logs may not be turned on. Send them to a workspace, then reproduce the failure:

APIM_ID=$(az apim show -g rg-api -n apim-prod --query id -o tsv)
WS_ID=$(az monitor log-analytics workspace show -g rg-mon -n law-prod --query id -o tsv)

az monitor diagnostic-settings create --name apim-gateway-logs \
  --resource "$APIM_ID" --workspace "$WS_ID" \
  --logs '[{"category":"GatewayLogs","enabled":true}]' \
  --export-to-resource-specific true

Logs take a few minutes to arrive. If you need an answer now, skip to the trace in Step 3. The Consumption tier doesn't support resource logs, so use Application Insights there.

#Step 2: Is it one API or the whole gateway?

Split the Requests metric by response category. If everything is failing, look at the platform and network. If one API is failing, go to that API's config.

az monitor metrics list --resource "$APIM_ID" --metric Requests \
  --aggregation Total --interval PT5M --offset 1h \
  --filter "GatewayResponseCodeCategory eq '*' and BackendResponseCodeCategory eq '*'"

Check the gateway itself, then its load. The status endpoint returns 200 if the gateway is up, without calling any backend.

curl -sS -o /dev/null -w "%{http_code}\n" https://apim-prod.azure-api.net/status-0123456789abcdef

# Classic tiers
az monitor metrics list --resource "$APIM_ID" --metric Capacity \
  --aggregation Average --interval PT5M --offset 1h

On v2 tiers, use CpuPercent_Gateway and MemoryPercent_Gateway instead of Capacity.

If the instance is injected into a VNet, check its connectivity to its dependencies:

az rest --method get \
  --url "https://management.azure.com${APIM_ID}/networkstatus?api-version=2024-05-01"

#Step 3: Trace one failing request

A trace shows each policy step and where the request stopped. The old Ocp-Apim-Trace header no longer works. You now request a debug token per API, which needs Contributor or equivalent write access on the API. Workspaces aren't supported.

az apim api list -g rg-api --service-name apim-prod -o table

TOKEN=$(az rest --method post \
  --url "https://management.azure.com${APIM_ID}/gateways/managed/listDebugCredentials?api-version=2023-05-01-preview" \
  --body "{\"credentialsExpireAfter\":\"PT1H\",\"apiId\":\"${APIM_ID}/apis/orders-api\",\"purposes\":[\"tracing\"]}" \
  --query token -o tsv)

# Note the Apim-Trace-Id value in the response headers
curl -sS -D - -o /dev/null \
  -H "Apim-Debug-Authorization: $TOKEN" \
  -H "Ocp-Apim-Subscription-Key: $KEY" \
  https://apim-prod.azure-api.net/orders/v1/items

az rest --method post \
  --url "https://management.azure.com${APIM_ID}/gateways/managed/listTrace?api-version=2023-05-01-preview" \
  --body "{\"traceId\":\"$TRACE_ID\"}"

The token lasts one hour at most, and it only works for the API you requested it for.

Warning

Traces can include request headers, resolved named values and tokens. Don't paste a raw trace into a ticket or chat without checking it first.

#Symptom, cause and fix

Error text can change between releases, so match on LastErrorReason rather than the message body.

#401 and 404: check the API definition

# Expected path, backend URL and whether a key is required
az apim api show -g rg-api --service-name apim-prod --api-id orders-api \
  --query "{path:path, backend:serviceUrl, keyRequired:subscriptionRequired}"

# Operations: compare the method and URL template with the failing call
az apim api operation list -g rg-api --service-name apim-prod --api-id orders-api -o table

# Revisions: a change in a non-current revision isn't live
az apim api revision list -g rg-api --service-name apim-prod --api-id orders-api -o table

# Products that contain this API
az rest --method get \
  --url "https://management.azure.com${APIM_ID}/apis/orders-api/products?api-version=2024-05-01" \
  --query "value[].name"

Most 404s come from a missing URL suffix or version segment, the wrong HTTP method, or an operation that only exists in a non-current revision.

#Backend failures

Run the test from where APIM runs, not from your laptop. If APIM is in a VNet, use a VM in that VNet. Then test name resolution, the port and the certificate:

nslookup orders-backend.internal.contoso.com
curl -v --max-time 10 https://orders-backend.internal.contoso.com/health

#Slow calls and timeouts

Work out whether the time goes on the backend or in APIM:

ApiManagementGatewayLogs
| where TimeGenerated > ago(1h)
| summarize Calls = count(),
    P95Total = percentile(TotalTime, 95),
    P95Backend = percentile(BackendTime, 95),
    P95Client = percentile(ClientTime, 95),
    Timeouts = countif(LastErrorReason == "Timeout"),
    ClientDrops = countif(LastErrorReason == "ClientConnectionFailure")
    by ApiId
| order by P95Total desc

• High BackendTime: the backend is slow. Raising the APIM timeout won't fix that.
• High TotalTime with low BackendTime and ClientTime: look at policies, especially send-request calls, and at gateway capacity.
• ClientConnectionFailure rising alongside BackendTime: the client's timeout is shorter than the backend's response time.

The forward-request timeout defaults to 300 seconds, but values above 240 may not be honoured because idle connections can be dropped by the network. Set it explicitly for each API when you need a different value:

<backend>
    <forward-request timeout="60" />
</backend>

#Prevent it next time

• Enable GatewayLogs on every instance with --export-to-resource-specific true, so Step 1 works during an incident.
• Alert on Requests split by BackendResponseCodeCategory and GatewayResponseCodeCategory. A jump in gateway 5xx without a matching backend 5xx points at APIM rather than the backend.
• Track backend certificate expiry and subscription key rotation with named owners.