OCI API Gateway token authentication failing: 401s, 500s and JWKS, issuer and audience mismatches

You turned on token validation for an API Gateway deployment and now either every caller gets 401 Unauthorized with a WWW-Authenticate header, or every request gets 500 whether a token is present or not. The backend never sees the traffic, so the backend logs are empty. The gateway's execution log has the reason. This post assumes a TOKEN_AUTHENTICATION policy. The older JWT_AUTHENTICATION type still works but Oracle no longer recommends it. Move to the new type when you next touch the spec.

#Symptom to cause

#Step 1: get the execution log

Turn on execution logging at INFO while you debug. In the deployment spec:

"loggingPolicies": {
  "executionLog": { "isEnabled": true, "logLevel": "INFO" }
}

Then create the service log object for the deployment:

oci logging log create \
  --log-group-id ocid1.loggroup.oc1.iad.aaaa... \
  --display-name apigw-orders-exec \
  --log-type SERVICE \
  --configuration '{
    "source": {
      "sourceType": "OCISERVICE",
      "service": "apigateway",
      "resource": "ocid1.apideployment.oc1.iad.aaaa...",
      "category": "execution"
    }
  }'

Search it from the Console's Logging Search, or with oci logging-search search-logs. Authentication failures are logged per request with a reason string. Message wording varies between releases, so search for the request ID returned to the client in the opc-request-id response header rather than for a fixed phrase.

Warning

At INFO the execution log records every request. Drop back to WARN or ERROR after the incident, or the log volume on a busy API will show up on the bill.

#Step 2: read the deployed policy, not the file in git

oci api-gateway deployment get \
  --deployment-id ocid1.apideployment.oc1.iad.aaaa... \
  --query 'data.specification."request-policies".authentication'

A working REMOTE_JWKS policy looks like this:

"authentication": {
  "type": "TOKEN_AUTHENTICATION",
  "tokenHeader": "Authorization",
  "tokenAuthScheme": "Bearer",
  "isAnonymousAccessAllowed": false,
  "maxClockSkewInSeconds": 60,
  "validationPolicy": {
    "type": "REMOTE_JWKS",
    "uri": "https://login.microsoftonline.com/<tenant-id>/discovery/v2.0/keys",
    "isSslVerifyDisabled": false,
    "maxCacheDurationInHours": 1,
    "additionalValidationPolicy": {
      "issuers": ["https://login.microsoftonline.com/<tenant-id>/v2.0"],
      "audiences": ["api://orders-api"]
    }
  }
}

The constraints that matter:

• tokenHeader and tokenQueryParam are mutually exclusive.
• issuers and audiences each take up to five values. verifyClaims takes up to ten.
• maxCacheDurationInHours is 1 to 24. The JWKS can contain up to ten keys.
• verifyClaims is exact string matching. There's no pattern matching.

#Step 3: decode a real token

Take a token the client actually sends. Don't use one you generated by hand. Decode the header and payload without verifying:

TOKEN='eyJ0eXAiOi...'
python3 - "$TOKEN" <<'EOF'
import sys, json, base64
for part in sys.argv[1].split('.')[:2]:
    part += '=' * (-len(part) % 4)
    print(json.dumps(json.loads(base64.urlsafe_b64decode(part)), indent=2))
EOF

Compare these fields with the policy:

#Entra ID: v1 vs v2 tokens

This mismatch catches most teams. An Entra app registration issues v1 access tokens unless its manifest sets accessTokenAcceptedVersion to 2. The issuer differs:

• v1: https://sts.windows.net/<tenant-id>/
• v2: https://login.microsoftonline.com/<tenant-id>/v2.0

The audience also changes. v1 tokens often carry the App ID URI (api://...), while v2 tokens carry the application (client) ID GUID. Either list both forms in issuers and audiences, or fix the app manifest so every client gets the same version. Entra also puts delegated scopes in scp, not scope, and app roles in roles. If the route uses ANY_OF authorization, which checks scopes, test it with an Entra token before you rely on it. A verifyClaims rule on scp only works if the token carries exactly one scope, because the match is exact on the whole string.

#OCI IAM identity domains

The JWKS URI is https://<domain-host>/admin/v1/SigningCert/jwk. By default an identity domain doesn't serve its signing certificate to anonymous callers, and the gateway calls anonymously. In the domain's settings, allow client access to the signing certificate. Otherwise the fetch fails and every request returns 500. Check it from any machine:

curl -s -o /dev/null -w '%{http_code}\n' \
  https://idcs-abc123.identity.oraclecloud.com/admin/v1/SigningCert/jwk

200 is good. 401 means the setting is off.

#Step 4: when everything returns 500, test the gateway's network path

If the JWKS fetch fails, every request on the deployment returns 500. The fetch comes from the gateway, so it uses the gateway subnet's routing:

• Private gateway, public IdP: the subnet needs a NAT gateway route to reach login.microsoftonline.com or the identity domain host. A service gateway route isn't enough, because the IdP isn't an OCI service behind the service gateway.
• Security list or NSG on the gateway subnet: must allow egress TCP 443.
• DNS: the VCN resolver must resolve the IdP host. Custom private views or forwarders that don't forward public names break it.
• TLS interception: a firewall that re-signs TLS fails certificate checks. Don't set isSslVerifyDisabled to true to get around this in production.

Launch a test instance in the gateway's subnet and run the same curl against the JWKS URI. If it hangs, the problem is routing. If it returns an HTTP error, the problem is the URI or the IdP.

Caution

Rolling back a deployment spec that removes authentication exposes the backend to anonymous callers. If you need to restore service fast, fix the egress route or switch temporarily to STATIC_KEYS with the IdP's current public keys. Don't set isAnonymousAccessAllowed to true.

#Step 5: clock skew and key rotation

Tokens are checked against exp and nbf. An IdP whose clock is a few seconds ahead issues tokens that look not yet valid. The default skew is 0. Set maxClockSkewInSeconds to 30 to 60 rather than the maximum of 120.

For key rotation, the gateway caches the JWKS for up to maxCacheDurationInHours. Tokens signed with a key the gateway hasn't fetched yet can fail until the cache refreshes. Mainstream IdPs publish new keys before they sign with them, which hides this. Custom IdPs often don't. A one-hour cache is a sensible default.

Apply changes with an update and wait for the work request:

oci api-gateway deployment update \
  --deployment-id ocid1.apideployment.oc1.iad.aaaa... \
  --specification file://spec.json --force

# Confirm it went ACTIVE, not FAILED
oci api-gateway deployment get \
  --deployment-id ocid1.apideployment.oc1.iad.aaaa... \
  --query 'data."lifecycle-state"' --raw-output

#Prevent it next time

• Keep the deployment spec in source control and diff it against deployment get output during reviews.
• Add a synthetic check that calls the API with a real token from each IdP and client type, and alarm on 401 and 5xx rates from the oci_apigateway metrics namespace.
• Pin Entra apps to one access token version and document the expected iss and aud next to the policy.
• Give private gateways a NAT route for JWKS fetches and treat that route as a dependency of the API.
• Customize the validationFailurePolicy only after the defaults work. A custom response makes the client error less clear while you're still debugging.