EKS IRSA: not authorized to perform sts:AssumeRoleWithWebIdentity - check it in the right order

#Symptom

A pod that should use an IAM role through its service account fails with one of these:

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation:
Not authorized to perform sts:AssumeRoleWithWebIdentity

An error occurred (InvalidIdentityToken) ... No OpenIDConnect provider found in your account

User: arn:aws:sts::111122223333:assumed-role/eks-node-role/i-0abc is not authorized to perform: s3:GetObject

The third message looks different, but it's often the same feature failing: the pod is using the node role, because IRSA never applied to it.

#How the pieces fit

1. The service account has an eks.amazonaws.com/role-arn annotation.
2. When the pod is created, a mutating webhook injects two environment variables and a projected token.
3. The AWS SDK calls AssumeRoleWithWebIdentity with that token.
4. IAM checks the token against the account's OIDC provider, then the role's trust policy.

Work through them in that order.

#Step 1: see what the pod has

NS=payments; POD=api-6f7c9; SA=api

kubectl get sa "$SA" -n "$NS" -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}{"\n"}'
kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.serviceAccountName}{"\n"}'
kubectl exec -n "$NS" "$POD" -- env | grep -E 'AWS_ROLE_ARN|AWS_WEB_IDENTITY_TOKEN_FILE|AWS_REGION'

If the image has the AWS CLI:

kubectl exec -n "$NS" "$POD" -- aws sts get-caller-identity

#Step 2: fix missing injection

The webhook acts only when a pod is created, so annotating a service account has no effect on running pods.

kubectl rollout restart deployment/api -n "$NS"

If the env vars are still missing:

• The pod's serviceAccountName may not be the one you annotated, or it may be default.
• A misspelled annotation is ignored without warning.
• A very old SDK may not support web identity credentials, or the app may be passing static credentials first. Check for AWS_ACCESS_KEY_ID in the environment, which takes priority.
• Check that the pod identity webhook works: kubectl get mutatingwebhookconfigurations pod-identity-webhook.

#Step 3: check the token

kubectl exec -n "$NS" "$POD" -- sh -c 'cat $AWS_WEB_IDENTITY_TOKEN_FILE' \
  | cut -d. -f2 | tr '_-' '/+' | base64 -d 2>/dev/null; echo

Look at three fields:

{
  "aud": ["sts.amazonaws.com"],
  "iss": "https://oidc.eks.ap-south-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE",
  "sub": "system:serviceaccount:payments:api"
}

The trust policy must match these exact values. Keep this output handy for Step 5.

Warning

The token is a live credential. Decode only the middle section as shown here, and don't paste the full token into tickets or chat.

#Step 4: check the OIDC provider

The provider must exist in the same account as the role, with the same issuer.

ISSUER=$(aws eks describe-cluster --name prod --query "cluster.identity.oidc.issuer" --output text)
echo "$ISSUER"
aws iam list-open-id-connect-providers --output text | grep "${ISSUER##*/}"

No match gives you No OpenIDConnect provider found. Create it:

eksctl utils associate-iam-oidc-provider --cluster prod --approve

Check that it trusts the right audience:

aws iam get-open-id-connect-provider \
  --open-id-connect-provider-arn "arn:aws:iam::111122223333:oidc-provider/${ISSUER#https://}" \
  --query ClientIDList

It should include sts.amazonaws.com.

Caution

Every EKS cluster has its own issuer. If you rebuild a cluster, or copy a role from another one, the old trust policy points to the wrong issuer. This is the most common cause of a sudden failure after a cluster replacement.

#Step 5: check the trust policy

aws iam get-role --role-name api-irsa --query "Role.AssumeRolePolicyDocument"

A correct policy looks like this:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.ap-south-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.ap-south-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:payments:api",
        "oidc.eks.ap-south-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
      }
    }
  }]
}

Common errors:

Update the policy, then test again. There's no need to restart the pod for a trust policy change:

aws iam update-assume-role-policy --role-name api-irsa --policy-document file://trust.json

#Step 6: use CloudTrail for the final answer

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRoleWithWebIdentity \
  --max-results 20 \
  --query "Events[].CloudTrailEvent" --output text | head -c 3000

Check errorCode, the role ARN and the subjectFromWebIdentityToken value in the event, if your account's events show it.

#Cross-account access

There are two working patterns:

• Provider in the target account: create the OIDC provider in account B using account A's cluster issuer, and put the trust policy on the role in B.
• Role chaining: the IRSA role in A calls sts:AssumeRole on a role in B. B's trust policy names A's role, and A's role has permission to assume it.

#Prevent it next time

• Build roles with a module that takes the cluster's issuer as an input, so it can't drift when a cluster is rebuilt.
• Require a metadata hop limit of 1 on nodes, so pods can't silently fall back to the node role. See the IMDSv2 hop limit post.
• Run aws sts get-caller-identity as a deployment smoke test.
• For new work, consider EKS Pod Identity, which doesn't need an OIDC provider per cluster.