Home / AWS

AWSTroubleshootingAdvanced

Containers on EC2 can't get IAM role credentials after enforcing IMDSv2: the hop limit, and other reasons the SDK falls back to nothing

Apps inside Docker or ECS-on-EC2 containers fail with "Unable to locate credentials" after IMDSv2 was required. Raise the hop limit or use task roles, and rule out network policies and proxies.

Deepak Bhardwaj5 min read

Applies toAmazon EC2 instance metadata service (IMDSv1/IMDSv2), Docker bridge networking, Amazon ECS on EC2, self-managed Kubernetes and EKS managed node groups, AWS SDKs and CLI v2

Symptoms

Right after a security change that set HttpTokens=required on instances or launch templates:

  • The AWS CLI works on the host, but inside a container it fails with Unable to locate credentials / NoCredentialProviders / Could not connect to the endpoint URL: "http://169.254.169.254/...".
  • SDK calls hang for a few seconds before failing (the SDK retries IMDS, then gives up).
  • Pods on EKS nodes that relied on the node role lose access to S3 or ECR.

Why it happens

IMDSv2 starts with a PUT /latest/api/token request. The response comes back with an IP TTL equal to the hop limit (default 1). A container on a bridge network is one extra network hop away from the host. With a hop limit of 1, the token response is dropped before it reaches the container, so IMDSv2 fails. With IMDSv1 disabled, there's no fallback.

Where the caller runs Hops to IMDS Works with hop limit 1?
Host process 1 Yes
Docker --network host 1 Yes
Docker bridge / ECS bridge mode 2 No
Kubernetes pod (most CNIs, non-host network) 2 No

Step 1: Confirm the metadata options

aws ec2 describe-instances --instance-ids i-0123456789abcdef0 \
  --query 'Reservations[].Instances[].MetadataOptions'

Look for "HttpTokens": "required" and "HttpPutResponseHopLimit": 1.

Reproduce from inside the container:

TOKEN=$(curl -s -m 3 -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
echo "token length: ${#TOKEN}"
curl -s -m 3 -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

Empty token from the container but a real token from the host = hop limit.

Step 2a: Raise the hop limit (quick fix)

aws ec2 modify-instance-metadata-options --instance-id i-0123456789abcdef0 \
  --http-tokens required --http-put-response-hop-limit 2 --http-endpoint enabled

Make it permanent in the launch template so new instances from an Auto Scaling group get it:

aws ec2 create-launch-template-version --launch-template-id lt-0abc \
  --source-version '$Latest' \
  --launch-template-data '{"MetadataOptions":{"HttpTokens":"required","HttpPutResponseHopLimit":2,"HttpEndpoint":"enabled"}}'
aws ec2 modify-launch-template --launch-template-id lt-0abc --default-version '$Latest'

Then roll the ASG (instance refresh) so running instances pick it up, or modify them directly as above.

Warning

Hop limit 2 lets every container on the host use the instance role. On shared nodes that is often more access than you want. Treat it as a stop-gap and move to per-workload roles (Step 2b).

Step 2b: Give workloads their own role (proper fix)

Platform Use The SDK reads credentials from
ECS on EC2 Task role (taskRoleArn) ECS agent endpoint 169.254.170.2 via AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
EKS EKS Pod Identity or IRSA Pod Identity agent / web identity token file
Plain Docker Pass credentials from a sidecar or use host networking for one trusted container —

With task roles or Pod Identity, containers never need IMDS, and you can keep hop limit 1 to block containers from the node role. For ECS tasks in awsvpc mode, you can also set ECS_AWSVPC_BLOCK_IMDS=true in /etc/ecs/ecs.config to block tasks from IMDS entirely; for bridge mode, the ECS documentation describes an iptables rule that does the same. Check the ECS agent configuration docs for your agent version.

Check what credentials a container is actually using:

aws sts get-caller-identity
env | grep -E 'AWS_CONTAINER_CREDENTIALS|AWS_WEB_IDENTITY|AWS_ROLE_ARN'

Step 3: Other causes that look the same

Cause How to spot it Fix
HttpEndpoint=disabled MetadataOptions shows it Enable, or move to task roles
Proxy env vars in the container HTTP_PROXY set, no NO_PROXY for 169.254.169.254 Add 169.254.169.254,169.254.170.2 to NO_PROXY
iptables rule blocking metadata iptables -t nat -L -n shows a DROP/REDIRECT for 169.254.169.254 Intended by your platform? Then use task/pod roles
Kubernetes NetworkPolicy blocks egress to 169.254.169.254 Policy egress rules Allow it, or use Pod Identity
Old SDK that only speaks IMDSv1 SDK version older than IMDSv2 support Upgrade the SDK/CLI
No role attached to the instance iam/security-credentials/ is empty from the host too Attach an instance profile

Check SDK minimum versions in the AWS IMDSv2 documentation before blaming the hop limit on very old images.

Step 4: Find who still needs IMDSv1 before you enforce

Before switching more fleets to required, look at the MetadataNoToken CloudWatch metric per instance. Non-zero means something still makes IMDSv1 calls:

aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name MetadataNoToken \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --start-time $(date -u -d '-1 day' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 3600 --statistics Sum

Prevent it next time

  • Roll out IMDSv2 in this order: watch MetadataNoToken, upgrade SDKs, move containers to task/pod roles, then enforce required.
  • Set the hop limit explicitly in every launch template (1 for VMs only, 2 only where containers truly need the node role).
  • Use an account-level default (aws ec2 modify-instance-metadata-defaults) so new instances launch with IMDSv2 and the hop limit you chose.
  • Add a container start-up check that calls sts get-caller-identity and fails fast with a clear message.

Related fixes