KMS AccessDeniedException and disabled keys: fixing encrypted resource failures

#Symptoms that are really KMS

aws ec2 describe-instances --instance-ids i-0abc --query "Reservations[].Instances[].StateReason"

#1. What state is the key in?

aws kms describe-key --key-id arn:aws:kms:ap-south-1:111122223333:key/1234abcd-... \
  --query "KeyMetadata.{state:KeyState,enabled:Enabled,manager:KeyManager,deletion:DeletionDate}"

#2. Key policy (the primary control)

aws kms get-key-policy --key-id <id> --policy-name default --output text | python3 -m json.tool

• The key policy must allow the principal, or delegate to IAM with the statement allowing the account root (arn:aws:iam::111122223333:root) kms:*. Only then do IAM policies count.
• For EBS through Auto Scaling, the key must allow the AWSServiceRoleForAutoScaling role to CreateGrant, Decrypt, GenerateDataKey*, ReEncrypt* and DescribeKey.

#3. Find the denied call

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=kms.amazonaws.com \
  --max-results 50 --query "Events[].CloudTrailEvent" --output text \
  | jq -r 'select(.errorCode != null) | [.eventTime, .eventName, .userIdentity.arn, .errorCode] | @tsv'

The userIdentity.arn is often a service role you didn't expect (EBS, Auto Scaling, RDS) acting on the user's behalf.

#4. Cross-account use

Both sides must allow it:

1. The key policy in account A allows account B's principal (or the account B root).
2. The IAM policy in account B allows kms:Decrypt and the other needed actions on the key ARN.
3. AWS-managed keys (aws/ebs, aws/s3) can't be shared across accounts. Re-encrypt with a customer managed key before sharing snapshots or AMIs.

#5. Conditions that silently deny

• kms:ViaService limits use to a specific service, for example s3.ap-south-1.amazonaws.com.
• kms:EncryptionContext:* must match exactly what the service sends.
• aws:SourceVpce in the key policy blocks calls that don't come through the VPC endpoint.

#Prevent it next time

• Protect keys from accidental deletion: limit kms:ScheduleKeyDeletion with SCPs, and use the maximum 30-day waiting period.
• Alarm (EventBridge) on DisableKey and ScheduleKeyDeletion events.
• Keep key policies in IaC, next to the resources that use them.