Home / AWS

AWSRunbookAdvanced

Sev1: KMS key disabled or scheduled for deletion — encrypted data suddenly unreadable

Restore access to EBS, S3, RDS and Secrets Manager data when a customer managed KMS key is disabled or pending deletion. Cancel the deletion, re-enable the key, find who did it and stop it happening again.

Deepak Bhardwaj2 min read

Applies toAWS KMS customer managed keys, Amazon EBS, S3 (SSE-KMS), RDS, Secrets Manager, AWS CLI v2

Symptoms

  • KMSInvalidStateException, DisabledException or KMS key is pending deletion.
  • EC2 instances with encrypted volumes fail to start; S3 GetObject on SSE-KMS objects fails; RDS goes to inaccessible-encryption-credentials.
  • Secrets Manager GetSecretValue fails for secrets encrypted with that key.

1. Check the key state

aws kms describe-key --key-id arn:aws:kms:eu-west-1:111122223333:key/1234abcd-... \
  --query "KeyMetadata.{state:KeyState, enabled:Enabled, deletionDate:DeletionDate, manager:KeyManager, multiRegion:MultiRegion}"
KeyState Fix
Disabled Enable it (Step 3)
PendingDeletion Cancel deletion, then enable (Step 2)
PendingImport Imported key material expired or deleted — re-import the same material
Not found Deleted. Data encrypted only under this key is unrecoverable

2. Cancel a scheduled deletion

aws kms cancel-key-deletion --key-id 1234abcd-...

After cancelling, the key is Disabled — enable it next. You can only cancel during the waiting period (7–30 days, set when deletion was scheduled).

3. Enable the key

aws kms enable-key --key-id 1234abcd-...

4. Check the key policy still allows your services

If someone edited the key policy, services can fail even with an enabled key.

aws kms get-key-policy --key-id 1234abcd-... --policy-name default --output text

Make sure the account root principal statement (which lets IAM policies grant access) is still present, and that service-linked or service roles (EBS via EC2, RDS, Auto Scaling's service-linked role for encrypted launch templates) have the needed kms:Decrypt, kms:GenerateDataKey* and kms:CreateGrant permissions.

5. Restore dependent services

  • RDS: an instance in inaccessible-encryption-credentials may need to be restored from a backup once the key is available again; check the instance status and the RDS documentation for your engine.
  • EC2: start instances that failed; for Auto Scaling, check activities for KMS errors.
  • S3, Secrets Manager: retry; access returns once the key is usable.

6. Who did it

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=ScheduleKeyDeletion --max-results 20
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=DisableKey --max-results 20

Prevent it next time

  • EventBridge rule / CloudWatch alarm on ScheduleKeyDeletion and DisableKey to on-call — AWS documents this pattern.
  • SCP that denies kms:ScheduleKeyDeletion and kms:DisableKey except to a security break-glass role.
  • Use a 30-day deletion window by default.
  • Tag keys with owner and the data they protect; never "clean up" keys without checking usage (kms:Decrypt events in CloudTrail).

Related fixes