Home / AWS

AWSTroubleshootingAdvanced

Secrets Manager rotation failed: Lambda, network, permissions and stuck AWSPENDING

Diagnose a failed Secrets Manager rotation step by step, clear a stuck AWSPENDING version safely, and make sure applications are still using a working password.

Deepak Bhardwaj6 min read

Applies toAWS Secrets Manager with Lambda rotation functions (RDS, Aurora, Redshift, DocumentDB and custom secrets), AWS CLI v2

Symptoms

  • The console shows Last rotation failed, or an EventBridge rule reports a RotationFailed event.
  • Applications start failing with password authentication failed or Access denied for user right after a rotation window.
  • rotate-secret returns A previous rotation isn't complete. That rotation will be reattempted.

First: is production broken?

A failed rotation does not always break anything. Secrets Manager only moves the AWSCURRENT label in the final step, so if rotation failed earlier, AWSCURRENT usually still holds the old, working password. Check that before you change anything:

SECRET=prod/app/db

aws secretsmanager describe-secret --secret-id $SECRET \
  --query "{enabled:RotationEnabled,lambda:RotationLambdaARN,rules:RotationRules,last:LastRotatedDate,next:NextRotationDate,stages:VersionIdsToStages}"

Then test that AWSCURRENT actually logs in (PostgreSQL example; use mysql or sqlcmd for other engines):

aws secretsmanager get-secret-value --secret-id $SECRET --version-stage AWSCURRENT \
  --query SecretString --output text > /tmp/s.json
PGPASSWORD=$(jq -r .password /tmp/s.json) psql -h "$(jq -r .host /tmp/s.json)" \
  -U "$(jq -r .username /tmp/s.json)" -d postgres -c "select 1"
rm /tmp/s.json

Caution

If AWSCURRENT does not log in, the database password and the secret no longer match, and applications will fail as their connection pools reconnect. Reset the database password to the value in AWSCURRENT (for RDS: aws rds modify-db-instance --db-instance-identifier <id> --master-user-password '<value>' --apply-immediately), or move AWSCURRENT back to the version that works (see "Clear a stuck rotation"). Fix the outage first, then the rotation.

How rotation works

The rotation Lambda function is called four times, once per step. Its CloudWatch log shows which step failed:

Step What it does Typical failure
createSecret Generates a new password and stores it as AWSPENDING Can't reach the Secrets Manager endpoint; KMS denied
setSecret Changes the password in the database Can't reach the database; user lacks privilege; password breaks a character rule
testSecret Logs in with AWSPENDING Wrong host/port in the secret; TLS required but not configured
finishSecret Moves AWSCURRENT to the new version Rarely fails; check for timeouts 
FN=$(aws secretsmanager describe-secret --secret-id $SECRET --query RotationLambdaARN --output text)
aws logs tail /aws/lambda/${FN##*:} --since 24h --format short

You can also see the failure event in CloudTrail:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=RotationFailed \
  --max-results 10 --query "Events[].CloudTrailEvent" --output text

Symptom → cause → fix

What you see Cause Fix
Secrets Manager can't invoke the specified Lambda function The function's resource policy doesn't allow secretsmanager.amazonaws.com Add the permission (below)
Log stops after Found credentials in environment variables, then Task timed out Function is in a VPC with no route to Secrets Manager Add a secretsmanager interface endpoint or a NAT route
Timeout in setSecret or testSecret Function security group can't reach the database port Allow the function's SG on the DB SG inbound rule
AccessDeniedException on GetSecretValue / PutSecretValue Execution role missing Secrets Manager actions Grant the actions on the secret ARN
KMS ... AccessDeniedException Secret uses a customer managed key the role can't use Allow kms:Decrypt and kms:GenerateDataKey in the key policy or IAM
password authentication failed in setSecret The current password in the secret is already wrong Fix the mismatch first (see above)
permission denied on ALTER USER Alternating-users strategy, but the superuser secret is wrong or lacks privilege Check the secret referenced by masterarn
A previous rotation isn't complete An AWSPENDING version from a failed run is still there Fix the cause, then clear it (below)
Rotation fails only some of the time Function hits its Lambda timeout, or is throttled by a low concurrency limit Raise the timeout; check the function's reserved concurrency and account concurrency

Lambda resource policy

aws lambda add-permission --function-name ${FN##*:} \
  --statement-id SecretsManagerInvoke \
  --action lambda:InvokeFunction \
  --principal secretsmanager.amazonaws.com \
  --source-account 111122223333

Network path from a VPC function

aws lambda get-function-configuration --function-name ${FN##*:} \
  --query "{vpc:VpcConfig,timeout:Timeout}"

aws ec2 describe-vpc-endpoints \
  --filters Name=service-name,Values=com.amazonaws.eu-west-1.secretsmanager \
  --query "VpcEndpoints[].{id:VpcEndpointId,vpc:VpcId,dns:PrivateDnsEnabled,sgs:Groups[].GroupId}"

The endpoint needs private DNS turned on, and its security group must allow inbound 443 from the function's subnets or security group. Run the Reachability Analyzer from the function's ENI to the database ENI if the problem is the database path.

Clear a stuck rotation

When the cause is fixed, remove the leftover AWSPENDING label and rotate again. Get the pending version ID first:

aws secretsmanager list-secret-version-ids --secret-id $SECRET \
  --query "Versions[].{id:VersionId,stages:VersionStages,created:CreatedDate}" --output table

aws secretsmanager update-secret-version-stage --secret-id $SECRET \
  --version-stage AWSPENDING --remove-from-version-id <pending-version-id>

aws secretsmanager rotate-secret --secret-id $SECRET

Warning

If setSecret succeeded but testSecret or finishSecret failed, the database may already have the AWSPENDING password. Test the AWSPENDING value (--version-stage AWSPENDING) before you remove it. If it works and AWSCURRENT doesn't, promote it instead:

aws secretsmanager update-secret-version-stage --secret-id $SECRET \
  --version-stage AWSCURRENT --move-to-version-id <pending-version-id> \
  --remove-from-version-id <current-version-id>

Watch the new run in the function's logs. All four steps should log and finish within a few seconds.

RDS-managed master passwords are different

If the instance uses --manage-master-user-password, RDS owns the secret and its rotation. There is no rotation Lambda for you to fix. Don't edit that secret by hand. Check the RDS events and the secret's OwningService:

aws secretsmanager describe-secret --secret-id $SECRET --query OwningService
aws rds describe-events --source-type db-instance --source-identifier prod-db --duration 1440

Prevent it next time

  • Create an EventBridge rule on Secrets Manager RotationFailed events (delivered via CloudTrail) and send it to your on-call channel, so you know before the old password ages out.
  • Use the alternating-users strategy for application credentials, so a failed rotation never invalidates the login that's in use.
  • Make applications read the secret on authentication failure and reconnect, rather than caching the password until restart.
  • Keep the rotation function's timeout at 30 seconds or more and give it a VPC endpoint rather than relying on a NAT gateway.
  • After any change to security groups, KMS key policies or the database user, trigger rotate-secret in a test environment first.

Related fixes