Home / AWS

AWSRunbookAdvanced

Sev1: Secrets Manager rotation broke database logins — apps fail auth after a rotation

Applications fail with password authentication errors after an automatic rotation. Check rotation state and staging labels, fix the stuck version, restore a working password and stop apps caching old secrets.

Deepak Bhardwaj3 min read

Applies toAWS Secrets Manager with RDS/Aurora rotation Lambda functions, AWS CLI v2

Symptoms

  • password authentication failed for user "app" (PostgreSQL) or Access denied for user (MySQL) across many app instances.
  • Started at the rotation schedule time.
  • Secrets Manager console shows a rotation failure, or AWSPENDING stuck on a version.

1. Look at versions and labels

aws secretsmanager describe-secret --secret-id prod/orders/db \
  --query "{rotation:RotationEnabled, last:LastRotatedDate, versions:VersionIdsToStages}"

Healthy state: one version with AWSCURRENT, maybe one with AWSPREVIOUS. A version labelled AWSPENDING means rotation started and did not finish.

2. Read the rotation function logs

FN=$(aws secretsmanager describe-secret --secret-id prod/orders/db --query RotationLambdaARN --output text)
aws logs tail /aws/lambda/${FN##*:} --since 3h | grep -iE "error|fail|step"

The rotation runs four steps: createSecret, setSecret, testSecret, finishSecret. Where it failed tells you the state:

Failed step Database password AWSCURRENT Effect
createSecret Unchanged Old, correct Usually no outage
setSecret Maybe unchanged Old No outage unless partially applied
testSecret Changed to pending Old (now wrong) Apps using AWSCURRENT fail
finishSecret Changed Old (wrong) Apps fail

Common causes: the Lambda can't reach the database (VPC, security group, no Secrets Manager VPC endpoint or NAT), the master user secret is wrong, or the user lacks permission to change its own password.

3. Restore a working login quickly

Option A, the pending value is already in the database (failed at testSecret/finishSecret): promote the pending version.

aws secretsmanager update-secret-version-stage --secret-id prod/orders/db \
  --version-stage AWSCURRENT --move-to-version-id <pending-version-id> --remove-from-version-id <current-version-id>

Option B, unsure what the database has: set the database password to the AWSCURRENT value using the master account.

CUR=$(aws secretsmanager get-secret-value --secret-id prod/orders/db --query SecretString --output text)
PW=$(echo "$CUR" | jq -r .password)
psql "host=aurora-prod.cluster-xxx user=postgres dbname=orders" \
  -c "ALTER USER app WITH PASSWORD '$PW';"

Caution

Don't paste passwords into shared terminals or tickets. Run the command from a secure admin host and clear shell history if your process requires it.

Then remove the stuck pending label so the next rotation can start cleanly:

aws secretsmanager update-secret-version-stage --secret-id prod/orders/db \
  --version-stage AWSPENDING --remove-from-version-id <pending-version-id>

4. Pause rotation until fixed

aws secretsmanager cancel-rotate-secret --secret-id prod/orders/db

Fix the Lambda network path or permissions, then test with an immediate rotation in a quiet period:

aws secretsmanager rotate-secret --secret-id prod/orders/db

5. Fix the apps

  • Apps that read the secret once at startup and cache forever fail after every rotation. Use the AWS secrets caching libraries with a refresh interval, and on an auth error refresh the secret and retry once.
  • For zero-downtime rotation use the alternating users strategy, so one user stays valid while the other changes.
  • Better still for RDS/Aurora: IAM database authentication or RDS-managed master passwords where they fit.

Prevent it next time

  • Alarm on rotation failures (EventBridge rule on Secrets Manager RotationFailed events).
  • Give the rotation Lambda a VPC endpoint for Secrets Manager and access to the DB security group.
  • Rotate first in staging on the same schedule and code.
  • Choose alternating-user rotation for high-traffic apps.

Related fixes