Home / AWS

AWSTroubleshootingAdvanced

Sev1: after an Aurora failover apps still write to the old instance — DNS caching, endpoints and read-only errors

Aurora failed over in seconds but the app keeps failing with read-only or connection errors for many minutes. Fix endpoint usage, DNS caching and connection pools so failovers become short blips.

Deepak Bhardwaj3 min read

Applies toAmazon Aurora MySQL and PostgreSQL, RDS Proxy, JDBC/psycopg/mysql drivers

Symptoms

  • After a failover event, the app logs The MySQL server is running with the --read-only option or PostgreSQL cannot execute INSERT in a read-only transaction.
  • Some app instances recover, others keep failing until restarted.

1. Confirm the failover and the new writer

aws rds describe-events --source-type db-cluster --source-identifier aurora-prod --duration 120 \
  --query "Events[].{t:Date, m:Message}" --output table
aws rds describe-db-clusters --db-cluster-identifier aurora-prod \
  --query "DBClusters[0].{writer:DBClusterMembers[?IsClusterWriter].DBInstanceIdentifier, endpoint:Endpoint, reader:ReaderEndpoint}"

2. Check which endpoint the app uses

App connects to What happens on failover
Cluster endpoint (aurora-prod.cluster-xxx) DNS repoints to new writer; OK if DNS is not cached
Instance endpoint (aurora-prod-instance-1.xxx) Keeps pointing at the old writer, now a reader. Wrong
Reader endpoint for writes Always fails writes
RDS Proxy endpoint Proxy handles failover; fastest recovery

Find the endpoint in the app's config or secret. If it is an instance endpoint, switch it to the cluster endpoint now.

3. DNS caching

The cluster endpoint is a CNAME with a short TTL (about 5 seconds). Clients that cache DNS longer keep connecting to the old IP.

  • Java: the JVM may cache DNS indefinitely with a security manager, otherwise for 30 seconds by default. Set:
networkaddress.cache.ttl=5

in java.security, or -Dsun.net.inetaddr.ttl=5 on older JVMs.

  • Connection pools keep existing connections open to the old writer. Even with correct DNS, those connections now point to a reader.

Quick check from an app host:

dig +short aurora-prod.cluster-xxxx.eu-west-1.rds.amazonaws.com

4. Make the pool drop bad connections

Immediate relief: restart app instances or recycle the pool. Permanent fix:

  • Validate connections and treat read-only errors as fatal. For MySQL, the AWS Advanced JDBC Wrapper (or the MariaDB Connector/J Aurora mode in older stacks) understands Aurora topology. For PostgreSQL, the AWS Advanced JDBC Wrapper or target_session_attrs=read-write in libpq-based drivers rejects connections to readers:
postgresql://app@aurora-prod.cluster-xxxx:5432/orders?target_session_attrs=read-write
  • Set a max connection lifetime (for example 5-10 minutes) so stale connections are recycled.
  • Retry transactions on failover-type errors with backoff.

5. Put RDS Proxy in front

aws rds create-db-proxy --db-proxy-name aurora-prod-proxy --engine-family POSTGRESQL \
  --auth "AuthScheme=SECRETS,SecretArn=arn:aws:secretsmanager:...:secret:aurora-app,IAMAuth=DISABLED" \
  --role-arn arn:aws:iam::111122223333:role/rds-proxy-secrets \
  --vpc-subnet-ids subnet-a subnet-b --vpc-security-group-ids sg-0proxy
aws rds register-db-proxy-targets --db-proxy-name aurora-prod-proxy --db-cluster-identifiers aurora-prod

The proxy keeps client connections open and reroutes to the new writer, which usually cuts failover impact to seconds.

Tip

Test before you need it: aws rds failover-db-cluster --db-cluster-identifier aurora-staging in staging during load, and measure how long the app errors.

6. If failovers keep happening

Repeated failovers point at an underlying issue: memory pressure on the writer, long-running transactions blocking recovery, or instance class too small. Check FreeableMemory, CPUUtilization and the error log events around each failover.

Prevent it next time

  • Always use the cluster endpoint (or RDS Proxy) for writes.
  • Short DNS TTL in clients, connection max lifetime, and read-only-error detection in pools.
  • Set failover priority tiers so the right reader is promoted.
  • Run a failover game day each quarter.

Related fixes