Sev1: Route 53 records deleted or changed and the site stops resolving — find the change and restore DNS

#Symptoms

• NXDOMAIN or SERVFAIL for production names; or the name resolves to an old or wrong address.
• Monitoring from several regions fails at the same time; the app itself is healthy.
• A Terraform apply, script or console edit ran against the hosted zone.

#1. See what the world sees

dig +short www.example.com
dig +short www.example.com @8.8.8.8
dig NS example.com +short
dig +trace www.example.com | tail -20

Ask Route 53 directly, bypassing caches:

ZONE=$(aws route53 list-hosted-zones-by-name --dns-name example.com \
  --query "HostedZones[0].Id" --output text)
aws route53 test-dns-answer --hosted-zone-id $ZONE --record-name www.example.com --record-type A

#2. Find the change

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=ChangeResourceRecordSets \
  --start-time "$(date -u -d '-24 hours' +%Y-%m-%dT%H:%M:%SZ)" \
  --query "Events[].{time:EventTime, user:Username}" --output table

Route 53 is a global service, so its CloudTrail events are recorded in us-east-1. Add --region us-east-1 if your default region differs. The full event (CloudTrailEvent) contains the exact change batch, including the deleted record values. That is your restore source.

aws cloudtrail lookup-events --region us-east-1 \
  --lookup-attributes AttributeKey=EventName,AttributeValue=ChangeResourceRecordSets \
  --max-results 5 --query "Events[].CloudTrailEvent" --output text | jq '.requestParameters.changeBatch'

Also check whether the hosted zone itself was deleted (DeleteHostedZone) or the domain's NS changed at the registrar (UpdateDomainNameservers in Route 53 Domains).

#3. Restore the records

Build a change batch with UPSERT, which is safe to rerun:

cat > restore.json <<'EOF'
{
  "Comment": "INC12345 restore www",
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "www.example.com",
      "Type": "A",
      "AliasTarget": {
        "HostedZoneId": "<alb-canonical-hosted-zone-id>",
        "DNSName": "dualstack.my-alb-123456.eu-west-1.elb.amazonaws.com",
        "EvaluateTargetHealth": true
      }
    }
  }]
}
EOF
aws route53 change-resource-record-sets --hosted-zone-id $ZONE --change-batch file://restore.json
aws route53 get-change --id <change-id>   # wait for INSYNC

For an ALB alias, get the canonical hosted zone ID from aws elbv2 describe-load-balancers.

#4. If the hosted zone was deleted

Recreating the zone gives you new name servers. You must then:

1. Recreate all records from IaC, an export or the CloudTrail change history.
2. Update the NS records at the registrar (or the parent zone for subdomains) to the new set.

aws route53 create-hosted-zone --name example.com --caller-reference inc12345-$(date +%s)
aws route53domains update-domain-nameservers --region us-east-1 --domain-name example.com \
  --nameservers Name=ns-1.awsdns-01.org Name=ns-2.awsdns-02.co.uk ...

Warning

Resolvers cache the old NS records for the parent zone's NS TTL (often 48 hours at TLD level). Recovery after a zone deletion can be slow. This is why deleting a production zone must be blocked.

#5. Verify and communicate

Check from several resolvers and give support teams the expected time for caches to clear (the record TTL).

#Prevent it next time

• Keep every record in IaC and export zones daily (aws route53 list-resource-record-sets to S3).
• Deny route53:DeleteHostedZone and limit ChangeResourceRecordSets on production zones with SCPs or IAM conditions.
• Use Route 53 health checks plus synthetic DNS monitoring from outside AWS.
• Turn on DNSSEC only with a documented key rollover runbook; a broken DNSSEC chain looks exactly like this outage.