DynamoDB global tables production checklist: MREC vs MRSC, write conflicts, replication latency and regional failover

#Who this is for

You're adding regions to a DynamoDB table, or you've inherited a global table and need to know whether it will survive a region outage without losing or corrupting data. Work through each section before you send production writes to more than one region.

Warning

Global tables replicate data, not intent. Two regions writing the same item concurrently in multi-Region eventual consistency (MREC) mode won't raise an error. One write silently replaces the other. Most global table incidents are application design problems, not replication failures.

#1. Choose the consistency mode deliberately

• Ledger-style data that can't tolerate any loss: MRSC, provided you can live with the latency and the missing features.
• Everything else: MREC with a write-ownership design (section 2).
• Feature support for MRSC is still changing. Check the current documentation for your regions before you commit to it.

#2. Design writes for last-writer-wins (MREC)

• Single writer per item. Route all writes for a given key to one "home" region (user's region, tenant's region) and use the other replicas for reads and failover. This removes conflicts entirely.
• No cross-region read-modify-write. A ConditionExpression is evaluated only against the local replica. Two regions can both pass attribute_not_exists(pk) and both "create" the item.
• No counters incremented from several regions. ADD and SET x = x + 1 from two regions in the same window lose increments, because the whole item is replaced. Use per-region counter items and sum them when you read.
• Transactions are regional. Another region can briefly see some items of a TransactWriteItems and not others. Don't make cross-item invariants depend on reads in a different region.
• Item-level version attribute. Write version and updatedRegion yourself. The 2019.11.21 version doesn't add the aws:rep:* attributes that the legacy version did, so without your own fields you can't tell after the fact which write won.

#3. Keep capacity and settings in step

• Every replica receives every write, including replicated writes. Size write capacity for global write traffic, not regional traffic.
• With provisioned mode, enable auto scaling for write capacity. DynamoDB keeps write capacity settings in step across replicas, but read capacity and GSI read capacity are per replica. Check each region.
• A throttled replica falls behind on replication. Alarm on WriteThrottleEvents in every region, not only the primary.
• GSIs, TTL, stream settings, encryption key choice and point-in-time recovery: confirm each replica is configured as you expect.

aws dynamodb describe-table --table-name orders --region us-east-1 \
  --query 'Table.{Replicas:Replicas[].{Region:RegionName,Status:ReplicaStatus,KMS:KMSMasterKeyId},Billing:BillingModeSummary.BillingMode,Stream:StreamSpecification}'

for R in us-east-1 us-west-2 eu-west-1; do
  echo "== $R"
  aws dynamodb describe-continuous-backups --table-name orders --region $R \
    --query 'ContinuousBackupsDescription.PointInTimeRecoveryDescription.PointInTimeRecoveryStatus'
done

Caution

Point-in-time recovery restores to a new single-region table, not to the global table. Restoring after a bad write that has already replicated everywhere means: restore, verify, then copy the corrected items back into the live global table. Write that runbook before you need it.

#4. Adding and removing replicas safely

# Add a replica (MREC). Existing data is backfilled; wait for ACTIVE before routing traffic.
aws dynamodb update-table --table-name orders --region us-east-1 \
  --replica-updates '[{"Create":{"RegionName":"eu-west-1"}}]'

aws dynamodb wait table-exists --table-name orders --region eu-west-1
aws dynamodb describe-table --table-name orders --region us-east-1 \
  --query 'Table.Replicas[].[RegionName,ReplicaStatus]' --output table

• Don't route traffic to a new replica until ReplicaStatus is ACTIVE and backfill has finished. Reads during backfill can return missing items.
• Removing a replica ({"Delete":{"RegionName":"..."}}) deletes that region's table. Treat it like delete-table and use change control.
• Turn on deletion protection for every replica: aws dynamodb update-table --table-name orders --deletion-protection-enabled --region <r>.
• IAM: anyone who can create or remove replicas needs the matching dynamodb:CreateTableReplica / dynamodb:DeleteTableReplica permissions (plus dynamodb:CreateGlobalTableWitness for MRSC witnesses). Restrict those permissions with SCPs.

#5. Monitor replication

• Alarm on ReplicationLatency (namespace AWS/DynamoDB, dimensions TableName and ReceivingRegion) in each source region. Set a baseline first. Normal values are usually around a second, and alarm on sustained deviation.
• PendingReplicationCount applies only to the legacy 2017.11.29 version. Don't build alarms on it for current tables.

aws cloudwatch put-metric-alarm --region us-east-1 \
  --alarm-name ddb-orders-replication-latency-euw1 \
  --namespace AWS/DynamoDB --metric-name ReplicationLatency \
  --dimensions Name=TableName,Value=orders Name=ReceivingRegion,Value=eu-west-1 \
  --statistic Average --period 60 --evaluation-periods 5 --datapoints-to-alarm 4 \
  --threshold 30000 --comparison-operator GreaterThanThreshold \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:ddb-oncall

treat-missing-data breaching matters here. If replication stops completely, the metric can stop publishing, and the default missing treatment would leave the alarm quiet.

#6. Failover and failback

• Failover is your routing change, not a DynamoDB operation. DynamoDB has no "promote" step. Every MREC replica is already writable. Failover means moving the application's write traffic (Route 53, config, ARC routing controls).
• Decide in advance what happens to writes that were in flight in the failed region. With MREC they arrive whenever the region recovers, and last-writer-wins applies. Writes made in the surviving region during the outage normally carry later timestamps, so they win.
• Rehearse a replication outage in a test account. AWS Fault Injection Service has a DynamoDB action that pauses global table replication to one replica. Check the current FIS action list for its name and supported modes.
• Failback: wait until ReplicationLatency from the recovered region is back to its baseline before you move writes back.
• Consumers of DynamoDB Streams in each region see replicated writes as well as local ones. Make sure Lambda triggers don't run side effects (emails, payments) once per region. Filter on an updatedRegion attribute you write yourself.

#7. Before go-live sign-off

• Consistency mode chosen and recorded with the reason.
• Write ownership per key documented. No multi-region counters or conditional creates.
• Capacity, GSIs, PITR and deletion protection checked in every region.
• Replication latency alarms in every source region, with missing data treated as breaching.
• Failover and failback rehearsed, with timings recorded.
• Restore-from-PITR-and-copy-back runbook tested.