Home / AWS

AWSRunbookIntermediate

Sev1: SQS backlog growing and dead-letter queue filling — find the poison, drain safely, redrive

Handle an SQS incident where messages pile up and land in the DLQ. Read the queue metrics, separate poison messages from a broken consumer, fix visibility timeout and batch settings, and redrive the DLQ without duplicating work.

Deepak Bhardwaj2 min read

Applies toAmazon SQS standard and FIFO queues, Lambda event source mappings, ECS/EC2 consumers, AWS CLI v2

Symptoms

  • ApproximateNumberOfMessagesVisible and ApproximateAgeOfOldestMessage keep rising.
  • The DLQ count grows; orders, emails or jobs are delayed.

1. Read the queue

Q=https://sqs.eu-west-1.amazonaws.com/111122223333/orders
aws sqs get-queue-attributes --queue-url $Q --attribute-names All \
  --query "Attributes.{visible:ApproximateNumberOfMessages, inFlight:ApproximateNumberOfMessagesNotVisible, vt:VisibilityTimeout, redrive:RedrivePolicy}"
Pattern Meaning
Visible rising, in-flight low Consumers not polling (down, scaled to zero, permissions)
In-flight high, visible rising, DLQ growing Consumers receive but fail — bug, dependency down, poison messages
In-flight high, duplicates processed Visibility timeout shorter than processing time
DLQ growing, main queue normal A subset of messages always fails (poison)

2. Look at a DLQ message (without consuming it for good)

DLQ=https://sqs.eu-west-1.amazonaws.com/111122223333/orders-dlq
aws sqs receive-message --queue-url $DLQ --max-number-of-messages 5 --visibility-timeout 30 \
  --attribute-names All --message-attribute-names All

Messages become visible again after 30 seconds. Correlate their IDs with consumer logs to find the exception. Common causes: schema change by the producer, a record that references a deleted entity, payload too large for a downstream API.

3. Fix the consumer

Lambda consumers

  • Visibility timeout on the queue should be at least 6× the function timeout (AWS guidance) so retries don't overlap.
  • Use partial batch responses (ReportBatchItemFailures) so one bad message doesn't fail the whole batch and get good ones retried.
aws lambda update-event-source-mapping --uuid <uuid> --function-response-types ReportBatchItemFailures
aws sqs set-queue-attributes --queue-url $Q --attributes VisibilityTimeout=360
  • If a downstream dependency is down, reduce pressure: lower the mapping's MaximumConcurrency rather than letting everything fail into the DLQ.

Container/EC2 consumers

  • Check the worker count, crash loops and IAM permissions (sqs:ReceiveMessage, sqs:DeleteMessage, KMS Decrypt for encrypted queues).

4. Redrive the DLQ

After the fix is deployed:

aws sqs start-message-move-task --source-arn arn:aws:sqs:eu-west-1:111122223333:orders-dlq \
  --max-number-of-messages-per-second 50
aws sqs list-message-move-tasks --source-arn arn:aws:sqs:eu-west-1:111122223333:orders-dlq

Without a destination, messages go back to their original source queue. A rate limit protects downstream systems.

Warning

Redriving replays work. Make sure consumers are idempotent (dedupe on a business key or message ID) before you redrive orders, payments or emails.

5. Poison messages that can never succeed

Move them aside to an archive (S3) with their attributes, then delete from the DLQ. Raise a bug with the producer team.

Prevent it next time

  • Alarms on ApproximateAgeOfOldestMessage (main queue) and on any message in the DLQ.
  • maxReceiveCount of 3–5 in the redrive policy; DLQ retention longer than the main queue.
  • Idempotent consumers and schema validation at the producer.

Related fixes