SQS FIFO in depth: message groups, deduplication, high-throughput mode and ordering without bottlenecks

FIFO queues promise ordering and exactly-once processing within a deduplication window. Teams often get one of three things wrong: they use a single message group (no parallelism), they rely on deduplication they never configured, or a single failing message stalls everything behind it.

#Ordering is per message group

Order is guaranteed only within a MessageGroupId. Messages in different groups can be processed in parallel. While a message in a group is in flight, no other message from that group is delivered.

Pick the narrowest entity whose events must stay in order.

#Deduplication

Within a 5-minute window, SQS drops messages with a repeated MessageDeduplicationId.

• Content-based deduplication: SQS hashes the body. Two different events with identical bodies (for example two "+1 stock" messages) get deduplicated by mistake.
• Explicit IDs: use the producer's event ID. Safer.

aws sqs send-message --queue-url $Q \
  --message-group-id customer-1042 \
  --message-deduplication-id evt-7f3a2c \
  --message-body '{"type":"address_changed","customer":1042}'

Warning

Deduplication only covers 5 minutes. A producer that retries after a long outage can still send a duplicate. Consumers must be idempotent anyway.

#High-throughput mode

FIFO queues have lower default throughput than standard queues. High-throughput mode raises it per partition, as long as traffic spreads over many message groups.

aws sqs set-queue-attributes --queue-url $Q --attributes \
  DeduplicationScope=messageGroup,FifoThroughputLimit=perMessageGroupId

With DeduplicationScope=messageGroup, dedup IDs only need to be unique within a group. Batching (SendMessageBatch, up to 10 messages) multiplies effective throughput. Check the current quota table for your region rather than assuming a number.

#Lambda consumers

The event source mapping receives batches and keeps per-group order: it processes groups in parallel, but one batch per group at a time.

aws lambda create-event-source-mapping --function-name process-orders \
  --event-source-arn arn:aws:sqs:eu-west-1:111111111111:orders.fifo \
  --batch-size 10 --function-response-types ReportBatchItemFailures \
  --scaling-config MaximumConcurrency=50

Report partial failures so successful messages are deleted:

def handler(event, context):
    failures = []
    for i, record in enumerate(event["Records"]):
        try:
            process(record)
        except Exception:
            # FIFO: stop at the first failure and return it plus everything after it in this batch,
            # so later messages in the same group are not processed out of order
            failures = [{"itemIdentifier": r["messageId"]} for r in event["Records"][i:]]
            break
    return {"batchItemFailures": failures}

#The poison message problem

A message that always fails stays at the head of its group. Every retry blocks the group until it reaches maxReceiveCount and moves to the DLQ. Set a redrive policy with a FIFO DLQ:

aws sqs set-queue-attributes --queue-url $Q --attributes '{
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:111111111111:orders-dlq.fifo\",\"maxReceiveCount\":\"5\"}",
  "VisibilityTimeout": "180"
}'

Moving a message to the DLQ breaks ordering for that group: later messages proceed. Decide whether that is acceptable. If not, the consumer should park the whole group (record it as blocked) and alert.

Visibility timeout should be at least six times the Lambda timeout when using Lambda, so in-flight messages are not redelivered while still processing.

#Monitor

aws cloudwatch get-metric-statistics --namespace AWS/SQS --metric-name ApproximateAgeOfOldestMessage \
  --dimensions Name=QueueName,Value=orders.fifo --statistics Maximum --period 60 \
  --start-time $(date -u -d '-1 hour' +%FT%TZ) --end-time $(date -u +%FT%TZ)

#Redrive from the DLQ

aws sqs start-message-move-task --source-arn arn:aws:sqs:eu-west-1:111111111111:orders-dlq.fifo \
  --destination-arn arn:aws:sqs:eu-west-1:111111111111:orders.fifo

Fix the cause first; redriving into the same bug just cycles the messages.

#Prevent it next time

• Document the message group key and why; review it when throughput needs grow.
• Explicit deduplication IDs from producers, idempotent consumers.
• DLQ on every FIFO queue, with an alarm and a runbook for whether to unblock or hold a group.