Lambda with SQS: one bad message retries the whole batch - fix it with ReportBatchItemFailures

#Symptom

Messages get processed several times, ApproximateAgeOfOldestMessage keeps rising and good messages end up in the dead-letter queue alongside the bad one. Lambda's error rate looks small, yet the queue never drains.

#Why it happens

Lambda receives messages in batches. By default, if the function raises an exception or times out, all the messages in that batch return to the queue after the visibility timeout, including those it already handled. Each one's receive count goes up, so a good message that shares a batch with a poison one can reach maxReceiveCount and land in the DLQ.

#Step 1: check the current configuration

aws lambda list-event-source-mappings --function-name orders-worker \
  --query "EventSourceMappings[].{UUID:UUID,State:State,Batch:BatchSize,Window:MaximumBatchingWindowInSeconds,Resp:FunctionResponseTypes,Scale:ScalingConfig}"

aws lambda get-function-configuration --function-name orders-worker \
  --query "{Timeout:Timeout,Reserved:ReservedConcurrentExecutions}"

aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
  --attribute-names VisibilityTimeout RedrivePolicy ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

A low visibility timeout causes duplicates even with no failures: the message becomes visible again while the first invocation is still running.

#Step 2: return only the failed messages

Update the handler first, then turn on the setting.

import json
import logging

log = logging.getLogger()
log.setLevel(logging.INFO)

def handler(event, context):
    failures = []
    for record in event["Records"]:
        try:
            process(json.loads(record["body"]))
        except Exception:
            log.exception("failed messageId=%s", record["messageId"])
            failures.append({"itemIdentifier": record["messageId"]})
    return {"batchItemFailures": failures}

Turn it on:

aws lambda update-event-source-mapping \
  --uuid "$MAPPING_UUID" \
  --function-response-types ReportBatchItemFailures

Warning

The order matters. If you turn on the setting with the old handler, a function that returns nothing counts as complete success, but exceptions still fail the whole batch. Handler errors that are caught and only logged, with no batchItemFailures returned, will delete those messages.

How Lambda reads the response:

#Step 3: FIFO queues need different handling

To keep order, stop at the first failure and return that message and everything after it:

def handler(event, context):
    records = event["Records"]
    for i, record in enumerate(records):
        try:
            process(json.loads(record["body"]))
        except Exception:
            log.exception("failed messageId=%s", record["messageId"])
            return {"batchItemFailures": [
                {"itemIdentifier": r["messageId"]} for r in records[i:]
            ]}
    return {"batchItemFailures": []}

A poison message blocks its message group until it reaches the DLQ. Keep maxReceiveCount low enough that one bad message doesn't hold up that group for long.

#Step 4: set the timeouts and the limits

# Function timeout 60 s, so visibility timeout 360 s
aws sqs set-queue-attributes --queue-url "$QUEUE_URL" \
  --attributes VisibilityTimeout=360

aws sqs set-queue-attributes --queue-url "$QUEUE_URL" --attributes \
  '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"arn:aws:sqs:ap-south-1:111122223333:orders-dlq\",\"maxReceiveCount\":\"5\"}"}'

# Protect the database downstream
aws lambda update-event-source-mapping --uuid "$MAPPING_UUID" \
  --scaling-config MaximumConcurrency=20

Use MaximumConcurrency on the mapping to limit concurrency rather than a low reserved concurrency on the function. With reserved concurrency alone, the poller can get throttled by Lambda, messages return to the queue and receive counts rise for no real failure.

Batch time matters too. If a batch of 100 needs 2 seconds each, a 60-second timeout will fail every time. Keep batch size × P99 message time well inside the function timeout, or stop before the time runs out:

for i, record in enumerate(records):
    if context.get_remaining_time_in_millis() < 5000:
        return {"batchItemFailures": [{"itemIdentifier": r["messageId"]} for r in records[i:]]}

#Step 5: make the work idempotent

SQS standard queues deliver at least once, so partial batch responses reduce duplicates but don't remove them. Use a conditional write on a unique business key:

import boto3
from botocore.exceptions import ClientError

table = boto3.resource("dynamodb").Table("processed-orders")

def claim(order_id):
    try:
        table.put_item(
            Item={"order_id": order_id},
            ConditionExpression="attribute_not_exists(order_id)",
        )
        return True
    except ClientError as err:
        if err.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise

#Step 6: clear the backlog and the DLQ

Look at the messages first, and find the error in the logs:

aws logs start-query --log-group-name /aws/lambda/orders-worker \
  --start-time $(date -d '-1 hour' +%s) --end-time $(date +%s) \
  --query-string 'fields @timestamp, @message | filter @message like /failed messageId/ | sort @timestamp desc | limit 50'

After you've deployed the fix, move messages back to the source queue:

aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:ap-south-1:111122223333:orders-dlq \
  --destination-arn arn:aws:sqs:ap-south-1:111122223333:orders \
  --max-number-of-messages-per-second 10

aws sqs list-message-move-tasks \
  --source-arn arn:aws:sqs:ap-south-1:111122223333:orders-dlq

Caution

Redriving before the fix sends the same messages through the same failure, and may repeat their side effects. Redrive slowly and watch the first few minutes.

#Prevent it next time

• Alert on ApproximateAgeOfOldestMessage, and on ApproximateNumberOfMessagesVisible above zero on the DLQ.
• Add a test with one poison message in a batch, and check that only that one retries.
• Log messageId and approximateReceiveCount for each message.
• Related: SQS backlog and DLQ filling and account concurrency exhausted.