Kinesis Data Streams consumers falling behind: iterator age, ReadProvisionedThroughputExceeded and Lambda event source mapping tuning

#Symptoms

• GetRecords.IteratorAgeMilliseconds (or the Lambda IteratorAge metric) rises steadily and doesn't recover after the traffic peak.
• Consumers log ProvisionedThroughputExceededException on GetRecords, or the stream shows ReadProvisionedThroughputExceeded above zero.
• Downstream data is minutes or hours stale. In the worst case records age past the retention period and are lost.

Caution

Records older than the stream's retention period are deleted whether or not anyone read them. If iterator age is getting close to retention (24 hours by default), raise retention first with aws kinesis increase-stream-retention-period and diagnose afterwards. Retention above the default is billed, so check current pricing.

#Find out where the bottleneck is

Iterator age is a symptom. The cause is one of four things, and each has a different fix.

#Stream-level metrics

STREAM=orders-stream
for M in GetRecords.IteratorAgeMilliseconds ReadProvisionedThroughputExceeded WriteProvisionedThroughputExceeded; do
  echo "== $M"
  aws cloudwatch get-metric-statistics --namespace AWS/Kinesis --metric-name "$M" \
    --dimensions Name=StreamName,Value=$STREAM \
    --start-time "$(date -u -d '-3 hours' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
    --period 300 --statistics Maximum \
    --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Maximum]' --output text
done

#Shard-level metrics (find the hot shard)

The stream-level metrics hide skew. Turn on enhanced shard-level monitoring. It's billed per shard metric, so turn it off once the incident is over.

aws kinesis enable-enhanced-monitoring --stream-name $STREAM \
  --shard-level-metrics IncomingBytes IncomingRecords IteratorAgeMilliseconds ReadProvisionedThroughputExceeded

Then compare IteratorAgeMilliseconds and IncomingBytes per ShardId in a CloudWatch Metrics Insights query:

SELECT MAX(IteratorAgeMilliseconds) FROM SCHEMA("AWS/Kinesis", ShardId, StreamName)
WHERE StreamName = 'orders-stream' GROUP BY ShardId ORDER BY MAX() DESC LIMIT 10

If one shard gets far more IncomingBytes than the rest, the partition key is skewed. Splitting that shard only helps if the hot traffic is spread across many keys. A single hot key always maps to one shard.

#Fix 1: give consumers their own read throughput

Each shard supports 2 MB/s of reads and 5 GetRecords calls per second, shared by every standard consumer. Three applications polling once a second each already use most of that. With enhanced fan-out, each registered consumer gets its own 2 MB/s per shard, pushed over HTTP/2 SubscribeToShard.

STREAM_ARN=$(aws kinesis describe-stream-summary --stream-name $STREAM \
  --query StreamDescriptionSummary.StreamARN --output text)
aws kinesis register-stream-consumer --stream-arn "$STREAM_ARN" --consumer-name billing-efo
aws kinesis list-stream-consumers --stream-arn "$STREAM_ARN"

For Lambda, pass the consumer ARN (not the stream ARN) as the event source of the event source mapping. For enhanced fan-out consumers, watch SubscribeToShardEvent.MillisBehindLatest instead of the GetRecords metrics. Enhanced fan-out is billed per consumer-shard hour and per GB read, so use it for the consumers that need low latency and keep batch consumers on shared throughput.

#Fix 2: tune the Lambda event source mapping

The defaults are risky in production. A failing batch is retried until the records expire, and the shard is blocked while that happens.

UUID=$(aws lambda list-event-source-mappings --function-name orders-consumer \
  --query 'EventSourceMappings[0].UUID' --output text)

aws lambda update-event-source-mapping --uuid "$UUID" \
  --parallelization-factor 4 \
  --batch-size 500 \
  --maximum-batching-window-in-seconds 1 \
  --bisect-batch-on-function-error \
  --maximum-retry-attempts 3 \
  --maximum-record-age-in-seconds 3600 \
  --function-response-types ReportBatchItemFailures \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:eu-west-1:111122223333:orders-dlq"}}'

What each setting does:

• --parallelization-factor (1–10): concurrent batches per shard. Ordering is still kept per partition key, not per shard. Concurrency becomes shards × factor, so check the account's concurrency headroom before you raise it.
• --bisect-batch-on-function-error: splits a failing batch in half on each retry, so one poison record ends up alone instead of blocking 500 good ones.
• --maximum-retry-attempts / --maximum-record-age-in-seconds: stop retrying a batch forever. Records that run out of retries go to the on-failure destination as metadata (shard ID and sequence numbers), not as the payload. Your redrive tool must read the records back from the stream before retention removes them.
• ReportBatchItemFailures: return {"batchItemFailures":[{"itemIdentifier":"<sequenceNumber>"}]} so the retry starts from the first failed record instead of replaying the whole batch.

Warning

Bisecting and partial-batch retries both replay records the function has already processed. Kinesis delivers at least once, so the handler must be idempotent (conditional writes, or dedupe on sequence number).

#Fix 3: add shards or switch capacity mode

# Provisioned: UNIFORM_SCALING is the only scaling type; there are limits on how often and how far you can scale
aws kinesis update-shard-count --stream-name $STREAM --target-shard-count 16 --scaling-type UNIFORM_SCALING

# Or let Kinesis manage capacity
aws kinesis update-stream-mode --stream-arn "$STREAM_ARN" --stream-mode-details StreamMode=ON_DEMAND

Resharding closes parent shards and opens child shards. KCL and Lambda finish reading the parent before they start on the children, so iterator age can briefly rise during a reshard. To target one hot shard, use split-shard with a hash key in the middle of its range. Check the current limits on resharding frequency and on mode switches per day before you plan an emergency change.

#KCL consumers: check the lease table

The Kinesis Client Library keeps shard leases and checkpoints in a DynamoDB table named after the application. Common failure modes:

• Throttled lease table: checkpoints fail and workers lose leases, so shards get re-processed or stall. Check ThrottledRequests on the table. On-demand capacity avoids this.
• Workers > shards: extra workers sit idle. A worker processes a shard serially, so adding workers beyond the shard count doesn't add throughput.
• Blocking record processor: a slow downstream call inside processRecords holds up the whole shard. Batch downstream writes and checkpoint after each batch.

aws dynamodb scan --table-name orders-kcl-app \
  --projection-expression "leaseKey,leaseOwner,checkpoint,leaseCounter" --output table

A lease whose checkpoint doesn't move while its leaseCounter keeps increasing is a shard where the worker is alive but making no progress. Look at that worker's logs for the record it's stuck on.

#Prevent it next time

• Alarm on iterator age as a fraction of retention (for example 10 minutes on a 24-hour stream), not only on errors.
• Give every Lambda event source mapping a retry cap, a maximum record age, bisect on error and an on-failure destination from the start.
• Register enhanced fan-out for any consumer that needs low latency, once a stream has more than two consumers.
• Load-test with realistic partition keys. Key skew is the most common cause of a single lagging shard.
• Keep enough retention to cover your worst consumer outage plus the time it takes to recover.