S3 event notifications not triggering Lambda, SQS or SNS: filters, destination policies, KMS and EventBridge

#Symptom

Files arrive in the bucket, but the Lambda function never runs, the SQS queue stays empty or the SNS topic publishes nothing. Sometimes it worked yesterday and stopped after a deployment. Sometimes some objects trigger and others don't.

Work through the checks below in order. Most cases are a missing or overwritten notification config, a filter that doesn't match the real key, or a destination policy that won't let S3 deliver.

#Quick map: symptom, cause, fix

#1. Is there a notification config at all?

aws s3api get-bucket-notification-configuration --bucket my-bucket

Empty output means there is no configuration. The most common cause: put-bucket-notification-configuration replaces the whole configuration. If two stacks or pipelines each write their own notification to the same bucket, the last one wins and the other one is silently removed.

Find out who changed it last:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutBucketNotification \
  --max-results 10 \
  --query 'Events[].{Time:EventTime,User:Username}'

Warning

Always read the current config first, merge your change into it, then put the full document back. Pushing only your own rule deletes everyone else's triggers on that bucket.

aws s3api get-bucket-notification-configuration --bucket my-bucket > current.json
# edit current.json to add or fix your rule, then:
aws s3api put-bucket-notification-configuration \
  --bucket my-bucket \
  --notification-configuration file://current.json

If more than one team needs events from one bucket, use EventBridge instead (section 5) so each team owns its own rule.

#2. Does the filter match the real key?

Compare the rule filter with keys that actually arrived:

aws s3api get-bucket-notification-configuration --bucket my-bucket \
  --query 'LambdaFunctionConfigurations[].{Events:Events,Filter:Filter.Key.FilterRules}'

aws s3api list-objects-v2 --bucket my-bucket --prefix incoming/ \
  --max-items 5 --query 'Contents[].Key'

Things that commonly break matching:

• Case. Matching is case-sensitive. A .csv suffix doesn't match REPORT.CSV.
• Leading slash or wrong folder. The filter incoming/ doesn't match /incoming/file.csv or Incoming/file.csv.
• Special characters. Prefix and suffix values with special characters must be URL-encoded in the filter.
• Event type. Large uploads finish as s3:ObjectCreated:CompleteMultipartUpload, and copies as s3:ObjectCreated:Copy. A rule with only s3:ObjectCreated:Put misses both. Use s3:ObjectCreated:* unless you need something narrower.
• Deletes on a versioned bucket. A plain delete creates a delete marker, which fires s3:ObjectRemoved:DeleteMarkerCreated, not s3:ObjectRemoved:Delete.

#3. Can S3 deliver to the destination?

S3 checks destination permissions when you save the config. If the check fails you get Unable to validate the following destination configurations. If the permission is removed later, events just stop.

Lambda needs a resource-based policy that allows S3 to invoke it:

aws lambda get-policy --function-name process-upload --query Policy --output text

aws lambda add-permission \
  --function-name process-upload \
  --statement-id s3-invoke-my-bucket \
  --action lambda:InvokeFunction \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::my-bucket \
  --source-account 111122223333

If the trigger points at an alias or version, the permission must be on that same qualifier (--qualifier live).

SQS needs a queue policy that allows sqs:SendMessage from S3:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "s3.amazonaws.com" },
    "Action": "sqs:SendMessage",
    "Resource": "arn:aws:sqs:eu-west-1:111122223333:uploads",
    "Condition": {
      "ArnLike": { "aws:SourceArn": "arn:aws:s3:::my-bucket" },
      "StringEquals": { "aws:SourceAccount": "111122223333" }
    }
  }]
}

SNS needs the same shape with "Action": "SNS:Publish" on the topic ARN.

Check what's there now:

aws sqs get-queue-attributes --queue-url "$QUEUE_URL" \
  --attribute-names Policy KmsMasterKeyId SqsManagedSseEnabled
aws sns get-topic-attributes --topic-arn "$TOPIC_ARN" \
  --query 'Attributes.{Policy:Policy,Kms:KmsMasterKeyId}'

#4. Encrypted queue or topic

If KmsMasterKeyId is alias/aws/sqs or alias/aws/sns, S3 can't deliver. You can't edit the key policy of an AWS managed key. Switch the queue or topic to a customer managed key (or SQS managed encryption for queues) and add this statement to the key policy:

{
  "Sid": "AllowS3EventNotifications",
  "Effect": "Allow",
  "Principal": { "Service": "s3.amazonaws.com" },
  "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
  "Resource": "*"
}

This one is easy to miss. The config saves, nothing errors, and messages never arrive.

#5. FIFO queues, many consumers, richer filters: use EventBridge

S3 event notifications can't target SQS FIFO queues. They also allow only non-overlapping rules per event type. EventBridge removes both limits. Turn it on for the bucket. The EventBridgeConfiguration key goes in the same document as your other rules, so merge it into current.json as in section 1:

{
  "EventBridgeConfiguration": {},
  "LambdaFunctionConfigurations": [ "...existing rules..." ]
}

Then create a rule per consumer:

aws events put-rule --name uploads-incoming-csv --event-pattern '{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": { "name": ["my-bucket"] },
    "object": { "key": [{ "prefix": "incoming/" }] }
  }
}'

Add the target with aws events put-targets and give the target its own resource policy (queue policy with principal events.amazonaws.com). If the EventBridge rule then doesn't fire, see the existing post on EventBridge FailedInvocations.

#6. "Configuration is ambiguously defined"

S3 rejects two rules for the same event type when their filters overlap: overlapping prefixes with overlapping suffixes, or one rule with no filter at all. Suffixes overlap when one string can end with both, so json and on overlap but .png and .jpg don't.

Fixes:

• Make the prefixes distinct (raw/ vs curated/).
• Send one event to one SNS topic and fan out to several queues from there.
• Move to EventBridge, where overlapping rules are allowed.

The same error appears if an existing rule points at a destination that no longer exists, such as a deleted function. Remove the stale rule and save again. If you have to save while a destination is temporarily missing, --skip-destination-validation is available on put-bucket-notification-configuration. Use it knowingly.

#7. The event arrives but the function "does nothing"

S3 invokes Lambda asynchronously. Failures are retried and then dropped unless you have an on-failure destination. Check:

aws cloudwatch get-metric-statistics --namespace AWS/Lambda \
  --metric-name AsyncEventsDropped --dimensions Name=FunctionName,Value=process-upload \
  --start-time "$(date -u -d '-6 hours' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 300 --statistics Sum

Also look at Errors and AsyncEventAge. A classic bug: object keys in the event are URL-encoded (spaces become +). Decode them before calling GetObject, or you get NoSuchKey:

from urllib.parse import unquote_plus

def handler(event, context):
    for record in event["Records"]:
        bucket = record["s3"]["bucket"]["name"]
        key = unquote_plus(record["s3"]["object"]["key"])
        print(f"processing s3://{bucket}/{key}")

Caution

A function that writes its output back to the same bucket and prefix it is triggered from can run in a loop. That costs money fast. Write output to a different bucket or a prefix the trigger filter excludes.

You can also run the AWSSupport-TroubleshootS3EventNotifications Automation runbook. It checks the bucket's Lambda, SNS and SQS destinations for you:

aws ssm start-automation-execution \
  --document-name AWSSupport-TroubleshootS3EventNotifications \
  --parameters S3BucketName=my-bucket

#Prevent it next time

• Have one owner for each bucket's notification config. If several teams need events, use EventBridge rules instead of shared S3 notification rules.
• In IaC, manage the bucket notification as one resource. Never let two stacks write it.
• Use s3:ObjectCreated:* unless you have a reason to be narrower, and keep filters lowercase and documented.
• Encrypt queues and topics with customer managed keys whose policies already allow s3.amazonaws.com.
• Give async Lambda triggers an on-failure destination (SQS or SNS) and alarm on AsyncEventsDropped and Errors.
• Expect duplicates and out-of-order delivery. Make consumers idempotent, and use the sequencer field when order matters.