Home / AWS

AWSTroubleshootingAdvanced

AWS WAF blocking legitimate requests with 403: find the rule and add a narrow exception

Users get 403 and the app logs show nothing. Find the AWS WAF rule that matched, stop the false positive with a Count override, and replace it with an exception limited to one path.

Deepak Bhardwaj7 min read

Applies toAWS WAF (wafv2) web ACLs on Application Load Balancer, CloudFront and API Gateway; AWS Managed Rules; AWS CLI v2

Users report 403 on one form, upload or API call. The request never appears in the application log. Find the exact rule that matched, relieve the users with a single-rule override, then narrow the exception so the rest of the rule group keeps protecting you.

Confirm WAF is the source

Find the web ACL attached to the resource:

# ALB, API Gateway and other regional resources
aws wafv2 get-web-acl-for-resource --resource-arn "$ALB_ARN" \
  --query 'WebACL.{Name:Name,Id:Id,Arn:ARN,Default:DefaultAction}'

# CloudFront
aws cloudfront get-distribution-config --id E1234567890ABC \
  --query 'DistributionConfig.WebACLId' --output text

On an ALB, the access log confirms it. A blocked request has waf in actions_executed, an elb_status_code of 403 by default, and - as the target. See ALB access logs in Athena for the query.

Not every WAF response is a 403. A custom response can return another code, and these actions have their own:

Client sees Likely WAF action Check
403 Block Rule and logs, below
Custom code Block with a custom response The rule's CustomResponse setting
405 CAPTCHA, no valid token API clients can't solve a CAPTCHA, so exclude their paths
202 Challenge, no valid token Non-browser clients can't complete a challenge

Find the rule that matched

If logging is on

aws wafv2 get-logging-configuration --resource-arn "$ACL_ARN"

For a CloudWatch Logs destination, run this in Logs Insights on the aws-waf-logs-* log group:

fields @timestamp, httpRequest.clientIp, httpRequest.uri, terminatingRuleId,
       terminatingRuleType, ruleGroupList.0.terminatingRule.ruleId as rule0,
       ruleGroupList.1.terminatingRule.ruleId as rule1
| filter action = "BLOCK" and httpRequest.uri like /\/api\/upload/
| sort @timestamp desc
| limit 50

The fields tell you what happened:

  • terminatingRuleId is the web ACL level rule that ended evaluation. Default_Action means no rule blocked it and the ACL's default action applied.
  • terminatingRuleType is RATE_BASED, REGULAR, GROUP or MANAGED_RULE_GROUP.
  • ruleGroupList[].terminatingRule.ruleId is the rule inside the group. Look at the full event, because the list can hold more than one group.
  • oversizeFields shows whether the body, headers or cookies exceeded the inspection limit.

If logging is off

Sampled requests cover only the last three hours, picked at random from the first 5,000 requests in the time range:

aws wafv2 get-sampled-requests \
  --web-acl-arn "$ACL_ARN" \
  --rule-metric-name AWS-CommonRuleSet \
  --scope REGIONAL \
  --time-window StartTime=2026-11-06T09:00Z,EndTime=2026-11-06T10:00Z \
  --max-items 100 \
  --query 'SampledRequests[?Action==`BLOCK`].{ip:Request.ClientIP,uri:Request.URI,rule:RuleNameWithinRuleGroup}'

--rule-metric-name is the MetricName in the rule's VisibilityConfig. For CloudFront, use --scope CLOUDFRONT --region us-east-1.

Then turn logging on for next time. The log group name must start with aws-waf-logs-, and it must be in the same account and Region as the web ACL:

aws wafv2 put-logging-configuration --logging-configuration \
  ResourceArn=$ACL_ARN,LogDestinationConfigs=arn:aws:logs:eu-west-1:111122223333:log-group:aws-waf-logs-prod

Common false positives

Rule names below are from AWSManagedRulesCommonRuleSet.

Symptom Rule Cause Fix
Uploads or large JSON fail SizeRestrictions_BODY Body over the rule's size limit Exempt the upload path
CMS or rich-text saves fail CrossSiteScripting_BODY HTML in the body looks like XSS Exempt the edit path, keep app-side sanitising
Callback or redirect links fail GenericRFI_QUERYARGUMENTS Parameter value looks like a remote file include Exempt that path
Long sign-in URLs fail SizeRestrictions_QUERYSTRING Long tokens in the query string Exempt the sign-in path
Users with large sessions fail SizeRestrictions_Cookie_HEADER Large cookies Reduce cookie size, or exempt the path
Scripts or monitors fail NoUserAgent_HEADER No User-Agent header Set one in the client
Everyone behind one NAT fails Your rate-based rule (RATE_BASED) Clients share a source IP Review the limit and aggregation key

The body inspection limit depends on the resource. It is fixed at 8 KB for ALB and AppSync. CloudFront and API Gateway default to 16 KB and can be raised to 64 KB in the web ACL. Check the current limits for other resource types.

Immediate relief: override one rule to Count

Caution

update-web-acl replaces the web ACL's mutable settings with what you send. Fetch the current ACL first and send back everything you want to keep. That includes Description, CustomResponseBodies, CaptchaConfig, ChallengeConfig, TokenDomains and AssociationConfig if you use them. Keep acl.json as your rollback copy.

Warning

Don't switch the whole rule group to Count or remove it. That turns off every rule in the group. Overriding one rule also removes that rule's protection on all paths, so follow up with the narrow exception below.

aws wafv2 get-web-acl --name prod-acl --scope REGIONAL --id "$ACL_ID" > acl.json
jq '.WebACL.Rules' acl.json > rules.json

jq '(.[] | select(.Name=="AWS-CommonRuleSet")
     | .Statement.ManagedRuleGroupStatement.RuleActionOverrides) |=
    ((. // []) + [{"Name":"SizeRestrictions_BODY","ActionToUse":{"Count":{}}}])' \
  rules.json > rules-new.json

aws wafv2 update-web-acl --name prod-acl --scope REGIONAL --id "$ACL_ID" \
  --lock-token "$(jq -r .LockToken acl.json)" \
  --default-action "$(jq -c .WebACL.DefaultAction acl.json)" \
  --visibility-config "$(jq -c .WebACL.VisibilityConfig acl.json)" \
  --rules file://rules-new.json

If someone changed the ACL after your read, the call fails with WAFOptimisticLockException. Fetch it again and redo the edit. Changes take from a few seconds to a few minutes to propagate, so old and new behaviour can mix briefly.

Durable fix: re-block everywhere except the one path

A matching rule still adds its label when it is set to Count. Add a rule after the group that blocks on that label unless the request is for the exempt path.

Label names don't always match rule names. Rule SizeRestrictions_BODY emits awswaf:managed:aws:core-rule-set:SizeRestrictions_Body, so copy the exact label from the rule group's documentation.

The new rule's Priority number must be higher than the group's, because a label match only sees labels from rules evaluated earlier. Append this object to the array in rules-new.json:

{
  "Name": "block-large-body-except-upload",
  "Priority": 20,
  "Statement": {
    "AndStatement": {
      "Statements": [
        { "LabelMatchStatement": {
            "Scope": "LABEL",
            "Key": "awswaf:managed:aws:core-rule-set:SizeRestrictions_Body" } },
        { "NotStatement": { "Statement": {
            "ByteMatchStatement": {
              "SearchString": "L2FwaS91cGxvYWQ=",
              "FieldToMatch": { "UriPath": {} },
              "PositionalConstraint": "STARTS_WITH",
              "TextTransformations": [ { "Priority": 0, "Type": "NONE" } ] } } } }
      ]
    }
  },
  "Action": { "Block": {} },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "block-large-body-except-upload"
  }
}

SearchString is base64 here because CLI v2 reads blob fields as base64 by default, and get-web-acl returns them that way too. Generate the value with printf '/api/upload' | base64. Don't add --cli-binary-format raw-in-base64-out when you send rules copied from get-web-acl, because that double-encodes the existing values.

Fetch a fresh lock token and run the update again with this file. The other body rules in the group still inspect the upload path, since only one rule was overridden.

A scope-down statement on the managed group is the blunter option. Requests that fall outside the scope-down statement skip the entire group, so use it only when a path needs no managed inspection at all.

Verify

Test both sides of the exception, against your own endpoint and with your organisation's approval:

head -c 20000 /dev/zero | tr '\0' 'a' > body.txt

# Exempt path: should reach the app
curl -s -o /dev/null -w '%{http_code}\n' -X POST --data-binary @body.txt https://app.example.com/api/upload

# Any other path: should still return 403
curl -s -o /dev/null -w '%{http_code}\n' -X POST --data-binary @body.txt https://app.example.com/api/orders

Prevent it next time

  • Turn logging on for every web ACL before you need it.
  • Test a new rule group in Count mode on production traffic, review the matches, then enable blocking.
  • Do the same when a managed rule group version changes, if you pin versions.
  • Alarm on a jump in blocked requests. Metric dimensions vary by rule type, so list them with aws cloudwatch list-metrics --namespace AWS/WAFV2 --metric-name BlockedRequests.
  • Keep the web ACL in Terraform or CloudFormation, so each exception has a reviewed change and an owner.