How do I allow requests from a bot blocked by AWS WAF Bot Control managed rule group?

Short description

To allow requests from a bot blocked by AWS WAF Bot Control rule group, do the following:

1. Identify the Bot Control rule that's blocking the requests from AWS WAF logs by Querying AWS WAF logs.
2. Set the Bot Control rule that's blocking the requests to count.
3. Create a custom rule to match against the excluded rule's label and to block all matching requests except for the bot that you want to allow.
4. Validate that the bot traffic is allowed.

The Bot Control managed rule group verifies bots using the IP addresses from AWS WAF. If you have verified bots that route through a proxy or a CDN that doesn't preserve the client IP address while forwarding the requests, then you must specifically allow the bot.

Resolution

Identify the Bot Control rule that's blocking the requests

Analyze the AWS WAF logs to identify the Bot Control rule that's blocking requests from the required bot.

1.    To analyze AWS WAF logs using Amazon Athena, create a table for AWS WAF logs in Athena using partition projection. For instructions, see Creating the table for AWS WAF logs in Athena using partition projection.

2.    Run the following Athena query to find the details of the request blocked by the Bot Control rule group:

Note: Replace waf_logs with your table name. The time interval time > now() - interval '3' day can be replaced with your specified time interval.

WITH waf_data AS
    (SELECT from_unixtime(waf.timestamp / 1000) as time,
    waf.terminatingRuleId,
    waf.action,
    waf.httprequest.clientip as clientip,
    waf.httprequest.requestid as requestid,
    waf.httprequest.country as country,
    rulegroup.terminatingrule.ruleid as matchedRule,
labels as Labels,
         map_agg(LOWER(f.name),
         f.value) AS kv
    FROM waf_logs waf,
    UNNEST(waf.httprequest.headers)
AS t(f), UNNEST(waf.rulegrouplist) AS t(rulegroup)
    WHERE rulegroup.terminatingrule.ruleid IS NOT NULL
    GROUP BY 1, 2, 3, 4, 5, 6, 7,8)
SELECT waf_data.time,
       waf_data.action,
       waf_data.terminatingRuleId,
       waf_data.matchedRule,
       waf_data.kv['user-agent'] as UserAgent,
waf_data.clientip,
       waf_data.country,
       waf_data.Labels
FROM waf_data
Where terminatingRuleId='AWS-AWSManagedRulesBotControlRuleSet' and time > now() - interval '3' day
ORDER BY time
DESC

For sample Amazon Athena queries to filter records for a specified time range, see Example queries for AWS WAF logs.

3.    (Optional) To further narrow down your search, add an additional filter on UserAgent using the AND operator in the Where clause. For a description of the fields in WAF logs, see Log Fields. For example, you can add the filter kv['user-agent'] like 'Postman%' to narrow your results.

4.    Check the matchedRule column to identify the rule which is blocking the requests. Note: For additional information on Bot Control rules, see AWS WAF Bot Control rule group.

Set the Bot Control rule that's blocking the requests to count

Edit the Bot Control Rule group to set the rule that's blocking the requests to count. To set a rule to count, see Setting rule actions to count in a rule group. This allows the rule to apply its label to matching requests and to allow the bot that isn't blocked.

Create a custom rule to match against the excluded rule's label and to block all matching requests except for the bot that you want to allow

Add a label matching rule to your web ACL based on the rule label that is blocking the request. The label matching rule must come after the Bot Control managed rule group. For information on Bot Control managed rule group labels, see AWS WAF Bot Control rule group.

If a rule with the category label is blocking the request

Configure your custom rule to allow a specific blocked bot. Important: Replace the bot category and bot name labels in the rule configuration with the bot category and bot name labels from the Athena query results.

For all other rule labels

Create a custom rule to Create an exception for a blocked user agent.
Important: Replace the bot signal label and the UserAgent value in the field SearchString in the rule configuration with the bot signal label and UserAgent value from the labels and UserAgent columns of Athena query results.

Validate that the bot traffic is allowed

Check the AWS WAF logs again to verify that the bot is now being allowed. If the bot is still blocked, repeat the preceding process to identify additional rules that are blocking the requests.

---

Related information

False positives with AWS WAF Bot Control

AWS WAF Bot Control examples