Feedback
Feedback has been submitted! Thank you :)
#AI Crawler/Search/Assistant "allow"ed when I have block rules for AI
2 messages · Page 1 of 1 (latest)
harsh otterBOT
harsh otterBOT
Support AI
With Block AI bots, Bot Fight Mode, and Managed Robots.txt for AI Crawl Control enabled, you have a robust setup to block AI crawlers.
Given that ChatGPT and Google AI mode are still able to access your site, it's possible that they are using advanced techniques to evade detection. However, it's also possible that there's another explanation.
According to the Cloudflare documentation, AI Crawl Control's pay per crawl feature takes place after WAF custom rules. (Source: https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/) This means that if you have a WAF custom rule that allows or skips certain traffic, it might be bypassing the AI Crawl Control rules.
I would recommend checking your WAF custom rules to ensure that they're not inadvertently allowing ChatGPT and Google AI mode to access your site. You can also try checking the Cloudflare logs to see if there are any clues about how these AI crawlers are accessing your site.
Additionally, you might want to consider reaching out to Cloudflare support to see if they can provide more insight into why these AI crawlers are able to access your site despite your security setup.
This is an AI generated response, it may not be 100% accurate. Please validate the output. You can ask followups by replying to this message or mentioning me.