Bot protection blocking AI agents
BOT_PROTECTION_BLOCKING_AGENTS · v1.0
What it detects
A WAF or bot-protection layer (Cloudflare, Akamai, DataDome, etc.) challenges or blocks requests that carry an AI-agent user agent, independent of robots.txt.
Why it matters
robots.txt is a request, not an enforcement mechanism — even a fully permissive robots.txt is moot if the edge infrastructure challenges or drops the agent's request before it ever reaches the origin.
How to fix
- Identify the WAF/bot-protection provider in front of the site (Cloudflare, Akamai, DataDome, etc.) and check its dashboard for a challenge/CAPTCHA rule that would catch a headless HTTP client with a bot-like or unrecognized user-agent. - Add the AI crawler user-agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot) to the provider's "verified bots" / known-bots allowlist, or write a custom rule that skips the challenge when the user-agent matches those strings. - If the provider supports IP-range allowlisting instead of/in addition to UA matching, allowlist the published IP ranges for each AI crawler as a more robust fix. - Verify: fetch the page with a matching AI-crawler user-agent (or ask Scovant to re-scan) and confirm real page content is returned instead of a challenge page.
References
Last reviewed: 2026-09-01 · rule v1.0