AI crawlers blocked in robots.txt

What it detects

robots.txt disallows one or more named AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot) from the site.

Why it matters

A robots.txt block is a direct, deliberate-looking signal to shut an AI crawler out — it prevents the site's content from ever reaching AI search or agent traffic, even traffic the site owner might actually want.

How to fix

- Open `robots.txt` at the domain root and look for `Disallow: /` rules under `User-agent: GPTBot`, `User-agent: ClaudeBot`, `User-agent: Google-Extended`, or `User-agent: PerplexityBot`. - Replace any blanket disallow for these agents with an explicit allow, e.g. `User-agent: GPTBot` / `Allow: /`, repeated per AI crawler user-agent (or remove the blocking rule entirely if it targets `User-agent: *`). - Coordinate with whoever owns the CDN/WAF config too — a robots.txt allow is useless if the same crawler is challenged or blocked at the edge (see `BOT_PROTECTION_BLOCKING_AGENTS`). - Verify: fetch `/robots.txt` and confirm none of GPTBot/ClaudeBot/Google-Extended/PerplexityBot has a `Disallow: /` rule applying to them.

References

• OpenAI crawler documentation
• Anthropic crawler documentation
• RFC 9309 — Robots Exclusion Protocol
• Sitemaps protocol