Declared AI-crawler policy

What it detects

This site's robots.txt declares an AI-crawler policy — captured per AI user-agent across search, agent, and training intents.

Why it matters

Blocking a search-intent crawler (e.g. OAI-SearchBot, PerplexityBot) removes the site from the indexes those answer engines cite; blocking agent-intent fetchers stops live user-triggered retrieval. Captured for visibility — informational, not a defect judgement.

How to fix

- Review your robots.txt AI user-agent directives by intent: search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot), live agent fetchers (ChatGPT-User, Claude-User, Perplexity-User), and training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) are three separate decisions. - Blocking a search-intent crawler removes you from that answer engine's citable index — allow it if you want to be cited; block training crawlers separately if that's your only concern. - Verify: fetch /robots.txt and confirm each AI user-agent's Allow/Disallow matches your intent for that class.

References

• RFC 9309 — Robots Exclusion Protocol
• Sitemaps protocol