A monthly measurement of how ready websites actually are for AI agents — computed from Scovant's own production scans: declared crawler policy, observed WAF behavior toward an honestly-identified agent, protocol adoption, and the compatibility score distribution. Published as an open, machine-readable dataset.
Share of sites whose robots.txt allows each AI crawler at /, grouped by the crawler's intent. Measured on 7,433 sites.
| Crawler | Intent | Allowed |
|---|---|---|
| CCBot | Training | 93.4% |
| GPTBot | Training | 93.2% |
| Applebot | Search | 98.1% |
| ClaudeBot | Training | 93.6% |
| Claude-User | Agent / user | 98.1% |
| ChatGPT-User | Agent / user | 97.5% |
| DuckAssistBot | Agent / user | 98% |
| OAI-SearchBot | Search | 97.9% |
| PerplexityBot | Search | 97.6% |
| Google-Extended | Training | 94% |
| Perplexity-User | Agent / user | 98.1% |
| Claude-SearchBot | Search | 98% |
| Applebot-Extended | Training | 94% |
Machine-readable agent interfaces detected on 9,113 sites.
| Period | Index | Agent-ready | WAF-blocked | MCP declared | Sample |
|---|---|---|---|---|---|
| Sep 2026 | 88.5 | 69.4% | 10.6% | 3% | 9,120 |
| Aug 2026 | 88.5 | 69.4% | 10.7% | 3% | 9,098 |
The full series and the latest snapshot are published as data, so agents, researchers, and journalists can read the numbers directly instead of quoting this page. Open data, licensed CC BY 4.0 — reuse freely with attribution to Scovant.
Each monthly snapshot aggregates the latest scan of every site in Scovant's research corpus — independent, publicly reachable websites scanned by Scovant's own crawler. Corpus sites did not opt in and are not endorsed by or affiliated with Scovant; a site can request removal at any time, and removed sites leave the index.
One site, one vote: rates are computed per site over its most recent scan, never per scan. Metrics that depend on newer probes report their own measured-site count rather than assuming missing data means "no". Scores follow Scovant's public scoring model; the headline index is the corpus median overall score — not a novel composite. Each snapshot records the exact rule-set version that produced it.
Scovant is an AI agent readiness testing platform that scans, simulates, and continuously monitors websites to determine how reliably AI agents can discover, understand, and interact with them.