A monthly measurement of how ready websites actually are for AI agents — computed from Scovant's own production scans: declared crawler policy, observed WAF behavior toward an honestly-identified agent, protocol adoption, and the rating distribution. Published as an open, machine-readable dataset.
Share of sites whose robots.txt allows each AI crawler at /, grouped by the crawler's intent. Measured on 8,412 sites.
| Crawler | Intent | Allowed |
|---|---|---|
| CCBot | Training | 96% |
| GPTBot | Training | 95.9% |
| Applebot | Search | 98% |
| ClaudeBot | Training | 96.2% |
| Claude-User | Agent / user | 97.7% |
| ChatGPT-User | Agent / user | 97.1% |
| DuckAssistBot | Agent / user | 97.7% |
| OAI-SearchBot | Search | 97.7% |
| PerplexityBot | Search | 97.2% |
| Google-Extended | Training | 96.7% |
| Perplexity-User | Agent / user | 97.8% |
| Claude-SearchBot | Search | 97.7% |
| Applebot-Extended | Training | 96.7% |
Machine-readable agent interfaces detected on 8,412 sites.
| Period | Index | Δ index | Methodology | Scoring ≥ 80 | WAF-blocked | MCP declared | Sample |
|---|---|---|---|---|---|---|---|
| Oct 2026 | 88.9 | +0.4 | readiness-r1 | 71% | 21.6% | 2.9% | 8,638 |
| Sep 2026 | 88.5 | 0 | readiness-r1 | 69.4% | 10.6% | 3% | 9,126 |
| Aug 2026 | 88.5 | — | readiness-r1 | 69.4% | 10.7% | 3% | 9,098 |
Three independent snapshots of the marketing corpus around Cloudflare's 2026-09-15 AI-bot default change, taken with the same crawler identity (ScovantBot honest UA + a browser-UA baseline). This is not a before/after comparison: each two-day window reached a different set of sites (see `overlap`), and from 2026-09-21 the crawler follows redirects to the final page before judging a block, so the first two snapshots (first hop) and the third (redirects followed) measure differently. Each snapshot's shares describe that snapshot only. Edge provider labelled by a one-off header probe or by the crawler's own headers; 'none' = no CDN marker recognised. Layers under 20 domains report null. Causality is not claimed.
| Layer | Before · 09-13–09-15 first hop · 452 sites | After 1 · 09-20–09-22 first hop · 646 sites | After 2 · 09-27–09-29 redirects followed · 7,972 sites |
|---|---|---|---|
| cloudflare | 17.4% of 201 | 16.2% of 173 | 33.3% of 2,961 |
| akamai | not enough domains (< 20) | 3.0% of 33 | 40.6% of 362 |
| fastly | not enough domains (< 20) | 2.9% of 34 | 19.7% of 295 |
| cloudfront | 3.2% of 31 | 6.6% of 76 | 15.6% of 946 |
| other | not enough domains (< 20) | not enough domains (< 20) | 13.9% of 158 |
| none | 8.4% of 202 | 7.4% of 324 | 11.7% of 3,250 |
| Cloudflare: declared crawler policy and observed edge policy differ | 16.9% | 15.6% | 33.3% |
The snapshots are different samples, not the same sites re-measured. Domains shared: before∩after 1 0 · before∩after 2 439 · after 1∩after 2 550.
The full series and the latest snapshot are published as data, so agents, researchers, and journalists can read the numbers directly instead of quoting this page. Open data, licensed CC BY 4.0 — reuse freely with attribution to Scovant.
Each monthly snapshot aggregates the latest scan of every site in Scovant's research corpus — independent, publicly reachable websites scanned by Scovant's own crawler. Corpus sites did not opt in and are not endorsed by or affiliated with Scovant; a site can request removal at any time, and removed sites leave the index.
One site, one vote: rates are computed per site over its most recent scan, never per scan. Metrics that depend on newer probes report their own measured-site count rather than assuming missing data means "no". Scores follow Scovant's public scoring model; the headline index is the corpus median overall score — not a novel composite. Each snapshot records the exact rule-set version that produced it.
Scovant is an AI agent readiness testing platform that scans, simulates, and continuously monitors websites to determine how reliably AI agents can discover, understand, and interact with them.