Every month we scan a fixed corpus of public websites for the signals AI agents rely on. The November snapshot of the Agent Web Index covers 8,529 sites. The dataset is CC BY 4.0, and you can download the JSON or CSV and check every number below.
The finding that surprised me most is not about robots.txt at all.
What sites say in robots.txt
Very few sites block AI agents on paper:
| Crawler group | Sites that disallow at least one of them |
|---|---|
| Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) | 6.0% |
| Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot) | 3.0% |
| Agents acting for a user (ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot) | 3.0% |
The single most-blocked token is GPTBot, at 5.3%. A quarter of sites (25.2%) say anything about AI crawlers at all. The rest have no AI-specific rule, which means everything is allowed by default.
So by policy, the web is wide open to agents.
What actually happens when an agent knocks
We also fetch each homepage the way an honest agent would: a normal request with an identifying user agent. No spoofing, no headless-browser tricks.
On 22.0% of sites, that request was refused by the edge. A bot-protection challenge, a 403, or an equivalent wall came back instead of the page.
Compare the two numbers: 3% of sites ask agents to stay away in robots.txt, and about seven times as many turn them away at the firewall. On most of those sites robots.txt says nothing against agents. The block isn't a policy anyone wrote down. It's a default in a WAF or a bot-management product.
This also isn't stable. Across repeated scans of the same site, 14.7% flipped at least once between "served" and "refused". For a site owner that means an agent may get your page today and a challenge tomorrow, with no change on your side.
The new agent protocols are still rare
| Signal | Share of sites |
|---|---|
/llms.txt present |
7.4% |
| UCP commerce profile | 3.1% |
Valid MCP discovery file (/.well-known/mcp.json listing an endpoint) |
0 sites |
| WebMCP tools in the page | 3 sites |
So if you're deciding whether to add llms.txt or an MCP server card this quarter, you'd be early, not late.
Caveats
- The corpus is a sample of popular public sites, not a random sample of the web. These sites did not opt in. Removal requests are honoured.
- The firewall result comes from our own honest request. A different client, IP range or day can get a different answer — that's what the 14.7% means.
- These are passive measurements. Nothing here ran a real agent through a purchase or a booking.
- The methodology is public: scovant.com/research/methodology.
Check your own site
The scanner behind the passive part of this index is open source, Apache-2.0: Scovant Core. It runs 69 checks (51 readiness, 18 security), covering robots.txt AI-bot policy, llms.txt, sitemaps, JSON-LD, MCP/OAuth/UCP discovery and security posture. It shows the evidence for each finding.
uvx --from scovant-core scovant scan https://your-site.com
# or: pipx run --spec scovant-core scovant scan https://your-site.com
# or: npx @scovant/core scan https://your-site.com
It is passive by design: it never runs a browser, never logs in and never invokes a tool.
To keep it from regressing, run it as a GitHub Action on every pull request:
- uses: Scovant/scovant-core@v0
with:
url: https://staging.example.com
min-core-score: '70'
fail-on: fail
It also ships as an MCP server (scovant mcp) and as a Claude Code plugin. Details are in the README.
If your scan disagrees with what you know about your own site, open an issue. That kind of report is the most useful feedback I can get.