Every few months a new file format shows up and someone declares it the fix for AI discoverability. First it was structured data, then it was robots.txt directives for crawlers, now it's llms.txt. Each one is useful. None of them is the whole story.
This article breaks down what each layer actually does, where it stops, and what you need beyond declarations if you want AI agents to work reliably on your site.
What These Files Actually Do
robots.txt
robots.txt is a crawl-permission file. It tells automated clients which paths they are and aren't allowed to fetch. That's it. It doesn't describe your site's capabilities, it doesn't say whether a checkout flow works, and it doesn't tell an agent anything about what actions are available.
For AI agents, robots.txt matters in one narrow way: if you block the user-agent of an AI crawler, that crawler won't index your content. But a shopping agent executing a purchase flow isn't a crawler. It's a client making real requests. robots.txt doesn't govern that at all.
Structured Data
Structured data — JSON-LD, schema.org markup — tells a consuming system what a page represents. A product, a price, an organization, an event. Structured data is genuinely useful. Agents can parse it to understand what's on a page without reading prose.
The limit is the same as robots.txt: it's a declaration. It says what the page claims to be, not what the page actually does when an agent tries to interact with it. A product page can declare a price in JSON-LD and still have a broken add-to-cart button.
llms.txt
llms.txt is a newer convention. The idea is to give language models a plain-text summary of a site — what it does, how it's organized, what actions are available — so an LLM can orient itself before crawling or acting.
This is useful for content-heavy sites and documentation. It's a good-faith signal to AI systems. But it shares the same fundamental limit: it's static. It describes intent, not behavior. An agent reading your llms.txt learns what you say your site does. It doesn't learn whether your booking flow actually completes, whether your cart survives a session, or whether a cheaper model can navigate your checkout the same way a premium one can.
The Gap: Declarations vs. Behavior
Static readiness checkers — the kind that scan your robots.txt, sitemaps, and structured data and hand you a score — can only see what a site declares. That's the ceiling of what file-based discovery can tell you.
The actual question agents care about is: does this site work when I try to use it?
That's a behavioral question. You only answer it by running real agents against real flows.
Some concrete gaps that static files can't surface:
- A checkout flow that breaks mid-session. Your structured data is perfect. Your llms.txt is accurate. An agent adds an item to the cart and the session drops. No declaration catches that.
- Model-specific failures. A premium LLM might handle ambiguous navigation just fine. A lighter model running in a cost-sensitive deployment might not parse the same UI. If you've only tested with one model, you don't know your actual compatibility surface.
- Regressions introduced by deploys. Agent-readiness isn't a one-time audit. Every deploy can change how an agent interacts with your site. A score from last month doesn't protect you from a regression you shipped yesterday.
What Actually Matters for Agent Compatibility
Declarations are table stakes. You should have clean robots.txt, accurate structured data, and a useful llms.txt if your site is a candidate for agent interaction. Get these right. They're not hard.
But after that, three things matter that no static file can address:
1. Real Agent Simulations
AI agents need to complete actual purchase and booking flows on your site to know whether those flows work. Not a crawl. Not a parse. An agent that attempts the task end-to-end and reports what happened. That's the only way to know what happens, not just what's declared.
2. CI Regression Detection
Agent-readiness is a regression problem, not a score problem. Every deploy should be compared against a baseline. If a deploy breaks something an agent depends on, you want to catch that before an agent hits it in production — not after a user reports that a bot failed to complete their booking.
3. Multi-Model Coverage
One site tested against ten different LLM agents tells you something a single-model test can't. What a premium model handles fine can still trip up a cheaper one. Your compatibility surface is defined by the least capable model your users might send at you, not the most capable one you tested with.
How Scovant Fits In
Scovant is infrastructure for the agentic web. It's not a static SEO scanner or a checklist tool.
Here's what it does that file-based discovery can't:
- Real AI agents attempt actual purchases and bookings on your site. You see what happens, not what's declared.
- CI regression detection compares every deploy against a baseline. Agent-readiness regressions are caught before agents hit them in production.
- Multi-model testing across ten different LLM agents. Not just one.
- Full-site crawl and scoring, not a single-page check.
- Honest, public scoring methodology and an embeddable trust badge so your users can see your agent-readiness standing.
You can read the full scoring methodology at scovant.com/scoring, check the docs at scovant.com/docs, or look at pricing at scovant.com/pricing.
For agent discovery specifically, Scovant also publishes its own MCP discovery endpoint at scovant.com/.well-known/mcp.json.
The Short Version
llms.txt, robots.txt, and structured data are declarations. They tell agents what your site claims to be and what it claims to allow. Get them right — they matter.
But declarations are only the first step. Scores tell you what's declared. Simulations tell you what happens. Agent-readiness is a regression, not a score. And one site tested against ten models gives you a real compatibility surface, not a best-case result.
If you're building for the agentic web, file hygiene is necessary but not sufficient. The rest of the work is behavioral.