vans.com's robots.txt explicitly welcomes AI agents. The agent-intent crawlers — ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot — are all allowed. By its own declared policy, this is a site that wants agent traffic.
Then an honest agent actually knocks, and the door says no — twice, at two different layers.
What the scan found
- Bot protection blocking AI agents (critical). An honest ScovantBot request — a stable, declared identity that respects robots.txt — received HTTP 403 Forbidden from the WAF/bot manager.
- Declared allow but blocked (medium). Scovant flags the divergence itself: the robots policy allows the agent, the enforcement layer returns 403. Policy and edge contradict each other.
The headline isn't the score. It's that vans.com's declared policy says "yes" and its edge says "403" — to the exact honest, well-behaved agents its robots.txt names.
What the simulation showed — blocked at the door
We then pointed a real browser-driving agent (Claude Sonnet 4.6, full Chromium) at vans.com. It got one step. On the first navigation the page returned "Access Denied — automated access refused," and the run ended: failure owner WAF / bot protection, not the site's content and not the agent.
So the block isn't just a crawler-UA filter that a real browser slips past. The WAF refused a genuine browser session too. Both layers — the honest crawler and the browser agent — hit the same wall, while robots.txt held the door open the whole time.
How common is this?
This isn't a one-off. In Scovant's Agent Web Index, a substantial minority of measured sites WAF-block an honest agent identity — the current share, its denominator and the measurement window are published together on the index page. Many of them, like vans.com, block while their robots.txt formally allows it. Declared-versus-observed divergence is a systemic, mostly-accidental failure mode of the agent-era web.
Why it happens
Nobody decides "welcome agents in robots, block them at the WAF." It's an org-chart artifact: the marketing/SEO team owns robots.txt and adds the friendly AI-crawler allowances; the security/infra team owns the bot manager and turns on aggressive defaults; the two documents are never reconciled. The site makes two contradictory promises, and the enforcement layer wins.
The self-defeating part
Aggressive bot protection lands hardest on the agents you'd want. The traffic vans.com's robots.txt names — an answer engine's crawler, a shopping assistant that identifies itself and respects robots — is the easiest thing in the world to refuse, because it announces itself honestly. A site that goes all the way, as vans.com does, blocks a real browser agent too. Either way the outcome is the same: the well-behaved, reputation-bearing agents that would send real customers or cite the brand in an AI answer simply can't get in — to a site that formally invited them.
The fix list
- Reconcile the two documents. Decide what you actually mean, and make robots.txt and the WAF say the same thing. The contradiction is the bug.
- If you want agents, allow-list honest identities — Cloudflare's verified-bot program, or cryptographic Web Bot Auth (RFC 9421 request signatures) — so a declared, verifiable agent is admitted while anonymous abuse is still challenged. For the record: Scovant signs with Web Bot Auth, and we applied to Cloudflare's verified-bot program in August 2026; the application is still under review. Until it is approved, our crawler reaches a site as an honest, declared user-agent that respects robots.txt and rate-limits itself to roughly ten requests a minute per domain.
- If you really mean to block them, say so in robots.txt — an honest
Disallowis machine-legible and lets well-behaved agents skip you cleanly, instead of burning a request to earn a 403. - Then fix the commerce gaps behind the door (structured Product data, an agent surface) so the agents you let in can actually act.
The honest takeaway
vans.com isn't hostile to agents on paper — its robots.txt says the opposite. It's inconsistent, and the inconsistency defaults to the strictest layer, so the site turns away the very agents it invited: the honest crawler with a 403, a real browser agent with "Access Denied." No static SEO audit catches this, because the divergence only appears when you fetch as an honest agent and compare what the site said to what it did. Declared versus done — this time at the edge.
Scovant scanned vans.com as part of its public agent-readiness research corpus. The site is not affiliated with or endorsed by Scovant; removal requests are honored. See the Agent Web Index and the research methodology.