Scovant
Sign in Scan free →
Scovant Blog

Web Bot Auth Explained: Preparing Your Site for Cloudflare Verified Bots

If you run a site behind Cloudflare, you have probably seen the Verified Bots list. It is a curated set of crawlers and agents that Cloudflare considers legitimate — search engines, monitoring tools, and increasingly, AI agents. When your firewall rules or bot fight mode is misconfigured, even a verified bot can get a 403 or a JS challenge it cannot solve. That is a silent failure. The bot logs an error, skips your site, and you never know it happened.

This article explains how Cloudflare Verified Bots works, what web bot auth actually means in practice, and what you need to do so legitimate AI agents can reach your site cleanly.

What Is Cloudflare Verified Bots?

Cloudflare maintains a list of bots it has independently verified. Verification means Cloudflare has confirmed the bot's IP ranges actually belong to the organization that claims to operate them. When a request comes in from a verified bot, Cloudflare tags it with cf.client.bot = true in its firewall expression language.

Your firewall rules can then explicitly allow verified bots rather than treating them the same as unknown scrapers. Without that explicit allowance, a rule like "block all bots" will catch verified bots too.

Why This Matters for AI Agents

AI agents are not just crawlers anymore. They navigate, authenticate, and transact. An agent trying to complete a booking or purchase on your site needs to:

If your Cloudflare configuration does not account for verified AI agents, those flows break at the network layer before any of your application logic even runs.

The Three Layers Where Bot Auth Can Break

1. Firewall Rules

This is the most common failure point. A blanket "managed challenge" or "block" action on anything Cloudflare classifies as a bot will catch verified bots unless you add an explicit bypass rule.

The correct pattern is to check cf.client.bot first:

(cf.client.bot) => Allow
(http.user_agent contains "suspicious-pattern") => Block

Order matters. Cloudflare evaluates rules in priority order. If your block rule runs before your allow rule, verified bots are blocked regardless.

2. Bot Fight Mode and Super Bot Fight Mode

Cloudflare's Bot Fight Mode is a one-click toggle that is easy to enable and easy to forget about. Super Bot Fight Mode on Pro and Business plans gives you more granular control, including a setting specifically for "verified bots" — you can set it to Allow, Block, or Challenge independently.

If you are on a plan with Super Bot Fight Mode, check that verified bots are set to Allow before you do anything else.

3. Rate Limiting Rules

Even if a bot passes the firewall and bot fight mode, a rate limiting rule can still throttle or block it. AI agents that crawl your site or run multi-step flows can trigger rate limits that were designed for human users.

Consider separate rate limit thresholds for verified bots, or use a custom header that your backend recognizes to exempt known agent traffic.

What "Web Bot Auth" Actually Means

Web bot auth is not a single protocol. It is a catch-all phrase for the mechanisms by which a bot proves its identity to a server so the server can make an access decision. Depending on who you ask, it covers:

For most sites, the practical starting point is Cloudflare IP verification plus a well-known discovery file for AI agents.

Setting Up a Well-Known Discovery File

AI agents increasingly look for a /.well-known/mcp.json or similar file before they interact with a site. This file tells the agent what the site can do, how to interact with it, and any auth requirements.

Publishing this file does two things:

  1. It gives well-behaved agents a structured starting point instead of having them guess at your site's structure.
  2. It signals to agent operators that your site is intentionally agent-ready.

Scovant publishes its own discovery file at https://scovant.com/.well-known/mcp.json as a working example.

A minimal file looks something like this:

{
  "name": "Your Site Name",
  "description": "What your site does",
  "contact": "ops@yoursite.com",
  "capabilities": ["browse", "search"],
  "auth": {
    "required": false
  }
}

The exact schema is still evolving, but publishing something is better than publishing nothing. Agents that respect the well-known convention will find it. Agents that do not were probably not going to respect your robots.txt either.

Robots.txt Is Not Enough

Robots.txt tells crawlers which paths to avoid. It does not tell them how to authenticate, how to handle multi-step flows, or what your site actually supports. It is a declaration, not a handshake.

The same applies to static readiness scores. A score can tell you that your robots.txt is well-formed and your sitemap is reachable. It cannot tell you whether an AI agent successfully completed a purchase on your site last Tuesday, or whether your last deploy introduced a regression that breaks booking flows for a specific class of LLM agent.

That gap is exactly what Scovant is built to close. Scovant runs real AI agents against your site — agents that attempt actual purchases and bookings — and it compares every deploy against a CI baseline so regressions are caught before they hit production. It also tests across ten different LLM agents, because a configuration that works fine for a premium model can still break a cheaper one.

Practical Checklist

Here is a concrete list of things to verify:

Cloudflare Configuration - [ ] Super Bot Fight Mode is set to Allow for verified bots (if on Pro or higher) - [ ] Firewall rules check cf.client.bot before any block or challenge action - [ ] Rate limiting rules have separate thresholds or exemptions for verified bot traffic - [ ] JS challenge is not applied globally — verified bots often cannot solve them

Discovery and Declarations - [ ] robots.txt is reachable and correctly scoped - [ ] XML sitemap is present and linked from robots.txt - [ ] Structured data (JSON-LD or similar) is present on key pages - [ ] /.well-known/mcp.json or equivalent discovery file is published

Testing - [ ] You have verified that an actual AI agent can complete a purchase or booking flow end to end - [ ] You have a baseline so you can detect regressions after deploys - [ ] You have tested against more than one model — what works for one may not work for another

Where Static Scores Stop and Real Testing Starts

Static readiness scores are a useful first pass. They will catch missing sitemaps, broken robots.txt syntax, and absent structured data. That takes maybe five minutes and you should do it.

But scores tell you what your site declares. Simulations tell you what actually happens. Agent-readiness is a regression, not a score — it can change with every deploy, every dependency update, every A/B test you run on your checkout flow.

If you want to know whether your site is actually ready for AI agents, not just whether it looks ready, you need to run agents against it. Scovant does that, with CI integration so you catch regressions early and multi-model coverage so you are not optimizing for one agent at the expense of others.

Learn more at scovant.com, check the pricing page, or read the scoring methodology to understand exactly how results are calculated.

Summary

Cloudflare Verified Bots gives you a clean mechanism to let legitimate crawlers and agents through without opening the door to bad actors. The key steps are: enable the verified bots allow rule before any block rules, check your Bot Fight Mode settings, and audit your rate limiting configuration.

Beyond Cloudflare, publish a well-known discovery file, keep your robots.txt and sitemap current, and — critically — actually test with real agents. Declarations are easy. Working end-to-end flows under CI are what agent-readiness actually looks like.

← All posts