Scovant
Sign in Scan free →
Scovant Blog

Web Bot Auth Explained: Signed Requests and Cloudflare Verified Bots

For years, "is this bot who it says it is?" had only two answers: trust the user-agent string, or check whether the request came from an IP range the operator published. The first is trivially spoofable. The second works, but it makes every bot operator's network topology part of their identity, and it breaks the moment they move hosts.

Web Bot Auth is the replacement. It is no longer a catch-all phrase for "however a bot proves itself" — it names a specific, emerging IETF mechanism, and it is the thing Cloudflare's verified-bot program is being built around. If you run a site behind a WAF and you want AI agents to reach it, this is the mechanism you are configuring for.

What Web Bot Auth Actually Is

A bot signs every outbound request with an Ed25519 private key, using RFC 9421 HTTP Message Signatures. The signature travels in three headers:

The server fetches that directory — by convention /.well-known/http-message-signatures-directory on the operator's domain — and gets a JWKS of the operator's current public keys. It verifies the signature against the key named in Signature-Input, and now it knows, cryptographically, which operator sent the request.

Three properties matter here:

  1. Identity is not tied to an IP. The operator can change hosts, add regions, or run from a serverless edge without re-verifying anything.
  2. A signature cannot be replayed indefinitely. The covered created / expires window is short — ours is capped at sixty seconds — so a captured request is not a reusable credential.
  3. Keys rotate without a flag day. A directory can publish the outgoing and incoming keys simultaneously, so a rotation is an overlap, not a cutover.

How Cloudflare's verified-bot program uses it

Cloudflare's Verified Bots list is the curated set of crawlers and agents it considers legitimate. Historically an operator got onto it by handing over IP ranges for verification; a request from one of those ranges is tagged cf.client.bot = true in the firewall expression language, and your rules can allow it explicitly.

Signature-based identity is the direction that list is moving. Rather than confirming that a request came from a declared IP range, the edge verifies a signature against the operator's published key directory — the same trust decision, without the network-topology coupling. Either way, the output your rules consume is the same: a request that the edge is willing to attribute to a named operator.

The part that has not changed: the allow decision is still yours. A bot can be perfectly verified and still be blocked by a rule that fires first.

What Scovant itself does

We think an operator writing about bot identity should state its own posture plainly, so here is ours.

Scovant's crawler identifies itself honestly. It sends a declared ScovantBot user-agent with a contact URL, it reads and respects robots.txt for that token before it fetches anything, and it rate-limits itself to roughly ten requests a minute per domain. It never impersonates another crawler's user-agent, and it never authenticates to a site it scans.

We have implemented Web Bot Auth: the crawler can sign its requests with RFC 9421 signatures and we publish an Ed25519 key directory for verification. We applied to Cloudflare's verified-bot program in August 2026; the application is still under review. Until it is approved, our crawler reaches a site as an honest, declared user-agent that respects robots.txt and rate-limits itself to roughly ten requests a minute per domain. Nothing here should be read as a claim of verified-bot status.

That is not a footnote — it is the whole reason the fallback in the next section matters. Most legitimate agents hitting your site today are in exactly this position: honest, declared, and cryptographically capable, but not yet on anybody's allow-list.

The Three Layers Where Agent Traffic Dies

1. Firewall rules

The most common failure point. A blanket "managed challenge" or "block" on anything the edge classifies as a bot catches verified bots too, unless an allow rule runs first.

The correct pattern is to check the verified-bot signal before anything else:

(cf.client.bot) => Allow
(http.user_agent contains "suspicious-pattern") => Block

Order matters. Rules are evaluated in priority order; if the block rule runs first, the allow rule never executes.

2. Bot fight mode

Bot Fight Mode is a one-click toggle that is easy to enable and easy to forget. Super Bot Fight Mode, on the higher plans, exposes a separate control for verified bots — Allow, Block, or Challenge. Set it to Allow before you touch anything else, and remember that a JS challenge is not a soft "no" to an agent: most agents cannot solve one, so it is a block with extra latency.

3. Rate limiting

Even past the firewall, a rate limit written for human browsing will throttle an agent running a multi-step flow. Give verified or allow-listed agent traffic its own threshold — and when you do throttle, throttle honestly: see the Retry-After point below.

The checklist: not blocking legitimate agents

At the edge

In your HTTP semantics — this is the part almost nobody audits, and agents act on status codes far more literally than browsers do:

In your declarations

Testing

Declaring an agent interface

Separate from identity, but adjacent to it: agents look for a discovery file at /.well-known/mcp.json to find out whether you expose an MCP endpoint they can call, rather than screen-driving your UI.

The shape is a map of servers:

{
  "mcpServers": {
    "your-site": {
      "url": "https://yourdomain.com/mcp",
      "transport": "http"
    }
  }
}

Serve it as application/json, unauthenticated and without redirects. Scovant validates its own discovery file — published at https://scovant.com/.well-known/mcp.json — with the same check it runs on yours.

Robots.txt Is Not Enough

Robots.txt tells crawlers which paths to avoid. It does not tell them how to authenticate, how to handle a multi-step flow, or what your site actually supports. It is a declaration, not a handshake — and, as the teardown above shows, a declaration your edge is free to contradict.

The same limit applies to static readiness scores. A score tells you your robots.txt is well-formed and your sitemap is reachable. It cannot tell you whether an AI agent completed a purchase on your site last Tuesday, or whether your last deploy broke booking flows for one class of LLM agent.

That gap is what Scovant is built to close. Scovant runs real AI agents against your site — agents that attempt actual purchases and bookings, stopping at checkout without ever entering payment details or creating accounts — and compares every deploy against a CI baseline so regressions surface before production. Scovant runs the same flow against every LLM agent in its catalog — Claude Sonnet 4.6 is the default — so readiness reflects a distribution, not one lucky data point.

Summary

Web Bot Auth is a specific thing now: RFC 9421 request signatures, an Ed25519 key published in a directory the verifier can fetch, and an identity that survives a change of IP. Cloudflare's verified-bot program is where most site owners will meet it, but the trust decision stays on your side of the edge.

The practical work splits in two. Configure the edge so verified and honestly-declared agents are allowed before any block rule runs. Then fix the HTTP semantics underneath it, because an agent that gets a 200-wrapped challenge page, a soft-404, or a bare 429 has been refused without being told — and it will not come back to ask why.

A free check runs from the box on scovant.com — one per visitor per day, up to 10 pages, no account. It will tell you within a minute whether an honest, declared agent identity gets a 403 from your edge. For the full picture, see how scoring works, the docs, and pricing.

← All posts