Declared allow but agent request blocked

What it detects

The site's robots.txt permits AI search/agent crawlers, but an honest agent request (Scovant's own declared user-agent, no impersonation) was blocked at the network layer.

Why it matters

Declared preference and observed behavior diverge (DECLARED_ALLOW_BUT_BLOCKED): a WAF/CDN rule enforces a stricter policy than robots.txt advertises, so agents that respect robots are still turned away.

How to fix

- Your robots.txt allows AI search/agent crawlers, but an honest agent request was blocked by a WAF/CDN rule — reconcile the two so declared and enforced policy agree. - If you intend to allow AI agents, add the AI user-agents (or a cryptographic verified-bot allow path) to the CDN/WAF allow-list so they aren't challenged. - If you intend to block them, declare that in robots.txt so agents don't waste requests discovering a hidden block. - Verify: request the homepage with an AI-agent user-agent through your CDN and confirm the response matches what robots.txt advertises.

References

• RFC 9309 — Robots Exclusion Protocol
• Sitemaps protocol