Page crawlability blocked

What it detects

The page is not reachable by a plain crawl — a noindex directive, a non-200 status, or an auth wall in front of otherwise-public content.

Why it matters

A page an agent cannot fetch at all cannot be evaluated, indexed, or acted on, regardless of how good its content is.

How to fix

- Check the page's `<meta name="robots">` tag and `X-Robots-Tag` HTTP header for a `noindex` directive, and check the HTTP status code returned on a direct GET. - Remove any unintended `noindex` directive from pages meant to be indexed, and set `<meta name="robots" content="index, follow">` explicitly if it's currently absent-but-defaulting incorrectly. - If the page returns a non-200 status, investigate the cause: misconfigured redirect chains, an auth wall in front of public content, or geo-blocking rules. - Verify: `curl -I` the page and confirm a `200` status with no `noindex` in either the meta tag or response headers.

References

• RFC 9309 — Robots Exclusion Protocol
• Sitemaps protocol