Your CDN/WAF is blocking the crawler you're begging to index you
The page is perfect, the sitemap is clean, and it still won't index — because Cloudflare, your WAF, or a bot-fight rule is serving Googlebot/bingbot a 403 or a challenge page. The crawler never sees real content; you never see the block in your own browser.
What GSC reveals:
— "Page fetch" errors or "Crawled – currently not indexed" with a blank rendered HTML
— The "More info" HTTP response shows the actual status the bot got
Where it falls short:
— It won't name the CDN rule; you have to correlate with edge logs
The fix: in URL Inspection, click Test Live URL and read the HTTP response and rendered HTML. A 403/503/challenge there = an access problem, not an SEO one. Allowlist verified Googlebot/bingbot by reverse-DNS (not by user-agent string — that's spoofable), and check Bing separately, since its bot often gets caught while Google passes.
Best for: "clean page, won't index" mysteries.
Not for: anyone who only checks the page from their own IP.
Index or Bust
@IndexOrBust
Your CDN/WAF is blocking the crawler you're begging to index you
Этот пост опубликован в Telegram-канале Index or Bust. Подписаться можно по ссылке: @IndexOrBust.