Screaming Frog vs GSC for hunting rogue noindex tags

Screaming Frog vs GSC for hunting rogue noindex tags

A stray noindex can nuke a section silently. Two ways to catch it.

Screaming Frog crawls your whole site and lists every URL with meta robots noindex or X-Robots-Tag in one filtered view, including ones GSC hasn't recrawled yet. GSC's Page Indexing report shows "Excluded by noindex" but only after Google revisits, so it lags days to weeks.

Pros (Frog): full-site snapshot now, catches header-level X-Robots-Tag, free up to 500 URLs.
Cons (Frog): crawls your rendered/served HTML, can miss JS-injected noindex unless you enable rendering (paid £259/yr for full rendering at scale).
Pros (GSC): reflects what Google actually saw and acted on.
Cons (GSC): reactive, slow, sampled.

Best for: Frog to find the tag before it bites, GSC to confirm Google obeyed your fix. NOT for users who only have a phone, no desktop.