Case study: a noindex audit that recovered 30k pages

Case study: a noindex audit that recovered 30k pages

Setup: A media archive lost 60% of indexed pages over two months. I crawled 80k URLs with Screaming Frog (free up to 500, paid £239/yr for the full crawl) filtering on the robots meta column.

What it did well:
— Found a templating bug stamping noindex on every paginated archive page (page 2+)
— Cross-checked against GSC 'Excluded by noindex tag' which confirmed 31,400 affected URLs

Where it fell short:
— Screaming Frog won't render some JS-injected meta tags without the JS-rendering mode toggled on (slower, more RAM)
— It only finds the problem; the fix was a separate dev sprint

Verdict: 5/5 for diagnosis. Recovery took ~7 weeks post-fix to re-index ~30k pages.

Best for: large sites after a traffic cliff. Not for: tiny sites where a manual GSC check is enough.