They deleted 22K tag pages. Crawl on real content rose 38%. The advice is always "add more content." A WordPress news site did the opposite. 22,000 auto-generated tag and author archive pages, most wi…
A clean lastmod field recrawled 90K pages in 9 days. Free. The consensus is that sitemaps don't help crawling — "Google ignores them." Mostly true. But one publisher tested the one part Google actuall…
They noindexed 400K filter URLs. Traffic didn't move. Everyone says faceted nav is eating your crawl budget. So a fashion retailer I looked at slapped noindex,follow on 412,000 color/size filter combo…
"noindex, follow" doesn't preserve crawl the way you think The clever-sounding move: noindex,follow on thin pages to keep link equity flowing while staying out of the index. The catch nobody mentions …
Duplicate URLs don't just hurt indexing — they tax crawl first The usual worry about duplicates is split signals and canonical confusion. The earlier, quieter cost is crawl: Google has to fetch every …
Search Console's crawl report hides most of what Googlebot does People diagnose crawl from the Crawl Stats report. It's aggregated, sampled, and rounded — useful for trend lines, useless for "which ex…
