Flattening click-depth from 9 to 3 got 140K orphan-ish pages crawled.
"Just submit a sitemap and Google will find everything." A retailer with 600,000 SKUs believed that. 140,000 deep pages sat un-crawled for months despite being in the sitemap.
Reality: sitemaps aid discovery, but crawl scheduling follows internal link signals and PageRank flow. Pages buried 9 clicks deep get a near-zero crawl priority. The sitemap listing didn't override that.
The team rebuilt category pagination and added hub pages so nothing sat deeper than 3 clicks. No new external links.
In eight weeks, log files showed first-time crawls on 138K of those URLs. Indexation on the long tail went from 41% to 88%.
No, a sitemap is not a crawl guarantee. It's a hint.
Depth is a budget allocator. Shallow pages get the spend.
Budget Myths
@CrawlBudgetMyths
Flattening click-depth from 9 to 3 got 140K orphan-ish pages crawled.
Этот пост опубликован в Telegram-канале Budget Myths. Подписаться можно по ссылке: @CrawlBudgetMyths.