Killing session IDs in URLs doubled product recrawl. No content changed.
People blame crawl budget on Google. This site's problem was its own URLs: every visit appended a ?sid= session parameter, and internal links carried it. Googlebot saw the same product as dozens of distinct URLs.
Log files told the story: 84% of crawl requests hit parameterized duplicates of pages it had already seen.
The fix was boring — strip the session param server-side for bots, canonicalize, and the duplicates stopped propagating through internal links. No redesign.
Thirty days later: unique-URL crawl efficiency went from 16% to 51%. Core product pages got recrawled twice as often, so price and stock updates showed in search faster.
No, Googlebot isn't wasteful. Your URL structure is generating the waste.
Check what fraction of crawl hits unique URLs. That number is the whole game.
Budget Myths
@CrawlBudgetMyths
Killing session IDs in URLs doubled product recrawl. No content changed.
Этот пост опубликован в Telegram-канале Budget Myths. Подписаться можно по ссылке: @CrawlBudgetMyths.