<b>Finding crawl traps in your logs</b>

<b>Finding crawl traps in your logs</b>

When Googlebot burns budget on infinite URLs, logs show it first. References for spotting the leak.

→ Google's faceted-navigation guidance — the canonical list of trap patterns (sort/filter params, session IDs, calendars).
Takeaway: ?sort= and ?sessionid= are budget black holes.

⭐ Pick of the week: the directory-hit-count technique shared by many log analysts —
<code>awk '{print $7}' access.log | grep Googlebot-only | cut -d'?' -f1 | sort | uniq -c | sort -rn</code>
Spikes on one template = your trap.
Takeaway: rank paths by bot hits, investigate the top oddities.

→ Screaming Frog's guide on matching log hits to crawl depth — deep, rarely-linked URLs eating crawl.

→ OnCrawl's posts on parameter explosion in logs.

Credit to Google Search Central, Screaming Frog, and OnCrawl.