Finding wasted crawl in your logs, in 6 grep-able steps

Finding wasted crawl in your logs, in 6 grep-able steps

You can't fix "wasted crawl" until you can point at the URLs. From a verified-Googlebot log:

— Count hits to URLs with query strings (? in the path). High share = parameter sprawl.
— Count hits returning non-200. Every 404/301/500 Googlebot fetches is a request that taught it nothing.
— Count hits to /tag/, /page/2+, /?sort=, /?filter=, internal search results.
— Count hits to URLs that are noindex or canonicalized away.
— Tally those four buckets as a % of total Googlebot hits.
— If junk is over ~30%, you found your problem and your fix list.

Here's the thing: the fix is rarely "crawl budget tooling." It's killing the URLs that shouldn't exist via internal links, robots.txt, or canonical hygiene.

Measure the waste in percent. A vibe is not a number.