Mistake: counting URLs without normalizing query strings

Mistake: counting URLs without normalizing query strings

Faceted and tracking parameters explode one page into thousands of "unique" crawled URLs in your reports. Sources:

→ Google Search Central (faceted navigation) — Parameter permutations are the top crawl-budget sink on large sites.
Takeaway: in logs, fold ?color=red&sort=asc back to the base path to see real coverage.

→ RFC 3986 (URI normalization) — Canonical rules for ordering, casing and default params.
Takeaway: normalize consistently before any dedupe or count.

★ Pick of the week — a sed/awk param-stripper — Drops the query for aggregation while keeping a raw copy.
Takeaway: report both — total fetches AND distinct base paths.