Counting crawl hits: wc -l vs sort | uniq -c

Counting crawl hits: wc -l vs sort | uniq -c

The humblest comparison, and people still pick the wrong one. Credit to the coreutils manual.

🔗 wc -l — answers one thing: how many lines total. Great for "how many Googlebot requests today," useless for which URLs.
🔗 sort | uniq -c — the workhorse for distribution: hits per URL, per status, per hour. The catch is uniq only collapses adjacent lines, so the sort first is mandatory, and on huge logs that sort is the slow step.

⭐ Pick of the week: the scaling fix experienced shell hands use — awk '{c[$7]++} END{for(u in c)print c[u],u}' counts per-URL in one pass, no sort, far faster on big files.

Takeaway: wc -l for a single total; sort|uniq -c for small distributions; an awk hash map once the file gets big.