Mining status codes: targeted grep vs a status dashboard

Mining status codes: targeted grep vs a status dashboard

Hunting 4xx/5xx the bots actually hit. Two approaches, each with a job.

🔗 Targeted grep — awk '$9 ~ /^5/ {print $7}' | sort | uniq -c | sort -rn answers "which URLs threw 5xx, how often" in one shot. Credit to the old Distilled log-analysis playbooks.
🔗 Status dashboard (GoAccess/Kibana) — better when you need the trend: is the 404 rate creeping up week over week, and is it Googlebot or scrapers eating them?

⭐ Pick of the week: the diagnostic from Google's docs — cross status code with user-agent. A 503 served only to Googlebot during recrawl is a very different fire than a 503 hitting everyone.

Takeaway: grep to find today's offending URLs; a dashboard to catch a slow status-code drift before it costs you crawl budget.