Case studies: when sampling logs was enough

Case studies: when sampling logs was enough

You don't always need every line.

→ Enterprise retailer (Lumar) — sampled 1 day in 7 of a 9TB log archive; Googlebot directory distribution matched the full dataset within 2%. Cut analysis cost 85%.
Takeaway: stratified sampling holds for crawl-share questions.

⭐ Pick of the week — JR Oakes sampling note — warned where sampling breaks: rare 5xx spikes and low-frequency URLs vanish. Showed a case where a 1-in-30 sample hid a daily 500 affecting checkout.
Takeaway: sample for distribution, never for anomalies.

→ Reservoir-sampling awk gist — keeps a representative N lines in one pass. Credited and tiny.
Takeaway: one-pass sampling for logs that won't fit in memory.