Mistake: BigQuery log tables that scan terabytes every query

Mistake: BigQuery log tables that scan terabytes every query

Loading logs into one unpartitioned table turns every bot query into a full-table bill. Curated:

→ Google Cloud BigQuery docs (partitioning) — Partition by ingestion date or a DATE column so a one-day crawl query reads one day.
Takeaway: PARTITION BY DATE(timestamp) on load, not after.

→ BigQuery clustering docs — Cluster by user-agent or status to prune within partitions.
Takeaway: cluster on the columns you filter bots by most.

★ Pick of the week — a cost-control pattern from the GCP blog — require_partition_filter forces every query to include a date bound.
Takeaway: turn it on so nobody accidentally scans all of history.