How to stand up log-level analysis from zero — a 6-step setup

How to stand up log-level analysis from zero — a 6-step setup

Log-level data (one row per bid opportunity, not aggregated reports) is the only ground truth in programmatic. Here's the order to build the pipeline.

1. Request the feed from your DSP and confirm the schema includes, at minimum: auction ID, timestamp, bid, win price, win flag, supply chain object, and deal ID.

2. Land raw files in object storage partitioned by date and hour — never overwrite, you'll need replays when a metric looks wrong.

3. Build one canonical join key. The auction ID ties bid requests to wins; without it you cannot connect a loss to its clearing price.

4. Reconcile against the dashboard first. Sum spend and impressions from logs and confirm they land within 1-2% of the UI. If they diverge wildly, you're missing a partition or double-counting multi-seat bids.

5. Materialize three core daily tables: bids, wins, and losses-with-clearing-price. Most analysis is a query against these.

6. Only now compute derived metrics (win rate, surplus, shave). Derived numbers built on an unreconciled feed are confidently wrong.

Why it matters: Teams rush to dashboards on top of log data before reconciling the raw feed, and then make six-figure decisions on numbers that silently disagree with billing. The reconciliation step is unglamorous and non-negotiable — it's what separates analysis from guessing.