Mistake: counting the same crawl twice behind a reverse proxy

Mistake: counting the same crawl twice behind a reverse proxy

Nginx-in-front-of-Apache stacks log every request twice and inflate crawl frequency. Sources:

→ Nginx access_log docs — Both the proxy and the origin write a line. If you concatenate logs, every Googlebot hit doubles.
Takeaway: pick one layer as the source of truth, usually the edge.

→ Apache mod_remoteip docs — Without it, your origin logs the proxy IP, so bot verification fails on internal addresses.
Takeaway: restore the real client IP, then dedupe across layers.

★ Pick of the week — a community awk dedupe pattern — Keys on request-id + timestamp to collapse duplicate edge/origin lines.
Takeaway: add an X-Request-ID and dedupe on it, not on the URL.