Your origin logs lie about Googlebot when there's a CDN in front

Your origin logs lie about Googlebot when there's a CDN in front

Myth: pull the server access logs, analyze the crawl.

Reality: if Cloudflare, Fastly, or any CDN caches your pages, Googlebot's request is often served from edge cache and never reaches your origin. Your origin log shows a fraction of the real crawl. You're analyzing the requests that happened to miss cache — a biased sample.

Here's the thing. The CDN's own logs hold the truth: every Googlebot hit, cached or not, at the edge. That's where you see the real crawl volume and what's eating it.

The tradeoff: origin logs are easy to reach but blind to cached traffic. CDN logs are complete but often locked behind the higher plan tier or a log-push setup nobody configured.

Analyzing origin logs behind a CDN is reading half the story and calling it the whole.