Roundup: CDN logs vs origin logs

Roundup: CDN logs vs origin logs

Where you read the log changes the answer.

→ Cloudflare crawl-analytics case — origin logs showed Googlebot hitting only 30% of pages, because the CDN served the rest from cache and never forwarded the request. Reading CDN logs revealed the bot crawled 3x more than origin suggested.
Takeaway: behind a CDN, origin logs undercount the bot badly.

⭐ Pick of the week — Fastly edge-log study — reconstructed true Googlebot crawl frequency from edge logs and found cached pages were being re-fetched far more than origin data implied, changing the budget verdict.
Takeaway: if you have a CDN, the edge log is the real crawl record.

→ Cloudflare crawler-analytics docs — the edge-log view, no setup.
Takeaway: check the edge before blaming origin.