<b>Log files vs GSC: which tells the crawl-to-index truth?</b>

<b>Log files vs GSC: which tells the crawl-to-index truth?</b>

GSC tells you what Google reports. Server logs tell you what Googlebot actually did. They diverge more than you'd think.

Setup: pull access logs, filter verified Googlebot (reverse-DNS, not just UA string), or use Screaming Frog Log Analyser / a tool like Logflare.

What logs do well:
— Show real crawl frequency per URL — GSC's 'last crawl' is a single timestamp
— Expose crawl waste: bot hammering parameter/404 URLs instead of your money pages
— Reveal if 'Discovered - not indexed' was ever fetched at all

Where they fall short:
— Fake Googlebot UAs pollute data unless you verify IPs
— No 'indexed?' signal — logs are crawl-only, pair with Inspection API
— Big sites = gigabytes to parse

Pros: ground truth on crawl behavior.
Cons: setup friction, crawl-only, easy to misread spoofed bots.

Best for: diagnosing why discovered URLs sit uncrawled.

Not for: confirming index status — logs can't.