Crawler simulation vs server logs for crawl diagnosis

Crawler simulation vs server logs for crawl diagnosis

Two ways to see your site as a bot does. They answer different questions — use the right one.

1. Crawler simulation (Screaming Frog / Sitebulb): shows what's crawlable — broken links, depth, canonical chains, what bots COULD reach.
— Use before launch and for structure audits.
2. Server logs: show what bots actually crawled — real Googlebot hits, frequency per path, wasted budget on junk URLs.
— Use after launch to find where crawl budget leaks.
3. Decision rule: "can it be crawled?" → simulator. "is it being crawled, and what's eating budget?" → logs. A simulator cannot tell you Googlebot spent 40% of hits on parameter URLs. Only logs can.
4. Guardrail: verify bot identity by reverse DNS in logs — fake Googlebots pollute the data.
5. Gate: monthly log review on any site over 10k URLs.

Rollback: n/a — diagnostic only.

Ship gate: don't publish until all boxes are checked.