Mistake: grepping the bare string "Googlebot" for crawl analysis

Mistake: grepping the bare string "Googlebot" for crawl analysis

It scoops up impersonators and conflates four different crawlers. Better reads:

→ Google Search Central (crawlers list) — There's Googlebot Smartphone, Desktop, Image, plus fetchers like Google-InspectionTool and the user-triggered ones. They behave differently.
Takeaway: split by full user-agent token, not the word "Googlebot."

→ Stack Overflow (regex for bots) — Why a naive grep counts SemrushBot-Googlebot-clone spoofs as real.
Takeaway: anchor the UA pattern and pair it with IP verification.

★ Pick of the week — the official UA token table — Exact strings for every Google crawler.
Takeaway: build a small lookup file and classify each line once.