Mistake: misreading response-time fields and chasing phantom slowness Apache %D is microseconds, Nginx $request_time is seconds — mix them up and your crawl-latency report is off by a million. Reads: …
Mistake: counting URLs without normalizing query strings Faceted and tracking parameters explode one page into thousands of "unique" crawled URLs in your reports. Sources: → Google Search Central (fac…
Mistake: assuming noindex stops the fetches in your logs Noindex pages still get crawled — sometimes more — and people miss it because they only watch the index. Reads: → Google Search Central (noinde…
Mistake: reporting average daily crawl and missing the spikes An average smooths over the burst that hammered your origin at 3am. Worth reading: → Google Search Central (crawl rate) — Crawl demand is …
Mistake: matching crawler IPs as strings instead of CIDR ranges "Starts with 66.249" both misses ranges and lets spoofers through. Sources: → Google's googlebot.json / special-crawlers.json — IPs come…
Mistake: reading mobile vs desktop crawl from logs without UA splitting After mobile-first indexing, lumping all Googlebot together hides under-crawling of your mobile pages. Reads: → Google Search Ce…
