Making lastmod honest: re-crawl latency dropped 61%

Making lastmod honest: re-crawl latency dropped 61%

Publisher had been stamping every sitemap entry with today's date on each build. 33,000 URLs, ~99% falsely 'fresh.'

We wired lastmod to true content-change timestamps. Of 33,000 URLs, only 4.1% had actually changed in 90 days.

Delta, measured by log timestamps on the genuinely-updated subset:
— Median crawl latency after edit: 8.9d → 3.5d (↓60.7%)
— p90 crawl latency: 26d → 9d
— Wasted re-crawls of unchanged pages: ↓ ~41% of total volume

So what: a sitemap that cries wolf gets ignored. Once lastmod correlated with reality (Pearson ≈ 0.0 → ~0.8), the crawler started trusting it as a scheduling hint.