On-Request vs Pre-Built Sitemap Generation: Choose By Crawl Cost

On-Request vs Pre-Built Sitemap Generation: Choose By Crawl Cost

Two ways to serve sitemap.xml. Pick by how expensive the query is.

Generate on request (DB query per hit) when:
— Under ~50,000 URLs total.
— Query returns in under 1 second.
— You need always-fresh lastmod within minutes.

Pre-build to a static file (cron/job) when:
— Over 50,000 URLs or multi-file index.
— The query touches large joins or scans.
— Bots hit the sitemap dozens of times daily.

The trap with on-request: Googlebot may fetch your sitemap 50+ times/day. A 4-second query × 50 = avoidable load and timeout risk. A timed-out sitemap = silent drop in discovery.

Definition of done:
— ✅ Sitemap responds in under 1 second at p95.
— ✅ If pre-built, regeneration runs on a schedule AND on publish.
— ✅ Caching layer returns last good file if the job fails.