<b>Cache your dynamic sitemap — but bound it</b>

<b>Cache your dynamic sitemap — but bound it</b>

Generating a sitemap from a live DB query on every bot hit will get you 500s under crawl load. Cache, with a hard ceiling.

SOP:
Step 1 — Generate to a static file or cache layer, not on-request from the database.
Step 2 — Set regeneration cadence to match content velocity: hourly for news, daily for catalogs.
Step 3 — Cap query: never <code>SELECT *</code> the whole table into memory — paginate generation in chunks of 10,000.

Pass/fail:
— ✅ Sitemap response time under 1 second.
— ✅ Regeneration runs on a schedule, decoupled from the crawler request.
— ❌ TTFB spikes when Googlebot hits → you're building it live.

Definition of done: the crawler is served a pre-built file; regeneration is a background job on a fixed schedule.