Case: Nightly sweep removed 8,700 dead URLs

Case: Nightly sweep removed 8,700 dead URLs
A UGC site let deleted posts linger in the sitemap. ~8,700 URLs returned 404 or 410, dragging crawl efficiency.
What we did:
— Added a nightly job: any URL whose status changed from the DB is dropped
— Kept the sitemap in sync with the live published flag
— Set deleted pages to return 410, not 404
Measured outcome over 4 weeks:
— Dead URLs in sitemap: 8,700 → under 50 at any time
— GSC 'Not found (404)' from sitemap: down 92%
Checklist:
✅ Sitemap regenerated from current live records only
✅ No 404/410 URL persists past one cycle
Definition of done: nightly job removes any URL whose status is no longer 200.