SOP: Sweep noindex conflicts out of your sitemap

SOP: Sweep noindex conflicts out of your sitemap
A URL in the sitemap that says noindex is a contradiction Google logs against you.

Step 1 — Export all sitemap URLs to a list.

Step 2 — Crawl that exact list (Screaming Frog: list mode). For each URL collect: HTTP status, meta robots, X-Robots-Tag header, canonical target.

Step 3 — Flag any URL that is NOT all four of these:
— ✅ Returns 200
— ✅ No 'noindex' in meta OR header
— ✅ Self-canonical (canonical points to itself)
— ✅ Not blocked in robots.txt

Step 4 — Remove every flagged URL from the sitemap. The sitemap is for indexable pages only — fix-or-remove, never both signals.

Step 5 — Re-crawl to confirm zero flags.

Definition of done: 100% of sitemap URLs pass all four checks, verified by a clean second crawl with no exceptions logged.