Case: 6,300 sitemap URLs blocked by robots.txt
A site listed URLs in its sitemap that Disallow rules in robots.txt blocked. GSC flagged the contradiction.
What we did:
— Ran every sitemap URL against the live robots.txt rules
— Found 6,300 URLs matching a Disallow path
— Decided per group: unblock genuine pages, drop the rest from the sitemap
Measured outcome over 4 weeks:
— 'Indexed, though blocked by robots.txt' warnings: 6,300 → 110
— Crawl budget redirected to indexable pages
Checklist:
✅ No sitemap URL matches a Disallow rule
✅ A URL is either crawlable AND in the sitemap, or in neither
Definition of done: zero sitemap URLs are blocked by robots.txt.
The Sitemap SOP
@SitemapSOP
Case: 6,300 sitemap URLs blocked by robots.txt
Этот пост опубликован в Telegram-канале The Sitemap SOP. Подписаться можно по ссылке: @SitemapSOP.