SOP: Wire sitemaps into robots.txt correctly
The one line every crawler reads first.
Step 1 — Add Sitemap: on its own line, absolute URL, at the bottom of robots.txt. Example: Sitemap: https://example.com/sitemap-index.xml
Step 2 — ✅ Point to the index file, not each child.
Step 3 — Multiple sitemaps from one host? List each on its own Sitemap: line. ✅ All on the same protocol+host as robots.txt.
Step 4 — The directive is host-independent of crawl rules — it ignores User-agent groups. ✅ Place it outside any User-agent block.
Step 5 — Confirm the path you reference is not itself Disallowed above.
Step 6 — Fetch robots.txt as a crawler. ✅ Returns 200, plain text, under 500 KB.
Definition of done: robots.txt names your index by absolute URL, returns 200, and GSC's robots tester reports the directive without warnings.
The Sitemap SOP
@SitemapSOP
SOP: Wire sitemaps into robots.txt correctly
Этот пост опубликован в Telegram-канале The Sitemap SOP. Подписаться можно по ссылке: @SitemapSOP.