robots.txt Disallow vs Sitemap Exclusion: They Are Not Interchangeable

robots.txt Disallow vs Sitemap Exclusion: They Are Not Interchangeable

Teams reach for the wrong one constantly. Match the tool to the goal.

robots.txt Disallow:
— Stops crawling. Does NOT reliably deindex. A disallowed URL can still rank as a bare link.
— Use to save crawl budget on infinite spaces (filters, search results).

Sitemap exclusion (just don't list it):
— Removes the "please crawl this" hint. Does nothing to stop crawling via internal links or deindex anything.
— Use for URLs that are fine but shouldn't be advertised as priority.

meta noindex:
— The only reliable deindex tool. Page must stay crawlable.

The fatal combo:
— robots Disallow + expecting deindex. Blocked = Google never sees your noindex.

Pass/fail:
— ✅ Want it gone from index → noindex, NOT robots block.
— ✅ Want to save crawl budget → robots block, accept it may still rank as a link.

Definition of done: right lever for the right outcome.