robots.txt disallow vs noindex for crawl-budget control

robots.txt disallow vs noindex for crawl-budget control

The most common mix-up in scaled SEO. They are not the same tool.

1. robots.txt disallow: stops the crawl. Saves crawl budget. But — a disallowed URL can still get indexed (URL-only) if linked, and Google can NEVER see a noindex inside a blocked page.
2. noindex meta/header: stops indexing, but the page must be crawlable for Google to read the tag. Costs crawl budget every visit.
3. Decision SOP:
— Want it gone from index AND it's already crawlable → noindex first, let it deindex, THEN consider disallow.
— Pure infinite-space junk (calendars, session params) never meant to index → disallow at robots.txt.
4. The classic bug: disallow + noindex together. Google can't read the noindex behind the disallow, so it stays indexed as a URL-only entry. Pick one.
5. Gate: never disallow a path you also want deindexed until it's already out of the index.

Rollback: robots.txt edits apply on next crawl; reversible in minutes.

Ship gate: don't publish until all boxes are checked.