robots.txt Disallow vs meta noindex: the mix-up that traps pages

robots.txt Disallow vs meta noindex: the mix-up that traps pages

This tradeoff confuses people weekly, so let's settle it.

A robots.txt Disallow blocks crawling. A noindex tag blocks indexing. The trap: if you Disallow a URL, Google can't crawl it to see your noindex, so it can stay indexed as a URL-only entry forever.

Pros (Disallow): saves crawl budget, instant, no page edit needed.
Cons (Disallow): does NOT remove a page from the index; can leave naked URL listings.
Pros (noindex): actually drops the page from results.
Cons (noindex): page must stay crawlable for Google to read the tag; spends crawl budget.

Best for Disallow: infinite faceted/filter URLs you never want crawled.
Best for noindex: real pages you want gone from results, kept crawlable.
Never combine them on the same URL if removal is the goal. This one's a 5/5 'must understand'.