robots.txt disallow vs noindex for facets — what's the difference?

robots.txt disallow vs noindex for facets — what's the difference?

'Q: Everyone tells me to block filters in robots.txt. Others say noindex. They can't both be right?'

Short answer: robots.txt saves crawl budget but can't remove already-indexed URLs; noindex removes them but burns crawl budget. They solve opposite problems.

The longer version: If Google has never indexed your filter URLs and you just want to stop it wasting crawls, robots.txt Disallow is right — it blocks the fetch entirely. But here's the trap: a robots-blocked URL can still get indexed (URL-only, no snippet) if it has external links, because Google never sees the noindex tag inside it. So if URLs are already in the index, robots.txt locks the problem in place.

Example: 40,000 indexed filter URLs polluting your site? Leave them crawlable, add noindex, wait for Google to drop them, then block in robots.txt to protect budget.

Rule of thumb: noindex first to clean up, robots.txt after to prevent re-crawl. Never both at once on the same URL.

Got an e-com SEO question? Drop it.