Blocking crawl waste with robots.txt — the careful version

Blocking crawl waste with robots.txt — the careful version

robots.txt is the only tool that actually stops a fetch before it happens. But people wreck their indexing with it. Do this:

— List the exact URL patterns wasting crawl (from your logs).
— Confirm those URLs have no indexable value AND aren't linked-to for ranking.
— Write Disallow rules for the patterns, not blanket folders.
— Test every rule in the robots.txt Tester / a parser. One bad wildcard can disallow your whole site.
— Check you're NOT blocking URLs that are already indexed — blocked pages can't be removed or recanonicalized, they just freeze.
— Never block CSS/JS Google needs to render.

No: robots.txt does not deindex. Google's own docs say blocked URLs can still appear in results. Block to save crawl, not to remove pages.

Disallow stops crawling. It does not stop indexing. Don't confuse the two.