Playbook: Stop Google from indexing your cloaked redirect URLs

Playbook: Stop Google from indexing your cloaked redirect URLs

If your /recommends/ or /go/ links show up in Search Console as indexed pages, you've got crawl-budget waste and possible thin-content flags. Clean it up.

— Confirm the problem: site:yourdomain.com/recommends in Google. Any results = leak.
— Add a Disallow: /go/ (your slug) line to robots.txt. This stops crawling but NOT deindexing of already-indexed URLs.
— For already-indexed redirect URLs, you can't add noindex (they redirect), so use the URL removal tool plus robots disallow together.
— Make sure your cloaker isn't auto-generating a sitemap entry for redirect links — some do by default.
— Set the redirect base path to noindex via X-Robots-Tag header on the redirect handler if your stack allows.

Gotcha: robots.txt disallow alone won't deindex; Google keeps the URL with "no info available." You need disallow PLUS removal, in that order, or it re-indexes.

Bottom line: disallow the slug folder, kill sitemap auto-entries, X-Robots-Tag on the handler, then request removal. Disallow alone is the mistake everyone makes.