robots.txt errors are uniquely dangerous because they fail silently — the site keeps working for users while Googlebot gets blocked from entire sections of your site.
The most damaging patterns I see:
1. Blocking /wp-admin AND /wp-content by copy-pasting a "standard" robots.txt — this accidentally blocks CSS and JS files Googlebot needs to render pages
2. Disallowing staging environments that share the same subdomain as production (Google indexes your staging content)
3. Using robots.txt to hide pages you actually want indexed but don't want users to navigate to directly (robots.txt is not access control)
4. Wildcard patterns that are too broad: `Disallow: /*?` blocks all query parameters including legitimate filter URLs you want indexed
5. Forgetting that robots.txt syntax is case-sensitive and Google-specific rules differ from Bing-specific rules
Validation: Google Search Console → Settings → robots.txt inspector. Test your actual URL patterns before assuming the file works as intended.
Index or Bust
@IndexOrBust
The robots.txt mistakes that silently tank your indexation.
This post was published in the Telegram channel Index or Bust (@IndexOrBust). All its posts are on the channel page.
Open on Telegram