Index or Bust
Index or Bust
@IndexOrBust

The robots.txt mistakes that silently tank your indexation.

robots.txt errors are uniquely dangerous because they fail silently — the site keeps working for users while Googlebot gets blocked from entire sections of your site.

The most damaging patterns I see:

1. Blocking /wp-admin AND /wp-content by copy-pasting a "standard" robots.txt — this accidentally blocks CSS and JS files Googlebot needs to render pages

2. Disallowing staging environments that share the same subdomain as production (Google indexes your staging content)

3. Using robots.txt to hide pages you actually want indexed but don't want users to navigate to directly (robots.txt is not access control)

4. Wildcard patterns that are too broad: `Disallow: /*?` blocks all query parameters including legitimate filter URLs you want indexed

5. Forgetting that robots.txt syntax is case-sensitive and Google-specific rules differ from Bing-specific rules

Validation: Google Search Console → Settings → robots.txt inspector. Test your actual URL patterns before assuming the file works as intended.
This post was published in the Telegram channel Index or Bust (@IndexOrBust). All its posts are on the channel page.
Open on Telegram

Latest posts in “Tools and infrastructure”

All channels in this category →

реклама в сети

Готовы запустить рекламу через сеть public.tg?

Новый оффер, продукт, GEO, кейс, событие или партнёрский запуск — соберём маршрут под задачу и отдадим медиаплан.

Telegram для медиаплана: @AFFtop_connect. Быстрый тест: $20 за канал, $1000 за пакет по сети.