Mistake: your noindex is below the fold of the HTML and Google skips it
Subtle one. A noindex meta tag injected by JavaScript, or a robots directive that only appears deep in the rendered DOM after a slow client-side load. You assume Google honors it. Sometimes it doesn't see it in time.
Here's the thing — Google needs to crawl, render, and reach that tag for it to count. If it's JS-injected and rendering is deferred or fails, Googlebot may index the page before the noindex ever materializes. Worse, conflicting signals (server header says index, JS says noindex) get resolved unpredictably.
The fix: serve robots directives in the raw HTML <head> or, better, as an X-Robots-Tag HTTP header. No rendering required, no ambiguity.
Google reads the header before it reads a word of your page.
Put the directive where the crawler looks first.
Budget Myths
@CrawlBudgetMyths
Mistake: your noindex is below the fold of the HTML and Google skips it
Этот пост опубликован в Telegram-канале Budget Myths. Подписаться можно по ссылке: @CrawlBudgetMyths.