robots.txt vs. meta robots vs. X-Robots-Tag: pick by file type
Three directive layers, three jobs. Audited mis-application across 81 domains.
— Trying to noindex a PDF/image with meta tag: fails (no HTML head) — found on 22.2% of sites ▓▓▓▓░░░░░░
— X-Robots-Tag header for non-HTML: correct, used by only 38% ▓▓▓▓░░░░░░
— robots.txt disallow used to "noindex": ineffective, found on 17.3% ▓▓▓░░░░░░░
So what: the directive has to match the resource. HTML pages → meta robots. PDFs, images, feeds, any non-HTML → X-Robots-Tag header (the only layer that reaches them). robots.txt controls crawling, never indexing. A fifth of sites try to noindex a PDF with a tag the file can't carry. Map directive to file type first; the tactic is only as right as the layer you put it on.
Crawl & Render
@CrawlAndRender
robots.txt vs. meta robots vs. X-Robots-Tag: pick by file type
Этот пост опубликован в Telegram-канале Crawl & Render. Подписаться можно по ссылке: @CrawlAndRender.