Robots.txt for AI crawlers: block, allow, or split access by bot

Robots.txt is not a switch for “AI” in general. It is a set of path rules for specific crawlers, so the decision has to be made bot by bot: GPTBot, ClaudeBot, PerplexityBot, and any other user agent you actually see in logs.

Use three access levels:
• Full allow: if you want models and answer engines to reach your public content.
• Partial allow: open your blog, docs, or landing pages, but block checkout, internal search, and account areas.
• Full block: for paywalled, private, or low-value pages that should not be crawled at all.

Keep the file simple. Put broad rules first, then carve out exceptions, and make sure the path list matches your real site structure. A common mistake is blocking the whole domain just to protect one section; that can hide useful pages from search-like AI systems too.

Remember: robots.txt is a crawler request, not a content firewall. If a page must stay out of AI responses, pair robots rules with noindex where appropriate, limit public links, and avoid exposing sensitive text in HTML.

Best default: allow crawlers on content you want cited, block them on pages you would not want quoted out of context.