A checklist to verify AI crawlers can actually reach your best content

A checklist to verify AI crawlers can actually reach your best content

Question: are you accidentally blocking the bots that feed AI search, or letting them waste budget on junk? A two-sided access audit.

— Step 1: open robots.txt and list every user-agent rule. Identify which AI crawlers you allow or block: GPTBot, ClaudeBot, PerplexityBot, Google-Extended.
— Step 2: decide deliberately per bot — blocking Google-Extended removes you from some AI training while keeping normal Search. Know which trade you're making.
— Step 3: confirm your key pages aren't blocked by a broad Disallow or a noindex meta tag.
— Step 4: check that important content isn't locked behind JavaScript that non-rendering crawlers can't execute — view the raw HTML source.
— Step 5: verify crawl budget isn't burned on faceted/filter URLs; Disallow those patterns.
— Step 6: spot-check server logs for which AI agents actually hit you and what they fetch.

Method note: robots rules read directly; real crawler behavior confirmed against server access logs.

Caveat: honoring robots.txt is voluntary and bot behavior changes; logs show what's truly happening better than the config alone.

Confidence: high