<b>AI crawlers in your logs: who's fetching and how to tell</b>

<b>AI crawlers in your logs: who's fetching and how to tell</b>
GPTBot, ClaudeBot, PerplexityBot now show up in access logs. A field guide.

→ <b>Dark Visitors agent list</b> — continuously updated catalog of AI/LLM crawler User-Agents and their declared purposes. The reference for labeling these hits.
★ <b>Pick of the week — Cloudflare Radar's AI-crawler verification</b> — publishes the IP ranges and verification methods for major AI bots, so you can confirm a "GPTBot" hit is real and not a scraper borrowing the name.
→ <b>OpenAI's GPTBot docs</b> — official IP ranges and robots directives.
→ <b>Originality.ai's AI-bot traffic studies</b> — useful benchmarks for how much of your crawl is now LLM training/retrieval.

Takeaway: AI crawlers verify like Googlebot — published IPs and UAs. Label them in your pipeline; their crawl patterns differ wildly from search bots.