LLM crawlers and user-agents: the bots that read you for AI
Not one bot — several, with different jobs
"AI crawler" isn't a single thing. Most AI companies run at least two kinds of bot, and they serve different purposes:
- Training crawlers gather text to help train or fine-tune a model. This happens on a slow cycle — what they read today might not affect an answer for months.
- Retrieval / search crawlers fetch content in real time, at the moment a user asks a question, so the model can ground its answer in current information and cite a source. This is the crawler traffic that matters for showing up in an answer today.
OpenAI, for example, runs GPTBot for training and OAI-SearchBot / ChatGPT-User for retrieval at answer time — three separate user-agents with three separate robots.txt rules.
The major crawlers by user-agent
| User-agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data collection |
| OAI-SearchBot | OpenAI | Real-time retrieval for ChatGPT search |
| ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT to browse it directly |
| PerplexityBot | Perplexity | Indexing and retrieval for Perplexity answers |
| Perplexity-User | Perplexity | Fetches a page when a user asks Perplexity to browse it directly |
| ClaudeBot | Anthropic | Training data collection |
| Claude-Web / anthropic-ai | Anthropic | Retrieval for Claude's web-browsing features |
| Google-Extended | Controls use of your content in Gemini and AI Overviews (separate from classic Googlebot) | |
| Applebot-Extended | Apple | Controls use of your content in Apple Intelligence features |
| CCBot | Common Crawl | Public web archive many AI labs train on indirectly |
Why this is different from Googlebot
Googlebot crawls to build a ranked index — your page earns a position, and a person decides whether to click. LLM crawlers feed a different output entirely: a synthesized answer with no ranked list. Being crawled by GPTBot doesn't guarantee a citation the way ranking on page one used to guarantee visibility — it's a prerequisite, not a result. A site an AI crawler can't read can never be cited, no matter how strong the content is otherwise.
How to check whether your site allows AI crawlers
Look at your site's robots.txt file, at yoursite.com/robots.txt. A blanket Disallow: / under User-agent: * blocks everything unless a specific user-agent is separately allowed. Here's what an explicit allow looks like:
Many default WordPress and site-builder configurations block everything by default, or were written before these user-agents existed and simply never mention them — which usually means they're allowed by omission, but it's worth confirming explicitly rather than assuming.
Frequently asked questions
What is an LLM crawler?
An LLM crawler is an automated bot operated by an AI company that visits web pages to gather text for training data, real-time retrieval, or both. Examples include GPTBot and OAI-SearchBot (OpenAI), PerplexityBot (Perplexity), ClaudeBot (Anthropic), and Google-Extended (Google's AI features).
How is an LLM crawler different from Googlebot?
Googlebot crawls to build a traditional search index of ranked links. LLM crawlers gather text either to train a model in advance, or to fetch fresh content at answer time so the AI can cite it in a generated response. Some AI companies run separate bots for each purpose.
How do I check if my site blocks AI crawlers?
Check your site's robots.txt file (yoursite.com/robots.txt) for Disallow rules under user-agents like GPTBot, PerplexityBot, ClaudeBot, or Google-Extended. If a bot is disallowed, or if your robots.txt blocks everything by default, that engine likely cannot read your site at all.
Should I block AI crawlers?
If you want AI answer engines to be able to name and cite your business, no — blocking them removes you from consideration entirely. Some publishers block AI crawlers deliberately to protect content from being reused without compensation; that's a legitimate but separate tradeoff, covered in the Legal & Ethics volume of this encyclopedia.
Is your robots.txt actually blocking AI?
Run a free scan across ChatGPT, Perplexity, Gemini and Claude — no credit card.