AI companies use several kinds of web fetchers: some collect material for model development, some build search indexes, and others open a page because a user explicitly asked an assistant to visit it. Each crawler identifies itself with a User-Agent that can be observed in server logs and, when the operator supports the Robots Exclusion Protocol, managed through robots.txt.
The important distinction in 2026
Do not treat every AI crawler as one category. OpenAI separates GPTBot,OAI-SearchBot, and user-initiated access. Anthropic documentsClaudeBot, Claude-SearchBot, and Claude-User. Perplexity similarly distinguishes PerplexityBot fromPerplexity-User.
Verify current behavior in the official documentation from OpenAI, Anthropic, and Perplexity before deploying rules.
Three crawler purposes
| Purpose | What it does | Examples |
|---|---|---|
| Model development | Collects web material that may be used to improve or train models. | GPTBot, ClaudeBot, CCBot |
| AI search | Indexes pages so an AI search product can retrieve and cite current sources. | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| User action | Fetches a URL because a person asked an assistant to open, summarize, or use it. | ChatGPT-User, Claude-User, Perplexity-User |
OpenAI crawlers
GPTBot
GPTBot is used for model-development crawling. Blocking it is separate from visibility in ChatGPT search.
User-agent: GPTBot
Disallow: /OAI-SearchBot
OAI-SearchBot supports discovery and inclusion in ChatGPT search results. Publishers seeking citations and referral traffic generally need to allow it on public pages.
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /private/ChatGPT-User
ChatGPT-User represents user-initiated fetching rather than broad automated indexing. A robots rule is not a substitute for authentication: sensitive content must be protected by access control.
Anthropic crawlers
| User-Agent | Purpose |
|---|---|
ClaudeBot | Automated crawling for model improvement and related uses described by Anthropic. |
Claude-SearchBot | Search discovery and retrieval. |
Claude-User | A fetch initiated by a Claude user. |
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /Perplexity crawlers
PerplexityBot builds and refreshes the search index used for answers with citations.Perplexity-User fetches pages in response to user actions. Allowing the search bot can improve the chance that eligible public pages appear as sources, but it does not guarantee a citation or ranking.
Google-Extended
Google-Extended is a robots.txt control token, not a separate HTTP crawler User-Agent. It lets publishers manage whether content crawled by Google may be used for certain Gemini model training and grounding. Google states that this control does not affect inclusion or ranking in Google Search.
User-agent: Google-Extended
Disallow: /See the current Google crawler documentation.
Other crawlers you may see
| User-Agent | Operator | Typical role |
|---|---|---|
CCBot | Common Crawl | Builds open web datasets used by research and industry. |
Bytespider | ByteDance | Automated web collection for search and AI products. |
Amazonbot | Amazon | Supports Amazon search and assistant products. |
Applebot-Extended | Apple | Publisher control related to generative AI uses. |
FacebookBot | Meta | Automated collection for Meta products. |
cohere-ai | Cohere | AI-related crawling. |
A practical access policy
Many publishers allow AI-search crawlers on public documentation while blocking broad model-development crawling and private paths. The exact policy should reflect your licensing, traffic goals, server capacity, and contractual obligations.
# Allow AI-search discovery
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Opt out of selected model-development crawling
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
# Never rely on robots.txt to secure private content
User-agent: *
Disallow: /admin/
Disallow: /account/robots.txt, llms.txt, and ai.txt solve different problems
robots.txtcommunicates crawler access preferences.llms.txtpresents a concise map of useful content and canonical links.ai.txtcan state a broader AI usage policy, but support is not universal.- Authentication, authorization, and a WAF enforce access; text files do not.
Verification checklist
- Open
/robots.txtpublicly and check syntax and conflicting groups. - Review access logs for the exact User-Agent and request pattern.
- Verify published IP ranges when the operator provides them.
- Protect non-public content with authentication instead of crawler directives.
- Run the llmsmap.me AI-readiness audit after deploying changes.
Summary
AI crawler policy is no longer a single allow-or-block switch. Separate model-development crawling, search indexing, and user-initiated retrieval; document the reason for each rule; monitor real traffic; and revisit the policy as providers update their crawlers.
Continue with the robots.txt configuration guide, then publish a structured context file with the llms.txt generator.