How AI crawlers work: training, search and user requests

Training crawlers, search crawlers and user-requested fetchers serve different purposes. Choose access for each purpose independently. Crawl access can enable discovery, but does not guarantee an answer, citation or recommendation.

Anthropic: three different agents

Anthropic documents three agents: ClaudeBot collects material for model development; Claude-SearchBot indexes content to improve search results; Claude-User retrieves pages in response to a user request. Their robots.txt controls are separate. Blocking training does not require blocking search or user-requested access.

Google Search and Google-Extended are separate controls

Google Search AI features, including AI Overviews and AI Mode, use Googlebot access and Search controls such as nosnippet, data-nosnippet, max-snippet and noindex. Googlebot must be able to crawl a page to read its preview directives.

Google-Extended is a robots.txt product token, not a separate HTTP crawler identity. It controls use for future Gemini model training and grounding in specified Gemini Apps and Vertex AI services. It does not control inclusion or ranking in Google Search.

Choose access by purpose

Check OpenAI's crawler documentation and Perplexity's crawler documentation for their current identities and verification guidance. Do not treat permission for model training as a prerequisite for search visibility.

A focused robots.txt example

This example allows Anthropic search and user-requested retrieval while declining ClaudeBot training crawls. Adapt it to your policy and existing private-path restrictions; do not replace an entire robots.txt file blindly.

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
Disallow: /account/

Sitemap: https://example.com/sitemap.xml

Specificity, not line order

Under Google's robots.txt specification, the most specific matching user-agent group applies. Repeated groups for that agent are combined; the wildcard group is not added to a specific group. Within the applicable rules, the longest matching path wins. An equally specific Allow/Disallow conflict resolves to Allow. Moving a rule earlier or later does not change that priority.

User-agent: ExampleBot
Disallow: /
Allow: /public/

# /public/article is allowed; /account is disallowed.
# Reversing these two rules does not change the result.

RFC 9309 defines the Robots Exclusion Protocol and its matching rules. Individual agents can have documented extensions. Robots.txt is a crawl preference, not authentication: protect private data with access controls.

Check the network edge too

A CDN or firewall can return a challenge even when robots.txt allows a request. Inspect verified bot traffic in your logs and test a public page. Sending a bot user-agent string with curl is only a diagnostic: it does not reproduce the provider's IP address, bot verification or rendering behavior.

curl -A "Claude-SearchBot" -i https://example.com/pricing

Look for the expected status and meaningful page content rather than a challenge or empty shell. A successful curl response does not prove that a provider indexed the page. Do not assign a universal JavaScript capability to all AI crawlers; consult current documentation and test your own rendering.