How AI crawlers actually work (and how to stop blocking them)

An AI answer engine that searches the live web before responding needs a copy of your page first. That copy comes from a crawler - a separate piece of infrastructure from the model itself, with its own user agent, its own crawl budget, and its own rules for what it's allowed to fetch. If that crawler can't reach your page, no amount of good content on it matters: it was never read.

The bots worth knowing by name

• GPTBot and OAI-SearchBot (OpenAI) - GPTBot mainly feeds model training; OAI-SearchBot powers ChatGPT's live web search and is the one that matters for being cited in an answer today.
• ClaudeBot (Anthropic) - fetches pages for both training and Claude's web-search tool.
• PerplexityBot - Perplexity's answers are built almost entirely from live retrieval, which makes this one of the highest-leverage crawlers to stay open to.
• Google-Extended - a separate opt-in from classic Googlebot, controlling whether Google's AI features (Gemini, AI Overviews) can use your content.

The robots.txt that actually allows them

This is the explicit version - naming each bot rather than relying on a wildcard, so there's no ambiguity about intent when you or a future teammate reads it back:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

Why sites block them without deciding to

Almost nobody sits down and decides to block AI crawlers on purpose while wanting AI visibility - it happens as a side effect of something else. A "block everything except the search engines we recognize" robots.txt template written before these bots existed. A bot-protection service (Cloudflare's Bot Fight Mode and similar) that challenges or blocks unrecognized user agents by default. A CDN's abuse detection flagging a crawler's request pattern as scraping. Each of these is a reasonable default that happens to catch AI crawlers in the net meant for something else.

If you want AI visibility, blocking the crawlers that produce it is a direct contradiction - and it's usually accidental.

How to actually check

Read your robots.txt line by line, not just its intent - a broad Disallow at the top can silently override a specific Allow further down depending on how it's ordered. Then check your edge: most bot-protection dashboards let you filter blocked requests by user agent, so search for GPTBot, ClaudeBot, and PerplexityBot in whatever's rejecting traffic before it reaches your app. A robots.txt that welcomes every bot means nothing if a WAF rule is returning 403 first.

Then confirm it directly, rather than trusting the dashboard - fetch a key page with the bot's real user-agent string and read back what actually comes through:

curl -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  -o - -s https://example.com/pricing | head -50

If that comes back with a 403, a CAPTCHA challenge page, or an empty shell instead of your pricing content, you've found the block - and you've found it faster than paging through a WAF's log viewer.

It's not only robots.txt

Googlebot has rendered JavaScript for years. Several AI crawlers still don't - they read the HTML that comes back from the first request and move on. If your pricing, your product description, or your comparison table only appears after a client-side fetch, a crawler that doesn't execute JavaScript sees an empty shell where your facts should be. Server-rendered content isn't just good practice here; for some of these bots, it's the only way they see your page at all.

This is exactly what the curl command above tests for, and it's worth running against every page you actually want cited - your pricing page, your comparison pages, your documentation - not just your homepage. A homepage that renders fine while your pricing page loads its numbers from a client-side API call is a common, easy-to-miss gap.

None of this requires guessing. Fetch your own key pages with each bot's exact user-agent string and read back what actually comes through - if the facts you want cited aren't in that response, they were never in the running.