Can ChatGPT, Claude and Perplexity crawl your site? What we found checking 2,000 sites

The bots you're actually dealing with

"Block AI" used to mean one line for GPTBot. Today each major AI company documents several user agents, and they do different jobs:

Google is different again: AI Overviews and AI Mode use the normal Search systems (Google's AI optimization guide), so Googlebot access is what matters there. Google-Extended is a separate robots.txt token that Google lists for controlling whether your content is used for its Gemini models (Google's crawler list); it does not affect Search.

The practical upshot: if you want to show up in AI answers, the search bots matter more than the training bots.

What we checked, and how

We wanted to know what real sites do, not what blog posts recommend. On 1 November 2026 we took two slices of the public Tranco top-sites list (list ID 746VX):

• Big sites: ranks 1–1,000.
• Smaller sites: ranks 200,001–201,000, which is a better proxy for a typical business website.

For each domain we fetched /robots.txt with a clearly labelled research user agent and ran it through an RFC 9309 matcher (the open standard for robots.txt, RFC 9309). The matcher picks the most specific user-agent group, uses longest-match rules, and handles * and $ wildcards. We tested the home page path / for 11 bots. 517 big sites and 587 smaller sites returned a readable text robots.txt; percentages below use those numbers.

What this does not measure: firewall or CDN blocks (we did not pretend to be GPTBot), per-page rules deeper than /, or what any company actually does with the content. The scripts and raw data are linked at the end.

What we found

1. Training bots get blocked about three times as often as search bots on smaller sites

Among smaller sites, 17.4% block GPTBot and 17.2% block ClaudeBot, but fewer than 6% block any of the search or user-fetch bots. Whether on purpose or not, most sites that block "AI" are blocking training and leaving AI search open.

2. Blocking GPTBot does not block ChatGPT search unless you say so

Of the 102 smaller sites that block GPTBot, 74 still allow OAI-SearchBot. For Anthropic it's 80 of 101 sites blocking ClaudeBot that still allow Claude-SearchBot. In most of these files the search bot isn't mentioned at all, so it falls back to the User-agent: * group, which usually allows everything. If your intent was "keep my site out of AI completely", this is the gap. If your intent was "don't train on me, but do send me visitors", this is exactly right. Either way, it should be a decision, not an accident.

3. Big sites are much stricter with AI search

12.2% of big sites block OAI-SearchBot and 17.2% block PerplexityBot, two to three times the rate on smaller sites. Big publishers have licensing reasons to say no. A small business usually has the opposite problem: it wants to be recommended.

4. Some sites block even a polite robots.txt request

62 of the big sites and 58 of the smaller sites returned HTTP 403 (forbidden) when we asked for robots.txt itself. That's a strong hint of a firewall or bot-protection layer sitting in front of the site. Robots.txt can say "welcome" while the firewall says "no". Cloudflare, for example, began asking new domains in July 2025 to choose whether to allow AI crawlers, with blocking as the default, and per-crawler controls now live in AI Crawl Control.

How to check your own site in 5 minutes

1. Open https://yourdomain.com/robots.txt in a browser. No file at all (a 404) means crawlers may fetch everything.
2. Find the groups. Each block starts with one or more User-agent: lines followed by Allow:/Disallow: lines. A bot uses the most specific group that names it, and only falls back to User-agent: * if nothing names it.
3. Check each bot by name: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot first, then the training bots. Disallow: / under a bot's group means that bot is blocked from the whole site.
4. Or run a script. Our free ai-crawler-check skill does steps 2–3 for 11 bots on any paths you choose. It runs inside Claude or on its own: bash python3 skills/ai-crawler-check/scripts/ai_robots_check.py https://yourdomain.com --paths / /blog/
5. Check the firewall layer. In your CDN or security plugin, look for "AI bots", "AI crawlers" or "bot fight" settings. On Cloudflare that's AI Crawl Control.
6. Check your logs. Search access logs for OAI-SearchBot, Claude-SearchBot and PerplexityBot. Look at the status code: 200 means it got in, 403 means something blocked it. User agents can be faked, so confirm against the published IP lists (OpenAI search bot IPs, PerplexityBot IPs).

A robots.txt that allows AI search but opts out of training

If that's the policy you want, name each bot explicitly rather than relying on *:

# AI search and user fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

# AI training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

# Everyone else (including Googlebot and Bingbot)
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml

Keep in mind that OpenAI says it can take about 24 hours for ChatGPT search to reflect a robots.txt change, and Perplexity says up to 24 hours too (both in the docs linked above). Anthropic asks you to repeat the rules on every subdomain.

What robots.txt can't do

• It's a request, not a lock. Well-behaved crawlers follow it. The user-triggered fetchers are explicitly treated differently by OpenAI and Perplexity.
• It doesn't make you get cited. Access is the entry ticket. Whether an AI answer picks your page depends on whether your page is the best answer, which is ordinary SEO work. Google says this plainly for AI Overviews and AI Mode in its AI optimization guide.
• It doesn't see your firewall. That's why step 5 matters.

Tools we built for this (disclosure)

We make two of the tools mentioned here, so weigh that accordingly:

• The free ai-crawler-check skill in our Oarsman Skills Sampler (MIT licence) reads robots.txt only.
• Our SEO + AI Visibility Skill Pack ($29) adds a live fetch test, schema and llms.txt checks and a fix-list report. Its 41 included tests check the scripts, not your rankings: we don't promise citations.

Next, read Does llms.txt actually work? before you spend time on one.

Method and data

• Tranco list ID 746VX, downloaded 1 Nov 2026. Slices: ranks 1–1,000 and 200,001–201,000.
• robots.txt fetched over HTTPS (apex, then www.), 12-second timeout. Denominator: sites returning a text robots.txt (517 and 587).
• Matcher: RFC 9309 longest-match with wildcards, path / only. A group applies when its user-agent token appears in the bot's name.
• Raw results and scripts: the dataset page with downloads. We'll re-run this every quarter and publish the change.