# Can ChatGPT, Claude and Perplexity crawl your site? What we found checking 2,000 sites

By Oarsman AI team · Published 2026-11-01 · Last checked 2026-11-01
Canonical: https://oarsmanai.com/blog/ai-crawlers-robots-txt-study/

**Short answer:** probably, but check two things. First, AI companies run separate bots for *training* and for *search*, and blocking one doesn't block the other. Second, your firewall or CDN can block AI bots even when robots.txt allows them. In our check of 2,000 websites, sites blocked training bots far more often than search bots: among smaller sites, 17.4% blocked OpenAI's GPTBot but only 4.8% blocked OAI-SearchBot, the bot that decides whether you can appear in ChatGPT search.

## The bots you're actually dealing with

"Block AI" used to mean one line for GPTBot. Today each major AI company documents several user agents, and they do different jobs:

| Bot | Company | What it does | What blocking it means |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train models | Your content shouldn't be used for training ([OpenAI docs](https://platform.openai.com/docs/bots)) |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers" ([OpenAI docs](https://platform.openai.com/docs/bots)) |
| ChatGPT-User | OpenAI | Visits a page when a user's request needs it | OpenAI says robots.txt rules "may not apply" because a user triggered it ([OpenAI docs](https://platform.openai.com/docs/bots)) |
| ClaudeBot | Anthropic | Training data | Future content excluded from training ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) |
| Claude-SearchBot | Anthropic | Improves Claude's search results | "May reduce your site's visibility" in Claude's search results ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) |
| Claude-User | Anthropic | Fetches pages when a user asks Claude | Claude can't fetch your pages for that user ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)) |
| PerplexityBot | Perplexity | Builds Perplexity's search index | Perplexity recommends allowing it "to ensure your site appears in search results" ([Perplexity docs](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)) |
| Perplexity-User | Perplexity | User-requested fetches | "Generally ignores robots.txt rules" ([Perplexity docs](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)) |

Google is different again: AI Overviews and AI Mode use the normal Search systems ([Google's AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)), so Googlebot access is what matters there. Google-Extended is a separate robots.txt token that Google lists for controlling whether your content is used for its Gemini models ([Google's crawler list](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)); it does not affect Search.

The practical upshot: if you want to show up in AI answers, the search bots matter more than the training bots.

## What we checked, and how

We wanted to know what real sites do, not what blog posts recommend. On 1 November 2026 we took two slices of the public [Tranco top-sites list](https://tranco-list.eu/) (list ID 746VX):

- **Big sites:** ranks 1–1,000.
- **Smaller sites:** ranks 200,001–201,000, which is a better proxy for a typical business website.

For each domain we fetched `/robots.txt` with a clearly labelled research user agent and ran it through an RFC 9309 matcher (the open standard for robots.txt, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)). The matcher picks the most specific user-agent group, uses longest-match rules, and handles `*` and `$` wildcards. We tested the home page path `/` for 11 bots. 517 big sites and 587 smaller sites returned a readable text robots.txt; percentages below use those numbers.

What this does *not* measure: firewall or CDN blocks (we did not pretend to be GPTBot), per-page rules deeper than `/`, or what any company actually does with the content. The scripts and raw data are linked at the end.

## What we found

| Bot | Big sites blocking it (n=517) | Smaller sites blocking it (n=587) |
|---|---|---|
| GPTBot (training) | 21.3% | 17.4% |
| OAI-SearchBot (ChatGPT search) | 12.2% | 4.8% |
| ChatGPT-User | 14.3% | 5.8% |
| ClaudeBot (training) | 21.9% | 17.2% |
| Claude-SearchBot | 13.7% | 3.7% |
| Claude-User | 14.3% | 3.6% |
| PerplexityBot | 17.2% | 5.8% |
| Google-Extended | 18.8% | 14.8% |
| CCBot (Common Crawl) | 23.4% | 17.7% |
| Googlebot | 3.7% | 1.9% |
| Bingbot | 3.7% | 2.2% |

### 1. Training bots get blocked about three times as often as search bots on smaller sites

Among smaller sites, 17.4% block GPTBot and 17.2% block ClaudeBot, but fewer than 6% block any of the search or user-fetch bots. Whether on purpose or not, most sites that block "AI" are blocking training and leaving AI search open.

### 2. Blocking GPTBot does not block ChatGPT search unless you say so

Of the 102 smaller sites that block GPTBot, 74 still allow OAI-SearchBot. For Anthropic it's 80 of 101 sites blocking ClaudeBot that still allow Claude-SearchBot. In most of these files the search bot isn't mentioned at all, so it falls back to the `User-agent: *` group, which usually allows everything. If your intent was "keep my site out of AI completely", this is the gap. If your intent was "don't train on me, but do send me visitors", this is exactly right. Either way, it should be a decision, not an accident.

### 3. Big sites are much stricter with AI search

12.2% of big sites block OAI-SearchBot and 17.2% block PerplexityBot, two to three times the rate on smaller sites. Big publishers have licensing reasons to say no. A small business usually has the opposite problem: it wants to be recommended.

### 4. Some sites block even a polite robots.txt request

62 of the big sites and 58 of the smaller sites returned **HTTP 403 (forbidden)** when we asked for robots.txt itself. That's a strong hint of a firewall or bot-protection layer sitting in front of the site. Robots.txt can say "welcome" while the firewall says "no". [Cloudflare, for example, began asking new domains in July 2025 to choose whether to allow AI crawlers, with blocking as the default](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/), and per-crawler controls now live in [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/).

## How to check your own site in 5 minutes

1. **Open `https://yourdomain.com/robots.txt`** in a browser. No file at all (a 404) means crawlers may fetch everything.
2. **Find the groups.** Each block starts with one or more `User-agent:` lines followed by `Allow:`/`Disallow:` lines. A bot uses the most specific group that names it, and only falls back to `User-agent: *` if nothing names it.
3. **Check each bot by name:** OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot first, then the training bots. `Disallow: /` under a bot's group means that bot is blocked from the whole site.
4. **Or run a script.** Our free [ai-crawler-check skill](https://github.com/vishalrwt1995/oarsman-skills-sampler) does steps 2–3 for 11 bots on any paths you choose. It runs inside Claude or on its own:
   ```bash
   python3 skills/ai-crawler-check/scripts/ai_robots_check.py https://yourdomain.com --paths / /blog/
   ```
5. **Check the firewall layer.** In your CDN or security plugin, look for "AI bots", "AI crawlers" or "bot fight" settings. On Cloudflare that's AI Crawl Control.
6. **Check your logs.** Search access logs for `OAI-SearchBot`, `Claude-SearchBot` and `PerplexityBot`. Look at the status code: 200 means it got in, 403 means something blocked it. User agents can be faked, so confirm against the published IP lists ([OpenAI search bot IPs](https://openai.com/searchbot.json), [PerplexityBot IPs](https://www.perplexity.ai/perplexitybot.json)).

## A robots.txt that allows AI search but opts out of training

If that's the policy you want, name each bot explicitly rather than relying on `*`:

```text
# AI search and user fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

# AI training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

# Everyone else (including Googlebot and Bingbot)
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
```

Keep in mind that OpenAI says it can take about 24 hours for ChatGPT search to reflect a robots.txt change, and Perplexity says up to 24 hours too (both in the docs linked above). Anthropic asks you to repeat the rules on every subdomain.

## What robots.txt can't do

- **It's a request, not a lock.** Well-behaved crawlers follow it. The user-triggered fetchers are explicitly treated differently by OpenAI and Perplexity.
- **It doesn't make you get cited.** Access is the entry ticket. Whether an AI answer picks your page depends on whether your page is the best answer, which is ordinary SEO work. Google says this plainly for AI Overviews and AI Mode in its [AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide).
- **It doesn't see your firewall.** That's why step 5 matters.

## Tools we built for this (disclosure)

We make two of the tools mentioned here, so weigh that accordingly:

- The **free ai-crawler-check skill** in our [Oarsman Skills Sampler](https://github.com/vishalrwt1995/oarsman-skills-sampler) (MIT licence) reads robots.txt only.
- Our [SEO + AI Visibility Skill Pack](https://oarsmanai.com/agents/seo-ai-visibility-skills/) ($29) adds a live fetch test, schema and llms.txt checks and a fix-list report. Its 41 included tests check the scripts, not your rankings: we don't promise citations.

Next, read [Does llms.txt actually work?](https://oarsmanai.com/blog/does-llms-txt-work/) before you spend time on one.

## Method and data

- Tranco list ID 746VX, downloaded 1 Nov 2026. Slices: ranks 1–1,000 and 200,001–201,000.
- robots.txt fetched over HTTPS (apex, then `www.`), 12-second timeout. Denominator: sites returning a text robots.txt (517 and 587).
- Matcher: RFC 9309 longest-match with wildcards, path `/` only. A group applies when its user-agent token appears in the bot's name.
- Raw results and scripts: [the dataset page with downloads](https://oarsmanai.com/blog/data/ai-crawlers-2026-11/). We'll re-run this every quarter and publish the change.

## Figures

![Horizontal bar chart. Share of sites blocking each crawler, big sites vs long tail: GPTBot 21.3% vs 17.4%, OAI-SearchBot 12.2% vs 4.8%, ClaudeBot 21.9% vs 17.2%, Claude-SearchBot 13.7% vs 3.7%, PerplexityBot 17.2% vs 5.8%, Googlebot 3.7% vs 1.9%, Bingbot 3.7% vs 2.2%.](https://oarsmanai.com/blog/ai-crawlers-robots-txt-study/chart-bots-blocked.png)
*Share of sites whose robots.txt blocks each crawler at the root path. Big sites: Tranco ranks 1–1,000 (517 sites with a text robots.txt). Long tail: ranks 200,001–201,000 (587 sites). Source: our study, 1 November 2026.*

![Bar chart. Of sites blocking GPTBot, 47 of 110 big sites and 74 of 102 long-tail sites still allow OAI-SearchBot. Of sites blocking ClaudeBot, 42 of 113 and 80 of 101 still allow Claude-SearchBot.](https://oarsmanai.com/blog/ai-crawlers-robots-txt-study/chart-training-vs-search.png)
*Of the sites that block a training bot, how many still let the same company's search bot in. Most long-tail sites never name the search bots, so those fall through to the * group. Source: our study, 1 November 2026.*

![Diagram table of AI crawlers by company and purpose. OpenAI: GPTBot (training), OAI-SearchBot (search), ChatGPT-User (user-triggered). Anthropic: ClaudeBot, Claude-SearchBot, Claude-User. Perplexity: no training bot listed, PerplexityBot (search), Perplexity-User (user-triggered).](https://oarsmanai.com/blog/ai-crawlers-robots-txt-study/diagram-bot-types.png)
*Each AI company runs separate bots for training, search and user-requested fetches, and robots.txt treats each one separately. Based on the OpenAI, Anthropic and Perplexity documentation linked in this guide.*

## FAQ

### Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI says GPTBot is for training and OAI-SearchBot is for ChatGPT search, and each setting is independent. To stay in ChatGPT search answers, allow OAI-SearchBot.

### Which AI crawlers should a small business allow?

If you want to be found in AI answers, allow the search bots: OAI-SearchBot, Claude-SearchBot and PerplexityBot, plus Googlebot and Bingbot. Whether to allow training bots like GPTBot, ClaudeBot and Google-Extended is a business choice.

### Can robots.txt stop every AI fetch?

No. OpenAI says robots.txt rules may not apply to user-initiated ChatGPT-User fetches, and Perplexity says Perplexity-User generally ignores robots.txt. robots.txt is a request, not a lock.

### My robots.txt allows the bots. Why don't I show up?

Your CDN or firewall may be blocking them. Since July 2025 Cloudflare has asked new domains to choose, with AI crawlers blocked by default. Check your CDN's bot settings and your server logs.

### How often should I re-check?

After any robots.txt, CDN or security-plugin change, and at least every quarter, because AI companies add and rename crawlers.

**Get it:** [SEO + AI Visibility Skills Pack](https://oarsmanai.com/agents/seo-ai-visibility-skills/) · $29 one-time
Launch code LAUNCH30: 30% off until 31 December 2026. The Gumroad buy link on the agent page already applies it.

---
*How this was written: drafted with AI assistance from our own data collection and the sources linked above, then checked against our sourcing rules by the Oarsman AI team. Facts last checked 1 November 2026.*