Blog › Data
AI crawler rules and llms.txt on 2,000 websites (November 2026)
The data behind our guide Can ChatGPT, Claude and Perplexity crawl your site? What we found checking 2,000 sites.
Method
- Sample: the Tranco top-1M list (list id 746VX, downloaded 1 Nov 2026). Two slices of 1,000 domains: ranks 1–1,000 and ranks 200,001–201,000.
- We fetched each site's robots.txt with a plainly labelled user agent and parsed it with the RFC 9309 matcher in our free ai-crawler-check skill, testing the root path / only.
- Percentages use only sites that returned a text robots.txt (517 and 587). 62 and 58 sites returned 403 to our fetcher.
- llms.txt counts as present only if it returned text with a Markdown H1 in its first 20 lines. This is a presence check, not full validation.
- We did not pretend to be GPTBot or any other crawler, so firewall and CDN blocks are not measured.
Summary: robots.txt blocks at /
| Bot | Big sites blocked (n=517) | Long tail blocked (n=587) |
|---|---|---|
| GPTBot | 110 (21.3%) | 102 (17.4%) |
| OAI-SearchBot | 63 (12.2%) | 28 (4.8%) |
| ChatGPT-User | 74 (14.3%) | 34 (5.8%) |
| ClaudeBot | 113 (21.9%) | 101 (17.2%) |
| Claude-SearchBot | 71 (13.7%) | 22 (3.7%) |
| Claude-User | 74 (14.3%) | 21 (3.6%) |
| PerplexityBot | 89 (17.2%) | 34 (5.8%) |
| Google-Extended | 97 (18.8%) | 87 (14.8%) |
| CCBot | 121 (23.4%) | 104 (17.7%) |
| Googlebot | 19 (3.7%) | 11 (1.9%) |
| Bingbot | 19 (3.7%) | 13 (2.2%) |
Downloads
- llms-study-s0-n1000-2026-11-01.json (14 KB)
- llms-study-s200000-n1000-2026-11-01.json (17 KB)
- llms_study.py (1 KB)
- robots-study-s0-n1000-2026-11-01.json (408 KB)
- robots-study-s200000-n1000-2026-11-01.json (449 KB)
- robots-summary-2026-11-01.csv (1 KB)
- robots-summary-textual.txt (1 KB)
- robots_study.py (3 KB)
Licence and how to cite
CC BY 4.0. Cite as: "Oarsman AI checked the robots.txt and llms.txt files of 2,000 domains from the Tranco list on 1 November 2026", with a link to this page.
Changelog
- 1 November 2026: first run.