Does llms.txt actually work? What Google says, who reads it, and when it's worth 10 minutes

What llms.txt is

llms.txt is a proposal by Jeremy Howard, first published on 3 September 2024 and revised as "v2" on 10 August 2026. The idea: put a short Markdown file at /llms.txt that tells AI agents what your site is and where the useful pages are, because "web pages are built for people" and turning HTML back into clean text is "difficult and imprecise" (quotes from the proposal).

The format is simple. The only required part is an H1 with the name of the site. The proposal then recommends a one-paragraph summary in a blockquote, optional notes, and sections of links:

# Acme Plumbing

> Family-run plumbing company in Leeds, UK. Emergency call-outs 24/7,
> boiler servicing and bathroom fitting. Prices on /pricing.

## Services
- [Emergency call-outs](https://acmeplumbing.example/emergency): areas covered, response times
- [Boiler servicing](https://acmeplumbing.example/boilers): makes we service, annual plans

## Policies
- [Pricing](https://acmeplumbing.example/pricing): call-out fee and hourly rates
- [Guarantee](https://acmeplumbing.example/guarantee): 12-month workmanship guarantee

## Optional
- [About us](https://acmeplumbing.example/about)

The v2 proposal also suggests offering clean Markdown copies of key pages (for example page.md next to page.html) and pointing to them with standard link headers. That part is more work and mostly useful for documentation sites.

What Google says

Google has been unusually direct about this. Its guide to optimizing for generative AI features in Google Search says you don't need "new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." After questions from site owners, Google added that it's "completely fine" to maintain llms.txt for other systems, and that doing so "won't harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them" (reported by Search Engine Land).

John Mueller of Google has explained the reasoning: a file in which a site describes itself can't help an AI system choose between sites, because every site can claim to be the best (Search Engine Journal). He did carve out one use: when an automated system is already on your site, a guide can help it get around.

There's one Google wrinkle. Chrome's Lighthouse added an llms.txt check to its experimental "agentic browsing" audits. It marks the audit not applicable if the file is missing and flags a server error if fetching it fails (Search Engine Journal on the Lighthouse change). That's about agents using your site, not about Search rankings.

Who actually reads it

• Documentation-heavy sites and coding agents. This is where llms.txt is used most, according to the proposal itself: agents fetch a library's docs to get an API call right. The AI labs publish llms.txt files for their own developer docs (OpenAI, Anthropic and Google's Gemini, per llmstxt.org). OpenAI's crawler documentation page, for example, opens with a pointer to its llms.txt index (OpenAI docs).
• Site builders and SEO plugins generate it automatically. The proposal lists Yoast SEO, AIOSEO and Wix among tools that create one. A Google panelist noted at a 2026 Search Central event that most files he'd seen came from a plugin setting (event notes).
• Crawlers, rarely. One agency reported on Hacker News that after putting llms.txt on 83 websites, OpenAI's crawler read it 7 times in 12 weeks. That's their claim and their sample, but it matches the general picture: crawlers fetch files they find, and a fetch isn't the same as the file mattering.

How many sites publish one? We checked

On 1 November 2026 we requested /llms.txt from websites in two slices of the public Tranco top-sites list (list ID 746VX). We only counted a file if it came back as plain text with a Markdown H1 near the top, so HTML error pages served at that URL didn't count.

So roughly one in eight smaller sites and one in five big sites already serve one. Adopters in the top slice include developer platforms such as GitHub, Stripe, Cloudflare and Shopify. Having one is not unusual; it also isn't a competitive edge. (Method and data: see our AI-crawler study.)

When llms.txt is worth your time

A 10-minute version for a business website

1. Write the H1 and one-paragraph summary: who you are, where you operate, what you do. Plain facts, no slogans. An agent can't use "best in town".
2. List 5–15 links that answer real customer questions: services, pricing, areas served, booking, policies, contact. Add a short note after each link.
3. Save it as /llms.txt at the site root with a text/plain or text/markdown content type.
4. Make sure it returns 200, not a redirect to your home page or an HTML error page. In our check, an HTML page served at /llms.txt was the most common result after a plain 404.
5. Put a calendar reminder to update it when prices or services change. A stale file that contradicts your site is worse than none.
6. Test it the way the proposal suggests: give an AI assistant only your llms.txt and ask it three customer questions.

Then go and do the things that actually affect AI search: make sure OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot can reach you (our robots.txt study shows how to check), and make each service page answer one question clearly.

Tools we built (disclosure)

We make tools that generate llms.txt, so here is the honest pitch. Our SEO + AI Visibility Skill Pack ($29, five Claude skills) builds an llms.txt from your site, checks AI-crawler access and schema, and writes a fix list. Our free WordPress plugin, Oarsman AI Visibility, does the same inside WordPress; it's in review at WordPress.org and we'll link it here once it's approved. Neither will make Google rank you higher, and we don't claim they will.