Does llms.txt actually do anything?

Somewhere between 2024 and now, adding /llms.txt to a site became a checkbox on every "get cited by AI" listicle. In 2026 we finally have server logs and citation data instead of opinions — and they point the same way. The file is almost never read by the AI search engines people are optimising for. It is still worth publishing for one specific kind of site. This is what the evidence says, and how to write a version that earns its keep.

What llms.txt actually is

llms.txt is a single Markdown file at the root of a site, proposed by Jeremy Howard (Answer.AI, fast.ai) in 2024. It is an index: an H1 with the project name, a blockquote summary, optional prose, then H2 sections of links with short descriptions. The idea is that a language model or agent can read one short file and orient itself instead of crawling twenty pages to work out what your site is.

Two things it is not. It is not a permissions file — that is robots.txt, and nothing about llms.txt grants or denies access. And it is not a sitemap: a sitemap enumerates every URL for indexing, whereas llms.txt is deliberately a curated shortlist. The spec allows one special section, an H2 titled Optional, whose links an agent may skip when its context budget is tight.

The 2026 evidence

Until this year the argument ran on single-site anecdotes. Then three larger datasets landed, and they agree:

• Ahrefs, June 2026 — server logs and bot analytics across 137,000 domains. About 28% of them publish an llms.txt, and 97% of those files received zero requests in the month studied. Of the requests that did land, 96% came from bots, and most of those bots were not AI tools at all: SEO audit tools, generic crawlers, technology profilers. Named AI bots made up roughly 19.5% of fetches — led by GPTBot and Claude-Code, i.e. training crawlers and coding agents, not answer engines. Around 12% came from the industry studying itself: llms.txt validators, GEO/AEO readiness scanners and researchers.
• SE Ranking — 300,000 domains, no correlation found between publishing llms.txt and how often a domain is cited by AI engines.
• Matched-pair citation tracking (MaxAEO, ~2,400 tracked domains over 90 days) — adopters versus matched non-adopters differed by about 0.2 percentage points, well inside noise. A separate 90-day log study saw roughly 0.1% of AI bot requests touch /llms.txt.

The single most damning line in the Ahrefs data is this one: zero requests arrived for llms.txt files that did not exist. AI bots never go looking. That kills the mental model where a model "checks for" your file before answering a question about you. It does not. It fetches whatever URLs its retrieval step already surfaced — normally your HTML.

Why the mixed messages?

Because two parts of Google say different things. Google Search's guidance on generative AI features includes an explicit mythbusting note that machine-readable files like llms.txt are not needed to appear in AI features. Days later, Chrome shipped an llms.txt check inside Lighthouse's experimental "agentic browsing" audits, on the reasoning that without such a file an agent spends longer crawling to understand a site's structure.

Pressed on the contradiction, Google's John Mueller framed it as: llms.txt is "not done for search". He called it a temporary crutch to save tokens for AI coding tools parsing developer documentation. That reading matches the log data exactly — the bots fetching it are coding agents, and the audit tooling is a browser team's agent concern, not a ranking signal.

So when is it worth it?

Publish one if you tick at least one of these boxes:

1. You ship developer docs, an SDK or an API. Coding agents genuinely fetch these files, and pointing them at a canonical quickstart and API reference measurably reduces the wrong-turn rate when someone asks an assistant to integrate your product.
2. Your docs are heavy client-rendered HTML. A Markdown index (ideally alongside plain .md versions of key pages) is far cheaper for an agent to parse than shipping it a wall of framework markup.
3. You have a large site with a small "actually important" core. Ten curated links beat 4,000 sitemap entries when the reader has a token budget.

If you run a marketing site, a blog or a shop, treat it as optional housekeeping. It costs twenty minutes and it will not move your AI citations — the studies are unanimous. Spend that effort on answer quality and page structure instead, which is what actually drives citation.

Writing one that isn't junk

The whole spec fits in a screen:

# Project Name
> One sentence saying what this is and who it's for.

Optional paragraph of context an agent should read first.

## Docs
- [Quickstart](https://example.com/docs/quickstart): install and first request in 5 min.
- [API reference](https://example.com/docs/api): every endpoint, with error codes.

## Optional
- [Changelog](https://example.com/changelog)

The H1 is the only required element. Beyond that, a few rules that separate useful from noise:

• Curate, don't dump. Auto-generators tend to emit your entire sitemap, which throws away the only advantage the format has. Ten to forty links is a healthy range.
• Write descriptions for the reader, not for keywords. "Webhook signing: HMAC scheme, replay window, retry semantics" tells an agent whether to open the page. "Learn more about our powerful webhooks" does not.
• Use absolute URLs. Agents resolve links out of context.
• Put secondary material under ## Optional so a context-constrained reader can skip it deliberately rather than randomly.
• Serve it as text/plain at the root, and give it a subpath version (/docs/llms.txt) if a section deserves its own index — the spec says the most specific applicable file wins.
• Keep it in CI. Generate it from the same source as your nav so it cannot drift. A file listing pages that 404 is actively worse than no file.

One more thing worth knowing: because these files are fetched by agents, they are a prompt-injection surface. Ahrefs noted the largest research crawler hitting llms.txt was running a prompt-injection survey. Treat the file as public, machine-consumed content: links and descriptions, no instructions to the model, no secrets.

How to test it on your own domain

Don't take anyone's word for it, including this post. The measurement is cheap because the file is a single URL:

1. Publish the file and log requests to /llms.txt with the user-agent string.
2. Run for 30 days. Bucket the hits: AI coding agents, training crawlers, AI retrieval bots, SEO tools, everything else.
3. Compare against fetches of an equivalent HTML page in the same period.

If your bucket for retrieval bots is empty — which the population data says is likely — you know the file is doing nothing for AI search on your site, and you can stop treating it as a visibility lever without guessing.

The short version

llms.txt is a reasonable idea with an honest use case that got sold as something else. As an orientation file for agents reading technical documentation, it works and costs almost nothing. As an AI-visibility tactic it currently has no measured effect, no platform commitment, and log data showing the relevant bots do not even look for it. Publish it if you're a docs-heavy site, keep it accurate, and don't put it on the roadmap as a ranking play.