llms.txt
A file that tells an agent what your site is and where to read it
llms.txt is a proposed convention: a markdown file at the root of a site summarising what it offers and linking to clean text versions of its pages. It is robots.txt for comprehension rather than for crawling, and it is cheap enough to be worth doing regardless of adoption.
Best for when
- your site is documentation or reference material agents should read
- you want answer engines to describe your product accurately
- you can generate the file from content you already maintain
Watch out for
- adoption by crawlers is not guaranteed, so treat it as cheap upside
- a hand maintained file goes stale, so generate it
- it is not a ranking mechanism and will not behave like one
What it is for
An agent arriving at a site has to infer what it is from navigation and markup, which is expensive and often wrong. A short file stating the purpose, the structure and where the plain text lives removes the guesswork.
The companion pattern is a markdown twin for each page, so an agent can read content without parsing a layout. This site serves one for every page, at the same URL with a .md suffix.
Honest status
This is a convention with real momentum among documentation sites and no guarantee that any particular crawler honours it. Nobody should rebuild a site around it.
It also costs almost nothing: a generated file from content you already have. That asymmetry is the whole argument for shipping one.
On the same shelf
Anthropic Skills
The reference implementation of what a skill should look like
Superpowers
A process framework that makes an agent plan before it builds
AGENTS.md
One instruction file that every coding agent agrees to read