Definition

What is llms.txt?

llms.txt is a plain markdown file served at the root of a site, such as example.com/llms.txt. It gives an AI agent a short description of what the site covers and a curated list of links to the pages worth reading, so the agent can find the right documentation without crawling a JavaScript shell or guessing at a sitemap. Jeremy Howard proposed the format in September 2024.

The format

The file is markdown, and the convention is loose. An H1 with the project name comes first, then an optional blockquote summary, then H2 sections holding bullet lists of links. Each bullet is a markdown link followed by a colon and a one-line description of what the linked page contains.

A section headed Optional marks links an agent can skip when it is working with a limited context budget. Nothing enforces any of this. A file that returns 200 with a plausible index is what the AI Agent Readiness Test counts.

# Example API

> Send transactional email over HTTP.

## Docs

- [Quickstart](https://example.com/docs/quickstart.md): Send the first message in five steps.
- [Rate limits](https://example.com/docs/limits.md): Per-plan quotas and the 429 response.

## Optional

- [Changelog](https://example.com/changelog.md): Release history.

llms.txt and llms-full.txt are different files

llms.txt is an index. It points at pages and stays small enough to read in one request. llms-full.txt is the whole corpus concatenated into a single markdown file, which can run to hundreds of kilobytes.

An agent with a large context window can load llms-full.txt and skip the fetching entirely. An agent working against a budget reads llms.txt and fetches only what the task needs. The two serve different readers, which is why the test scores them separately at 10 points each.

llms.txt is not robots.txt

robots.txt tells a crawler which paths it may request. It is a permission file, and it is enforced by convention among crawlers that choose to honour it.

llms.txt grants no permission and blocks nothing. It is an invitation that says which pages are worth reading. A site can serve both, and most sites that serve llms.txt also serve robots.txt.

How common is it?

Adoption among the products tested so far is recorded on the grades table, alongside the four other agent surface checks. The count updates as each report publishes.

The file is cheap to publish and it makes the markdown mirror discoverable, which is the part of the surface an agent uses most directly. Several documentation platforms generate llms.txt for their customers, so a team can end up serving one without having decided to.

Adoption across the products tested

16 of the 16 products tested serve an llms.txt, and 11 serve an llms-full.txt.

Mailgun serves llms.txt
Resend serves llms.txt
Stripe serves llms.txt
Browserbase serves llms.txt
Airbyte serves llms.txt
Arcade.dev serves llms.txt
CARTO serves llms.txt
Composio serves llms.txt
Convex serves llms.txt
Daytona serves llms.txt
DuckDB serves llms.txt
E2B serves llms.txt
Felt serves llms.txt
Firecrawl serves llms.txt
LangChain serves llms.txt
Railway serves llms.txt

Every score and surface check in one table

Common questions about llms.txt

Where does llms.txt go?

At the root of the domain serving the docs, so example.com/llms.txt or docs.example.com/llms.txt. An agent looks for it at the root and does not search for it elsewhere.

Should the links point at .md URLs?

Point them at markdown when a markdown mirror exists. An agent reading an HTML page has to strip navigation, scripts, and styling before it reaches the content, and a .md URL removes that step.

How large should llms.txt be?

Small enough to read in one request. It is an index, so it holds links and one-line descriptions. Put the full text in llms-full.txt instead.

Put another docs site through the battery.

Nominate a docs site