What is llms.txt and why AI agents need it
llms.txt is a plain text file that helps AI agents and language-model crawlers understand what a website is, which pages matter, and where canonical information can be found. It is usually published at/llms.txt and written in a compact Markdown-like format.
Why it exists
Traditional SEO files were not designed for model context. A sitemap lists URLs, robots.txt controls crawler access, and HTML pages are optimized for browsers. llms.txt gives an AI system a short, curated entry point before it spends tokens reading the rest of a site.
What goes into the file
A useful llms.txt normally includes:
- A clear project or company name.
- A short description of the product, audience, and scope.
- Links to core pages such as docs, pricing, API references, changelogs, or support.
- Optional notes for AI agents, including preferred source pages and update cadence.
How it differs from robots.txt
robots.txt answers the access question: which crawlers may fetch which paths. llms.txt answers the context question: once an AI system can read the site, what should it read first and how should it interpret the source?
How it differs from sitemap.xml
A sitemap is exhaustive and URL-oriented. llms.txt is editorial and context-oriented. It should not list every page unless that list is genuinely useful. The best files behave like a concise guide for a technical reader.
When to add llms-full.txt
If your site has documentation, policies, product pages, or reference material that an AI agent should process in depth, publish/llms-full.txt alongside the short file. Use llms.txt as the table of contents and llms-full.txt as the expanded body.
Practical checklist
- Publish the file at
/llms.txt. - Keep it text-only, not HTML, JSON, or XML.
- Use stable canonical URLs.
- Update it when important site sections change.
- Validate that it returns HTTP 200 and a readable text response.
After publishing, submit the site to the global llms.txt directory so AI agents and developers can discover it.