AI publishing conventions and standards: llms.txt, ai.txt, and robots.txt
llms.txt and ai.txt are emerging conventions rather than formal web standards. These files solve different problems: robots.txt for access, llms.txt for context, llms-full.txt for depth, and ai.txt for usage policy. Publishing them can reduce discovery friction for compatible tools, but it does not guarantee crawling, indexing, ranking, or citation.
llms.txt
- Purpose
- A compact, curated overview of a website for language models and AI agents.
- Location
- /llms.txt
- Audience
- LLMs, AI assistants, RAG systems, agentic crawlers.
llms-full.txt
- Purpose
- An expanded text context file for deeper analysis of large documentation or content sites.
- Location
- /llms-full.txt
- Audience
- LLMs, RAG pipelines, documentation processors.
ai.txt
- Purpose
- A policy-style file that explains how AI systems may use, summarize, or train on content.
- Location
- /ai.txt
- Audience
- AI crawlers, aggregators, AI product teams.
robots.txt
- Purpose
- The classic crawler access-control file for search and automated fetchers.
- Location
- /robots.txt
- Audience
- Search bots, web crawlers, AI crawlers that respect robots.txt.
Recommended publishing order
- Make sure robots.txt allows the crawlers you want to support.
- Publish llms.txt as the short canonical guide to the site.
- Add llms-full.txt when the site has dense documentation or long-form content.
- Use ai.txt if you need to state usage, attribution, or training rules.
Frequently asked questions
Do I need all four files?+
No. Start with a valid robots.txt and a concise llms.txt. Add llms-full.txt for documentation-heavy sites and ai.txt when you need an explicit AI usage policy.
Does llms.txt replace sitemap.xml or Schema.org?+
No. llms.txt provides curated context, sitemap.xml helps discover URLs, and Schema.org describes entities on individual pages. They work best together.
Does publishing llms.txt guarantee AI search visibility?+
No. It removes friction, but visibility still depends on crawl access, original content, citations, technical markup, and each AI product’s retrieval rules.
Where should the files be published?+
Publish them at the host root: /llms.txt, /llms-full.txt, /ai.txt, and /robots.txt. Configure each subdomain separately when it serves distinct content.