AI crawling standards: llms.txt, llms-full.txt, ai.txt, and robots.txt
These files solve different problems. Use them together: robots.txt for access, llms.txt for context, llms-full.txt for depth, and ai.txt for usage policy.
llms.txt
- Purpose
- A compact, curated overview of a website for language models and AI agents.
- Location
- /llms.txt
- Audience
- LLMs, AI assistants, RAG systems, agentic crawlers.
llms-full.txt
- Purpose
- An expanded text context file for deeper analysis of large documentation or content sites.
- Location
- /llms-full.txt
- Audience
- LLMs, RAG pipelines, documentation processors.
ai.txt
- Purpose
- A policy-style file that explains how AI systems may use, summarize, or train on content.
- Location
- /ai.txt
- Audience
- AI crawlers, aggregators, AI product teams.
robots.txt
- Purpose
- The classic crawler access-control file for search and automated fetchers.
- Location
- /robots.txt
- Audience
- Search bots, web crawlers, AI crawlers that respect robots.txt.
Recommended publishing order
- Make sure robots.txt allows the crawlers you want to support.
- Publish llms.txt as the short canonical guide to the site.
- Add llms-full.txt when the site has dense documentation or long-form content.
- Use ai.txt if you need to state usage, attribution, or training rules.