llms.txt
A proposed standard — a text file in the website root that briefly tells AI models what the site offers and where to find key content.
llms.txt is a proposed web standard (llmstxt.org): a markdown file placed at /llms.txt that gives large language models a concise, curated overview of a website — what the site offers, what its key sections are and where to find important content without wading through HTML templates.
Structure of llms.txt:
- A heading with the website name and a one-sentence description
- Sections with links to key pages (products, categories, FAQ, contact)
- Optionally /llms-full.txt with extended content
Why have one: AI crawlers process huge numbers of pages on a limited token budget. llms.txt gives them a shortcut to the essence of your website — it increases the chance that the model understands what you sell, correctly and quickly. For online stores it is useful to add information about shipping, returns and unique selling points — exactly the things customers ask AI about.
Relationship to robots.txt: llms.txt neither blocks nor allows anything — it is a content map. AI crawler access is controlled by robots.txt (GPTBot, ClaudeBot, PerplexityBot); the Google-Extended token in it also determines whether Google may use the content for Gemini.