llms.txt is a proposed Markdown file at the root of a website that gives language models LLM-friendly content. It works as a curated starting point, so AI crawlers can understand how your site is organised and find the content they need for an answer. Think of it as a short map of your site, written for agents instead of people.
Why it exists
Agents read sites badly. Most AI crawlers can only read a page's basic HTML, not content loaded by JavaScript. And when the proposal was written in 2024, agents using websites was largely a prediction. Today the heaviest use is software docs, where coding agents follow the file to find API references and tutorials. OpenAI, Anthropic and Gemini publish llms.txt files for their own developer docs.
What goes in the file
The spec at llmstxt.org sets the order:
- An H1 with the name of the project or site. This is the only required section.
- A blockquote with a short summary that holds the key information.
- Optional paragraphs, then sections of links, each link with a short note on what the page holds.
Save it as lowercase llms.txt, the filename the specification uses, and format it as Markdown. It usually sits at the root, but it can go at any path and covers the pages under that path, so /docs/llms.txt covers everything in /docs/.
An example
Here is a shortened copy of this site's own file, at shipfound.co/llms.txt:
# Shipfound
> Shipfound turns Claude Code or Codex into your growth engineer: site fixes for Google and AI search, content as pull requests, and proof of every result.
## Key pages
- [Pricing](https://www.shipfound.co/#pricing): plans and what credits pay for
- [Connect your agent](https://www.shipfound.co/mcp): the MCP server and the plugin commands
## Answers
- [Is writing an llms.txt file worth it for SEO?](https://www.shipfound.co/answers/is-writing-an-llms-txt-file-worth-it-for-seo): Not for rankings.
One H1, one line of summary, then links grouped by what a reader would come for. The file is built from the same data as the pages, so a new page shows up in it on the next deploy.
How to write one
Pick the pages an agent would need to explain your product: what it does, pricing, setup, docs, and the questions you answer. Write one plain line for each link. Leave out logins, legal pages and anything you would not publish on the page itself.
The proposal also asks for two extras. Pages an agent might need should have a clean Markdown copy at the same URL, with .md added. And two link tags help agents find things: rel="alternate" type="text/markdown" points to a page's Markdown copy, and rel="describedby" points to the llms.txt that covers the page.
Does any AI engine read it?
Not on the record. No major AI platform, including ChatGPT, Google's AI Overviews and Anthropic's Claude, has confirmed that it reads or relies on llms.txt. Semrush checked its own logs: from mid-August to late October 2025, its llms.txt got zero visits from Google-Extended, GPTBot, PerplexityBot or ClaudeBot.
Adoption is growing anyway. In July 2025, 951 domains had published an llms.txt. By September 2026 there were 16,670. And Chrome's Lighthouse now audits sites for llms.txt in its agentic browsing checks.
So treat it as cheap and useful for docs, not as a ranking lever. More on that in is writing an llms.txt file worth it for SEO.
How to check it is read
Open yoursite.com/llms.txt in a browser and confirm it loads as plain text. Then watch your server logs for AI user agents requesting it. An analytics script cannot see them, as explained in how to know if AI crawlers are hitting your site. If you build with Claude Code or Codex, Shipfound checks for the file in its audit, adds one as a pull request, and counts which AI bots fetch your pages.
Common mistakes
- A capitalised filename. Use llms.txt in lowercase.
- Links to pages that need a login, so the agent hits a wall.
- A file written once and never updated, pointing at pages that moved.
- Blocking AI bots in robots.txt. A map does not help a bot that is not allowed in. See how to get indexed by PerplexityBot and GPTBot.
Frequently asked questions
Is llms.txt a real standard?
It is a proposal, not a standard. It was proposed as a Markdown file for websites, and no AI platform has confirmed it reads one.
Should I add llms.txt to my website?
Yes if you have docs or an API, since that is where coding agents use it. For marketing pages it costs little, but clear pages matter more.
What is the difference between llms.txt and robots.txt?
robots.txt tells bots what they may crawl. llms.txt tells them what is worth reading.
Where do I put the file?
At your site root, so it loads at yoursite.com/llms.txt. A file at a subpath, like /docs/llms.txt, covers only the pages under it.