Key facts
| Tool type | Free llms.txt generator and validator |
| What it generates | Markdown file with page titles, URLs and short descriptions |
| Location | Served at the domain root, /llms.txt |
| Status of the convention | Emerging; adoption varies by assistant and crawler |
| Relationship to robots.txt | Complementary: robots controls access, llms.txt curates discovery |
| Relationship to sitemaps | Not a replacement; sitemaps list URLs, llms.txt prioritises them |
| Live example | Plugsky publishes llms.txt and llms-full.txt |
| Product status | Live |
TL;DR
- llms.txt is a curated Markdown map, not a ranking file or access control.
- It does not replace robots.txt, sitemaps or clean content structure.
- No crawler is obliged to read it, so treat it as an invitation, not a guarantee.
- Generate it from your best pages and keep descriptions short and factual.
- Pair it with the robots.txt tester so access and curation are both covered.
How it works, step by step
- List the pages that represent your product, pricing, docs and key articles.
- Open the llms.txt generator and paste the URLs with short titles.
- Write one-line descriptions that state what each page contains.
- Generate the file and review it for accuracy and dead links.
- Publish it at https://yourdomain.com/llms.txt and confirm it returns 200.
- Validate the output and fix broken or redirected URLs.
- Review quarterly as pages are published, moved or removed.
Try it yourself
What llms.txt is, and what it is not
llms.txt is a proposed convention: a Markdown file at your domain root that lists your most important pages with short descriptions, giving language models a cleaner entry point than crawling thousands of URLs. It is not a ranking factor, not a directive, and no assistant is required to read it. It does not block crawlers — robots.txt does that — and it does not list every URL the way a sitemap does. Think of it as a curated reading list you publish for machines, useful when it is accurate and harmless when it is ignored.
Generating a useful file
Quality beats completeness. Include the pages that answer questions about your product: home, pricing, documentation, key guides and a few high-value articles. Write descriptions that state what each page contains rather than marketing copy, because the description is what an assistant uses to decide whether to fetch the page. Keep the file small enough to read in one pass — a focused list of twenty to fifty URLs is more useful than a dump of every route. The generator produces the structure and a validator checks links and formatting so the published file does not rot.
Keeping it honest and current
An llms.txt that points at moved pages, abandoned docs or outdated pricing is worse than none. Assign an owner, regenerate when your information architecture changes and validate on a schedule. Layer it with the other discovery files rather than treating it as a silver bullet: robots.txt for access, an XML sitemap for complete coverage, structured data for facts and llms.txt for curation. If you are unsure whether AI crawlers can reach your content at all, run the robots.txt tester first — access is the prerequisite that a curated file cannot fix.
Honest comparison
| File | Purpose | Consumer | Guarantee |
|---|---|---|---|
| llms.txt | Curated map of key pages | AI assistants and crawlers that choose to read it | None; convention still emerging |
| robots.txt | Access rules per user-agent | Well-behaved crawlers | Advisory, widely respected |
| XML sitemap | Complete list of canonical URLs | Search engines | Discovery aid, not a ranking signal |
| Structured data | Machine-readable facts about pages | Search engines and AI systems | Improves extraction, no guaranteed display |
Frequently asked questions
Is llms.txt an official standard?
No. It is an emerging convention, not a ratified standard, and adoption varies. Publish it as a low-cost, low-risk complement to sitemaps and robots.txt.
Does llms.txt improve rankings?
No. It is not a ranking factor. Clean content, access rules and structure do the heavy lifting; llms.txt only helps assistants find your key pages faster.
Does it replace robots.txt?
No. robots.txt controls crawler access; llms.txt curates discovery. Use both, and test access first because a blocked crawler never reads your llms.txt.
Does it replace my sitemap?
No. Sitemaps list canonical URLs for discovery. llms.txt prioritises a small set of pages for assistants. Keep both.
What should the file contain?
A short H1, a one-line summary and sections of links with brief descriptions — usually twenty to fifty of your most important pages.
Can I block some pages and still use llms.txt?
Yes. Access rules live in robots.txt. Keep llms.txt curated to pages you want surfaced, and ensure nothing in it is blocked.
How do I publish it?
Serve the generated Markdown at https://yourdomain.com/llms.txt, confirm it returns 200, and keep it accessible without authentication.
Does Plugsky publish one?
Yes. Plugsky publishes llms.txt and llms-full.txt so assistants can discover the catalogue and documentation, and the generator builds one for your site.