LLMs.txt Generator

Enter your domain. LLMs.txt Generator crawls the site, picks the pages worth showing to AI systems, groups them into sections and writes a description for each one. You get a validated llms.txt you can edit before publishing.

The difference is in the middle of that process. The generator does not dump your sitemap into a text file: URLs are filtered, important pages are prioritised, and descriptions come from the actual page content rather than the meta tag.

How to create an llms.txt file

1. Enter your website. A domain, a sitemap URL, or your own URL list pasted in.
2. LLMs.txt Generator crawls and analyses. robots.txt, your sitemaps, and the content of each page.
3. It builds and validates the file. Sections, descriptions, and a compliance check.
4. You review, edit and publish. Nothing leaves the page until you approve it.

More than a sitemap-to-llms.txt converter

A simple llms.txt builder does one pass: sitemap in, URLs with meta descriptions out. That is how login pages, carts and a hundred empty blog entries end up in a published file.

Capability	DiagnoSEO	Other generators
Crawls actual page content	✅	Varies
Filters low-value URLs	✅	Varies
Prioritises important pages	✅	Varies
Creates logical sections	✅	Varies
Writes descriptions from page content	✅	Varies
Writes in the language of the site	✅	Varies
Built-in validation	✅	Varies
Edit before download	✅	Varies
Shows skipped URLs and reasons	✅	Varies
Compares your existing llms.txt	✅	Varies
Generates llms-full.txt	✅	Varies

What LLMs.txt Generator actually does

• Sitemap-first discovery. Reads robots.txt and handles sitemap index files and gzipped sitemaps.
• Fallback crawling. No sitemap? Pages are discovered through internal links from your homepage.
• Parallel fetching keeps larger crawls fast.
• Smart URL selection. Documentation, products, services, pricing, about and contact come first.
• Noise removal. Assets, login, cart, pagination, tag and date archives, filter parameters.
• Custom exclusions. Add your own URL fragments to skip, such as /tag/ or ?filter=.
• Section balancing stops a large blog from crowding out your documentation.
• Title cleanup. "Pricing | Brand" becomes "Pricing".
• H2 sections plus an Optional section for secondary content.
• Summary and per-page descriptions written from the real page content, in the language of your site.
• Meta description fallback if the model is unavailable, so you always get a usable file.
• Validation before download, including the Chrome Lighthouse criteria.
• Comparison with the llms.txt already published on your domain.
• Transparent sources. Every included URL is listed and can be unticked; every skipped one shows why.
• llms-full.txt with the full page content, from the same crawl.

What is llms.txt

llms.txt is a Markdown file described in the llmstxt.org proposal, published by Jeremy Howard (Answer.AI) in September 2024 and updated to v2 in August 2026. llms.txt is a community-driven proposal, not a W3C or IETF standard.

The structure is short: an H1 with the site name (the only required element), a blockquote summary, an optional paragraph of context, then H2 sections of Markdown links where a colon introduces a short note. A section named Optional marks content an agent may skip when context is tight.

The file can sit at the domain root or at any path within it. v2 allows scoped files, so /docs/llms.txt covers everything under /docs/, and agents should use the most specific file that applies.

What is llms-full.txt

llms-full.txt is a convention popularised by documentation platforms, not part of the specification. Instead of links with descriptions it carries the full content of your pages as Markdown, merged into one document.

llms-full.txt earns its place where users ask AI assistants detailed questions: documentation, SaaS products, knowledge bases. The llms-full.txt generator runs from the same crawl as the index file, so the two stay consistent.

llms.txt, sitemap.xml and robots.txt

Three files, three jobs. They do not replace each other, and a well-maintained site serves all three.

	robots.txt	sitemap.xml	llms.txt
Written for	Crawlers	Search engines	Language models and agents
Answers	What may be crawled	Which URLs exist	What key pages are about
Format	Directives	XML	Markdown
Includes descriptions	No	No	Yes
Controls access	Yes	No	No
Scope	Domain	Domain	Domain or path

Why publishers may want one

• Control. You choose which pages and descriptions you expose, instead of leaving a model to infer them from HTML full of navigation and scripts.
• Machine readability. Markdown is easier for compatible agents and tools to consume than a rendered page.
• Context efficiency. A concise map fits in a context window that your whole site never would.
• Low cost. Nothing to maintain in your stack, no effect on performance.

The format is published by the documentation teams at OpenAI, Anthropic and Gemini, and generated automatically by platforms such as Mintlify, GitBook and Wix, and by the Yoast SEO and AIOSEO plugins.

What llms.txt does not do

• It does not block crawlers. Access control belongs in robots.txt.
• It does not replace robots.txt or sitemap.xml.
• It does not guarantee citations in ChatGPT, Perplexity, Gemini or Claude.
• It is not a confirmed Google ranking factor.
• It is not required, and adoption varies between AI systems.
• It cannot compensate for thin content or a site crawlers cannot reach.

Anyone promising more citations from the file alone is promising something nobody can currently demonstrate. The value is control and readability.

Validation and the Chrome Lighthouse check

Chrome Lighthouse 13.3 added an experimental Agentic Browsing category. One of its audits validates basic properties of a published llms.txt: an H1 heading, at least one real Markdown link in the [Title](URL) form, and a length above 50 characters.

The generator mirrors the checks that audit runs. It is a structural check, not a Google Search ranking signal. For the audit itself a missing file is treated as Not Applicable, while a published file that fails it can be reported as a failure.

That is why validation runs before download: llmstxt.org structure, the three Lighthouse criteria, duplicate URLs, foreign-domain links and file size. The separate llms.txt Checker & Validator checks a file that is already live, including whether its links still resolve.

Where to put llms.txt and how to publish it

Serve it as plain text at yourdomain.com/llms.txt, next to robots.txt. To describe only part of a site, place it on that path instead, for example yourdomain.com/docs/llms.txt. After uploading, open the address and confirm you see Markdown rather than a 404 page.

Platform	Upload path
cPanel or shared hosting	/public_html/llms.txt
WordPress	Site root, not /wp-content/
Next.js, Nuxt, Astro, React	/public/llms.txt
Laravel, Symfony	/public/llms.txt
Shopify	Content → Files, then redirect /llms.txt to the uploaded file
Webflow	Site settings, custom code or file upload
Netlify, Vercel, GitHub Pages	Commit it to your public folder

What belongs in the file, by website type

An llms.txt is a shortlist, not an archive. A complete dump of URLs is what your sitemap is for. The same rules apply whether the site runs on WordPress, Shopify, Webflow or a custom stack.

• SaaS and documentation: product overview, pricing, getting started, API reference, changelog.
• E-commerce: categories, flagship products, shipping, returns, size guides. Not variants, filters or pagination.
• Local services: each service separately, service area, pricing or quote page, about and contact.
• Publishers: topic hubs and evergreen pieces rather than the latest fifty posts.

Frequently asked questions

• What is an llms.txt file?

  A Markdown file with your site name, a short summary and a list of key pages with descriptions. It gives language models and agents a concise map of your content instead of making them infer it from HTML.

• Is llms.txt an official standard?

  No. It is a community-driven proposal described at llmstxt.org, currently at v2, and has not been ratified by W3C or IETF. Spec-compliant here means the structure described there.

• Is llms.txt required?

  No. No system requires it. You publish it voluntarily, to control what you expose to compatible AI tools and how it is described.

• Does llms.txt improve AI visibility or rankings?

  There is no evidence for that, and this tool does not promise it. The file makes your content easier to read and lets you control the descriptions, but it is not a confirmed ranking factor or a guarantee of citations.

• Does Google use llms.txt?

  Google states the file is not needed for its AI features in Search. Separately, Chrome Lighthouse can validate basic properties of a published file in its experimental agentic browsing audits. That is a structural check, not a ranking signal.

• Where should llms.txt be hosted, and can it live in a subdirectory?

  Serve it as plain text at your domain root, next to robots.txt. v2 also allows files at any path: /docs/llms.txt covers the content under /docs/, and agents should use the most specific file that applies.

• What is the difference between llms.txt and llms-full.txt?

  llms.txt is a curated index of links with descriptions. llms-full.txt contains the full page content as Markdown. The first is a map, the second is the territory, and only the first is part of the specification.

• How is llms.txt different from sitemap.xml and robots.txt?

  A sitemap lists every URL for search engines, with no descriptions or hierarchy. robots.txt controls crawler access. llms.txt is a short curated map with context, sized to fit a model's context window.

• Does the generator work without a sitemap?

  Yes. Without a sitemap, pages are discovered through internal links from your homepage, prioritising documentation, products and services.

• Is the generator free?

  Yes. Within the daily limit you get the complete file, with no watermark and no truncated preview. Paid plans raise the page limit and add llms-full.txt.

Generate your llms.txt

Enter your domain and get a structured, validated and editable llms.txt file ready to publish. Free, no signup, no manual URL collection. Already have a file? Check it with the llms.txt Checker & Validator.