llms.txt is a plain-text file you place at your domain root that tells AI language models what your website covers, which pages matter, and how your content is organised. Think of it as robots.txt for AI crawlers — except instead of saying what not to crawl, it tells AI tools exactly what to read and why it matters.
The plain-English definition
llms.txt is a structured text file, placed at https://yourdomain.com/llms.txt, that provides AI systems with a curated map of your website. It tells them your site's purpose, lists your priority pages with short descriptions, and optionally includes sections covering your services, products, or key topics.
It does not replace your sitemap.xml. The sitemap tells search engine crawlers which URLs exist. llms.txt tells AI language models what those URLs are about and which ones matter most.
Who created llms.txt and where it came from
The llms.txt specification was proposed by Jeremy Howard (fast.ai) in 2024 as a practical standard for making websites more readable to large language models. It has since been adopted voluntarily by a growing number of sites, and AI crawlers from Perplexity, Anthropic, and others have begun reading it as part of their content acquisition processes.
It is not yet an official W3C standard, but it functions as a de facto signal that your site is AI-aware and structured for machine reading. AgentReady checks for its presence and format in every audit.
robots.txt vs llms.txt vs sitemap.xml: what each one does
| File | Audience | Purpose |
|---|---|---|
robots.txt | All crawlers | Tells crawlers what they can and cannot access |
sitemap.xml | Search engine crawlers | Lists all URLs on your site for indexing |
llms.txt | AI language models | Explains what your site covers and which pages to prioritise |
All three should exist on a well-optimised site. They serve different audiences and solve different problems.
What goes inside an llms.txt file
Required sections
A valid llms.txt file has two required elements: a site name or title at the top, and a short paragraph summary of what the site covers. Everything else is optional but recommended.
A real example you can copy
# AgentReady AgentReady is an AI readiness auditing platform that scores websites across six categories to determine whether AI tools like ChatGPT, Claude, and Perplexity can read, understand, and cite the site. It is built for e-commerce stores, service businesses, and SaaS companies that rely on online discovery. ## Core Pages - /: Homepage with free AI audit tool - /how-it-works: Explanation of the six audit categories and scoring methodology - /pricing: Pricing for free audit, full report ($99 AUD), and expert strategy session ($999 AUD) - /industry-insights/: GEO and AI visibility guides for business owners ## Services - Free AI Readiness Audit: Scores any public website across 50+ signals in under 60 seconds - Full AI Readiness Report ($99 AUD): Complete analysis with prioritised fix plan and schema templates - Expert Strategy Session ($999 AUD): Manual review, competitor analysis, and 90-day roadmap
How to create your llms.txt file
Step 1: Write a one-paragraph site summary
Open a plain-text editor. Write one paragraph describing what your site covers, who it serves, and what makes it authoritative. Keep it under 150 words. Do not write marketing copy — write as if you are briefing a researcher on what your site is and what it contains.
Step 2: List your priority pages with short descriptions
Add a ## Core Pages section. For each important page, add a line with the URL path and a one-sentence description. Prioritise pages that cover your core products, services, or topics — not every page needs to be listed.
Step 3: Add optional detail sections
Add sections for Services, Products, or Topics if your site has distinct content areas. These help AI tools understand the depth of your content without reading every page.
Step 4: Upload to your domain root
Save the file as llms.txt (plain text, UTF-8 encoding) and upload it so it is accessible at https://yourdomain.com/llms.txt. No server configuration required — treat it like any other static file.
AgentReady checks both the presence of your llms.txt and its formatting. A file that exists but is empty or malformed will be flagged. The audit also checks that the URLs listed in your llms.txt actually return HTTP 200.
Check if your llms.txt is correctly formatted
Run a free AI readiness audit to see whether your llms.txt exists and passes all formatting checks — plus your full score across all six GEO categories.
Run Free Audit →