Open your server logs. A growing slice of the traffic is not people. It is bots, crawlers, and AI assistants trying to read your site on behalf of someone else. When a developer asks Cursor for a SaaS to handle invoices, or a marketer asks ChatGPT for a competitor of Shopify, the model answers based on what it managed to read. If it could not read your site cleanly, you are not in the answer.
Most websites today are built for humans first and last. That is fine for humans. It is hard for machines. The pages render with JavaScript, the content is wrapped in three layers of components, the menus are dropdowns, the value proposition lives behind an animation. A language model has to fight through all of that to extract a sentence it can quote. Many times it gives up and uses a competitor whose copy was easier to parse.
The llms.txt standard exists to fix exactly this problem.
What llms.txt is, in plain words
llms.txt is a markdown file that lives at the root of your domain. The two common locations are /llms.txt and /.well-known/llms.txt. It is a curated, structured summary of your most important content, written in plain text, with links and short descriptions.
Think of it as a sitemap.xml for AI agents. A sitemap tells search engines which URLs exist. llms.txt tells AI assistants which URLs matter, what the site is about, and how the pieces fit together.
It is not a replacement for your homepage. It is a clean menu that an AI can read fast, with no rendering, no ads, no cookie banner, no chat widget popping up after three seconds.
Why this matters now
AI assistants are becoming the way many people search. ChatGPT browses the web. Claude browses the web. Perplexity is built around it. Cursor and Windsurf pull context from documentation when they generate code. Each of these tools fetches pages, parses them, and decides what to quote.
If your site is heavy with client side rendering, ad clutter, third party scripts, and a noisy DOM, the AI ends up with a confusing snapshot of what you sell. It might describe you wrong. It might skip you. It might recommend a competitor whose page was simpler to read. This is the same readability problem that hurts you with classic search, where security and performance also act as ranking signals.
llms.txt sidesteps all of that. The model fetches one small markdown file and gets a clean summary of who you are and what you offer. No guessing.
A second reason matters too: AI assistants prefer plain text representations of content because they are easier to parse. They are not against design. They just have a token budget and a parsing pipeline, and a 50 KB markdown file is faster and more reliable than a 2 MB rendered page.
The format, in five lines
The structure is simple. An H1 with the site name. A paragraph that says what the site is. Then H2 sections with link lists, where each link has a one line description.
# Acme Invoicing
Acme Invoicing is a SaaS that helps small businesses send invoices,
track payments, and reconcile with their bank in one place.
## Product
- [Features](https://acme.com/features): what the product does, with screenshots
- [Pricing](https://acme.com/pricing): three plans, monthly and yearly
- [Integrations](https://acme.com/integrations): list of supported banks and accounting tools
## Documentation
- [Getting started](https://acme.com/docs/start): account setup and first invoice
- [API reference](https://acme.com/docs/api): endpoints, auth, and rate limits
- [Webhooks](https://acme.com/docs/webhooks): events you can subscribe to
## Company
- [About](https://acme.com/about): team, location, mission
- [Contact](https://acme.com/contact): email, phone, support hoursThat is the whole format. No XML, no schema, no special syntax to learn. If you can write a README, you can write an llms.txt.
A second optional file, /llms-full.txt, holds your full documentation concatenated into a single plain text dump. Use it if you want AI tools like Cursor to ingest your entire knowledge base in one fetch. If you generate it from existing markdown content, our Markdown to HTML converter is handy when you need to preview the rendered output. Skip it if your site is small.
Universal guide: how to implement it on your platform
Below are working snippets for the four cases that cover most of the web. Pick the one that matches your stack.
Astro
Astro can serve llms.txt in two ways. The simplest is a static file in the public folder.
Drop a file at public/llms.txt with your markdown content. Astro will serve it at https://yourdomain.com/llms.txt with no extra config. Done.
If you want to generate the file dynamically (for example, list every blog post automatically), create an endpoint at src/pages/llms.txt.ts:
import type { APIRoute } from "astro";
export const GET: APIRoute = async () => {
const body = `# Your Site
Your one paragraph description here.
## Pages
- [Home](https://yourdomain.com/): main landing page
- [Blog](https://yourdomain.com/blog): articles on topic X
`;
return new Response(body, {
headers: { "Content-Type": "text/plain; charset=utf-8" },
});
};That is five lines of logic. The endpoint runs at build time (or on demand if you use SSR) and returns plain text.
WordPress
WordPress does not let you drop a file at the root unless you have FTP access, which you usually do. The two clean options are a plugin or a custom rewrite rule.
The plugin route: search the plugin directory for “LLMs Full Text” or “llms.txt” and install one that is actively maintained. Most of them auto generate the file from your published posts and pages. Configure the categories you want to include, save, done.
The DIY route: add this to your theme’s functions.php or a small custom plugin:
add_action('init', function () {
add_rewrite_rule('^llms\.txt$', 'index.php?llms_txt=1', 'top');
});
add_filter('query_vars', function ($vars) {
$vars[] = 'llms_txt';
return $vars;
});
add_action('template_redirect', function () {
if (get_query_var('llms_txt')) {
header('Content-Type: text/plain; charset=utf-8');
echo "# " . get_bloginfo('name') . "\n\n";
echo get_bloginfo('description') . "\n\n";
echo "## Posts\n\n";
$posts = get_posts(['numberposts' => 50]);
foreach ($posts as $p) {
echo "- [" . $p->post_title . "](" . get_permalink($p) . "): "
. wp_trim_words($p->post_excerpt ?: $p->post_content, 15)
. "\n";
}
exit;
}
});After saving, go to Settings, Permalinks, and click Save (this flushes the rewrite rules). Visit /llms.txt and you should see your posts listed.
Shopify
Shopify is the trickiest because it does not let you drop arbitrary files at the root of your domain. Three options, in order of preference.
Option one, an app. Search the Shopify App Store for “llms.txt” and install one. Several apps now generate the file from your products, collections, and pages. This is the path of least resistance.
Option two, a Cloudflare Worker in front of your store. If your domain runs through Cloudflare, add a Worker that intercepts requests to /llms.txt and returns plain text:
export default {
async fetch(request) {
const url = new URL(request.url);
if (url.pathname === "/llms.txt") {
const body = `# Your Store\n\nWhat you sell, in one paragraph.\n\n## Collections\n\n- [Featured](https://yourstore.com/collections/featured): top products\n`;
return new Response(body, {
headers: { "Content-Type": "text/plain; charset=utf-8" },
});
}
return fetch(request);
},
};Option three, a fallback page at /pages/llms. Shopify lets you create a regular page with that handle. The URL becomes https://yourstore.com/pages/llms, which is not the canonical location but is better than nothing. Some AI crawlers will find it via your sitemap.
Generic Nginx
If you run your own server, the cleanest path is a static file plus one location block.
Save your markdown content to /var/www/llms.txt. Then in your Nginx config:
server {
listen 443 ssl;
server_name yourdomain.com;
location = /llms.txt {
alias /var/www/llms.txt;
default_type text/plain;
add_header Content-Type "text/plain; charset=utf-8";
}
# rest of your server block
}Reload Nginx with sudo nginx -s reload and visit /llms.txt. The same pattern works for /.well-known/llms.txt if you prefer that path.
A short example for a SaaS site
Here is what a small SaaS llms.txt might look like in practice. Copy this as a starting point and edit.
# Linkpipe
Linkpipe is a link in bio tool for creators who want analytics, custom
domains, and webhooks without paying for a marketing suite.
## Product
- [Features](https://linkpipe.io/features): full feature list with screenshots
- [Pricing](https://linkpipe.io/pricing): free, pro at $9/month, team at $29/month
- [Compare](https://linkpipe.io/vs/linktree): how Linkpipe differs from Linktree
## Documentation
- [Quickstart](https://linkpipe.io/docs/start): create an account and your first link page
- [Custom domains](https://linkpipe.io/docs/domains): connect your own domain
- [Webhooks](https://linkpipe.io/docs/webhooks): list of events and payload format
- [API](https://linkpipe.io/docs/api): REST endpoints with examples in curl
## Company
- [About](https://linkpipe.io/about): team and origin story
- [Blog](https://linkpipe.io/blog): articles on creator economy and growth
- [Contact](https://linkpipe.io/contact): email and DiscordThat is around 20 lines of markdown. An AI assistant fetching this file gets a complete map of the product in one request.
Practical advice
A few rules that come from running this in production.
Keep the file under 50 KB. Bigger files start to lose useful signal and may be truncated by some crawlers. If you need to expose more, use /llms-full.txt for the heavy dump and keep /llms.txt as the index.
Link only to your most important pages. The whole point is curation. If you list 200 URLs, the model learns nothing about what you actually want it to remember. Pick the 10 to 30 pages that define your value.
Write the descriptions in plain language. Skip marketing copy. “Pricing: three plans, monthly and yearly” is more useful than “Pricing: discover our flexible plans tailored to your needs.” The first sentence tells the model a fact. The second tells it nothing.
Update the file when you ship something material. New product, new pricing tier, new major feature. Treat it like a living document, not a one time setup. A /llms.txt from 18 months ago that still says “beta” is worse than no file at all.
Add the file to your sitemap and link to it from your robots.txt. Some crawlers find it on their own, some need a hint.
If you write good developer docs, generating /llms-full.txt from them is usually a one afternoon job. The payoff is that tools like Cursor can pull your full context in one fetch when a developer points them at your domain.
Where this fits with the rest of your stack
llms.txt does not replace anything. Your sitemap.xml still serves classic search engines. Your robots.txt still controls crawl behavior. Your structured data still helps Google build rich results. llms.txt sits next to those, aimed at a different audience: language models that read your site to answer questions on behalf of users.
If you have a clean information architecture already, with good headings and a solid semantic hierarchy, you are most of the way there. llms.txt is just the explicit version of what you already do implicitly.