llms.txt: What it is and why ChatGPT is ignoring your site

Check your server logs. Most of what you see isn’t humans anymore; it’s agents, crawlers, and LLMs trying to parse your value proposition. The reality is simple: if I’m looking for a software provider today, I don’t browse through ten pages of Google ads.

I ask Perplexity, Claude, or my IDE’s AI directly.

And here lies the problem. If your website or technical documentation is designed exclusively for humans to read, you are forcing Artificial Intelligence to do a dirty job it hates doing. The result? They ignore you and recommend your competitors instead.

The Problem with “Over-Designed” Websites

Humans love dropdown menus, glassmorphism effects, and smooth animations (we love them too, but we use them with technical purpose). However, AIs detest visual clutter.

For a Large Language Model (LLM), every unnecessary <div> tag or Tailwind class on your site is dead weight. It’s “visual noise” that consumes processing tokens, distracts from what truly matters (your content), and increases the chances of hallucinations. Just like a broken semantic hierarchy confuses Googlebot, the lack of a machine-readable version confuses GPT-4.

To solve this, the developer community created a shockingly simple standard: the llms.txt file.

Universal Guide: How to Implement it on Your Platform

Regardless of the technology you use, implementing llms.txt is a matter of minutes. Here are the best strategies for the most common systems:

// In Astro: src/pages/llms.txt.ts
export async function GET() {
  const content = `# Your Brand\n\nAI summary of your services...`;
  return new Response(content, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' }
  });
}

// Add to your theme's functions.php
add_action('init', function() {
    add_rewrite_rule('^llms\.txt$', 'index.php?llms_txt=1', 'top');
});

add_filter('query_vars', function($vars) {
    $vars[] = 'llms_txt'; return $vars;
});

add_action('template_redirect', function() {
    if (get_query_var('llms_txt')) {
        header('Content-Type: text/plain; charset=utf-8');
        echo "# My WordPress Site\n\nAI-readable content goes here...";
        exit;
    }
});

Conclusion: SEO is No Longer Just for Humans

Traditional SEO—chasing keywords and praying for the first page of Google—isn’t dead, but it’s no longer enough. Today, you are either in your client’s chat context, or you don’t exist. Optimize your site for humans because they pay, but leave the llms.txt file ready for the machines because they decide what to recommend.

If you are already using advanced tools like our SEO Expert or ProText Expander, you know that efficiency is the only way forward.

Technical FAQ