What Is llms.txt? How to Create One for Your Website (With Examples)

llms.txt is a Markdown file you put at yoursite.com/llms.txt to give AI tools a short, curated map of your site: what it is, and links to the pages that matter, ideally as clean Markdown. It was proposed by Jeremy Howard of Answer.AI in September 2024, and thousands of sites now publish one, mostly for documentation that coding agents read. It is not a ranking signal: Google says its search and AI features don't use it.

This guide explains the format, shows real examples, and walks you through creating, hosting, testing and maintaining your own llms.txt file.

What is llms.txt?

Web pages are built for people. An AI agent reading one has to wade through menus, cookie banners and scripts to find the useful text, and every wasted token costs time and money. An llms.txt file tells it, in plain Markdown, "here's what this site is, and here are the pages worth reading."

The idea comes from the llms.txt proposal by Jeremy Howard, co-founder of Answer.AI and fast.ai, first published on September 3, 2024. It's a community proposal, not an official standard from a body like the W3C or IETF. A revised version 2 in August 2026 added a few things based on two years of real use:

• An llms.txt file can live at the root (/llms.txt) or in a subfolder (/docs/llms.txt). A file covers the pages under its path, and agents should use the most specific one.
• Pages can offer a Markdown version at the same URL with .md added (page.html.md) or swapped in (page.md).
• Pages can point to their Markdown version and their llms.txt with standard link relations: rel="alternate" type="text/markdown" and rel="describedby", either as <link> tags or an HTTP Link: header.

The spec says llms.txt is used mostly at inference time, when an agent is helping someone right now, rather than for training models.

llms.txt vs robots.txt vs sitemap.xml

llms.txt doesn't control access. If you want to block or allow AI crawlers such as GPTBot or ClaudeBot, that's still robots.txt. And it's not a sitemap: a sitemap lists everything, while llms.txt is a short, hand-picked guide that fits in an AI model's context window.

llms.txt vs llms-full.txt

llms-full.txt isn't part of the official spec, but it's a widely used companion. Where llms.txt is a short index of links, llms-full.txt puts the full text of every page into one big Markdown file, so an agent can load everything at once instead of following links.

It's popular with docs platforms: Mintlify generates both files for every site it hosts. The trade-off is size. Anthropic's llms-full.txt is about 47 MB and Cloudflare's is about 69 MB, far too big to fit in any model's context. For a small site, llms-full.txt is handy. For a large one, the index plus per-page .md files works better.

Who uses llms.txt?

We checked these live in October 2026. Open any of them to see how real companies structure the file:

• Anthropic: docs.anthropic.com/llms.txt (now redirects to platform.claude.com) and an llms-full.txt
• OpenAI: developers.openai.com/llms.txt, which links to a separate llms.txt per product
• Google Gemini API: ai.google.dev/gemini-api/docs/llms.txt
• Cloudflare: developers.cloudflare.com/llms.txt, with one llms.txt per product and .md pages
• Stripe: docs.stripe.com/llms.txt, with every link pointing to a .md page
• Vercel: vercel.com/llms.txt
• Others: Perplexity's docs, Supabase, Svelte, Zapier, Yoast, and every Mintlify-hosted docs site

Most are developer documentation. That's where llms.txt is really used today, because coding agents like Claude Code, Cursor and Codex fetch docs constantly.

Does anyone actually read llms.txt?

Be clear-eyed about this before you spend time on it.

• Google Search doesn't use it. Google's guide to optimizing for generative AI features says you don't need "AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them," and lists llms.txt among tactics you can ignore. It adds that creating one is "completely fine" for other services. Google's Gary Illyes and John Mueller have said the same publicly.
• No major AI company has said its crawler reads it. As of October 2026, OpenAI, Anthropic, Perplexity and Microsoft haven't announced that their search or training crawlers use llms.txt as a signal. Many blog posts claim otherwise without a source. Treat those claims with suspicion.
• Agents do use it when pointed at it. Developers routinely paste an llms.txt URL into Claude, ChatGPT or Cursor, or connect it through a docs tool, and the agent follows its links. That's the use the spec now describes.
• Chrome checks for it. Chrome's Lighthouse added an experimental llms.txt audit in its agentic browsing checks. A missing file is marked "Not Applicable," not a failure. Only a server error when fetching it gets flagged.

So llms.txt won't boost your rankings or get you into AI Overviews. It can make your content easier for agents to use when someone asks them about you.

1Look at a few real llms.txt files

Before writing your own, open two or three of the files above in a browser. llms.txt is plain text, so it displays right in the tab.

What to notice:

• Each starts with # Name and usually a > one-line summary.
• Links are grouped under ## headings such as Docs, API reference or Guides.
• Each link has a short description after a colon, written for an AI reader: what the page covers and when it's useful.
• Several include direct instructions for agents (Stripe tells them to check the latest package version before installing).

2Learn the llms.txt format

The format is defined at llmstxt.org. A valid file contains these parts, in this order:

1. An H1 with the name of the site or project. This is the only required part.
2. A blockquote (>) with a short summary containing the key information needed to understand the rest of the file.
3. Optional details: any Markdown except headings, such as paragraphs or bullet lists. Use this for important notes, like "our API is not compatible with X."
4. Zero or more H2 sections, each containing a "file list": a Markdown list where every item is a link [name](url), optionally followed by : and a note.
5. An optional section called ## Optional, by convention for secondary links an agent can skip when it's short on context.

The spec's own mock example is just:

# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)

The strict structure means simple code can parse it, while people and AI read it as normal Markdown. The spec's advice: concise language, a brief description for each link, no unexplained jargon.

3Make clean Markdown versions of your pages

llms.txt links work best when they point to clean Markdown rather than full HTML pages. If your site doesn't produce .md files already (Mintlify, GitBook and many docs generators do), you can make them with URL to Text.

One page at a time

1. Go to urltotext.com and paste the page's URL.
2. Set Output Format to Markdown.
3. Tick Extract Only Main Content with AI so the menus, footers and cookie banners are dropped.
4. Leave Render JavaScript on if your site builds pages in the browser.
5. Click Convert, check the result, and copy it into a file such as pricing.md.

On Chrome, the free URL to Text extension does the same for the page you're on: right-click and choose URL to Text›Download page as Markdown.

Many pages at once with the API

For more than a handful of pages, use the URL to Text API. Sign up, open your dashboard, click Create API key and then Generate API Key. This Python script reads a list of URLs, saves each page as a .md file, and combines them all into llms-full.txt:

# pip install requests
# Usage: URLTOTEXT_API_TOKEN=your_key python make_llms_full.py
import os
import re
import requests

API_URL = "https://urltotext.com/api/v1/urltotext/"
TOKEN = os.environ["URLTOTEXT_API_TOKEN"]

SITE_NAME = "Brightline Coffee"
SUMMARY = "Brightline Coffee roasts and ships specialty coffee from Portland, Oregon."

# urls.txt: one URL per line
with open("urls.txt") as f:
    urls = [line.strip() for line in f if line.strip() and not line.startswith("#")]

os.makedirs("md", exist_ok=True)
sections = [f"# {SITE_NAME}\n\n> {SUMMARY}\n"]

for url in urls:
    response = requests.post(
        API_URL,
        headers={"Authorization": f"Token {TOKEN}"},
        json={
            "url": url,
            "output_format": "markdown",
            "extract_main_content": True,
            "render_javascript": True,
        },
        timeout=180,
    )
    if response.status_code != 200:
        print(f"Skipped {url}: {response.status_code} {response.text[:200]}")
        continue

    data = response.json()["data"]
    title = data.get("page_title") or url
    content = data["content"].strip()

    # Save a per-page copy, e.g. md/pricing.md, for llms.txt to link to.
    slug = re.sub(r"[^a-z0-9]+", "-", url.split("://", 1)[-1].lower()).strip("-")
    with open(f"md/{slug}.md", "w", encoding="utf-8") as page_file:
        page_file.write(f"# {title}\n\nSource: {url}\n\n{content}\n")

    sections.append(f"## {title}\n\nSource: {url}\n\n{content}\n")
    print(f"OK {url}")

with open("llms-full.txt", "w", encoding="utf-8") as out:
    out.write("\n\n".join(sections))
print(f"Wrote llms-full.txt with {len(sections) - 1} pages")

Each call returns JSON with the Markdown in data.content and the title in data.page_title. Requests use credits, and JavaScript rendering and AI extraction add a little to each one. See the API documentation for every option, including extract_css_selector to keep just one part of a page.

Tip: Upload the md/ files to your site so they sit next to the original pages (for example /pricing.md next to /pricing), then link to those .md URLs from llms.txt.

4Write your llms.txt file by hand

For most small and medium sites, a hand-written file is best: you decide what an AI reads first, in your own words.

1. List the pages that answer real questions. About, products or services, pricing, shipping and returns, contact, docs, FAQ. Skip tag archives, legal boilerplate and thin pages.
2. Group them under a few H2 headings.
3. Write one line per link saying what the page covers and when it's useful.
4. Put must-know facts in the details at the top: what you sell, where you operate, what you don't do.
5. Move nice-to-have links into ## Optional.
6. Save it as llms.txt in UTF-8 plain text (not a Word document).

Here's a complete example for a small business:

# Brightline Coffee

> Brightline Coffee is a specialty coffee roaster in Portland, Oregon. We sell whole-bean and ground coffee online, ship to the US and Canada, and run a monthly subscription.

Key facts:

- We roast to order every Monday and Thursday and ship within 2 business days.
- Free US shipping on orders over $40. Canada shipping is a flat $15.
- We don't sell coffee pods, espresso machines or wholesale to cafes outside Oregon.
- Customer support: hello@brightline.example, Monday to Friday, 9am to 5pm Pacific.

## Shop

- [All coffees](https://brightline.example/coffee.md): Every coffee we sell, with origin, roast level, tasting notes and price
- [Subscription](https://brightline.example/subscription.md): How the monthly plan works, pricing, skipping and canceling
- [Gift cards](https://brightline.example/gift-cards.md): Digital gift cards from $25 to $200

## Help

- [Shipping](https://brightline.example/shipping.md): Delivery times, costs and countries we ship to
- [Returns and refunds](https://brightline.example/returns.md): Our 30-day freshness guarantee and how to request a refund
- [Brewing guides](https://brightline.example/brew.md): Recipes for pour-over, French press, AeroPress and espresso
- [FAQ](https://brightline.example/faq.md): Answers about grind sizes, storage, caffeine and allergens

## Company

- [About us](https://brightline.example/about.md): Our story, sourcing principles and the farms we buy from
- [Contact](https://brightline.example/contact.md): Email, phone and our cafe address and hours

## Optional

- [Blog](https://brightline.example/blog.md): Coffee news, new releases and farm visits
- [Press](https://brightline.example/press.md): Media coverage and a downloadable press kit

If you don't have .md versions yet, you can link to the normal HTML pages. Agents can still read them, just less efficiently.

Tips for a good llms.txt

• Keep it short. The file should fit in an AI's context, with the detail behind the links. Dozens of links is fine. Thousands is a sitemap.
• Use absolute URLs (https://...) so links work wherever the file is read.
• State facts plainly. Prices, policies and limits are what people ask AI assistants about.
• Skip the keywords and marketing copy. Nothing ranks on it. Agents just need facts.

5Or generate llms.txt automatically

Many platforms can build and update the file for you.

• Mintlify: generates /llms.txt, /llms-full.txt and a .md version of every page automatically. To override it, add your own llms.txt file to the root of your docs project.
• GitBook: serves an llms.txt for published docs sites.
• Yoast SEO (WordPress): free in all versions. Go to Yoast SEO›Settings›Site Features, find LLMS.txt under AI tools, turn on the toggle and click Save changes. Yoast picks your key pages automatically, or you can switch to manual mode and choose them.
• AIOSEO (WordPress): turn it on from the AIOSEO dashboard. The free Lite version generates llms.txt, while llms-full.txt and the Markdown settings need Pro.
• Website LLMs.txt (WordPress): a dedicated free plugin from the WordPress plugin directory, if you don't use Yoast or AIOSEO.
• Wix: creates an llms.txt for every eligible site. In your dashboard open SEO & GEO, scroll to Tools and settings and click Go to llms.txt. To edit it, click More Actions›Edit file. Once you edit it, Wix stops updating it automatically, and Reset to Default brings automatic updates back.
• Docs generators: docusaurus-plugin-llms for Docusaurus, vitepress-plugin-llms for VitePress, and nbdev projects create .md pages by default.
• Firecrawl: its open-source create-llmstxt-py script crawls any site and writes llms.txt and llms-full.txt, using Firecrawl and OpenAI API keys. Its older hosted generator at llmstxt.firecrawl.dev is deprecated and no longer maintained.

Note: Generated files are a good start, but read the result. Automatic tools tend to include too many pages and use generic descriptions. A quick edit makes a big difference.

6Upload llms.txt to your site

The file must be reachable at https://yourdomain.com/llms.txt (or in a subfolder, such as /docs/llms.txt, if it only covers that section).

• Static sites, Next.js, Astro, Hugo, Vite: put llms.txt in the public/ or static/ folder and deploy.
• WordPress: use one of the plugins above, or upload the file by FTP or your host's file manager to the folder that contains wp-config.php.
• Your own server: copy it to the web root. In nginx or Apache, make sure it's served as text/plain or text/markdown with charset=utf-8, so accented characters don't turn into gibberish.
• Wix, Mintlify, GitBook: the platform hosts it for you.

Optionally, help agents find it from any page with an HTTP header, as llms.txt v2 suggests:

Link: </llms.txt>; rel="describedby"

7Test your llms.txt file

1. Open it in a browser. Visit https://yourdomain.com/llms.txt. You should see your Markdown as plain text, not a 404 page or a download prompt.

2. Check the response from a terminal:

   curl -I https://yourdomain.com/llms.txt
   

   Look for HTTP/2 200 and a content-type of text/plain or text/markdown. A 301 redirect to another domain or a 403 from your firewall means agents may not get it.

3. Check the structure. The first line must be # Name, and every list item in an H2 section should be a working [name](url) link. Click a few.

4. Run Lighthouse. In Chrome, open DevTools (F12), go to the Lighthouse tab and run a report. The llms.txt check sits in the agentic browsing category.

5. Ask an AI. This is the test the spec recommends. Give Claude or ChatGPT only your llms.txt URL and ask it real questions ("What's the return policy?", "Do they ship to Canada?"). If it answers correctly by following your links, the file works.

Tip: Want to know if anything reads it? Search your server or CDN logs for requests to /llms.txt and check the user agents.

8Keep llms.txt up to date

A stale llms.txt that links to deleted pages or old prices is worse than none.

• Plugin or platform: Yoast, AIOSEO, Mintlify and Wix (until you edit it) refresh the file for you.
• Hand-written: add it to your checklist whenever you add, rename or remove an important page, or change prices or policies.
• Scripted: run the API script from step 3 on a schedule (a weekly cron job or a step in your deploy pipeline) to regenerate the .md files and llms-full.txt.

Should you bother with llms.txt?

Worth doing if:

• You publish developer docs, an API or a software product. This is where agents use llms.txt the most.
• You have facts people ask AI assistants about (pricing, policies, specs) and want a clean, authoritative source for them.
• You can set it up in minutes with a plugin or platform you already use.

Skip it, or don't expect much, if:

• You're hoping for better Google rankings or AI Overviews citations. Google says it doesn't use the file.
• You'd have to maintain it by hand and probably won't. An outdated file can mislead agents.

The cost is low and it won't hurt your SEO. Treat it as a convenience for AI agents, not a growth hack.

Frequently asked questions

What is llms.txt used for?

llms.txt gives AI agents and assistants a short Markdown summary of a website with links to its most useful pages. It's used mostly for developer documentation, so coding agents can find the right API reference or guide without crawling the whole site.

Is llms.txt an official standard?

No. It's an open community proposal by Jeremy Howard of Answer.AI, published at llmstxt.org in September 2024 and revised as version 2 in August 2026. No standards body has adopted it.

Does Google use llms.txt?

No. Google's AI optimization guide says Google Search, including AI Overviews and AI Mode, doesn't use llms.txt or other AI text files, and that you don't need one to appear in Search. Creating one won't hurt your site either.

What's the difference between llms.txt and llms-full.txt?

llms.txt is a short index with links to your key pages. llms-full.txt contains the full text of all those pages in one Markdown file. llms-full.txt isn't in the official spec, but many docs platforms generate both.

Does llms.txt replace robots.txt?

No. robots.txt controls which crawlers may access your site. llms.txt only describes your content. To block or allow AI crawlers, keep using robots.txt.

How do I create an llms.txt file for WordPress?

Turn it on in Yoast SEO under Settings›Site Features›LLMS.txt, use AIOSEO's llms.txt generator, or install the Website LLMs.txt plugin. You can also upload a hand-written file to your site's root folder.

How do I convert my pages to Markdown for llms.txt?

Paste each URL into URL to Text, choose Markdown and tick Extract Only Main Content with AI. For many pages, use the URL to Text API with the script in this guide.

Wrapping up

An llms.txt file is a few minutes of work: a # Name, a one-line summary, and a short list of your best pages with clear descriptions, saved at /llms.txt. Point the links at clean Markdown copies (paste your pages into URL to Text or script it with the API), test it by asking an AI about your site, and keep it current. Just don't expect it to move your Google rankings. It's for the agents that come looking for answers.