llms.txt and robots.txt for AI Crawlers: What Actually Matters

TL;DR: robots.txt decides which crawlers may fetch your pages, and for AI search it matters a lot: block OAI-SearchBot and you drop out of ChatGPT search answers, block PerplexityBot and Perplexity's index loses you. Training crawlers (GPTBot, ClaudeBot, Google-Extended) are a separate choice and blocking them does not remove you from search features. llms.txt is a proposed summary file for language models that no major AI search engine has said it reads. Publish one if you like; our free AI intro file tool makes it in a minute. Spend your real effort on robots.txt.

Of the two files in the title, one controls whether AI assistants can see your site at all, and the other is mostly a conversation topic. Guess which one gets more LinkedIn posts.

What does robots.txt control?

robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may fetch, one group of rules per user agent. It was standardised in 2022 as RFC 9309, and Google documents how it reads the file in its robots.txt guide. Two limits matter. First, it controls crawling, not indexing: a blocked URL can still appear in Google if other pages link to it, just without a description, so to keep a page out of search you use a noindex tag and let it be crawled. Second, it is a request that well-behaved crawlers honour, not access control. The major AI companies all document that their crawlers follow it, which makes it the right tool for choosing who gets in, and the wrong tool for protecting anything secret. A crawler follows the most specific group that names it, and falls back to the * group only if none does.

Which AI crawlers are there, and what does each one do?

Each company now separates its crawlers by job, which is good news, because it lets you say yes to search and no to training. These are the user agents documented by the companies themselves.

Google-Extended and Applebot-Extended are not separate crawlers that visit your site; they are tokens you name in robots.txt to set a policy for content the main crawler already fetched.

Does blocking training crawlers hurt your AI search visibility?

Not according to the companies' own documentation. OpenAI says each of its settings is independent, so a site can disallow GPTBot and allow OAI-SearchBot, and it will still be eligible to appear in ChatGPT search. Google says Google-Extended has no effect on inclusion or ranking in Google Search, and AI Overviews and AI Mode are Search features that use what Googlebot crawls. Anthropic and Perplexity likewise separate their training and search crawlers.

So the decision splits in two. Allowing the search and user-fetch crawlers is how you stay visible in AI answers, and there is little reason for a business site to refuse them. Allowing the training crawlers is a question about whether you want your text used to train models, which is a business and values decision rather than an SEO one. Our default for client sites is to allow search crawlers and leave training as the owner's explicit choice.

What robots.txt should a business site use?

This is the file we start from on business sites. It lets every search and user-fetch crawler in, keeps private paths out, and leaves the training decision as a clearly marked block you can flip:

# Search engines and AI search: welcome
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /admin/
Disallow: /cart/

# Model training: your choice. Delete this block to allow it.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Grouping several User-agent lines over one set of rules is valid under RFC 9309. Note that a crawler named in a specific group ignores the * group entirely, which is why the private paths are repeated in the search group. Check the result with our free AI visibility check, which reads the file the way each bot would.

Where does robots.txt have to live?

At the root of each host, exactly. https://example.com/robots.txt covers https://example.com and nothing else: www.example.com, shop.example.com and the plain http:// version each need their own file, or a redirect to one that applies. The path in each rule is case-sensitive, so Disallow: /Admin/ does not block /admin/. Google reads only the first 500 KiB of the file and usually caches it for up to a day, so a fix you publish at noon may not be respected until tomorrow. Keep the file short enough to read in one screen, and write a comment above each group saying why it exists, because the next person to edit it will not remember. Search Console has a robots.txt report that shows the version Google last fetched and any lines it could not parse, which is the fastest way to confirm Google sees what you think it sees.

Why are AI crawlers blocked when robots.txt allows them?

Because robots.txt is not the only gatekeeper. The most common cause we find in audits is a CDN or security layer rejecting bots before robots.txt is ever consulted. Cloudflare, for example, offers one-click blocking of AI crawlers, and security plugins for WordPress do similar things. The crawler gets a 403 or a challenge page, your robots.txt says "welcome", and nobody notices because the site works fine in a browser.

Other culprits: a staging Disallow: / that shipped to production, a * group that disallows everything with a specific group meant to override it that has a typo in the user agent, and robots.txt returning a 5xx error. Google treats a server error on robots.txt as a reason to pause crawling, as its documentation explains. Check the response code and your CDN's bot settings as well as the file itself.

What is llms.txt?

llms.txt is a proposal, published at llmstxt.org in September 2024 by Jeremy Howard, for a Markdown file at /llms.txt that gives language models a curated summary of a site. The format is an H1 with the site's name, a blockquote with a short summary, then sections of links with one-line descriptions:

# Harbor Dental Clinic

> Harbor Dental Clinic is a family dental clinic in Harbor Town
> offering check-ups, fillings, crowns and teeth whitening.

## Treatments
- [Check-ups and cleaning](https://example.com/check-ups): for adults and children, twice a year
- [Teeth whitening](https://example.com/whitening): in-clinic whitening in one or two visits

## Contact
- [Contact](https://example.com/contact): address, phone and WhatsApp

A companion llms-full.txt can hold the full text of the important pages in one file.

Do AI search engines actually read llms.txt?

As of September 2026, none of the major AI search products has documented using llms.txt when deciding what to cite. Google's guidance for AI features says there is nothing extra to create beyond normal SEO, and Google's John Mueller has said in public that no AI system currently uses the file, comparing it to the old keywords meta tag. OpenAI, Anthropic and Perplexity document their crawlers and robots.txt handling in detail and say nothing about reading llms.txt from other sites.

Where it does get used is developer documentation. AI coding assistants load llms.txt files to pull a library's docs into context, and several documentation platforms generate one automatically. If you run a developer product, publish one. If you run a dental clinic, it is harmless and unlikely to change anything. We check for it in audits and give it no weight in the score.

So what should you actually do?

1. Open yoursite.com/robots.txt and read it. Look for Disallow: / under any AI search user agent or under *.
2. Run it through the free AI visibility check to see how each bot interprets it.
3. Check your CDN, firewall and security plugin for AI bot blocking, and make it match what robots.txt says.
4. Decide deliberately on training crawlers and write the decision down in the file as a comment.
5. If you want an llms.txt, generate it with the free AI intro file tool, keep it accurate, and move on.

For the wider picture of how AI assistants choose sources, read What is GEO?. If you want every crawler rule, CDN response and AI readiness check tested across your site, the full report does it and tells you what to change.