robots.txt
The standard for telling crawlers what they may visit.
llms.txt and AI crawlers
Who reads your site for the assistants.
llms.txt is a text file, proposed in 2024, that gives language models a readable index of a site’s most useful pages; AI crawlers are the programs AI assistant providers use to read the web. Find out how to control them with robots.txt.
In brief
llms.txt and AI crawlers in a few lines.
AI crawlers visit pages to train models, power assistants’ search or answer a user’s request. robots.txt tells each of them what it may read; llms.txt proposes an ordered map of the content.
llms.txt
A proposal for pointing models to the pages to read first.
Server logs
The only way to know who really visited.
Who visits the site
The most common AI crawlers and what they are for.
Each provider declares its agents by name, the user agent. Some collect pages for training, others for search inside the assistant, and others only when a user asks to open a page.
Six AI crawler agents publicly documented by their providers, with the provider and the declared purpose, for use in robots.txt rules.
| Agent | Provider | Declared purpose |
|---|---|---|
| GPTBot | OpenAI | Collect pages that may be used to train models. |
| OAI-SearchBot | OpenAI | Index pages for search inside ChatGPT. |
| ChatGPT-User | OpenAI | Open a page when a user asks for it in a conversation. |
| Google-Extended | Control the use of content for Gemini models, not for Search. | |
| ClaudeBot | Anthropic | Collect pages that may be used to train models. |
| PerplexityBot | Perplexity | Index pages for Perplexity’s answers. |
The rules
What to write in robots.txt.
robots.txt is read from the top, one block per agent. You can open assistants’ search and close training, or protect only some folders. The file is a request: declared crawlers respect it; it is not a technical barrier.
The Robots Exclusion Protocol is described in IETF RFC 9309 (September 2022). The agents and their meaning are documented by the providers on their webmaster pages.
An example robots.txt for a fictional site that lets assistants’ search read it, closes training and protects the private area.
example.com/robots.txt · fictional site
- Assistants’ search
- User-agent: OAI-SearchBot · Allow: /
- Training
- User-agent: GPTBot · Disallow: /
- Use for Gemini models
- User-agent: Google-Extended · Disallow: /
- All the others
- User-agent: * · Disallow: /private-area/
The llms.txt file
What llms.txt does, and what it doesn’t.
llms.txt was proposed in September 2024 on llmstxt.org as a Markdown file at the site root, with a summary and links to the main pages. It is a voluntary convention: worth looking after, but with no assurance that an assistant reads it.
The two text files at a site’s root compared on purpose, status, audience and the result you can expect for presence in AI answers.
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Say what may be visited and what may not. | Point to the most useful pages, with a summary. |
| Status | An IETF standard respected by declared crawlers. | A voluntary proposal, unevenly adopted. |
| Audience | All crawlers, one per block. | Language models and the tools that use them. |
| What to expect | Compliance with the rules by those who declare themselves. | Extra order, with no measurable effect assured. |
Decide and verify
From the choice to the proof in the logs.
The decision on what to open to AI crawlers is made page by page and checked in the server logs: only there can you see who came, when and on which addresses.
The steps by which a company decides which parts of its site to open to AI crawlers, writes the rules and checks in the logs that they are respected.
List the pages
Those you want assistants to be able to read and cite.
Decide per agent
Search, training, visits started by a person: three different choices.
Write the rules
One block per agent in robots.txt, with the protected folders.
Check the logs
Which agents come, on which pages and how often.
Review every quarter
New agents appear: the list needs updating.
Frequently asked questions
Frequently asked questions about llms.txt and AI crawlers.
Whether llms.txt is useful, whether blocking AI crawlers reduces presence in answers, how to know who visits the site and whether robots.txt is protection.
What is the llms.txt file and who proposed it?
A Markdown file at the site root, proposed in 2024, that gives language models a summary and links to the most useful pages. It is a voluntary convention, not a standard.
Do you really need an llms.txt file?
It costs little and brings order, but no one can promise that an assistant will read it. Clear, reachable pages matter more.
If I block AI crawlers, do I disappear from answers?
It depends on which agent you block. Closing an assistant’s search can remove your pages from its citations; closing training concerns a different use.
Does robots.txt protect content?
No. It is a request that declared crawlers respect. Private content needs protected access and authentication.
Related paths
After “llms.txt and AI crawlers”, the next step.
The guides explain the concepts with examples on fictional brands. From here you continue with the other guides, the methodology and the features that apply them to your brand.
Guides to reputation, trust and AI
Guides on GEO, AEO, AI visibility, online reputation, reviews and monitoring: what to check, a worked example and the steps to act.
ReadFrom question to action
How TrustInsider works in six steps: brand, questions, quote, evidence, page-by-page intervention and a check on the same questions.
ReadA transparent method, not a black box
How every TrustInsider measure is built: sampling of AI answers, sources and rights, index calculation and declared limits.
ReadFind the right answer
Help with access, credits, collections, integrations and data: guides, FAQs, the site assistant and support requests from your workspace.
Read