robots.txt check
AI crawler check: paste your site’s robots.txt and find out, crawler by crawler, who can read a path, who is blocked and which reads requested by a person do not follow robots.txt.
Free check
Paste the content of your robots.txt: it is read in your browser and the text never leaves the page.
What it controls
The tool applies your robots.txt rules to every AI crawler TrustInsider recognises. For each one you find the provider, the declared purpose and whether the path you choose is allowed or blocked.
The purposes for which providers declare their crawlers, with what the crawler does in each case and an agent from the registry that carries it, so you can read the purpose column in the results table.
| Declared purpose | What the crawler does | An agent with this purpose |
|---|---|---|
| Model training | Collects pages that may go into a model’s training data. | GPTBot by OpenAI |
| Index for answers with search | Builds the index from which an assistant chooses sources when it searches. | OAI-SearchBot by OpenAI |
| Read requested by a person | Opens a page when a user asks the assistant to read it. | ChatGPT-User by OpenAI |
| Search engine | Indexes the site for search results and the features that use them. | Googlebot by Google |
Reading the rules
A crawler follows the User-agent group that names it; only if no group names it does it follow the one with the asterisk. Among the group’s rules, the one with the longest path applies, and on a tie Allow wins.
A sample robots.txt for a fictional brand: a general group that allows everything, a group dedicated to a training crawler that blocks a folder, and the sitemap declared at the end.
User-agent: *
Allow: /User-agent: GPTBot
Disallow: /prodotti/Sitemap: https://ilmarde-audio.example/sitemap.xmlStated limitations
robots.txt asks crawlers what not to read; respecting it is a choice of whoever runs the crawler. Some providers declare that reads requested by a person do not follow it: the tool marks them as not applicable.
The tool does not open your site or download the file: if the published robots.txt has changed, paste the live version.
A crawler allowed by the rules may never have come by; a blocked crawler may have requested the pages anyway.
Rules and visits
With the platform’s AI agent analysis you compare the rules with recorded requests: which agents read the pages, when, and whether whoever uses an agent’s name comes from the addresses its provider publishes.
What the free check says about the rules and what the analysis of requests recorded by the site adds, on three questions: permission, actual passage and agent identity.
| robots.txt check | AI agent analysis | |
|---|---|---|
| Permission | Says whether the rules allow a crawler on a path. | Puts the rules next to the pages each agent requested. |
| Step | Does not know whether the crawler ever reached the site. | Counts each AI agent’s requests in the chosen period. |
| Identity | Takes the name written in the rules at face value. | Compares the source address with the ranges published by the provider. |
Frequently asked questions
The answers clarify what an allowed or blocked crawler means, why some reads show as not applicable and how to check a single section of the site with the path field.
Not necessarily. Many providers use different crawlers for training and for the index of answers with search: blocking one says nothing about the others. That is why the table shows each crawler’s purpose.
They are reads started by a person, when they ask the assistant to open a page. The providers we flag as not applicable declare that these reads do not follow robots.txt.
Type the section’s address in the path field, starting from the slash, for example /blog/. The tool applies the rules to that path for every crawler.
Related paths
Having seen which AI crawlers your robots.txt lets in, try the other free tools, find out how the platform tracks the brand over time, read the measurement methodology and find support.
Free tools for AI visibility: check how Gemini describes the brand, the AI crawlers in robots.txt, your llms.txt and your share of voice.
ReadLearn how to use TrustInsider: choose the brand and the questions, read the collected answers, update your content and compare the results.
ReadHow every TrustInsider measure is built: sampling of AI answers, sources and rights, index calculation and declared limits.
ReadHelp with access, credits, collections, integrations and data: guides, FAQs, the site assistant and support requests from your workspace.
Read