Free tool
Which AI crawlers is your site blocking?
We read your robots.txt and tell you, for each AI crawler separately, whether it is allowed or blocked — and which line in the file decided it.
- No sign-up, no email
- One read of one file
- Check a specific path, not only the home page
Who each one is
This table stands on its own: the crawler name as it appears in the User-agent line, who operates it, and what for.
| User-agent | Operated by | What for |
|---|---|---|
GPTBot | OpenAI | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
OAI-SearchBot | OpenAI | IndexBuilds a search index, which AI answers are built on top of. |
ChatGPT-User | OpenAI | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
ClaudeBot | Anthropic | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
Claude-SearchBot | Anthropic | IndexBuilds a search index, which AI answers are built on top of. |
Claude-User | Anthropic | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
Google-Extended | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. | |
PerplexityBot | Perplexity | IndexBuilds a search index, which AI answers are built on top of. |
Perplexity-User | Perplexity | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
Applebot-Extended | Apple | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
Meta-ExternalAgent | Meta | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
meta-externalfetcher | Meta | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
Amazonbot | Amazon | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
CCBot | Common Crawl | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
Bytespider | ByteDance | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
cohere-ai | Cohere | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
Diffbot | Diffbot | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
AI2Bot | Allen Institute for AI | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
Timpibot | Timpi | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
omgili | Webz.io | TrainingCollects pages to train a model. Blocking it keeps you out of the next model. |
MistralAI-User | Mistral AI | Live answerFetches the page the moment someone asks, to cite you in an answer with a link. |
Googlebot | IndexBuilds a search index, which AI answers are built on top of. | |
Bingbot | Microsoft | IndexBuilds a search index, which AI answers are built on top of. |
Applebot | Apple | IndexBuilds a search index, which AI answers are built on top of. |
What this check cannot tell you
- robots.txt is a request, not a wall. A crawler can ignore it.
- A firewall or CDN can block a crawler that robots.txt allows.
- The check describes one moment. Editing the file changes the answer at once.
- Blocking a training crawler and blocking a live-answer crawler are two entirely different decisions.
See whether AI actually mentions you
Access is only the first step. Go Top SEO checks automatically whether your business shows up in answers from ChatGPT, Gemini and Google AI, and tracks your Google and Maps rankings every month.
Try it free for 7 days, no card