Go TopSEO
Free tool

Which AI crawlers is your site blocking?

We read your robots.txt and tell you, for each AI crawler separately, whether it is allowed or blocked — and which line in the file decided it.

  • No sign-up, no email
  • One read of one file
  • Check a specific path, not only the home page

Empty checks the home page. robots.txt rules change from one path to another.

Who each one is

This table stands on its own: the crawler name as it appears in the User-agent line, who operates it, and what for.

User-agentOperated byWhat for
GPTBotOpenAITrainingCollects pages to train a model. Blocking it keeps you out of the next model.
OAI-SearchBotOpenAIIndexBuilds a search index, which AI answers are built on top of.
ChatGPT-UserOpenAILive answerFetches the page the moment someone asks, to cite you in an answer with a link.
OAI-AdsBotOpenAIAdvertsChecks the landing pages of adverts. Blocking it affects neither answers nor the index.
ClaudeBotAnthropicTrainingCollects pages to train a model. Blocking it keeps you out of the next model.
Claude-SearchBotAnthropicIndexBuilds a search index, which AI answers are built on top of.
Claude-UserAnthropicLive answerFetches the page the moment someone asks, to cite you in an answer with a link.
Claude-AgentAnthropicAgent in a browserA browser an AI agent drives for one person. Blocking it changes nothing a model knows — it stops your own visitors asking an assistant to do something on your site.
Google-ExtendedGoogleTrainingCollects pages to train a model. Blocking it keeps you out of the next model.
GoogleOtherGoogleTrainingCollects pages to train a model. Blocking it keeps you out of the next model.
Google-CloudVertexBotGoogleIndexBuilds a search index, which AI answers are built on top of.
PerplexityBotPerplexityIndexBuilds a search index, which AI answers are built on top of.
Perplexity-UserPerplexityLive answerFetches the page the moment someone asks, to cite you in an answer with a link.
Applebot-ExtendedAppleTrainingCollects pages to train a model. Blocking it keeps you out of the next model.
Meta-ExternalAgentMetaTrainingCollects pages to train a model. Blocking it keeps you out of the next model.
Meta-WebIndexerMetaIndexBuilds a search index, which AI answers are built on top of.
Meta-ExternalFetcherMetaLive answerFetches the page the moment someone asks, to cite you in an answer with a link.
Meta-ExternalAdsMetaAdvertsChecks the landing pages of adverts. Blocking it affects neither answers nor the index.
FacebookExternalHitMetaLink previewBuilds the title and image shown when someone shares the link. Blocking it leaves the share bare.
AmazonbotAmazonLive answerFetches the page the moment someone asks, to cite you in an answer with a link.
CCBotCommon CrawlTrainingCollects pages to train a model. Blocking it keeps you out of the next model.
AI2BotAllen Institute for AITrainingCollects pages to train a model. Blocking it keeps you out of the next model.
DiffbotDiffbotTrainingCollects pages to train a model. Blocking it keeps you out of the next model.
MistralAI-IndexMistral AIIndexBuilds a search index, which AI answers are built on top of.
MistralAI-TrainingMistral AITrainingCollects pages to train a model. Blocking it keeps you out of the next model.
MistralAI-UserMistral AILive answerFetches the page the moment someone asks, to cite you in an answer with a link.
DuckAssistBotDuckDuckGoLive answerFetches the page the moment someone asks, to cite you in an answer with a link.
KagibotKagiIndexBuilds a search index, which AI answers are built on top of.
GooglebotGoogleIndexBuilds a search index, which AI answers are built on top of.
ApplebotAppleIndexBuilds a search index, which AI answers are built on top of.
BingbotMicrosoftIndexBuilds a search index, which AI answers are built on top of.
Bytespider †ByteDanceTrainingCollects pages to train a model. Blocking it keeps you out of the next model.
cohere-ai †CohereTrainingCollects pages to train a model. Blocking it keeps you out of the next model.
Timpibot †TimpiTrainingCollects pages to train a model. Blocking it keeps you out of the next model.
omgili †Webz.ioTrainingCollects pages to train a model. Blocking it keeps you out of the next model.

† No operator page documents this token. It is listed because it appears in server logs, and kept apart from the rows that rest on an operator’s own documentation.

Build the robots.txt lines

Decide crawler by crawler, and we write the exact lines. Nothing is sent anywhere: the file is built in your browser.

  • GPTBotOpenAI · Training
  • OAI-SearchBotOpenAI · Index
  • ChatGPT-UserOpenAI · Live answer
  • OAI-AdsBotOpenAI · Adverts
  • ClaudeBotAnthropic · Training
  • Claude-SearchBotAnthropic · Index
  • Claude-UserAnthropic · Live answer
  • Claude-AgentAnthropic · Agent in a browserThis crawler has not committed to honouring robots.txt
  • Google-ExtendedGoogle · Training
  • GoogleOtherGoogle · Training
  • Google-CloudVertexBotGoogle · Index
  • PerplexityBotPerplexity · Index
  • Perplexity-UserPerplexity · Live answerThis crawler has not committed to honouring robots.txt
  • Applebot-ExtendedApple · Training
  • Meta-ExternalAgentMeta · Training
  • Meta-WebIndexerMeta · Index
  • Meta-ExternalFetcherMeta · Live answerThis crawler has not committed to honouring robots.txt
  • Meta-ExternalAdsMeta · Adverts
  • FacebookExternalHitMeta · Link previewThis crawler has not committed to honouring robots.txt
  • AmazonbotAmazon · Live answer
  • CCBotCommon Crawl · Training
  • AI2BotAllen Institute for AI · Training
  • DiffbotDiffbot · Training
  • MistralAI-IndexMistral AI · Index
  • MistralAI-TrainingMistral AI · Training
  • MistralAI-UserMistral AI · Live answer
  • DuckAssistBotDuckDuckGo · Live answer
  • KagibotKagi · Index
  • BytespiderByteDance · TrainingThis crawler has not committed to honouring robots.txt
  • cohere-aiCohere · TrainingThis crawler has not committed to honouring robots.txt
  • TimpibotTimpi · TrainingThis crawler has not committed to honouring robots.txt
  • omgiliWebz.io · TrainingThis crawler has not committed to honouring robots.txt

Your robots.txt lines

No crawler has a decision yet, so there are no lines to add. Choose above and the lines appear.

How many sites actually block them

On the date below we read the robots.txt of the highest-ranked sites in the markets listed here, and matched them against the table above.

Of the sites that block GPTBot, 66.5% also block an AI crawler that would have brought them visitors.

These are large, high-ranked sites, not small businesses. Among small businesses the picture is probably different, with fewer deliberate blocks and more neglect. Do not read these figures as a picture of small businesses.

MarketBlock an AI crawler that brings visitors
.nl31.5%
.com29.7%
.fr25.8%
.it16.2%
.es15%
.jp13.7%
.de12.3%

Percentages are of 528 robots.txt files actually read, not of the domains requested.

Brazil and Portugal were not in the sample. The .com figure is the global English-language slice and is not a figure for either country.

Measured on

The whole measurement, as open data: JSON · CSV

We read them again on 2026-11-09, keeping every domain this time. Of the sites that block GPTBot, 80.3% wrote a rule that names it, so the block was a decision. Of the sites that block OAI-SearchBot — the crawler behind the answers people are shown — 43.6% never chose it: the block came from a rule aimed at every bot alike.

That second reading covers 407 readable files out of 700 domains requested. 145 hosts refused this reader outright, and a host that refuses an unknown automated reader is plausibly a host that restricts AI crawlers, so this reading probably under-states blocking rather than over-stating it.

By country suffix

The same reading, cut by suffix. Every share in a row is of the files read in that suffix, not of the hundred domains requested.

MarketFiles readBlock GPTBotBlock OAI-SearchBot
.it65 / 10015 (23.1%)15 of these deliberate5 (7.7%)0 of these inherited
.es64 / 10011 (17.2%)9 of these deliberate5 (7.8%)2 of these inherited
.fr59 / 1007 (11.9%)6 of these deliberate4 (6.8%)1 of these inherited
.nl59 / 10010 (16.9%)8 of these deliberate4 (6.8%)2 of these inherited
.com56 / 10017 (30.4%)10 of these deliberate12 (21.4%)9 of these inherited
.de55 / 1008 (14.5%)7 of these deliberate4 (7.3%)1 of these inherited
.jp49 / 1008 (16.3%)6 of these deliberate5 (10.2%)2 of these inherited

Every domain, with its verdict, the rule group that decided it and the exact line: JSON · CSV

A second sample, five more markets

On 2026-11-17 we read 350 more robots.txt files in markets the first reading never touched. They block less: 42.3% of the sites blocking GPTBot there also shut out a crawler that brings visitors, against 66.5% in the table above. The two samples are never added together.

MarketBlock an AI crawler that brings visitors
.kr14.8%
.se11.3%
.pl9.1%
.br6.1%
.gr4.1%

The same caveat holds here: large, high-ranked sites, not small businesses. Korea contributed the fewest files of the five, so its figure moves more than the others.

What this check cannot tell you

  • robots.txt is a request, not a wall. A crawler can ignore it.
  • A firewall or CDN can block a crawler that robots.txt allows.
  • The check describes one moment. Editing the file changes the answer at once.
  • Blocking a training crawler and blocking a live-answer crawler are two entirely different decisions.

See whether AI actually mentions you

Access is only the first step. Go Top SEO checks automatically whether your business shows up in answers from ChatGPT, Gemini and Google AI, and tracks your Google and Maps rankings every month.

Try it free for 7 days, no card
Free demo