MobileCheck

AI Crawler Access Checker: Which AI Bots Can Read Your Site?

Enter a website to see which of 20 AI and search crawlers its robots.txt allows — for any page — with the exact rule that decides, what each crawler is used for, and whether the site has an llms.txt file.

Free, no sign-up. Our server reads the address with plain requests (as MobileCheckBot, no browser) and keeps the result for 10 minutes; up to 30 checks an hour. Only /robots.txt and /llms.txt are read.

Or paste a robots.txt to test it before publishing

Tested in your browser; nothing is sent to us.

20 crawlers and what they do

  • GPTBot OpenAI AI training

    Collects public pages that may be used to train OpenAI's generative AI models. Source

  • OAI-SearchBot OpenAI Search / AI answers

    Indexes pages so they can appear in ChatGPT search results. OpenAI says changes take about 24 hours to apply. Source

  • ChatGPT-User OpenAI User-triggered fetch

    Opens a page when a ChatGPT user or a custom GPT asks for it. OpenAI says robots.txt rules may not apply to these user-initiated visits. Source

  • ClaudeBot Anthropic AI training

    Collects public web pages that may be used to train Anthropic's Claude models. Source

  • Claude-SearchBot Anthropic Search / AI answers

    Indexes pages to improve the relevance and accuracy of Claude's search results. Source

  • Claude-User Anthropic User-triggered fetch

    Fetches a page when a Claude user's question needs it. Anthropic says its bots honor robots.txt; blocking this one stops Claude from opening your pages for users. Source

  • anthropic-ai Anthropic Legacy token

    An older Anthropic token that the current documentation no longer lists. Rules for it are harmless but no longer needed; use ClaudeBot instead. Source

  • PerplexityBot Perplexity Search / AI answers

    Indexes and links pages in Perplexity search answers. Perplexity says it is not used to train AI foundation models. Source

  • Perplexity-User Perplexity User-triggered fetch

    Visits a page when a Perplexity user asks a question, to answer it with a link. Perplexity says this fetcher generally ignores robots.txt, because a user asked for the page. Source

  • Google-Extended Google Control token

    Not a crawler: a robots.txt switch for whether pages Google crawls may train Gemini models and ground answers in Gemini apps and Vertex AI. Google says it has no effect on Google Search or ranking, and it does not control AI Overviews. Source

  • Googlebot Google Search / AI answers

    Google Search's crawler — Search, Images, News, Discover, and also AI Overviews and AI Mode. Blocking it removes the page from Google Search. Use nosnippet or max-snippet to limit AI Overviews instead. Source

  • Bingbot Microsoft Search / AI answers

    Bing's search crawler; Bing's index also feeds Copilot answers. Bing controls use in Copilot with the NOARCHIVE and NOCACHE meta tags, not robots.txt. Source

  • Applebot Apple Search / AI answers

    Apple's crawler for Spotlight, Siri and Safari search features. Apple says its data may also help train Apple's models. With no Applebot group, it follows the rules for Googlebot. Source

  • Applebot-Extended Apple Control token

    Not a crawler: a robots.txt switch that keeps pages Applebot collects out of training Apple's foundation models. Apple says it does not affect search. Source

  • CCBot Common Crawl AI training

    Builds Common Crawl's free, open archive of the web — one of the most widely used sources of AI training data. Common Crawl's page describes the archive, not AI training; the label reflects how the data is used. Source

  • Bytespider ByteDance Undocumented

    ByteDance's (TikTok's owner) crawler; press reports link it to ByteDance's AI models. ByteDance publishes no documentation for it, and there are reports that it ignores robots.txt.

  • Meta-ExternalAgent Meta AI training

    Meta's crawler for uses such as training AI models and improving products by indexing content. Meta may cache robots.txt for up to 24 hours. Source

  • Amazonbot Amazon AI training

    Amazon's crawler for improving its products and services; Amazon says the data may be used to train its AI models. Amazon may use a robots.txt copy up to 30 days old. Source

  • DuckAssistBot DuckDuckGo Search / AI answers

    Fetches pages for DuckDuckGo's AI-assisted answers, which cite their sources. DuckDuckGo says the data is not used to train AI models. Changes take effect after about 72 hours. Source

  • cohere-ai Cohere Undocumented

    A token often copied into robots.txt block lists. Cohere documents no crawler by this name and says it does not currently crawl the web for training. Source

Training, search and user fetches

AI companies now run several crawlers each, and blocking one does not block the others:

Each description above comes from the vendor's own documentation, linked as "Source" (checked October 2026). Where a vendor publishes nothing, we say so.

How the checker reads robots.txt

It follows RFC 9309, the robots.txt standard that Google, Bing and the AI vendors refer to. A crawler uses the group whose User-agent names it (case does not matter, and several groups naming it are combined); only if none does, the User-agent: * group. Inside the group, the rule with the longest matching path wins, and Allow wins a tie. * matches any characters and $ marks the end of the address. A missing robots.txt (a 4xx answer) means everything is allowed; a server error (5xx) means crawlers must assume everything is blocked. Applebot is a special case: Apple says it follows the Googlebot rules when no group names Applebot.

A robots.txt that blocks AI training but keeps search

# AI training: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /

# Everyone else, including search and AI search crawlers
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Several User-agent lines in a row share one set of rules. Remember that a crawler with its own group ignores the * group completely — if you add a group for Googlebot, repeat in it every rule Googlebot should follow.

About llms.txt and "noai"

llms.txt is a proposal from September 2024 for a Markdown file at /llms.txt that summarizes a site for AI tools. It is an emerging, unofficial convention: no major AI vendor documents it as a way to allow or block crawling, and Google says you don't need special AI files to appear in its AI features. The checker only tells you whether you have one. The noai and noimageai meta tags are not supported by Google's robots meta tag rules or documented by OpenAI or Anthropic, so the checker does not count them.

Frequently asked questions

How do I block AI training but stay in search results?

Disallow the training crawlers and control tokens — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent — and leave Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. Google-Extended does not affect Google Search.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google says Google-Extended controls use in Gemini models and Gemini apps, not Search. AI Overviews and AI Mode are part of Google Search and use Googlebot; to limit them, use the nosnippet or max-snippet rules — or noindex, which removes the page from Search entirely.

Do AI crawlers obey robots.txt?

The crawlers documented by OpenAI, Anthropic, Google, Apple, Perplexity, Amazon, Meta, DuckDuckGo and Common Crawl say they do. Some user-triggered fetchers say robots.txt may not apply when a person asks for a page (OpenAI's ChatGPT-User, Perplexity-User). robots.txt is a request, not access control: to really keep content private, put it behind a login.

Is llms.txt or a "noai" meta tag a way to block AI?

No. llms.txt is a 2024 proposal for a Markdown summary that helps AI tools use a site; it does not control crawling. "noai" and "noimageai" meta tags started on DeviantArt and are not documented by Google, OpenAI or Anthropic. robots.txt is the control the vendors document.

More free tools: color contrast checker · tap target size checker · mobile font size calculator · css media query generator · device mockup generator · link preview checker · image alt text checker · all tools