AI Crawl Visibility Suite

Enter a domain or a full URL. We read your robots.txt rules for 30 AI and search crawlers, test both the homepage and the exact path, validate llms.txt and llms-full.txt, and flag noindex or noai directives that quietly hide pages from AI answers.

Secure
Instant
Accurate

AI Crawl report Sample

robots.txt analysisFound
LLMs.txt checkMissing
GPTBotAllowed
ClaudeBotAllowed
PerplexityBotBlocked
Shareable report linkReady

30AI and search bots checked

2Paths tested per run: homepage and your URL

2LLM files validated: llms.txt and llms-full.txt

3Page directives read: noindex, noai, noimageai

What we check

One run, four layers of AI bot visibility

A bot can be blocked at several levels. The report shows each one so you know exactly which rule is responsible.

TXT

robots.txt rules

Every user-agent group is parsed and matched per bot, including wildcard fallbacks, Allow versus Disallow precedence and Sitemap lines.

LLM

llms.txt files

We confirm whether llms.txt and llms-full.txt exist and follow the proposed format: an H1 title, a blockquote summary and sections of links.

Meta robots

Page level robots meta tags are read for noindex, noai and noimageai, which can override a permissive robots.txt.

HDR

X-Robots-Tag

HTTP response headers are inspected for the same directives, a common blind spot because they never appear in the page source.

Bot directory

AI crawlers and what they actually do

Not every AI bot is the same. Some collect training data, some build a search index that powers citations, and some fetch a page only because a user asked. Blocking decisions should follow the purpose.

User-agent token	Company	Purpose	Notes
GPTBot	OpenAI	Training	Collects content that may be used to train OpenAI models.
OAI-SearchBot	OpenAI	Search	Indexes pages for ChatGPT search results and citations. Separate from GPTBot.
ChatGPT-User	OpenAI	User-triggered fetch	Visits a page when a ChatGPT user or custom GPT requests it. OpenAI notes robots.txt may not apply to user initiated requests.
ClaudeBot	Anthropic	Training	Collects public web content that may be used to train Claude models.
Claude-SearchBot	Anthropic	Search	Indexes content to improve the quality of Claude's search results.
Claude-User	Anthropic	User-triggered fetch	Fetches pages when a Claude user asks a question that needs them.
anthropic-ai	Anthropic	Legacy token	Older token still found in many robots.txt files; worth keeping alongside ClaudeBot if you block training.
PerplexityBot	Perplexity	Search index	Builds the index behind Perplexity answers and citations. Perplexity says it is not used to train foundation models.
Perplexity-User	Perplexity	User-triggered fetch	Fetches a page in response to a user question. Perplexity states this agent generally ignores robots.txt because a person requested the page.
Google-Extended	Google	Control token (not a crawler)	Tells Google whether content crawled by Googlebot may be used for Gemini model training and for grounding in Gemini Apps and Vertex AI. No separate bot visits your site, and it does not affect Google Search.
Googlebot	Google	Search	Google Search crawler. AI Overviews and AI Mode draw on the Search index, so they are governed by Googlebot and snippet controls, not Google-Extended.
Applebot	Apple	Search	Powers Siri and Spotlight suggestions.
Applebot-Extended	Apple	Control token (not a crawler)	Lets you opt out of Apple using Applebot crawled content to train its generative models, without leaving Apple search features.
Bingbot	Microsoft	Search	Bing index, which also feeds Microsoft Copilot answers.
CCBot	Common Crawl	Open dataset	Builds the public Common Crawl archive, which many AI labs use as training data.
Bytespider	ByteDance	Training	Collects data for ByteDance products and models.
Amazonbot	Amazon	Mixed	Used to improve Amazon services such as Alexa answers; Amazon says content may also be used to train its AI models.
Meta-ExternalAgent	Meta	Training	Crawls content for training Meta AI models and improving related products.
cohere-ai	Cohere	AI assistant and models	Token associated with Cohere's crawling for its language models.
DuckAssistBot	DuckDuckGo	Search answers	Fetches pages to source DuckAssist answers; DuckDuckGo says the data is not used to train models.
MistralAI-User	Mistral AI	User-triggered fetch	Retrieves pages when a Le Chat user needs them in an answer.

Purposes reflect each company's public documentation at the time of writing. Vendors add and rename agents, so check their crawler pages before making permanent policy decisions.

robots.txt examples

Copy ready rules for common AI policies

Adapt these to your site, test them with the checker above, and keep a Sitemap line at the end.

Allow AI search, block AI training

Stay eligible for citations in ChatGPT search, Claude and Perplexity while opting out of model training datasets.

# Block training crawlers and control tokens
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /

# Allow AI search and answer crawlers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Block all known AI bots

For publishers who want no AI use at all. Traditional search stays open through the wildcard group.

# Block AI training, search and fetch agents
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: cohere-ai
User-agent: DuckAssistBot
User-agent: MistralAI-User
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Know the trade-off: the second policy removes you from AI search citations too, and blocking Google-Extended also stops your pages from grounding answers in the Gemini app. Robots.txt is a request, not an access control, and user-triggered agents may not honor it. Use server or CDN rules if you need enforcement.

llms.txt checker

What llms.txt is and how to write one

llms.txt is a plain Markdown file at the root of your domain that gives language models a curated map of your most useful content. It was proposed in September 2024 by Jeremy Howard of Answer.AI. An optional companion, llms-full.txt, contains the full text of key pages in one file.

The format is simple: one H1 with the site or project name, a blockquote with a short summary, optional notes, then H2 sections containing Markdown link lists with a brief description after each link. A section titled "Optional" marks links that can be skipped when context is limited.

Our checker confirms the file exists, returns a 200 status as text, and has the H1, blockquote and link sections. If it is missing, the report generates a starter file from your site.

Direct answer: llms.txt is a community proposal, not an official web standard, and major AI search engines have not confirmed they use it for retrieval or ranking. It is cheap to add and useful for AI coding tools and agents, but it will not replace good robots.txt rules or crawlable HTML.

example.com/llms.txt

# Example Plumbing Co.

> Licensed residential plumbing company serving
> Austin, Texas since 2009. Emergency repairs,
> water heaters and repiping.

Service area and pricing pages are updated monthly.

## Services
- [Water heater installation](https://example.com/water-heaters/): tank and tankless options, typical costs
- [Emergency plumbing](https://example.com/emergency/): 24/7 response area and call-out fees
- [Repiping](https://example.com/repiping/): PEX vs copper, timelines

## Company
- [About us](https://example.com/about/): license numbers, team
- [Contact](https://example.com/contact/): phone, hours, address

## Optional
- [Blog](https://example.com/blog/): maintenance guides

Guide

AI bot visibility: a practical guide

How to decide which AI crawlers to allow, avoid the mistakes that silently block them, and verify the result.

Three kinds of AI bots

The biggest mistake site owners make is treating "AI bots" as one group. In practice there are three jobs, and major vendors now use separate user agents for each so you can decide independently.

Training crawlers such as GPTBot, ClaudeBot, CCBot and Meta-ExternalAgent collect content that may end up in future model training data. Blocking them does not remove you from AI search answers.
Search and index crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes that answer engines search at question time. These are the bots that make citations possible. Block them and you largely disappear from that engine's linked sources.
User-triggered fetchers such as ChatGPT-User, Claude-User, Perplexity-User and MistralAI-User load a page because a person asked the assistant to read it or needed it for an answer. They behave more like a browser than a crawler, and some vendors say robots.txt does not fully apply to them.

Then there are control tokens. Google-Extended and Applebot-Extended never visit your site. They are names you use in robots.txt to tell Google and Apple how content already fetched by Googlebot or Applebot may be used for AI.

How robots.txt matching works

Robots.txt is standardized in RFC 9309. A few rules explain most surprises in AI bot visibility reports:

A bot obeys only its most specific group. If you write a group for GPTBot, GPTBot ignores everything in the User-agent: * group. Rules do not merge, so anything you want applied to GPTBot must be repeated in its own group.
Several user agents can share one group. Stacking User-agent lines above one set of rules, as in the examples on this page, keeps files short and consistent.
The longest matching path wins. With Disallow: /blog/ and Allow: /blog/guides/, the guides folder stays open. When an Allow and a Disallow match equally, Allow wins.
Tokens are case insensitive, paths are not. gptbot matches GPTBot, but /Blog/ and /blog/ are different paths.
A missing file means everything is allowed. A 404 on robots.txt is read as no restrictions. A 5xx error is different: many crawlers treat it as a temporary full block.

This is why the checker tests both your homepage and the exact path you enter. A site can welcome AI bots at the root while a Disallow rule on /resources/ hides the one guide you most want cited.

Is GPTBot blocked on your site?

Plenty of sites block GPTBot without anyone on the team deciding to. Common causes include a security or SEO plugin that added AI blocking rules during an update, a hosting provider or CDN setting that blocks AI crawlers by default, a copy and paste robots.txt from a news publisher, or a staging rule like Disallow: / that went live.

Remember that blocking GPTBot only affects training. If your goal is to appear in ChatGPT search answers, the bot that matters is OAI-SearchBot. Many sites block both by accident while intending to block only one. Run the checker above and look at each token individually rather than assuming the answer.

Hidden blocks outside robots.txt

A clean robots.txt does not guarantee access. Check these layers too:

Meta robots and X-Robots-Tag. A noindex keeps pages out of search indexes, including those used by AI search. The noai and noimageai values are informal signals that some platforms honor and many crawlers ignore, but they show your intent. The checker reads both the HTML tag and the HTTP header.
Firewalls and bot management. CDNs and security services can block or challenge AI user agents at the network level. These blocks never appear in robots.txt, so a page may look open while bots receive a 403 or a challenge page.
JavaScript-only content. Many AI crawlers do not render JavaScript. If your main text loads only after scripts run, a crawler may be allowed in and still see an empty page.
Login walls and geo restrictions. Content behind a sign in or blocked in the crawler's region is invisible to it.

Choosing a policy

There is no single correct setting. The right policy depends on how your business earns value from content.

Businesses that sell services or products usually benefit from being cited, since AI answers can send qualified visitors. Allowing search and user fetch agents is the common choice; training is a judgment call.
Publishers whose content is the product often block training crawlers and weigh the value of AI search referrals against lost page views, sometimes as part of licensing negotiations.
Documentation and developer sites tend to allow everything and add llms.txt, because being understood by AI coding assistants helps adoption.

A useful middle path is the first robots.txt example above: allow search crawlers, block training crawlers, and leave traditional search untouched. Blocking AI bots in robots.txt does not affect your Google or Bing rankings, as long as you leave Googlebot and Bingbot alone. Google-Extended in particular has no effect on Search ranking or inclusion.

llms.txt in practice

Think of llms.txt as a curated table of contents for machines. Keep it short, list the 10 to 40 pages that best explain what you do, and write one line of plain description per link. Link to clean, canonical HTML pages or Markdown versions, not to pages full of navigation and pop-ups.

Be realistic about impact. The proposal is widely discussed, many developer platforms publish the file, and AI coding tools often read it, but mainstream AI answer engines have not publicly committed to using it. Treat it as low cost preparation rather than a ranking lever, and put your main effort into crawl access and content quality.

Reading your AI Crawl report

The report is organized so you can move from a quick verdict to the rule behind it, then to a fix.

Bot access grid

Each of the 30 bots is shown as allowed or blocked for both the homepage and the path you entered. When a bot is blocked, open your robots.txt and find the group that applies to it using the matching rules above: either a group naming that bot or the User-agent: * fallback. That tells you whether the block is intentional and targeted or a side effect of a broad wildcard rule written years ago for a different purpose.

Directive findings

Meta robots and X-Robots-Tag findings are listed separately from robots.txt, because they work differently. Robots.txt controls whether a bot may fetch a URL. A noindex directive only works if the bot is allowed to fetch the page and read it. Blocking a URL in robots.txt and adding noindex to the same page is a classic conflict: the crawler never sees the noindex, so a search engine may still list the bare URL from links elsewhere.

Sitemap lines and llms files

We list every Sitemap: line found in robots.txt. Search crawlers use these to discover pages faster, and a sitemap that points to an old domain or a 404 is worth fixing while you are there. The llms.txt section shows whether each file exists and which format elements passed or failed.

Suggested fixes

At the end, the report proposes a robots.txt block based on the bots you currently block and allow, plus an llms.txt starter built from your site. Review both before publishing. They are starting points that reflect common policies, not a decision the tool makes for you.

Verify and monitor

After any change, run the checker again for your homepage and two or three important deep URLs. Then keep an eye on it over time:

Re-check after CMS, plugin, theme or CDN updates, since these are the usual sources of accidental blocks.
Review your server logs for requests from the user agents in the directory above, and confirm them against each vendor's published IP ranges where available, because user agent strings are easy to fake.
Revisit your policy a few times a year. New agents appear regularly, and a group you wrote for yesterday's bots will not cover tomorrow's.

Once bots can reach you, the next question is whether they cite you. Use the AI citation analyzer to see which sources answer engines choose for your topics, and the AI SEO score checker to make each page easier to quote.

FAQ

AI crawler checker questions

Does AI Crawl change anything on my website?

No. The tool is read only. It requests your public robots.txt, llms.txt, llms-full.txt and the page you enter, exactly as any visitor could, then analyzes the responses. It never logs in, writes files or edits settings. Suggested robots.txt and llms.txt blocks are provided as text for you to review and add yourself.

Should I enter a domain or a full URL?

Either works. With a domain, we test the homepage and site level files. With a full URL, we test the homepage and that exact path, which matters because robots.txt rules often block specific folders. Use a full URL when you want to confirm that an important article or product page is reachable by AI crawlers.

Why do some results show missing or unavailable data?

Usually the server did not return a usable file. Common reasons are a robots.txt or llms.txt that does not exist, a firewall or CDN blocking automated requests, a redirect loop, a timeout, or an HTML error page served with a 200 status. The report labels each case so you can tell a missing file from a blocked request.

How do I rerun the check after making changes?

Update your robots.txt or llms.txt, clear any page or CDN cache that serves those files, then run the checker again with the same URL. AI Crawl runs are unlimited on free and paid accounts, and guests get 5 per day. Compare the new share link with the old one to confirm each bot changed as intended.

Is GPTBot blocked if it is not mentioned in my robots.txt?

Not by name. If GPTBot has no group of its own, it follows the User-agent: * group. So if that wildcard group disallows a path, GPTBot is blocked there too. If there is no wildcard rule and no GPTBot group, it is allowed. The checker resolves this for every bot automatically.

Does blocking AI bots hurt my Google rankings?

Blocking GPTBot, ClaudeBot, CCBot or Google-Extended does not affect Google Search rankings or indexing. Google-Extended is a separate control token and Google states it is not a ranking signal. Blocking Googlebot itself is different: that removes you from Search and from AI Overviews. Keep Googlebot and Bingbot allowed unless you truly want to leave search.

Is llms.txt an official standard?

No. llms.txt is a community proposal published in 2024, not a standard from the IETF or W3C, and major AI search engines have not confirmed that they use it for retrieval or ranking. It is still worth adding if you run documentation or a product site, since some AI tools and agents read it, but it does not replace robots.txt.

Do all AI bots respect robots.txt?

The major training and search crawlers from OpenAI, Anthropic, Google, Apple and Perplexity state that they follow robots.txt. User-triggered agents are a gray area, and some vendors say those requests may not follow it. Less reputable scrapers may ignore it entirely or fake a user agent. For hard enforcement, use server, firewall or CDN rules.

Still have questions? Contact our team

Make sure the right AI bots get in

Check your robots.txt and llms.txt for free, then see which sources AI engines cite for your topic.

Run AI crawl check Analyze AI citations