Claude-SearchBot has to be open
Allowing ClaudeBot while omitting the search crawler drops you out of the candidate pool for web-search answers. The names are similar enough that this is easy to miss.
CLAUDE
Claude citation reference —
the search crawler is
a separate opt-in
Anthropic also splits its crawlers by purpose. Allowing only the training crawler ClaudeBot while blocking the search crawler Claude-SearchBot drops you out of web-search answers.
SHORT ANSWER
Claude is an answer engine operated by Anthropic. When it uses web search it shows the documents it referenced as sources, and its crawlers are split three ways by purpose — ClaudeBot for training data, Claude-SearchBot for the search index, and Claude-User for fetching a link a user asked about. To appear in search-backed answers, Claude-SearchBot has to be open.
Three distinct crawlersSearch runs on Claude-SearchBotTraining can be blocked selectively
SPEC
Claude specifications
Only items that can be verified are listed. Rows with a verification method on the right can be reproduced yourself.
- Operator
- Anthropic
- Crawlers
- ClaudeBot (training data) · Claude-SearchBot (search index) · Claude-User (fetch on user request) Check the User-Agent in your server access log.
- Gate for search visibility
- Claude-SearchBot. Allowing only the training crawler can drop you out of web-search answers Check whether a Claude-SearchBot group exists in robots.txt and, if so, whether it allows access.
- Blocking training while allowing search
- Possible. The names are separate, so selective configuration works
- How citations appear
- Answers that used web search show the referenced documents as sources
- Korean
- Supported
- Common configuration error
- Writing only ClaudeBot into robots.txt and omitting Claude-SearchBot. A per-User-Agent group replaces the * group, so each name has to be stated explicitly
CONDITIONS
What it takes to be cited in Claude
These are the items that differ most in this engine. Principles common to every engine are collected on the answer engines hub.
Each User-Agent group has to be stated explicitly
In robots.txt a specific User-Agent group replaces the `*` group. Rules written only under `*` do not apply to a crawler that has its own group.
Paragraphs have to be complete with their evidence
A paragraph becomes a citable unit when it still makes sense after being cut and carries its evidence with it. A sentence with a conclusion and no basis is hard to use.
MYTHS
Common misconceptions
Only the ones we meet repeatedly in audits.
-
“Blocking ClaudeBot removes you from Claude answers entirely”
Blocking the training crawler still leaves you able to appear in web-search answers if the search crawler is open. Blocking the search crawler closes that route instead. The judgement has to be made per purpose.
-
“Claude does not search, so it is not an AEO target”
Answers that used web search do display sources. That is why Navirang includes it in the seven-engine set and measures it under the same conditions as the rest.
HOW TO CHECK
How to check it yourself
A Navirang audit follows the same order. There is nothing stopping you from running it internally first.
- 01
Check how each of the three groups — ClaudeBot, Claude-SearchBot, Claude-User — is configured in robots.txt.
- 02
Confirm you have not written rules only under the `*` group (a per-User-Agent group replaces `*`).
- 03
Ask the target question under conditions that use web search and record whether your domain appears in the sources.
- 04
Check your server log for Claude-SearchBot visits.
FAQ
Questions about Claude
Each answer is written to be quoted as it stands.
What is the difference between ClaudeBot and Claude-SearchBot?
Their purposes differ. ClaudeBot is the crawler that collects training data; Claude-SearchBot is the crawler that builds the search index. The route into web-search-backed answers is the latter. Because the names are similar, writing one into robots.txt and omitting the other is a common mistake. A Navirang audit checks all three, including Claude-User for user requests.
We do not want to be used for training, but we do want to be cited.
That works. Anthropic separates its crawler names by purpose, so you can Disallow ClaudeBot while allowing Claude-SearchBot. It is a compromise publishers whose content is itself the product often choose. Note that not every company separates them this way, so it has to be checked engine by engine.
Isn't allowing everything under `*` in robots.txt enough?
Not always. In robots.txt, if a group exists for a specific User-Agent, that crawler ignores the `*` group and reads only its own. So if `*` allows everything but a ClaudeBot group is left somewhere with a Disallow, only the Claude side is blocked. This is why Navirang's own robots.txt repeats its rules under each group.
RELATED
Related reading
The canonical crawler list is kept as a table in a separate article — this page carries only the rows for this engine.
Last verified August 27, 2026
How does your brand look in Claude right now?
Send us a URL and our free audit asks the questions in all 7 engines and reports what actually came back, recorded engine by engine.
We reply within one business day.