Should you allow AI crawlers in robots.txt?

The conclusion first

If the goal is being cited in AI answers, you should allow them. Blocking is a measure that removes the path to citation itself.

Audit a site whose owner says AI never mentions their company and it is genuinely common to find AI crawlers blocked in robots.txt. Usually the block was never intended — it came from a template copied from somewhere.

Check first

Open yourdomain.com/robots.txt in a browser and look for lines like this.

User-agent: GPTBot
Disallow: /

With a line like that, the crawler does not read the site. If Disallow: / sits under User-agent: *, everything is blocked.

The main AI crawlers

Crawler	Operator	Primary purpose
`GPTBot`	OpenAI	Training data collection
`OAI-SearchBot`	OpenAI	Search indexing
`ChatGPT-User`	OpenAI	Live visit on a user request
`ClaudeBot`	Anthropic	Training data collection
`Claude-User`	Anthropic	Live visit on a user request
`Claude-SearchBot`	Anthropic	Search indexing
`PerplexityBot`	Perplexity	Search indexing
`Google-Extended`	Google	Controls Gemini training and grounding (unrelated to search and AI Overviews)
`Applebot-Extended`	Apple	Training control
`CCBot`	Common Crawl	Public archive (a training source for several models)

Google-Extended has nothing to do with ordinary search indexing. Blocking it does not affect Google search visibility; it only restricts use on the Gemini side. Block Googlebot, on the other hand, and you disappear from search itself. Do not confuse the two.

Deciding by business type

When full allow is right

Services, B2B, professional practices, manufacturing, local businesses — most cases sit here.

For these, the website is not the product but a means of promotion. More people knowing about you is the benefit, so AI reading and citing your information is a benefit too. There is no reason to block.

When selective blocking is worth considering

Publishers whose content is the product — news organisations, paid archives, course platforms, webtoons and fiction.

Here “being read” can equal lost revenue. Even so, blocking everything deserves caution. Block the search crawlers too and the publication’s own name disappears from AI answers.

This middle course is available only for companies that separate crawlers by purpose.

# Block the training crawler
User-agent: GPTBot
Disallow: /

# Allow the search and citation crawler (the publication's name and link appear in answers)
User-agent: OAI-SearchBot
Allow: /

Partial blocking

Opening everything while blocking specific paths is also an option.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /members/
Disallow: /api/

Login-only areas, admin screens and API endpoints have nothing citable in them anyway, so blocking them is the better move.

An example allow configuration

For an ordinary site aiming at AEO, this is enough.

User-agent: *
Allow: /

# OpenAI
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /

# Anthropic
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

# Google — controls Gemini training and grounding. Unrelated to search indexing and AI Overviews
User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap-index.xml

With Allow: / under User-agent: *, individual crawlers are allowed without being named — but naming them acts as a safeguard if someone later adds a blanket block rule. It also puts the intent on the record.

The limits to know about

robots.txt is a voluntary protocol, not a technical block. The major AI companies state that their official crawlers comply, but there is no means of enforcement. Third-party scrapers that are not official crawlers may ignore it.

Content that genuinely has to be blocked belongs behind authentication or behind a server-level block, not behind robots.txt.

If it is open and you still do not appear

With no block in place and still no appearance in AI, the cause is elsewhere. Usually one of these.

The body text is drawn only by JavaScript, so there is nothing for a crawler to read
The site is not in the index that engine sees (Copilot uses the Bing search service; ChatGPT uses the index OAI-SearchBot collects)
No document answering that question exists at all

The full order of checks is set out in why ChatGPT does not mention your company.

Summary

For most businesses, full allow is right
Publishers whose content is the product should consider blocking training crawlers while opening search ones
Google-Extended and Googlebot are different — blocking the latter removes you from search
robots.txt is voluntary compliance, so real blocking needs another mechanism
If it is open and you still do not appear, the cause is not crawler access

For reference, you can read this site’s robots.txt as it stands.

Frequently asked questions

Q If we block it in robots.txt, does AI really not read the site?

A The major AI companies state that their official crawlers respect robots.txt. But it is voluntary compliance rather than an enforceable protocol, and third-party scrapers that are not official crawlers may not honour it. If you need certain blocking, refuse by User-Agent at the server.

Q Can we block training but allow citation in search?

A Some companies separate their crawlers by purpose. OpenAI, for instance, uses different names for the training and search crawlers. Where the names are separated, selective blocking is possible — but not every company has split them.

Q We were blocking and have just opened it. Will we be cited straight away?

A Not straight away. It takes time for a crawler to revisit, read the documents and have them reflected in an index, and the revisit interval differs by engine and by site. Submitting a sitemap and using IndexNow can bring the revisit forward.