AI CRAWLER ACCESS
Before an answer engine can use us as evidence, it has to be able to fetch our documents. We find where the crawler is blocked across three layers — robots.txt, infrastructure and rendering — open the blocked point, and confirm from the logs that it actually came back.
In one paragraph
AI crawler optimization is the technical work of getting a site into a state where answer engine collection bots can actually fetch it. Citation is the question after that — an engine that could not read a document cannot use it as evidence. A block in any one of three places produces the same result: robots.txt rules, a firewall, WAF or bot-mitigation service, or rendering that depends entirely on JavaScript.
That said, a crawler having visited and AI citing us are different facts. We separate crawler visits (A), citation in an AI answer (B) and visits arriving via AI (C) into different metrics from the outset. The scope this work is responsible for is A. B is confirmed in the audit and in monitoring; C is confirmed in traffic analysis.
Last verified
WHEN YOU NEED THIS
These are the situations we hear repeatedly in consultations. If any of them apply, start by measuring.
Blocking does not only happen in robots.txt. If hosting, a CDN, a WAF or a bot-mitigation service refuses by user-agent or IP range at a separate layer, the document is never delivered regardless of what robots.txt says.
The most common cause is a block put in during development or staging that stayed after launch. Defaults inserted by a builder or platform get mixed in too, so the first thing to establish is who is able to edit that file at all.
Usually the two sides are talking about different bots. The bot collecting training data, the bot collecting documents to use as evidence, and the bot that goes and fetches a page when a user asks all have different names. Split the discussion per bot and most of it resolves.
This is not a block but an empty hand. The bot receives a normal 200 and there are no sentences inside it, so no amount of adjusting access rules changes the outcome. What needs fixing is the rendering method, not the allow rules.
WHAT WE DO
Written as units of work rather than abstract proposals. The scope of an engagement is set from this list.
We start by pinning down which layer is blocking. Without separating robots.txt, infrastructure and rendering, you end up fixing the wrong thing.
We express allow and block intent accurately per group. Because groups do not inherit, it is common for the applied result to differ from what was written.
If robots.txt allows but security equipment refuses the bot, the outcome equals a block. It is why OpenAI's documentation lists allowing the published IP ranges as a condition of inclusion.
We check page by page whether body text exists in the HTML source, because what a crawler receives is the server's first response rather than a browser's rendered screen.
Full allowance suits some businesses and selective blocking suits others. For most businesses where the website is a promotional channel, full allowance is right; for a publisher whose content is itself the product, blocking training bots only is worth considering.
Once it is open, we confirm from the logs that it actually came back. Stopping at 'we changed the setting' leaves nobody knowing whether the rule applied as intended, or whether a bot subsequently arrived.
PROCESS
What you receive at each stage is stated alongside it. Durations are the working time Navirang controls; they are not a promise about when results appear.
We sweep robots.txt, infrastructure rules and rendering in one pass to pin down which layer blocks. It starts from public state alone, with no documents needed.
Crawler access checklist
We decide full allowance or selective blocking to suit the business type, deciding training and search bots separately and recording the reasoning.
Crawler allow policy document
robots.txt is rewritten to the group rules and exceptions are registered in bot-mitigation rules. Items we cannot touch with our own access are handed over with the verification method attached.
Applied robots.txt · record of rule changes
Pages whose body is absent from the source are moved to server rendering or static output. Where the structural change is large, we discuss rebuilding first.
Pages with body text in the source · change record
We check revisits and response codes per bot in the logs to verify access actually opened. What is confirmed here is a visit, not a citation.
Crawler visit summary
DELIVERABLES
We do not do work that ends in conversation. The documents below remain, and become the baseline for the next measurement.
Pass or block state per bot, with the layer that blocked it marked (robots.txt, infrastructure, rendering). The place to fix is only decided once the layer is identified.
The before and after source, the rules per group, and the reasoning as comments. The purpose is to stop the next person reverting the rules.
The exceptions to register in a CDN, WAF or bot-mitigation service. Items we cannot apply for lack of access are marked separately.
A list splitting pages whose HTML source contains body text from those that do not. Judged against the response with JavaScript unexecuted.
Request counts, paths and status codes per bot. The limitation — that these are user-agent-based detections — is stated inside the document. Where access logs cannot be read, we agree an alternative before starting.
robots.txt is a voluntary convention and user-agent strings can be forged. So visit records are read only as detected AI crawlers, and content that genuinely must be blocked goes behind authentication or is blocked at the server. We collect and observe crawler requests on our own site under the same assumption.
HOW IT CONNECTS
Our work moves as one piece. SEO builds the foundation for being found by search engines, AEO raises the odds of that information being cited in an answer, structured data helps machines understand the facts, and content supplies the evidence there is to cite.
| Area | Relationship to this work |
|---|---|
| Citable content design | Once it is being read, rewriting the document structure so it can be cited |
| Structured data | The layer that lets a machine read facts out of the fetched document |
| Indexing | Confirming it actually entered the search index once crawling is open |
| Citation monitoring | Re-measuring whether AI citation (B) changed, not crawler visits (A) |
| AEO website build | The option of building from scratch when the rendering structure itself is the cause |
FAQ
A It is the technical work of arranging access conditions so answer engine collection bots can actually fetch a site's documents. We check allow rules per group in robots.txt, the user-agent and IP rules in hosting, CDN, WAF and bot-mitigation services, and whether body text exists in the HTML source. A block in any one of the three makes the site a place with nothing to read.
A No. Allowing access is a precondition for citation, not citation itself. Crawler visits (A), citation in an AI answer (B) and an actual visit arriving through an AI answer (C) are separate metrics, and this work is responsible for A. OpenAI likewise lists allowing OAI-SearchBot crawling and the published IP ranges as conditions of inclusion, while stating in its official documentation that there is no way to guarantee top placement.
A Only with companies that separated their bot names by purpose. OpenAI keeps training (GPTBot) and search (OAI-SearchBot) under different names, and its documentation carries a configuration example blocking the former while allowing the latter. Not every company separated them, and blocking the search bot too removes the brand name itself from that engine's answers.
A Because a user-agent group replaces the wildcard (*) group rather than inheriting from it. Create one group named for a specific crawler and that crawler reads only its own group, so the Disallow written under the wildcard does not apply to it. To exclude an admin screen or an API path you have to repeat the same Disallow inside each group.
A No. Google-Extended is not a separate crawler but a token used only in robots.txt, and its scope is training and grounding for the Gemini app and Vertex AI. It has no effect on Google's search index or ranking. Blocking Googlebot, by contrast, removes you from search itself — the two must not be confused.
A Open the three layers in order: robots.txt, infrastructure, rendering. First, add /robots.txt to the domain in a browser and see which group carries a Disallow: /. Next, infrastructure — whether a request with a bot user-agent returns 403 or 429, and whether CDN, WAF or bot-mitigation rules or IP block lists catch it. Finally, whether the HTML source with JavaScript unexecuted contains the body text. Whichever layer stops is the place to fix.
A Not immediately, and we cannot promise a time. Revisit intervals differ per engine and per site and are not a value we control. Instead we keep observing requests per bot in the access logs to confirm from the record that a revisit actually happened, and citation is measured separately in monitoring that repeats the same questions.
Send us a URL and we check all three layers — robots.txt, infrastructure blocking, rendering — and tell you where it stops. You get the checklist as it stands.
We reply within one business day.