llms.txt and AI crawlers: Google's official position, and the limits of four days of server logs

llms.txt is a markdown file at the site root, proposed as a way to give AI agents a concise background description and links so they can understand a site. It is not an official web standard but a proposal open for community input, and Google states that Google Search does not use the file.

What llms.txt was proposed for

llms.txt is a proposal first published by Jeremy Howard of Answer.AI in September 2024. It puts a markdown file at /llms.txt at the site root (or a sub-path) holding a concise background description, guidance, and links to detailed markdown documents.

The concern behind it runs as follows. Web pages are built for people, so information sits surrounded by navigation, advertising and JavaScript. Turning that back into clean text is difficult and imprecise, and context windows are still small relative to a whole site. So the proposal starts from the position that an agent is better served by concise information gathered in one place.

The proposal was updated to v2 in August 2026.

Is llms.txt a standard?

No. It is a proposal.

The original text itself states that the specification is “open for community input”, and calls the document hosted in its GitHub repository an “informal overview”. It is not a specification adopted by an official web standards body.

That does not mean the proposal is being ignored, either. Chrome’s Lighthouse describes llms.txt as an “emerging convention” and has added it to its agentic browsing audits. Where the file is absent and returns a 404, it marks the audit Not Applicable — stating explicitly that this is because providing the file is currently optional.

“A proposal that some tools have started checking for” is the accurate position today.

Does Google Search use llms.txt?

It does not. Google places llms.txt on the list of things that can be ignored for Google Search in its generative AI optimization guide.

Google’s explanation has three parts. You do not need to create a separate machine-readable file or markdown to appear in Google Search (including its generative AI features); Google Search itself does not use it; and having one brings neither benefit nor penalty to Google Search visibility or ranking.

It adds that creating and maintaining one for other services or systems is fine. So the message is not “do not make one” but “there is no reason to make one for Google Search.”

What about OpenAI? Its official crawler documentation states only that it provides an llms.txt for indexing its own documentation, and says nothing about whether OpenAI’s crawlers read other sites’ llms.txt files. The control mechanisms that document covers are the OAI-SearchBot and GPTBot tags in robots.txt. Silence is not “unsupported”; it is “unconfirmed”.

How AI crawlers actually visited Navirang

We aggregated Navirang’s server logs across a completed four-day window (from 2026-08-14T15:00:00Z up to 2026-08-18T15:00:00Z, which is 15 August 00:00 to 19 August 00:00 Korean time).

Requests arriving with AI or search crawler User-Agents totalled 352, across 8 crawler families, reaching 53 paths.

By crawler name, ClaudeBot was highest at 89 visits (17 paths), followed by ChatGPT-User at 62 (31 paths) and Yeti at 49 (25 paths). Even among crawlers from the same company, visit counts and path counts differed by name.

Two things to be clear about.

• These figures are observations by User-Agent string. No reverse DNS or IP cross-check was performed, so the verification rate is 0%. Whether the party sending the request really was that crawler was not confirmed.
• A visit record is a different metric from an answer citation.

Why these logs cannot judge the effect of llms.txt

Navirang operates an llms.txt, and the logs above are real measurements. Yet the two cannot be joined into a claim that llms.txt works or does not work.

What this data confirms is only which User-Agents made how many requests to how many paths. What it does not confirm:

• Whether visits rose because of llms.txt
• Whether the crawlers actually read the llms.txt
• Whether citations rose because of llms.txt
• What changed before and after llms.txt was adopted

There is no comparison group (a pre-adoption window, or a site without one) and no control of other variables. A high visit count gives no basis within this data for attributing the cause to llms.txt.

We publish the logs not to prove an effect but as material for confirming that crawler access is not blocked, and to show what “AI crawler visits” actually look like.

So what should a site owner do now?

Start by not treating llms.txt as an all-purpose AEO or GEO measure. On the established facts the judgement is simple.

• If Google Search visibility is the only goal, there is no need to create one separately. Google states it does not use it and that it brings neither benefit nor penalty to visibility or ranking.
• If you want to make one, go ahead. Google says keeping one for other systems is fine, and Chrome’s Lighthouse has added it to its checks. But the scope of what Google stated ends at neither benefit nor penalty to Google Search visibility and ranking. Production and maintenance cost, and any other effect, are not addressed by that evidence.
• Do not misjudge the priority, though. Before llms.txt come the basic technical elements: the crawl settings in robots.txt, URL discovery through the sitemap, and page accessibility and indexability. And indexing is a separate stage in itself, not guaranteed — Google states that meeting all the requirements still does not guarantee crawling, indexing or serving.

Above all, placing one file and being cited in an answer sit at different layers. On the evidence established so far, one llms.txt cannot be regarded as producing answer citations.

References

• The /llms.txt file, v2 — Jeremy Howard (Answer.AI). Primary source for the proposal’s definition, purpose and “open for community input” status (confirmed 20 August 2026)
• Google Search Central — optimizing for AI-powered features — primary source for Google Search not using llms.txt and it bringing neither benefit nor penalty
• Google Search Central — AI features and your website — the statement that meeting the requirements does not guarantee crawling, indexing or serving
• llms.txt — Lighthouse agentic browsing audits — the “emerging convention” wording and the statement that providing the file is optional (Chrome for Developers)
• Overview of OpenAI Crawlers — the OAI-SearchBot and GPTBot distinction. Whether other sites’ llms.txt files are consumed is not mentioned in that document
• Navirang server logs — from 2026-08-14T15:00:00Z up to 2026-08-18T15:00:00Z (15–19 August Korean time, four completed days). Observed by User-Agent; no reverse DNS or IP cross-check was performed