llms.txt Was Never Fetched by AI Crawlers in 29 Days: 19,347 Requests

Conclusion

I analyzed 29 days of access logs (August 3 to August 31, 2026) from a Japanese-language site I run. Of 19,347 requests from 8 AI search and AI training crawlers, 0 fetched /llms.txt.

The site actually serves /llms.txt. This is not a case of crawlers skipping a file that didn't exist. The file was there, and for 29 days not a single AI crawler touched it.

/llms.txt was fetched 7 times in the same period, but all 7 came from ordinary browser user agents (Chrome / Firefox). Those are visits from humans or browser extensions, not crawlers run by AI companies.

Meanwhile, /robots.txt, which sits in the same root directory, was fetched 132 times by OAI-SearchBot, 76 times by ClaudeBot, 59 times by Applebot, and 24 times by PerplexityBot. The behavior of fetching config files from the site root clearly exists, yet llms.txt alone gets passed over. That is the observation.

Figure

29 days, one site, measured

19,347

Total AI crawler requests

Sum across 8 crawlers

0

llms.txt fetches

By AI crawlers

7

llms.txt fetches

All from ordinary browser UAs

292

robots.txt fetches

By AI crawlers

What I measured

The subject is a single Japanese-language site that I run separately from this one. The application logs each request's user agent, path, and IP address, and I aggregated that table.

• Period: August 3, 2026 00:00 to August 31, 2026 23:59 (JST, 29 days)
• Scope: 8 crawlers identifiable by user agent that serve AI search or AI training purposes
• Counting: requests whose path exactly matches /llms.txt

The log records requests that reached the application without going through a CDN or WAF, so any access blocked before reaching it is not included. However, this site does not block AI crawlers, so the absence is not because they were refused.

Breakdown by crawler

Figure

AI crawler requests (29 days)

OAI-SearchBot

8391requests

robots.txt 132 times

PerplexityBot

2576requests

robots.txt 24 times

ClaudeBot

2302requests

robots.txt 76 times

meta-externalagent

2030requests

robots.txt 0 times

GPTBot

1614requests

robots.txt 0 times

ChatGPT-User

1599requests

robots.txt 1 time

Amazonbot

551requests

robots.txt 0 times

Applebot

284requests

robots.txt 59 times

What stands out is that GPTBot, meta-externalagent, and Amazonbot did not fetch even robots.txt during the period. robots.txt results are cached, so not fetching it for 29 days is not by itself evidence that a crawler ignores it. Still, one contrast holds: even the crawlers that did fetch robots.txt (OAI-SearchBot, ClaudeBot, PerplexityBot, Applebot) never once fetched llms.txt.

What the official documentation says

This observation is consistent with what AI companies and search engines say.

Google states explicitly that you don't need to create new machine-readable files, AI text files, or markup to appear in AI Overviews or AI Mode. It frames this as no additional requirements and no special optimization needed. It doesn't name llms.txt, but there is no official statement telling you to create new files for AI.

OpenAI's documentation on its crawlers covers only the user agent names and how to allow or disallow them in robots.txt. Nothing there says OpenAI reads, references, or recommends llms.txt.

Google's John Mueller has said, in effect, that nobody knows and that llms.txt is purely speculative at this point. His point was that the file has existed for a while, yet no AI system uses it.

Adoption, on the other hand, is growing. Rankability's measurements show 8.7% of the top 1,000 sites have an llms.txt (as of its August 23, 2026 update). But Rankability also states the limitation plainly: its own crawler only checks whether the file exists, and it does not measure whether actual AI assistants fetch it. Adoption has been measured, but nobody was measuring consumption. This analysis fills that gap for exactly one site.

What this observation shows

It shows exactly one thing: on this site, over these 29 days, AI crawlers did not fetch llms.txt. It is a record of an existing file being passed over. The sample is one site over 29 days.

In other words, the point is not that publishing it hurts you. It is that there was zero evidence, in the form of AI crawler fetches, that publishing it made any difference.

How to check

You can verify the same thing on your own site. If you have access logs, it takes 10 minutes.

Figure

How to check on your own site

1. 1

   1. Fix the period

   Pick a range whose edges aren't cut off, such as the first to the last day of a month. Logs keep growing, so without a fixed range the numbers shift

2. 2

   2. Count llms.txt

   Aggregate requests to the path /llms.txt by user agent

3. 3

   3. Count robots.txt too

   Compare using the same period and the same query. If this is 0, suspect gaps in your logging

4. 4

   4. Verify by IP, not UA

   When counts are high, match against the IP ranges published by each company to exclude spoofed traffic

Step 3 is the key. Even if llms.txt shows 0, a nonzero robots.txt count tells you your logging works. If both are 0, the measurement failed.

What to do / what to skip

Figure

How to approach llms.txt

What to do

• ✓State allow/disallow rules for AI crawlers in robots.txt (it is actually fetched)
• ✓If you publish llms.txt, treat it as an index for humans. It is cheap to set up and does no harm
• ✓Once a month, count in your own logs whether your llms.txt is being fetched
• ✓When evaluating any AI-focused tactic, check real crawler behavior in your logs first

What to skip

• ✕Make llms.txt the centerpiece of your AI search strategy
• ✕Claim that publishing llms.txt gets you cited by AI (no primary source supports this)
• ✕Spend time polishing llms.txt while putting off your robots.txt settings
• ✕Cite adoption statistics as evidence that the file is being consumed

FAQ

Should I avoid publishing llms.txt?

I haven't found any reason it would hurt you, and these logs show no negative effect from having it. The issue is return on effort. Rather than spending time on a file no AI crawler fetched in 29 days, stating allow/disallow rules for AI crawlers in robots.txt is the surer bet, because that file actually gets fetched.

Can AI cite my site even if AI crawlers don't fetch llms.txt?

Yes. In these same logs, OAI-SearchBot fetched actual pages 8,391 times over 29 days. That is the path AI actually uses. llms.txt not being read is a separate matter from your site not being read by AI.

Does a crawler that doesn't fetch robots.txt ignore it?

This log alone can't tell you. robots.txt results are commonly cached for some time, so going 29 days without fetching it is possible. To check whether a crawler ignores it, you would need to see whether it keeps accessing disallowed paths, which is outside the scope of this analysis.

How far can I trust results from a sample of one site?

Use them only as an existence proof: this is what happened on this site. They can't support ratios or general claims. That said, the procedure is only three steps, so producing the same numbers from your own logs should be faster. Your own site's numbers take priority over anyone else's statistics.