Does AI Read llms.txt? Measurements Across 4 Sites Show Mixed Results
Published
This article was first published in Korean by Intrix Lab on August 10, 2026.
Results varied by site. Comparing bot logs from 4 sites we operate directly, we found that GPTBot, OAI-SearchBot, and Amazonbot actually fetched llms.txt on one site, while another recorded 0 requests (as of 2026-08-10). This means we cannot look at just one site and conclude that "AI does not read llms.txt."
How much did results differ by site?
On the legal information site, llms-related files were fetched 12 times. GPTBot accounted for 1 request, OAI-SearchBot for 2, Amazonbot for 2, Claude-SearchBot for 1, and Google and Bing for the rest. Amazonbot even fetched a topic-specific file (llms-rehabilitation.txt).
Our company site recorded 0 requests. Across 1,622 visits by 10 types of AI crawlers, none requested the file. The restaurant information site recorded 17 requests, but a closer look showed that most came from measurement tools (WebPageTest and HeadlessChrome) and unclassified bots. Only 1 came from an actual AI bot.
We could not measure the remaining site because its measurement conditions were unsuitable. Although it appears to show 0, it was not measurable, so we did not use it as evidence.
Why did bots fetch the files on only some sites?
We checked the possible explanations one by one, but none fit.
We wondered whether listing the file location in robots.txt explained it. But the legal site where the files were fetched had 0 mentions of llms in robots.txt, while a site where they were not fetched had 30 mentions. We also considered links in HTML or sitemaps, but the pattern was again the opposite. The site where files were fetched had no links, while our site, with 0 requests, had links.
In other words, the discovery paths we know about do not explain the results. Domain history or differences in subject matter may be involved, but these data cannot distinguish between them.
Should you create the file?
Our conclusion is unchanged: it is not a priority. The reason, however, has changed. Rather than "nobody reads it," the finding is that "results vary by site, and even where it is read, the volume is extremely small."
Even on the legal site, which had the most fetches, robots.txt was fetched 1,290 times, compared with 12 for llms-related files. That is less than one-hundredth as many. The cost of creating the file is almost nothing, so there is no reason to delete it. But the same time would be better spent improving robots.txt and sitemaps.
How far can these results be trusted?
These are 4 sites we operate, each in a different industry. There is no guarantee that other sites will produce the same results.
Fetches on the legal site were also concentrated between mid- and late July 2026. Bots may have made a single pass during that period rather than fetching the files regularly. Without measuring again, we should not generalize this into "bots fetch these files."
This article corrects an earlier version that measured only our company site and reported "0 requests." Generalizing from a single sample was a mistake. We do not guarantee rankings or citations. This measurement answers only how often these files were fetched across 4 sites.
Frequently asked questions (FAQ)
Q. Is llms.txt a standard?
It is a proposed convention, not an official standard. We have not confirmed any documentation from search engines or AI providers stating that they will read this file.
Q. How can we check our own site?
Count requests for /llms.txt in your server or CDN logs, grouped by bot name. Requests from measurement tools and people can be mixed in, so you need to check the bot names as well.
Q. What should we do first?
Start by checking whether robots.txt allows AI crawlers and whether your sitemap is up to date. Across all four sites, those two were fetched far more often than llms-related files.