Home › Research › llms.txt and machine-readable access
SEMPITE Research · Ecosystem Audit, Part 2 · August 2026 · By Jacob Gerrish
1 in 8 llms.txt Files Is Empty. The Biggest Has 11,137 Links.
Our first audit asked whether llms.txt files work. This one asks what’s inside the ones that do. The spec is clear about the intent: a short, curated map of your most useful content, so an AI doesn’t have to guess. We opened all 1,318 working files and counted. In practice, the format has no shape at all. It’s either empty, a tidy little map, or a full-site firehose, with almost nothing enforcing the difference.
The shape of an llms.txt file
Links listed per working file, across all 1,318 that returned real text content:
The median file lists 30 links, exactly the curated map the format is for. But the mean is 147, dragged up by a long tail: 76 files list more than 500 links, and 107 files weigh over 100 KB. When a file lists 11,137 URLs, it has stopped being a map and become a sitemap with extra steps: the one thing llms.txt was supposed to replace.
The empty ones are the quiet failure
171 files, 13% of everything that loads, contain no links at all. Some are a lone heading; some are a title and a sentence. They pass every uptime check (200 OK, real text, not HTML), so no monitoring tool flags them. But to an AI trying to navigate the site, an empty llms.txt is indistinguishable from no llms.txt, except that its existence implies someone decided this was done. It’s the “we shipped it” checkbox with nothing behind it.
Two opposite mistakes, one root cause: the empty file says nothing; the 11,137-link file says everything and therefore nothing. Both happen because llms.txt has no editorial step. It’s generated (by a plugin, a build script, or a five-minute afternoon) and then never curated. The format assumes a human decided what matters. Mostly, no human did.
And the files still fight the crawler
Even where a file is well-shaped, delivery gets in the way. Of the 19,039 links we validated inside these files, 1.8% returned 429 Too Many Requests, the host rate-limiting the very act of reading its recommended pages, and 65 more were bot-blocked outright. A file built to invite AI in, served by infrastructure configured to keep bots out. The left hand publishes the manifest; the right hand throttles anyone who follows it.
The takeaway across both parts of this audit is the same: llms.txt is treated as an artifact you produce once, not a surface you maintain. The median file proves the format can work. The empty eighth, the firehose tail, and the rate-limited links prove that “can” and “does” are far apart, and that a little curation and one CI check would put a site ahead of most of the web.
Frequently asked questions
What did the second llms.txt audit find?
Of 1,318 working llms.txt files, 13% list zero links, the median file lists 30 links, 107 files exceed 100 KB, and the largest single file contains 11,137 links in 1.5 MB.
What does a typical llms.txt file look like?
The median file lists 30 links, which is the curated map the format is for. But 14.9% list 1 to 10 links, 35% list 11 to 50, 23.1% list 51 to 200, and 5.8% list more than 500.
Why are empty llms.txt files a problem?
171 files, 13% of those that load, contain no links at all. They pass every uptime check, so nothing flags them, but to an AI an empty file is indistinguishable from no file.
Do the files' own links work for crawlers?
Not always. Of the 19,039 links validated, 1.8% returned 429 Too Many Requests and 65 more were bot-blocked outright, so the host rate-limits the very pages its file recommends.
What is the right size for an llms.txt file?
A short curated map, on the order of the 30-link median. The study's point is that the format has no editorial step: files are generated and never curated, which produces both the empty ones and the 11,137-link firehose.
Want to know what an AI assistant actually finds when it reads your site?
Run a free AI visibility check