SEMPITE Research · Ecosystem Audit, Part 2 · August 2026
1 in 8 llms.txt Files Is Empty. The Biggest Has 11,137 Links.
Our first audit asked whether llms.txt files work. This one asks what’s inside the ones that do. The spec is clear about the intent: a short, curated map of your most useful content, so an AI doesn’t have to guess. We opened all 1,318 working files and counted. In practice, the format has no shape at all — it’s either empty, a tidy little map, or a full-site firehose, with almost nothing enforcing the difference.
The shape of an llms.txt file
Links listed per working file, across all 1,318 that returned real text content:
The median file lists 30 links — exactly the curated map the format is for. But the mean is 147, dragged up by a long tail: 76 files list more than 500 links, and 107 files weigh over 100 KB. When a file lists 11,137 URLs, it has stopped being a map and become a sitemap with extra steps — the one thing llms.txt was supposed to replace.
The empty ones are the quiet failure
171 files — 13% of everything that loads — contain no links at all. Some are a lone heading; some are a title and a sentence. They pass every uptime check (200 OK, real text, not HTML), so no monitoring tool flags them. But to an AI trying to navigate the site, an empty llms.txt is indistinguishable from no llms.txt — except that its existence implies someone decided this was done. It’s the “we shipped it” checkbox with nothing behind it.
Two opposite mistakes, one root cause: the empty file says nothing; the 11,137-link file says everything and therefore nothing. Both happen because llms.txt has no editorial step. It’s generated — by a plugin, a build script, or a five-minute afternoon — and then never curated. The format assumes a human decided what matters. Mostly, no human did.
And the files still fight the crawler
Even where a file is well-shaped, delivery gets in the way. Of the 19,039 links we validated inside these files, 1.8% returned 429 Too Many Requests — the host rate-limiting the very act of reading its recommended pages — and 65 more were bot-blocked outright. A file built to invite AI in, served by infrastructure configured to keep bots out. The left hand publishes the manifest; the right hand throttles anyone who follows it.
The takeaway across both parts of this audit is the same: llms.txt is treated as an artifact you produce once, not a surface you maintain. The median file proves the format can work. The empty eighth, the firehose tail, and the rate-limited links prove that “can” and “does” are far apart — and that a little curation and one CI check would put a site ahead of most of the web.
Want to know what an AI assistant actually finds when it reads your site?
Run a free AI visibility check