Does llms.txt Do Anything? A Study and One Broken File

Common Crawl, a nonprofit that crawls and archives a broad slice of the public web, read half a million llms.txt files last month. An llms.txt file is a plain-text page some sites publish at their root address to describe themselves to AI systems in ordinary language. Most of the ones Common Crawl found were template output nobody had ever opened, which makes the whole idea look like theater. I publish a hand-written one, so I checked mine against every failure the study measured. It passed all of them and was wrong anyway. For 13 days it told every automated visitor that my site had three topic sections when it had four, and the missing one was live and had published six articles.

Of 584,107 llms.txt files in Common Crawl’s July 2026 archive, 68% came from a website template or plugin, and more than a fifth contained no links at all.

What did Common Crawl actually measure?

Common Crawl gathered every llms.txt file it could find in its July 2026 archive and read what was inside. Two thirds of them, 68%, came straight from a website builder or a plugin, with the site builder Wix accounting for 41% on its own.¹ Those files were generated once, published, and never touched again.

Two further findings matter as much as the headline. Just over a fifth, 22.56%, contained no links at all.¹ For a file whose entire job is pointing an AI system at your best pages, that is close to publishing a blank sheet. And roughly one address in ten serving content at the llms.txt path was serving a robots.txt file instead.

Robots.txt is the older file, the one a crawler actually checks before fetching a page, and these were placed there, I would guess, by someone who thought the two were interchangeable.¹

Read together, those numbers describe a file most publishers installed and then forgot about, which is a different problem from a standard nobody adopted.

Why can’t an llms.txt file block a crawler?

Nothing is built to obey an llms.txt file, and the study caught people assuming otherwise in the clearest possible way. Among the 32 files that appeared to refuse Common Crawl’s own crawler, not one of those sites also refused it in robots.txt, the file a crawler actually obeys.¹ That gap contains the whole misunderstanding.

Google describes robots.txt as the place you say which addresses a crawler may request.³ Because a crawler already respecting that file was never consulting llms.txt for permission, there was never any permission sitting there to grant. What the proposal offers instead is a description, written for a system that has already decided to read.²

So a refusal typed into llms.txt stops nobody, and those 32 sites had left the door they meant to close standing wide open.

What went wrong with my own file?

My own llms.txt failed none of Common Crawl’s tests. Written by hand rather than produced by a plugin, it carries no robots.txt syntax anywhere in it and makes no attempt to block anyone, and on the morning I fixed it, its 173 lines pointed at 122 real pages. Measured against everything Common Crawl looked for, my file is the one those half a million were not.

It was wrong anyway, in a way the study has no category for. One line described the topic sections of my site and named three. There are four. Covering technical search fundamentals, the missing section had been live since the start of September and publishing articles for a week, and six of them sat inside a section my own machine-readable summary told crawlers did not exist.

Stranger still, the contradiction sat inside a single file. When the site builds, a program writes that summary, and further down the same program walks the real list of sections and prints every one. Three sections got named in one paragraph, while four got listed a few lines below.

Why did no check catch the stale llms.txt sentence?

Nobody edited the sentence naming my site’s sections, and that detail is the whole story. It was written on August 20, and on August 20 it was true, because the site genuinely had three sections then. When a fourth arrived on September 1, it was added in a different file by someone creating a section rather than describing one. The sentence stayed exactly as written. Underneath it, the site moved.

No check caught it either. This site runs an automated gate before every deploy that refuses to ship over a broken link, a missing image description, a malformed citation or a duplicate page title.

I have added a new check to that gate every time something slipped past it. Not one of them reads llms.txt. After the Common Crawl study I went looking, and no script in the project opens the file at all, so the gate held no opinion about it and never had.

Think of it as a floor plan pinned up beside a building that is still being extended. Every check I run inspects the building, but nothing compares the drawing to it. The comparison fails in one place, though, and the difference matters: a floor plan at least has a fixed notation you can check for errors, while this file has none.

What does documentation drift mean for anyone else’s file?

Sites running an unedited template file and my own hand-written one failed for opposite reasons and arrived at the same place. Theirs were never edited, while mine was written carefully and then went stale. Either way the published description no longer matched the site.

Common Crawl reaches a version of this from the outside. Looking at one site whose llms.txt claimed a block its real robots.txt never made, the study observes that the file “isn’t a policy” but “a description of a policy,” and that the description had drifted out of step with the thing it described.¹

Because I found the same shape in my own source rather than by inspecting a stranger’s site, I can say why it happened instead of only that it did.

Software teams already have a name for the parent problem. Documentation drift is what happens when code changes and the documentation describing it does not, and the vendors who sell documentation tools have written about it for years. What is different here is who does the reading. A developer can see that a stale document contradicts the code in front of them, whereas a machine reading your llms.txt has only the description.

Underneath, the mechanism is the same. An llms.txt file is prose a person writes about a site a program generates. Whenever anyone ships anything the program changes, while the prose changes only when somebody remembers it exists. Those two rates will never match, so drifting apart is the resting state and staying accurate is the part that costs effort.

No validator exists to lean on either, and the reason is subtler than the format being loose. In fact the llms.txt format is precise. Its own specification calls it “a precise format allowing fixed processing methods,” with a required title, an optional summary, and link lists under fixed headings, all of it readable by ordinary parsing code.² So a checker can confirm your file has the right shape.

What no checker can confirm is whether the prose inside that shape is still true, because the format fixes the container and leaves the contents entirely to you, and deleting the file is not the fix.

Generate the parts that describe structure. Any sentence counting your sections, listing your hubs or naming your address patterns should be built from the same source the site is built from. In my case the repair was to derive that one sentence from the real list rather than type it out, which makes the mistake impossible to repeat instead of merely correcting it this once.

Put the file inside whatever already blocks your deploys, even with a weak check. A test confirming that every address named in the file returns a real page would have caught nothing here. A test comparing the sections named in the prose against the sections the site actually builds would have caught it on September 1.

Treat the file as documentation of a moving system, which means giving it a review date rather than an author. One last number makes that case better than any advice can. Those 173 lines and 122 pages were accurate the morning I fixed the file, and they are already wrong, because publishing this article added a line and a link to the very file it describes.

Go and count them yourself and you will get a larger number than I did. Mine was accurate the day I wrote it, and a month from now that accuracy is exactly what will hide the next error.