Most of your bot log was never a crawler

A server log full of automated traffic looks like a crawler problem. On this site it mostly is not. I checked every bot request against the pages my site actually publishes, and 80.1% of them asked for something that has never existed here, including a database backup file and eleven variations on a hidden settings file. Every one of those requests is a guess at a filename that was never there, and the reflex they provoke, which is to block the AI crawlers, would not have stopped a single one.

Of 119,035 bot requests to this site over seven weeks, 95,327 asked for one of 1,454 pages the build has never produced.

What did the kernel.org numbers actually say?

On August 29, 2026, Konstantin Ryabitsev, who runs the servers behind the Linux kernel’s public code repository, published his traffic figures. The site takes about 6 million requests a day for random commits, and by his calculation “legitimate requests are only about 2% of git.kernel.org traffic.”¹ Of 90 processor cores spread across five machines, 14 to 16 do nothing but build pages for scrapers. A commit, in this context, is a single saved change to the code, and each one costs real computing time to render.

Those numbers are extraordinary, and they are first-party. Because they came from the operator himself rather than from a vendor with something to sell, they traveled fast. They also arrived in a week when plenty of people read them as a reason to start blocking.

Before copying that response, check whether you have the same problem.

Why does that not mean “block the AI crawlers”?

The traffic hitting git.kernel.org no longer identifies itself as an AI crawler, and Ryabitsev is explicit about when that changed. The bots once “helpfully told you who they were via their user-agent,” and then they “wised up and started pretending that they were random vanilla browsers.”¹ A user agent is the short line of text a program sends with every request to say what it is. Any program can type anything it likes there.

The traffic reaches him through what he calls proxy SDK monetization. Companies pay app developers to bundle a small piece of code that quietly turns the user’s device into a relay for someone else’s web requests. As Ryabitsev puts it bluntly, “your TV is probably doing it,” meaning an ordinary television set running an app that quietly resells its owner’s connection. He also reports that his challenge system now stops fewer of them than it once did, with 33% solving the puzzle and getting through.¹

So the most-cited crawler-load story of the month describes traffic that no crawler-blocking rule can see. If the requests arrive from thousands of home internet connections wearing ordinary browser identities, then a robots.txt line naming GPTBot is aimed at the wrong thing entirely. Worse, it is aimed at the bots that were honest enough to say who they were.

Carry that distinction into your own logs before you write a single blocking rule.

What is in my own bot log?

Four out of five requests are for pages that do not exist. Across the seven weeks from July 21 to September 7, 2026, my site logged 119,035 automated requests. Re-running the measurement on September 8 against a freshly built copy, I found real pages and files accounted for 23,708 requests across 369 addresses. The other 95,327 asked for one of 1,454 addresses my build has never produced, which is 80.1% of everything logged.

Look at the ten most-requested missing pages and the pattern is immediate. In my own run, every one was a probe for something a careless developer might have left behind, including /database.sql, /.git/config, and nine spellings of the hidden .env settings file that applications use to store passwords. Each of the ten came in at about 1,089 requests. Same number, ten filenames. Those ten are only the loudest. Sweeping the whole missing-page bucket for credential-shaped names, not just the top ten, my own rough pattern match puts close to three in ten of those requests in that category, and it almost certainly undercounts.

Sorting the same period by who is asking gives a second cut. I measured on-demand assistant fetches at 3.9% of requests and self-identified training crawlers at 8.8%. The remaining 87.3% is something else entirely.

Why did two methods disagree by 59 points?

I ran two methods over one log and got 21% and 80.1%, and that gap is the whole lesson. My first pass used a denylist, meaning a list of patterns known to be bad, checking each request against strings like .env and wp-. It told me 21% of the traffic was junk. It was wrong. How it was wrong is the useful part.

Reading the output, I found the denylist had filed /api/v1/env, /.npmrc, and a Google service-account credentials file under legitimate crawling, because none of them matched a pattern anybody had thought to add. Any list of bad things is only as complete as the last bad thing you saw. Miss a shape and it counts as normal.

Turning the test around fixes it. Instead of describing what junk looks like, ask whether the requested page exists in the finished build. The build is the site, so the list is exact and needs no judgment about attacker behavior. Scored that way, my same log came back at 80.1%.

How stable is that number?

The missing-page share moves a great deal, and pretending otherwise would be the easy mistake to make with it. I have now measured this site’s own share three times in seven weeks and recorded 43.4%, then 81.7%, then 80.1%. My read is that the swings track individual scanning campaigns arriving and leaving, since one tool sweeping 1,089 requests across a hundred filename guesses moves the total by itself.

Treat that instability as a finding, not a caveat. A number this volatile cannot support a threshold, so any rule of the form “block when junk exceeds X%” is built on sand. What survives is the ranking, which has never once reversed. Whatever the exact share this week, the missing-page bucket has been the larger one every time it has been measured here.

How do you tell a crawl trap from scanner noise?

A crawl trap is a part of your own site that generates near-endless addresses for crawlers to follow, and it differs from a scanner guessing filenames in one way that decides the fix. A calendar widget offering a link to next month forever is a crawl trap, because your own pages keep producing the links. A request for /database.sql is something else, since your site never linked to it and never will.

Sort the missing-page requests by address, then read the list against your own internal links. Three groups fall out.

Addresses nothing on your site links to are external probing. Fix those at your edge or firewall, or ignore them, because they cost you almost nothing beyond log noise. Addresses your own pages still link to but your build no longer produces are broken internal links, and those are worth repairing on any site. Addresses your site generates without limit are the actual trap, and that is a template problem, not a blocking problem.

Only the second and third groups touch crawl budget, which is the amount of fetching a search engine is willing to spend on your site before it stops. On that point Google is unambiguous, warning that it will “still request, but then drop the page,” and calling that “wasting crawling time.”² My own log holds very little of either group, which is the honest result of running a small static site with no database behind it.

On a large e-commerce catalog the balance would be expected to invert, and I want to flag that as reasoning rather than measurement. Faceted navigation, meaning the filter controls that let a shopper narrow by size and color and price, can mint more addresses than any crawler will ever finish. A site with that machinery should find its trap group swamping its probing group. I have not measured that here, and this site has no facets to measure.

What should you do before you block anything?

Classify the traffic first, then let that classification pick the tool. Sorting a bot log against the pages your build actually publishes produces three outcomes, each needing a different response, and blocking named AI crawlers is the right answer to none of the three. The sort itself takes minutes once you have the log.

For one thing, external probing for files you never published is a security-scanning question, not a search question, and a rule naming GPTBot will not touch it. For another, your own dead or infinite addresses are a site problem, and no amount of blocking repairs them. And where a named crawler genuinely is over-fetching, you can see that directly in the same log, without inference.

I wrote in blocking the AI crawlers did not stop the citations about how much bot activity this site sees and what it does. That piece asks how much. This one asks a different question, which is how to tell the kinds apart before spending anything on a response. The kernel.org case is the argument for doing that work: the operator with the most severe crawler bill on record is being hit by traffic that looks like a browser and arrives from someone’s television.