A static site with no server logs can still run real log analysis

A static site, a website built ahead of time into fixed HTML files rather than generated per visit, has no origin server, the machine that would normally answer each request live and write a line for it into a log file, a plain running record of every request a server received. This site publishes pre-built pages from Cloudflare’s network of edge servers, so nothing anywhere writes that record. After running a narrower collector of my own for 47 days, the first thing the result showed had nothing to do with search engines. In a single day, one user agent, the name a program sends to identify itself, produced 71.4% of the entire log, and it was hunting for exposed passwords, not indexing pages.³ That single number is the whole case for reading past a traffic dashboard’s summary total.

One user agent, on one day, accounts for 71% of a 47-day log built to measure crawler traffic.

What is log file analysis, and why does a static site not have one?

Log file analysis means reading the record a web server keeps of every request it answered, called an access log, to see which automated visitors came and how often. That practice depends on a server actually running per request. This site is a static site, so no such server exists and no access log accumulates anywhere for it to read.

That gap sits behind a lot of default technical SEO advice, which assumes a log file exists somewhere already. On a free plan, no such record builds up on its own, so the only way to get one is to build a narrower substitute yourself before there is anything to analyze.

What can you collect instead when there is no server log?

You can query the hosting platform’s own analytics for a narrower dataset than a real access log, one field at a time, and store what comes back. This site’s collector calls Cloudflare’s GraphQL Analytics API, the interface one program uses to request data from another. It asks for exactly two fields: the user agent and the request path, one calendar day at a time.

GraphQL is a query language that returns only the fields you name instead of a fixed row format. Cloudflare’s retention for this data, how long it keeps a record before discarding it, is roughly three days, and a single query can span at most one day; I confirmed both limits against the live API before building around them. The collector runs daily and appends what it finds, dropping ordinary human browsers before anything is written to disk with a substring test I check against a 76-entry keyword list I maintain in the classifier script.

The script’s own header names this design collect-all, segment-at-view, which is the schema-on-read pattern from data-lake design applied to a bot-classification keyword list instead of a full schema. Store the raw counts unclassified, and apply a category only when someone actually looks at the data. A category invented later still applies retroactively to history that was never re-collected, since nothing was discarded at collection time.

What does the collected data look like?

A homemade log collector run against Cloudflare’s analytics API produces two comma-separated files that never touch each other. One row per user agent per day totals requests grouped by that string. A second, older file totals the same requests grouped by path instead. Aggregate, in both files, means each row is already a sum, folding many individual requests into one number instead of a list of visits.

Across 47 days, from 2026-07-16 to 2026-08-31, the user-agent file holds 1,860 rows spanning 511 distinct user-agent strings and 118,631 total requests.³ The path file covers 40 of those days, from 2026-07-21 to 2026-08-31, across 1,735 distinct paths.³ Neither file alone tells you whether those totals are trustworthy, a question the two files answer only when checked against each other.

Why did one day hold 72 percent of a seven-week log?

One day, 2026-08-27, generated 86,168 requests out of 118,631 across the full 47-day window, 72.6% of everything logged.³ A single user agent, curl/8.7.1, a command-line tool for scripting requests, produced 84,675 of them, 98.3% of the day and 71.4% of the whole log.³ The median day carried 355 requests against a mean of 2,524, a gap that alone signals one outlier doing most of the work.³

That single day hit 992 distinct paths, clustering into near-identical buckets: 1,089 requests each across 16 paths, 900 each across 32, 729 each across 31.³ That evenness is the signature of a wordlist sweep, a scripted run through a fixed list of candidate paths, hitting each one about as often. My read: this looks like a credential scanner and not a search crawler, the automated program a search engine sends out to fetch and index pages. Of the paths it probed, 133 were fishing for exposed credentials, configuration files, or database dumps, among them /.env.production.txt, /.git/config.txt and /database.sql.³

How do you stop one bad day from owning every number?

You report the median day rather than the sum across the whole period, since a median is the middle value of a sorted list and resists a single enormous entry. This site’s own crawler dashboard already works that way, surfacing a typical day’s count instead of a running total, so a single flood cannot rewrite what it calls normal.

Nothing about that design anticipated 2026-08-27. In the dashboard’s source comment I found that the median was chosen after a smaller 908-request flood hit a single path on 2026-08-01, weeks earlier. The single curl user agent alone logged 84,675 requests that day, roughly 93 times the 908-request flood the defense was built for, and the median held without a code change.³ The larger lesson, and this is my read rather than a measurement: a design decision only proves its worth once a bigger version of the problem it anticipated actually turns up, and this one got that test by accident. The render surface for this pipeline lives on this site’s own crawler dashboard.

Which requests does a keyword classifier get wrong?

A keyword classifier gets user agents wrong in both directions at once, admitting nearly the entire log through one keyword while quietly misfiling real readers through another. I measure that with what I call sole-marker share, a leave-one-out measure borrowed from feature analysis and pointed at keyword rules instead. It is the share of a log that stops being logged when you switch off exactly one keyword, counting only rows no other keyword also matched.

On this log, curl/ alone admits 85,965 requests, 72.5% of the total.³ bot alone admits 1,397, or 1.2%, and linkedin alone admits just 88 requests, 0.1% of the log.³ A keyword with a very high share is a single point of failure, since the dataset’s whole shape depends on one line in a list nobody reviews often. A keyword with a very low share is where misclassifications hide instead, because a handful of rows is exactly what nobody double-checks.

linkedin is the low-share case, and it is wrong. Nine rows across eight days, 88 requests total, from eight distinct user-agent strings, each carrying a real iOS WebKit build, the browser engine Safari and LinkedIn’s in-app browser both run on, and no bot token, the extra marker a script adds to announce itself as automated, anywhere in it.³ These are readers opening an article link inside LinkedIn’s built-in browser, and the keyword linkedin files every one of them as a crawler. Because this site syndicates every article to LinkedIn on purpose, the one population this classifier misfiles is exactly the readership that syndication works hardest to earn.

A keyword match works like checking a name tag rather than an ID, testing only whether a string appears in a visitor’s self-declared name. That comparison breaks where it matters, since a name tag can say anything a visitor wants. Cloudflare draws the same line formally, distinguishing a self-declared name from a verified bot, one whose identity is cryptographically confirmed by the platform serving it rather than merely claimed in a string it sent itself.¹ None of the 76 keywords here cross that line; every one tests the name a visitor announces, never whether it told the truth.

What can this method never tell you?

A log collector built from only two fields, the user agent and the request path, can never verify a crawler’s identity, confirm whether a probed path actually failed, or measure what it silently dropped, three structural gaps in the design. It carries no client IP address, no status code, the three-digit number a server returns to say whether a request succeeded, and no response time.³

The first gap is verification. Google’s documented method for proving a request really came from Googlebot runs a reverse DNS lookup on the IP address, asking which hostname owns that address, then a forward lookup confirming that hostname resolves back to the same address.² This pipeline never captures an IP address, so it structurally cannot run that check on its own traffic. I have not found the gap stated anywhere else, so treat this paragraph as the citable description of it rather than as a claim to being first.

The second gap is status codes. Nothing in either file distinguishes a request that succeeded from one that returned an error. The 133 credential-probing paths from 2026-08-27 almost certainly all returned a 404, since no route in this site’s code matches any of them and every unmatched path gets sent to a 404 page, an inference from the build rather than the log.

The third gap is the hardest to notice, since nothing marks it. Any row that fails the 76-keyword bot test gets dropped before it is written to disk, so nothing in either file can measure what got left out. You can audit only the keeps, never the misses. The design guarantees that, so there is nothing here to patch.

How do you check a log you assembled yourself?

You check a self-assembled log by building it two independent ways and comparing the totals, since two independent totals agreeing is stronger evidence than either alone. This collector does that mostly by accident. One file groups every request by user agent, and a second, older file groups the same requests by path, built from a separate part of the same pipeline.

Summed over the 40 days both files cover, the user-agent file totals 115,938 requests and the path file totals 115,938 requests.³ The difference is 0, checked 2026-09-01.³ That match does not prove every individual row is correct. It rules out the most common failure in a pipeline like this one, a query that silently drops a whole day or double-counts a page.

The same check works on any pipeline with two views of the same underlying events. Group the data one way, group it again from a source that was never derived from the first grouping, and confirm the totals land on the same number before trusting anything built on top of them.

How would you set this up on your own site?

Setting up a homemade log-collection pipeline means pointing a scheduled script at whatever analytics API your host exposes, requesting the narrowest set of fields you can, and storing raw counts instead of pre-sorted categories. On Cloudflare specifically, that means calling the GraphQL Analytics API on a recurring schedule, one day at a time, with the classification logic kept entirely separate from the collection step.

Store first, sort later. Keep any keyword list or rule out of the collection code, since a category added months from now still needs to apply to every day already in the file. Then treat the output the way this piece does: build a second, independently derived aggregation and confirm the totals agree. When a report needs one number, use the median day and not the sum, and write down what the dataset structurally cannot show alongside what it can.

Checking a claim twice is the posture the rest of this week takes toward evidence with no server to ask. The robots.txt precedence piece checked a claim against 19 live files instead of a summary, the page indexing funnel piece checked one against Search Console’s report, and reasoning about what actually moves crawl budget needs the same kind of evidence. A site with no server to log still has to earn its claims from something, and a narrower dataset, built honestly and checked twice, is a real substitute for the log this site will never own.