Data Analysis Double Counting: Name the Method First

Five times in September 2026, I ran a full analysis, read the answer, and later found the number was wrong. Each miss had the same cause. Before the run, I had not written down what I was counting, what made two rows the same event, and what I was leaving out. The fix should take a few minutes: state the method, test it on a sample small enough to check by hand, and only then run the whole dataset. That was the missing step. Log files and Search Console exports are likely to invite the same errors, though I have only tested my own tooling. A wrong first number there can reach a slide before anyone rechecks it.

Five first-pass numbers from my own analysis work were wrong, and all five skipped the same step, which is stating the counting method before counting.

What did the five wrong first numbers look like?

Five analyses of my own work produced a wrong first number in September 2026, and each was caught only after the full run had finished. Four come from a usage report Claude Code generated about my sessions on 2026-09-18, and the fifth comes from this site’s deferred-work log.¹ The table shows what each pass counted, what the count should have been, and what the record says about the repair.

Eight terms first. A token is the unit an AI model bills by, roughly a short piece of a word. Base64 is a way of writing an image as plain text, so a pasted screenshot sits in a transcript as one very long string. A cache write is the charge for storing a conversation so later requests can reread it cheaply, and a cache miss is a request that had to pay that charge again.

A skill is a packaged instruction file for the AI tool, and a sub-agent is a helper session the main one launches. The grader is the script that scored whether a skill behaved correctly, and the scorer is the script that gave each skill file a quality score.

Number	What the first pass counted	What it should have counted	What the record says
Image cost	The base64 text that encodes each image, treated as tokens	The image tokens the model bills for	Images came to 331,200 tokens for the week, and the first pass overstated them about 50 times13
Sub-agent cache writes	Every transcript record	One record per API call	13,867 duplicate records in 7 days inflated cache writes by 65.5%. They also turned 15 real cache misses into an apparent 18012
Grader score	Text in the prompt’s own instruction	What the skill did in its output	The report proposes a test case designed to fool the grader, which the grader has to fail1
Skill-file scores	A score of zero for every file	Scores that differ by file	The report says it required redesign1
Script-to-article overlap	The video package’s production notes, counted as spoken words	Spoken lines only	The first pass read 61%, and the corrected median was 71% across 13 pairs4

The same usage report also recorded one further invalid experiment. It gives no figures and no cause, so I left it out of the table rather than invent detail.

What do the five errors have in common?

Of the five wrong numbers in this article (image cost, sub-agent cache writes, grader score, skill-file scores and script-to-article overlap), four counted something other than what I meant to count, and the fifth, the scorer, returned a zero for every file. The duplicate case used a wrong dedupe key, the field that says two rows are the same event, and the image case a wrong unit. The grader and overlap cases counted the wrong rows.

Four of the scripts returned a plausible total, and nothing in their output showed the problem. A payload was counted as tokens, a repeated record as a call, a prompt as a behavior, and a production note as a spoken word. A wrong unit raises no error. It returns a number.

The two errors with the largest figures are the easiest to see. My first pass counted the base64 string as tokens, when the real image cost was 331,200 tokens for the whole week.³ That is a mistake about the source and what to exclude.

The duplicate case is double counting, and it is a mistake about the dedupe key. One API call can write several records into a transcript, each carrying the same usage figures, so adding every record counted a call more than once.

The 65.5% inflation landed on cache writes.² Fifteen real cache misses read as 180. The transcript cost measurement this error sat inside is written up in Claude Code spend is the conversation being re-read, where the dedupe step is part of the method.

Does a fix to the counting method stay fixed?

A correction can carry its own error. The first duplicate rule I recorded kept the earliest record for each API call. On 2026-09-22 I found that rule was right for input and cache fields, which stay identical across a call’s duplicates, and wrong for output tokens, which grow as the response is written out. On one hand-checked call, the first record showed 3 output tokens and the last showed 220.²

So a dedupe key answers only half the question. It says which records are the same call, and a second rule must say whose values win. A hand check of one call exposed the gap, and no total would have shown it.

What rule should have caught all five?

State the counting method before counting, in three steps. First, name the data source, the dedupe key and what the count excludes. Second, run that method on a sample small enough to check by hand, and compare the result with an answer you work out yourself. Third, run the full dataset only after the sample matches.

1. Name the method. Write the source, the dedupe key and the exclusions in three lines. The image case needed the exclusion line, since the payload text was never a token, and the duplicate case needed the key line.
2. Check a small sample by hand. The usage report’s suggested prompt asks for a 20-record sample where the answer can be verified by eye. The transcript-overlap error shows why. The sample check caught the production notes leaking in, and the full-scale number had not.⁴ Confirmed for that one case.
3. Run at full scale. Only now does the paid run start, and the method from step one goes into the output beside the number, so a reader can see what was counted.

The usage report’s own verdict is that this rule would have caught the errors it lists cheaply.¹ I read that as suspected. Nobody re-ran the five analyses under the rule, so only the overlap case is confirmed. A hand-scored sample of a few skill files would very likely have disagreed with a script that gave every file a zero, but that is my inference about a run I did not repeat.

How does the rule apply to log files and Search Console exports?

Log files and Search Console exports are likely to carry the same hazards, though I have only tested my own tooling, so the same three steps apply. Before you count requests or clicks, decide what one row means, which rows are the same event, and which rows belong to a different audience. This section describes shapes only. I have no figures to attach to them, and any percentage here would be made up.

• Repeated rows. A content delivery network and an origin server can each log one visit, and a retried request can appear twice. Summing lines counts that visit twice, so the dedupe key is whatever identifies the request, such as a request ID when the log has one.
• A wrong unit. A log line exists for every image, script and stylesheet, so a request total is not a page view total. On this site’s own crawler log, a single credential scanner made up most of the requests, which is why the analysis of that log begins by defining what counts as a crawler.
• Rows that answer a different question. A branded query measures demand for the brand, not the value of a ranking. Pooling those rows with discovery queries overstated the top positions in my click-through-rate curve work, and the exclusion line is where that decision belongs.
• A result that is identical everywhere. If every page, day or file gets the same score, suspect the parser before you suspect the data. The scoring script that gave every skill file a zero had exactly this shape.
• A dedupe key you never asked about. Google’s Search Console Help says how the Performance report aggregates: “Data grouped by Queries, Countries, Devices, or Dates is aggregated by property. Data grouped by Pages or Search appearance is aggregated by page.”⁵ My reading of that line: the same impression can count once per property in one view and once per page in another. That decides whether page rows can be added up. Write which view you exported into step one.

What does the record of five wrong numbers not show?

The record of five wrong numbers in this article covers my own tooling and site work, not an SEO dataset. It shows how five errors happened without saying how often they happen elsewhere. Four of the five come from a summary that the usage report wrote about my sessions, so I took those errors from it and every figure from my own notes. My overlap measurement bounds shared vocabulary only, and shared vocabulary is not shared meaning.⁴

Before your next full run, write the three lines and take the sample. That costs minutes, and a wrong first number that ships costs a rerun and a correction.