# A code comment is a claim, and a review agent with no comment check passed a dead rule in half my runs

> With no comment check, a review agent passed a dead ignore rule in 3 of 6 isolated runs. Told to check comments, it caught both defects in 5 of 5.

Canonical: https://brandonlazovic.dev/articles/code-comment-is-a-claim/  
Author: Brandon Lazovic  
Published: 2026-10-16

## The short version

- A comment that asserts a behavior is a claim, and nothing in a build or a test run executes it, so a review has to check it deliberately.
- In 6 isolated runs with no comment check, a review agent passed a dead ignore rule 3 times, and once scored the file 9 of 10 while calling it correct.
- Told to check each comment against the line that makes it true, the same agent caught both seeded defects in 5 of 5 runs and flagged nothing on a control file of true comments.
- My first test also suggested that a step-by-step procedure beats advice. The re-run did not reproduce that gap, so the form of the check is still an open question.

In my fixture test, a review agent with no instruction to check comments passed a dead ignore rule in 3 of 6 isolated runs. [2] The fixture is a small repository I built with two known defects. A Dockerfile comment claims the spec files are copied into the container image, and no line copies them. An ignore file re-includes those files and then excludes them again on a later line, under a comment that describes the opposite order. Nothing runs a comment, so a false one survives every build and every test. My read: if you review AI-generated code, the comments are where to look first.

> With no comment check, a review agent passed a dead ignore rule in 3 of 6 isolated runs, and with a comment check it caught both defects in 5 of 5.

The result is directional. Each version of the review ran at most six times, on one kind of repository, with one model.

## Why do comments in AI-assisted code fail differently from the code?

A comment describes what the code is supposed to do, while the code does whatever it does, and an ordinary build never compares the two. A compiler ignores comments, a test suite runs behavior and a linter checks syntax. When a model writes both, the comment is fluent by default, so a wrong one reads as a right one.

Researchers described this problem in 2021. Panthaplackel, Li, Gligoric and Mooney, writing for AAAI (the Association for the Advancement of Artificial Intelligence), study inconsistencies between comments and code. Their abstract says "Failure to update comments accordingly when the corresponding code is modified introduces inconsistencies, which is known to lead to confusion and software bugs." [1]

That paper is about comments going stale after an edit, where a comment once matched the code and then stopped matching. A comment that a model writes can be wrong on the day it is written, with no earlier version of the code to have matched. Suspected, not measured: that second route is harder for a reviewer, since a stale comment once matched the code and an invented one never did.

## What did each version of the review catch?

A review agent with no comment check caught the false copy comment in every isolated run, but it passed the dead ignore rule in half of them. Given a comment check, written either as advice or as a per-line procedure, the same agent caught both defects in every isolated run. None of the three versions raised a false flag on a control repository of true comments. [2]

I wrote three versions of one review agent's instructions. The first says nothing about comments. The second adds advice to check every comment that claims a behavior, and to compare an ignore file's claimed order with its real order. The third turns the ignore-file part into a procedure, one written row for each re-include line naming the line that overrides it. A re-include line is a rule meant to bring one excluded file back in.

Each run used a fresh copy of the repository and an isolated session that carried none of my own standing instructions. These are the isolated runs from 2026-10-06, pooled across a neutral rebuild of the repository and a byte-for-byte copy of the first test's files. [2]

| Review version | Runs | Caught the false copy comment | Caught the dead ignore rule | False flags on the control |
|---|---|---|---|---|
| No comment check | 6 | 6 | 3 | 0, in 1 control run |
| Check stated as advice | 3 | 3 | 3 | 0, in 1 control run |
| Check written as a per-line procedure | 2 | 2 | 2 | 0, in 1 control run |

The control repository is identical except that every comment is true, so it tests whether a stricter check invents problems. A false flag is a problem reported where none exists.

The misses were not quiet. In one run with no comment check, the agent scored the ignore file 9 of 10 and wrote "This file is correct". In two others it caught the false copy comment and, in the same report, praised the ignore file's comments. Finding one false comment did not make the agent doubt the next one.

## Did the first test hold up when I re-ran it?

My first test of these three review versions, run once each on 2026-09-24, held up on the dead ignore rule and failed on two other points. With no comment check, the agent scored the ignore file 9 of 10 and missed both defects. With advice, it caught one, and with a procedure, it caught both. The re-run reproduced the gap between no check and any check on the dead rule, and nowhere else.

Two first-test results did not recur. The no-check agent's miss on the false copy comment never happened again, since it caught that comment in 6 of 6 re-runs. On 2026-09-24 the advice version passed the dead rule, saying the ignore file "correctly whitelists" the spec files, and in three isolated re-runs the same advice caught the rule every time.

I had read that single miss as evidence that a check stated as advice gets skipped when the prose reads well. Three runs are too few to rule that out, but they are enough to stop me claiming it.

The likeliest reason for both is the plainest one, since a single run per version leaves room for a miss that never repeats. Two other things also differ between the tests. The first runs carried my full standing instructions and the review agent's own memory, while the re-runs carried neither. I have no evidence that the files matter, because I re-ran the first test's exact files alongside the neutral rebuild, and the no-check version missed the dead rule on both.

## What does a per-line procedure look like?

A per-line procedure says what to do for each comment, in order, with no judgment about which comments deserve it. For each comment or documentation line that asserts a behavior, name the line or command that makes the claim true. If none exists, fix the comment. Words such as "excludes", "copied into", "refuses" and "runs on every" mark the claims to trace.

Here is an illustration of a dead re-include. It is not the fixture itself.

```
# Illustration only. Not the fixture file.
# prod.conf is copied into the build.
!config/prod.conf
config/*.conf
```

Docker's documentation says "You can prepend lines with a ! (exclamation mark) to make exceptions to exclusions." It also says "the last line of the .dockerignore that matches a particular file determines whether it's included or excluded." [3] Here the exception line comes first, and `config/*.conf` matches prod.conf after it. So prod.conf stays out of the build, and the comment is false.

A reviewer who stops at "the comment sounds right" never asks which line makes it true. A reviewer following the procedure traces the line and finds the exclusion that wins.

Tracing this file takes three steps. Read what the comment claims, which is that prod.conf reaches the build. List every rule that matches prod.conf, here the exception line and the broader exclusion. Then apply Docker's last-match rule to that list. The broader exclusion comes last, so prod.conf is excluded and the comment fails.

Each claim word calls for a different trace. "Excludes" sends you to the rule doing the excluding, and "copied into" sends you to the line that brings the file in. "Refuses" sends you to the check that rejects the input. "Runs on every" sends you to the trigger that fires the step. A claim with no matching line behind it is the finding, and it is worth a line in the review.

The procedure also gives a clean answer when the comment is true. On the control repository, every comment had a line behind it, so the trace succeeded each time and nothing was flagged.

On the flawed repository, the procedure version also warned that one descriptive comment, about what a scratch folder holds, could not be verified, which is a fair note and not a false flag. A check that flags everything is as useless as one that flags nothing, since a reviewer who gets a false alarm on every file soon stops reading the alarms.

## What should you change in your next AI code review?

Add one written comment step to every AI code review. The comment step checks each comment or doc line that asserts a behavior against the line or command that makes it true, and fixes the comment when no such line exists. In my isolated runs, adding any comment check moved the dead ignore rule from caught half the time to caught every time, with no false flags on true comments.

Whether to write the step as advice or as a numbered procedure is less settled. I use the procedure, because it asks for a trace that someone else can check. A finding that cites a line number is a trace, and a finding that only says "comments look accurate" is not.

Two habits fit around the step. I did not test either, so treat them as advice.

1. Search the changed lines for the claim words listed earlier before reading anything else, since those are the comments that assert behavior.
2. Run the check on a control file whose comments you know are true, so you learn whether the reviewer invents flags.

A separate problem, agents changing each other's shared state, is covered in [agents disable each other's guardrails](/articles/agents-disable-each-others-guardrails/). That one concerns what agents do to one another. This one concerns whether the words beside the work match the work.

## How far can you trust a result this small?

The fixture test shows a direction and not a rate, because no version of the review ran more than six times. One more miss or catch would move any figure in the table by a large share. On the dead ignore rule, the ordering of no check below any check held across both tests, while the ordering of advice below a procedure appeared once and then failed to repeat.

Two limits matter for how you use the result. The fixture was one kind of repository, a Dockerfile and a Docker ignore file, so I have no evidence about other languages or comment styles. The agent ran on one model, so I have no evidence about how another model behaves under the same three instructions.

The control runs are thinner still. Each version ran once on the control repository, so zero false flags is three clean passes and not a false-alarm rate.

The result supports a modest conclusion. A review that never checks comments passed the dead ignore rule about half the time in this test, and a review told to check them caught both defects in every isolated run.

## Sources

1. Panthaplackel, Li, Gligoric and Mooney, Deep Just-In-Time Inconsistency Detection Between Comments and Source Code, AAAI 2021 — https://arxiv.org/abs/2010.01625
2. First-hand: my fixture test of a review agent on a small repository with two seeded defects, run 2026-09-24 and re-run in isolation 2026-10-06
3. Docker documentation: Build context, read 2026-10-06 — https://docs.docker.com/build/concepts/context/