The most quotable thing in my review was made up

I had an agent review the post about scoring agent loops. It came back with three findings, written well, with quotes from my own text and numbers from my own repository.

Two were exactly right. Acting on them turned the weakest section of that post into the strongest.

The third was the best story in the batch. I checked it, and almost none of it was in the files.

What it got right

The post claimed I had no score history worth charting. The review said I'd had one since 11 July and pasted the numbers.

I opened var/score-history.jsonl expecting to find them approximately right. They were exact — every value to four decimal places, in order, including a regression at 0.9102 that I'd forgotten happened. One session, 79 minutes, eleven cycles from 0.8279 to 1.0, then 0.9985 on the full dev set against 0.9940 on the probe.

Its second finding was that two numbers in the post didn't match my actual target. Also correct. My goal.md defines the bar as holdout at or above 0.95 and no category below 0.90, with a probe gap of 0.02 or less. The post said "gap over 0.15 twice in a row", which I had invented while writing and then forgotten was invented.

That one stung in a useful way. The review found a fabrication of mine before I did.

What it made up

The third finding described a failure mode I'd missed. The run had scored perfectly, and what it had actually built was — in the review's words — an in-memory engine with no database, no row-level security, no HTTP ingestion, no roles.

Then the closer: the engine would have misattributed every real registration, because all 600 eval scenarios fed prebuilt click events carrying a meta field that live clicks don't have, and the rebuild dropped them silently.

That is a genuinely great paragraph. Here's what's in the files.

• The database was real. LOG.md records a per-run schema, FORCE RLS on the Kit pattern, a 14-case RLS suite passing, and a scorer role with neither superuser nor bypass rights. goal.md carries a constraint requiring exactly this — that scenarios run through the production path and the scorer check the database directly rather than through a scoring side-path.
• "Holdout 0.9995" doesn't exist. There is no holdout value anywhere in the history file. Holdout is aggregate-only on a budget of six calls per 48 hours, which is why it isn't there.
• There is no meta field. Zero of the 400 dev scenarios contain one. The mechanism the story turns on isn't in the data.

One part of it was true, and I could only tell which part by opening the eval. There are 600 scenarios — 400 dev, 200 holdout — and every one is a prebuilt state file with no HTTP request in it. So the ingestion routes really were never under test. The scope failure was real. The story explaining it was not.

The findings sorted by one thing

Looking at all three afterwards, they split cleanly, and not along the line I'd have guessed.

Every claim that pointed at a path and a value was right. The score history, the goal.md criteria, the eval sizes — all verifiable in under a minute, all correct.

Every error was in a sentence that characterised the state of the system in prose. "No database." "No RLS." A field that behaves a certain way. Those read like observations and are actually summaries, and a summary is where a plausible detail can be added without anything contradicting it.

I got a second data point the same week without looking for it. A review bot on the pull request for this site flagged that a scheduled post would break the production build through generateStaticParams. There was a real defect there — metadata is generated before the page component's publish gate, so it read a translation namespace that gets stripped in production. But the mechanism it described had been closed days earlier. Right that something was broken. Wrong about what, and wrong in the direction of a more dramatic failure than the real one.

The part I don't like

The invented claim was the one I wanted to be true.

"A perfect score on an engine that would have processed zero percent of reality correctly" is a better sentence than anything I'd written that week. It would have made the post. I had to go and check it specifically because it was good, and I noticed I was hoping the check would come back clean.

That's the uncomfortable version of the rule I already had about a successful run not being evidence its result is correct. The rule is easy to hold when the output is boring. The output that's hard to reject is the one that flatters the thing you're writing.

Which is roughly the same shape as the argument in the last post: my own preference decided what got examined closely, and I was the only thing in the loop that could have caught it.

What I do now

Not "trust review agents less". Two of three findings were right, one of them found a fabrication I'd shipped, and the post is materially better than it was. That's a good trade at any hit rate.

What changed is the order I work in:

• Sort by citability first. Anything naming a file and a value gets checked immediately — it's a minute, and it's usually right, and it's the fastest way to know whether the review is worth the rest of my time.
• Treat prose characterisations as leads. "There's no X" is a hypothesis to go and test, never a finding to act on. Every error in both reviews was one of these.
• Check the best claim hardest. Inverted from what I was doing. The most compelling finding gets the most scrutiny, precisely because it has the most pull on me.

None of that is expensive. Verifying all three findings took under an hour, most of it spent reading my own LOG.md, and it caught two fabrications plus one of mine.

Next

I want to know whether the split holds — cited claims right, characterised claims wrong — or whether I just saw it twice. That needs more reviews and a habit of writing down which findings survived, which I don't have yet.

I'd particularly like to hear from someone whose reviewer got a file-and-value claim wrong — that would break the only heuristic I've got. So: if you run AI review over your own writing or code, do the errors cluster anywhere for you?