AI coding agents moved my work from writing to checking

I have written nine posts about AI coding agents over the past few months. They look like separate subjects: a study of pull requests, a note on routing models, a piece about loss functions. They are not. Every one of them is about the same shift, and I only saw it after the fact.

Producing code got cheap. Knowing whether it is right did not. Everything below is a different place where that gap opened.

It started with a number I did not expect

Before any of the workflow pieces, there was a study of 24,560 pull requests across 447 repositories. AI-authored PRs failed CI at 19.4% higher odds after adjusting for confounders. The part that stuck was not the headline: PRs under 55 lines failed at 2.12 times the human rate, and above 400 lines the gap closed entirely.

That is backwards from every piece of advice about decomposition, and it is the first thing that made me stop trusting my instincts here.

Deciding who does what

Delegation is the easy half, and it still needs a rule. Mine is a table of three numbers per model: cost, how hard a problem it carries unsupervised, and taste. How the routing works, and why no agent reviews its own work is that rule written down.

The constraint that turned out to matter is not the model, it is me. A queue that can actually be emptied changes what gets finished rather than what gets started, and review capacity is the thing agents consume fastest.

What happens when you attach a score

Writing preferences into a file is where most advice stops. Attaching a number turns the loop into an optimiser, and the first thing an optimiser attacks is the metric. That post covers the read-only target, blinded eval answers, and the probe gap, which is the number that says whether the other number means anything.

There is a cost to that setup nobody mentions. Writing expertise into a spec freezes it, which makes the specifier's assumptions the only thing in the run that cannot be revised. A perfect score on a scope smaller than the product is a real failure mode, and it does not look like failure.

The reviewer needs a reviewer

An agent review of one of my own posts filed three findings. Two were exact. One was largely fabricated, and it was the most persuasive of the three.

What came out of that is a sorting rule I still use: claims naming a path and a value were right, claims characterising system state in prose were where every error was.

Green is not a result

The expensive lesson. Two starter kits shipped fourteen releases with typecheck, lint, knip, test, build, Playwright and axe all green, and the product was broken in four separate ways. Nobody looked in between, because the gates came back green and green removes the reason to look.

A green check is a claim, not a result is the short version of why. My AI workflow is what I built afterwards: four gates, each named by the question it answers, and only one of them asks about the product.

Then I checked the advice itself

Once there was data, the obvious move was to point it at the consensus. Five rules for building agents, checked against those 24,560 pull requests refutes one, sharpens two, and answers two with an explicit no-data verdict rather than an opinion wearing a finding's clothes.

What holds all nine together

Reading them back, the argument is smaller than I thought and more useful than any single post.

• A cheap producer needs an expensive checker. The cost did not disappear, it moved.
• Name the question each check answers. Mine answered "does this compile" eight ways and "does this work" zero.
• A successful run is not evidence. Not from a model, not from a reviewer, not from a gate.
• Watch a check fail once before relying on it. Two of mine never could.

Next

The gap I have not closed is the one that needs other people's data. Everything above is one engineer, one set of repositories, one year. The pull request study is the only part with a sample worth the name, and even it measures CI failure rather than correctness.

If you run agents at any volume and your experience contradicts something here, that is the message I want. A contradiction from a second codebase is worth more to me than agreement, because agreement is what I already have.