Five rules for building agents, checked against 24,560 pull requests

Search for how to build AI agents and you get the same article five times. The advice has converged, which usually means it is right. What struck me is that almost none of it is measured: the rules are asserted, repeated, and never checked.

One thing to say before I start, because it changed how I wrote this. The primary sources are careful. Anthropic's guide separates workflows, where the code path is fixed, from agents, where the model directs itself, and tells you to find the simplest thing that works, possibly not an agent at all. OpenAI's spends its length on when not to build one, and on guardrails. Chip Huyen, by way of Simon Willison, splits tools into read-only sensors and writing actuators, which is a distinction with consequences.

The advice degrades as it spreads. What arrives in the fifth-hand version is the rule with the hedge stripped off, and the hedge was the useful part. So I am testing the confident version, and where the source was more careful than its echo, I say so.

I have two things I can check them against: a study of 24,560 pull requests across 447 repositories, and a failure of my own that ran for fourteen releases. So here are five of the most repeated rules, and what my data says about each. Each gets one of four verdicts, and I want to define them before I use them. Confirmed: the data supports it. Refuted: the data contradicts it. Sharpened: the rule is right and incomplete, and my data supplies the missing condition. No data: I have measured nothing, so I owe you no verdict.

Nothing came out confirmed, and that is worth saying rather than hiding. It is not because the advice is bad. It is because a rule I simply agree with produces no section anyone needs to read, so those are not the five I picked.

1. Break work into small tasks: refuted

Decomposition is real advice, prompt chaining is one of the named patterns, and it works by splitting a task into fixed steps. The confident version that reaches you is different: smaller units are safer, so cut everything small. That is the one my data contradicts.

In the study, AI-authored pull requests under 55 lines failed CI at 2.12 times the rate of human ones. Between 200 and 400 lines the gap narrows to about 1.1. Above 400 lines it closes entirely: roughly 0.95, meaning AI-authored changes did as well or slightly better than human ones.

The advice points the wrong way. Small is where agents are least reliable, not most.

I want to be careful about what that does and does not show. It measures CI failure, not correctness. A plausible reading that has nothing to do with size: small changes are what you write when you are touching something delicate, so the small bucket is enriched with hard problems. I cannot separate those with this data, and I would not want anyone quoting the 2.12 as though I had.

What survives either reading is narrower and still useful: a small diff is not evidence of a safe one. If you are decomposing work into small tasks because small tasks feel reviewable, check whether you are actually reviewing them.

2. Never ship without a human in the loop: sharpened

Nobody argues with this one, and it is the rule I broke without noticing.

I ran the full spec-driven chain on two starter kits: requirements, design, plan, implementation. Fourteen releases went out with typecheck, lint, knip, test, test:boundaries, build, Playwright and axe all green. A fresh clone could not start. Every page behind /app crashed. The sign-in form put the password in the URL in plain text. Every onboarding write died on a foreign key.

There was no human in the loop for any of it. I triggered the next iteration, the gates came back green, and I triggered the next one.

That is the part the rule leaves out. Automation does not remove the human by overruling them. It removes the reason to look. Fourteen times in a row I had a signal that said everything was fine, and a signal that says everything is fine is indistinguishable from one that is not being asked the right question.

The study has a number that fits the same shape. When a human-authored PR fails CI, it gets repaired 23.4% of the time. When an AI-authored one fails, 9.3%. The remaining 91% are abandoned. Whatever else that measures, it is not a population being watched closely.

3. Write the eval before the code: sharpened

Good advice, and I had followed it. The kits had an eval suite before they had features. Playwright and axe, running on every commit, green throughout.

The suite visited routes and asserted that headings were present. It never signed anyone in. It passed while login silently did nothing, because rendering a login page and using one are different events and only one of them was being tested.

So the rule holds and it is incomplete. An eval written before the code tests the product you imagined. The missing clause: it has to perform the user's job and assert an outcome, a row exists, the redirect landed, rather than assert that a page rendered.

There is a second-order version of this I now take seriously. Two attempts at one lint rule in that repository were silently ineffective; one was schema-invalid, the other sat outside the config call. Both linted green and caught nothing for weeks. A gate you have never seen fail is untested, so I write the violating code as a fixture and watch the rule fire before I trust it.

4. Tools matter as much as models: no data

I believe this, and I have measured nothing about it. Saying so costs me a section and is the only honest option: an opinion dressed as a finding is exactly what makes the rest of a post like this unreadable. I have watched a review agent do it, and the fabricated claim was the most persuasive one in the file.

5. Multi-agent systems are the frontier: no data

Same answer. I run multi-agent workflows and have opinions about where they help. I have not measured one against a single-agent baseline on the same task, so I have nothing to add that you could check.

Worth noting that this is the rule where the gap between source and echo is widest. The secondary articles call multi-agent the frontier. Anthropic spends its space arguing for the simplest system that works and treats added agency as a cost you pay for a reason. Those are not the same claim, and only one of them is being repeated.

What I would keep

• Size is not safety. A small diff earns no trust it has not otherwise earned.
• Name the question each gate answers. Mine answered "does this compile" eight different ways and "does this work" zero.
• An eval has to operate the thing. Rendering is not using.
• Watch your gate fail once before you rely on it.

None of that contradicts the consensus. It is what the consensus looks like after something has gone wrong under it, which is the version I would have wanted to read first.

Next

The obvious follow-up is the one I cannot do from this data: whether the small-PR effect is about size at all, or about what people use small PRs for. That needs the diffs classified by what they touch, not just counted, and that is a different study rather than another chart.

If you have shipped agent-written code at any volume: does the size effect match what you have seen, or is your experience the opposite? I am more interested in a contradiction than in agreement here: one repository is one repository, however many pull requests it contains.