Five sets of agent engineering advice, run against one product that shipped

Over five months, five careful pieces of writing appeared about how to engineer for AI agents. A product team published five golden rules. An observability company adapted its platform-engineering playbook. The same company published a report with real production numbers. Three engineers wrote a manifesto with twelve principles. A data platform documented four design patterns.

Between them they are prescriptive about almost everything and measured against almost nothing. Not through carelessness: none of them is claiming to have tested its advice on a specific build, because that is not what a principle is for.

I had a specific build. So I ran all five against it, one at a time, and this is what survived.

The five, and what each one cost me

The subject in every case is the same product: a price index for GPU rental, LLM tokens and wholesale electricity, planned in a 1,153-line brief, built in three days across 57 commits, and live.

• PostHog's five golden rules. Four held for free, because a public index with no login wall is agent-first whether or not anyone used the phrase. The fifth, treat agents like real users, had been followed in the documentation and ignored in the infrastructure: the docs invited agents to poll without a key, and 120 requests in five seconds all returned 200 against a database with 100 connections shared across three projects. The full audit is here.
• Datadog's Golden Paths. Searching the build for runs that produced no evidence found none. It found four where the evidence was complete, platform-enforced, and wrong, including a test suite that printed "43 passed" while executing nothing. That one is here.
• Datadog's State of AI Engineering. Their number is that 5% of model requests fail. On a free endpoint I measured, nothing failed at all, and three of four models answered with 512 tokens and zero words. Here.
• The Agentic Engineering Manifesto. Its first value prefers steering over upfront specification. I bought the other side and it shipped, which is a disagreement rather than a finding. Here.
• Databricks' design patterns. Their continuum orders systems by delegated decision-making, and my build sits nowhere on it: fifteen agents investigated, one wrote code. Here.

What all five share

Each is written from the layer where its authors work. PostHog writes from the product surface, Datadog from the platform and from aggregate telemetry, the manifesto from the process, Databricks from the runtime.

Within its own layer, each is right. What none of them can do is see the failure that lives in a different one, and every serious defect in my build lived in a different one.

The rate limiter was a product-surface promise with a platform-layer consequence. The lying test suite was a runtime cache behaving correctly under a contract written at the process layer. The empty answers were a model-layer behaviour invisible to platform telemetry. In each case, following the advice of the layer where the problem appeared would not have prevented it, because the problem was not there.

The one thing I would add to all of them

If I could put a single sentence into each of these documents, it would be the same sentence.

Every control must be demonstrated failing before it is trusted.

Not tested. Demonstrated. Write the violating input, watch it go red, then fix it and watch it go green.

The reason this is the one is that it is the only requirement that catches the class of defect none of the five can see. A gate you have never watched fail is indistinguishable from an ornament, and everything downstream of it, including every principle these documents contain, is resting on it.

In one repository this caught: a test suite that passed without running, two lint rules that were silently inert, and an optimisation that was measurably dead in production while green in development. None of those produced an error. All of them produced evidence.

How to read prescriptive advice

Three things I would do differently now when a well-argued list of principles appears.

• Ask what layer it is written from, and assume it is blind to the others. That is not a criticism of the author, it is a property of where anybody stands.
• Look for the tradeoff format. Only one of these five phrases its items as "X over Y", and it is the only one I could disagree with cleanly. A list of requirements has no vocabulary for two of its items conflicting, so when they do, you are on your own.
• Check the items against something that shipped before adopting the list. Four of PostHog's rules cost me nothing because the product was already shaped that way. One cost a rate limiter, an edge cache, two subtle bugs and an afternoon. Lists rarely tell you which of their items is the expensive one, and it is nearly always exactly one of them.

Next

I am building a small probe that checks whether a site can actually be used by an agent: whether it has an llms.txt, a keyless read path, a rate limit with Retry-After, machine-readable errors and stable identifiers. It exists because the first post in this series makes a claim about my own product that a reader currently has to take on trust, and I would rather they could check it.

The build these five were tested against is described in the post about writing one plan and running it, which has the numbers and the failures.

If you adopt one of these lists this week, pick the item you are most confident you already satisfy, and try to prove it. That is the one that will be wrong.