The gate that reported 43 passing tests and ran none of them

Datadog published a piece last week on adapting Golden Paths for autonomous agents. It is careful work, and its central instruction is one I agree with: the platform, not the agent, must enforce the contracts. An agent asked to police itself is a witness testifying about its own alibi.

The article lists eight requirements for an agent-facing path. One of them is mandatory telemetry per run: model calls, tool selection, retrieved context, latency, errors, retries, evaluations, cost. Another is an audit event per action, correlated to the task identity that started it.

I went looking through my own build for runs that failed to produce evidence. I did not find any. What I found instead were four cases where the evidence was produced exactly as specified, enforced by a platform rather than by an agent, complete in every field, and wrong.

The suite that passed without running

The end-to-end suite drives every route in a real browser and asserts zero error pages, zero console errors, zero 4xx and 5xx. It is the one gate in the stack that asks about the product rather than about the code.

It declared no dependency on the application it tests. So its cache key never included the frontend, and the task runner did what a task runner is built to do: it recognised an unchanged input and skipped the work.

FULL TURBO
43 passed

Nothing had executed. The line is not a lie, either. Forty-three tests did pass, on the last run, whenever that was. The evidence is accurate about a run that happened at some point and silent about which one.

It went green on a broken chart, and then green again on the commit that fixed the chart. Two opposite states of the product, one verdict.

Everything Datadog asks for was present. There was a task identity, a recorded output, a platform enforcing the contract, and a run whose telemetry a dashboard would have accepted without complaint.

Three more of the same shape

• A limiter reporting a machine limit as a product failure. Playwright's default worker count killed the dev proxy: eight workers on a sixteen-core machine took it down in three runs out of three, response times climbing from 60 ms to 890 ms before every later request was refused. The suite recorded errors, retries and latency, all of them real. It attributed them to the application. Capped at four workers, it survived three of three.
• A linter red on its own exhaust. The formatter was checking test-results/.last-run.json, a file Playwright writes after every run. Lint went red on output no human had produced, which is an audit event about the audit system.
• An optimisation inert in the only environment that counted. Preload hints carried crossorigin only when the API sat on another origin. Development serves the API on its own port, so the conditional always took that branch and the test passed. In production, same-origin, all four hints reported "preloaded but not used". Fixing it moved LCP from 600 ms to 464 ms.

In each case the run produced evidence. In each case a platform enforced something. In each case the thing enforced was not the thing anyone wanted.

The requirement that is missing

Datadog's list is about evidence being produced, retained and correlated. Every item assumes that a signal, once emitted, means what its name says.

The ninth requirement, and I would put it above several of the other eight:

Every control must be demonstrated failing before it is trusted.

Not tested. Demonstrated. Write the violating input, watch the gate go red, then fix it and watch it go green. A gate you have never seen fail is not a control, it is an ornament that happens to be the right colour.

This is not a theoretical worry. In the same codebase, two attempts at a single lint rule were silently ineffective, one because the schema was invalid and one because it sat outside the config call. Both linted green. Both caught nothing. The only thing that distinguished them from a working rule was that nobody had watched either of them fire.

The cache-key defect was fixed the same way. Not by reading the config until it looked right, but by making the same edit twice and watching the task hash change. Two hashes, two runs, one observation. That is what "the platform enforces it" is worth once you have checked.

Why agents make this worse

A human who runs a suite that skipped everything usually notices, because the wall-clock time is wrong. Forty-three browser tests do not finish instantly, and the body knows it even when the eye does not.

An agent has no such intuition. It reads the exit code. That is the whole of its relationship with your gate, and it is precisely the relationship Datadog's design intends: deterministic controls between the probabilistic step and the consequential action.

Which means the blast radius of a lying gate scales with how much you have automated around it. The green tick that a human treats as one input among several is the only input the agent has.

Next

The advice in that article is right, and I would rather work in a system that follows it than one that does not. The gap is not in what it asks for. It is in the assumption underneath: that the difficulty lies in getting evidence produced, rather than in getting evidence that can be wrong in a way you would notice.

All four of these came out of one three-day build, which I wrote about in the post auditing PostHog's agent-first rules against the same product. The next in this series takes Datadog's other publication, the State of AI Engineering report, and holds its failure numbers against measurements of my own.

If you take one thing from this: pick the gate you trust most, and try to make it fail on purpose today. If you cannot, you have not found a gate that works. You have found one you have never seen work.