What the benchmarks cannot tell you about OpenCode, Claude Code and Codex

Two leaderboards, same benchmark, same week, different winner.

DeepSWE is a long-horizon agentic coding benchmark from Datacurve. On 31 August, llm-stats has GPT-5.6 Sol first at 0.727, with Claude Opus 5 nowhere in the top five. A snapshot dated 26 August has Opus 5 leading at 73.6% pass@1. Nobody is lying. The scores are close enough that the ordering depends on when you looked and who ran it.

You would know that immediately if the aggregators kept the one thing DeepSWE ships that almost nothing else does.

The benchmark publishes error bars. The copies drop them.

Here is DeepSWE's own launch table, from 26 May:

Every row carries an interval and a reasoning-effort setting. Both matter: a model at maximum effort and the same model at medium are different products with the same name.

The llm-stats page for the same benchmark carries neither. Twelve models, one decimal each, no uncertainty, no effort setting. The careful part is exactly the part that does not survive being copied, and what is left is a ranking that reads as though the gaps mean something.

Two models within three points of each other, with intervals of plus or minus three, are not ranked. They are tied, and the benchmark said so before anyone quoted it.

Why DeepSWE exists, in its own words

The reason to care about this one rather than the older names is in Datacurve's write-up, and it is an indictment of what most comparisons still cite.

On verifier reliability, they audited SWE-bench Pro and found 8% false positives and 24% false negatives. Close to a third of its pass and fail decisions are unreliable. DeepSWE's own verifiers, which test behavioural outcomes rather than implementation details, measure 0.3% and 1.1%.

On contamination, tasks built from merged commits leak: they report solution leakage and false positives affecting roughly 8% of audited rollouts, because an agent can go and read the fix in the repository's own history. DeepSWE's tasks are written from scratch and never merged upstream, so they are not in the public GitHub record at all.

And on scale, the gap is embarrassing: SWE-bench Pro tasks average around 120 lines. DeepSWE's average 668 lines across 7 files, from prompts averaging 2,158 characters, which is roughly what a real ticket looks like.

So when a comparison quotes a SWE-bench number to three significant figures, it is quoting a measurement whose own auditors put a quarter of its negatives in doubt.

What I would compare on instead

Single-shot code generation stopped being the interesting axis a while ago. The dimensions that separate these models in daily use are the ones where a task runs long enough to go wrong:

• Long-horizon agentic coding. Can it hold a goal across 668 lines and seven files without losing the thread. This is what DeepSWE measures and why it is worth reading.
• Tool use. Not whether it can call a tool, but whether it picks the right one and recovers when the call fails.
• Computer use. The failure modes here are qualitatively different from text, and a model can be excellent at code and useless at a browser.
• Visual reasoning. Reading a screenshot of a broken layout is a different skill from reading the CSS that broke it, and only one of them is on most leaderboards.

What none of them measure is the thing that decides which harness you keep: how many turns it takes, and how much of what comes back you throw away.

Meanwhile the question is dissolving

The harness-versus-harness framing assumes each tool comes with its model. That was true for about a year and is now mostly a default rather than a constraint.

Claude Code reads ANTHROPIC_BASE_URL. Point it at OpenRouter and you are running GPT or Gemini inside Anthropic's harness. There is a small ecosystem of local proxies that translate the Anthropic Messages API to the OpenAI chat format and back, and a hosted gateway or two that will do it without a local process. Anthropic's own changelog acknowledges the practice, in the entry that changes the commit trailer when "the active model isn't a recognized Claude model (e.g. third-party models behind a custom ANTHROPIC_BASE_URL)".

OpenCode was model-agnostic from the start, with 75-plus providers through Models.dev. Codex will point at a local model with --oss or at anything OpenAI-compatible through a custom provider block.

Which means a benchmark row labelled "Codex harness, GPT-5.6 Sol" is measuring a pairing, and the pairing is now a configuration choice rather than a product boundary.

What is actually a constraint

The real limits are commercial, and they move.

On 4 April 2026 Anthropic cut off Claude Pro and Max subscriptions in third-party agentic tools, OpenCode among them, citing the compute those tools consumed. Their legal page still says OAuth for Free, Pro and Max is intended exclusively for Claude Code and Claude.ai. In May they reversed it and introduced a separate credit system for third-party agent use.

OpenCode had already removed its Anthropic OAuth plugin in v1.3.0, on 22 March, before the cut-off landed.

That is a nine-week round trip on a question of "can I use the thing I pay for in the tool I prefer", and it is the kind of constraint no benchmark will ever show you.

The number nobody publishes

Here is my own position, and it is not a measurement.

Opus 5 is a letdown in daily use and it is load-bearing at the same time, which is an uncomfortable combination. It needs steering. I re-prompt it more than I expect to, and the work that gets done is often not the work I asked for but a larger, more thorough version of it.

I am not alone in that. The reception has been widely reported as a downgrade, with over-engineering and the need for re-prompting the recurring complaints, alongside strong benchmark scores. I want to be careful here: my evidence is my own hands and a pile of secondary write-ups summarising forum reaction. That is weaker evidence than a leaderboard, and I would rather say so than dress it up.

But notice what the two disagree about. The benchmark measures whether a task completed. My complaint is about how many turns it took and how much of the result I threw away. Those are different quantities, and only one of them has a leaderboard.

What the complaint actually is

Go looking on Hacker News and the grievance has a mascot. Claude says load-bearing, constantly, and people have been counting since spring. As a tic that is only funny. What makes it a defect is that the word is applied wrongly:

Minor inconvenience != load bearing, yet Claude consistently uses it while missing actual load bearing things.

SkyPuncher, 24 August 2026

The cost shows up in the only unit that matters, which is someone's afternoon:

My job improved significantly since I managed to get into their (managers) thick skulls to tell Claude to not be verbose. I'm tired of 1200-word tickets for "put the totals in the automated emails".

javcasas, 24 August 2026

I really love CC and have been using it exclusively for a year now but reading Claude's verbose and semantically-obfuscated writing style is wearing me out and I'm planning to move to another provider.

sockaddr, 24 August 2026

The sharpest version of the complaint is not "verbose"

I had this filed as a length problem until I read this, which I think is the correct diagnosis:

"I've noticed that most people seem to consider the core problem of Claude's output as 'too verbose' but I don't think this actually cuts to the heart of the matter at all. It's almost, in some weird way, the opposite: like the text is far too dense. It tries too hard to invent odd terminology."

cnity, 21 August 2026

That matches what I see. The problem is not that there are too many words for the idea. It is that the words are doing too much work each, in vocabulary invented for the occasion, so you have to decode rather than read. A thousand plain words would be faster than four hundred of these.

Someone else makes the case for the defence, and it is worth hearing:

"Some of the more unusual language it uses does result in more precise communication, if you can figure out what it means."

Myrmornis, 22 August 2026

If you can figure out what it means is doing the work in that sentence, and it is exactly the tax.

The agreement loop

The tics are cosmetic. This one is not.

I ask for five things. It does three, presents them as finished, and only when I point at the two missing does it produce the recognition, at length. Then it does one of the two and presents that as finished.

The recognition has a house style, and everyone has met it:

"You are absolutely right to push back!" ... I am really really really trying to wrap my head around this.

Bluestein, 19 August 2026

What makes it a defect rather than a manner is the order of events. The model is not checking its work against the request and reporting a shortfall. It is declaring completion, and then agreeing enthusiastically with whoever catches it. Enthusiasm on being corrected is not the same thing as accuracy, and it is a poor substitute for the check that did not happen.

People have started building the defence into their configuration. One developer's rules file opens with a section headed "Voice Rule #1: No AIisms", and he is explicit about what it buys:

Its impact is that I don't go insane, which is pretty high value.

tyre, 23 August 2026

The suggested remedy is a concise output style, and the reports I have seen from people who tried it in earnest are that it changes the prose and not the behaviour. That tracks: terseness is a formatting instruction, and declaring a job done that is three fifths done is not a formatting problem.

Which brings this back to the benchmarks. A task where the model completes three of five requirements, says it is finished, and is then corrected into completing a fourth will be scored by most harnesses as a pass, because it passed on the turn where it finally passed. The number of turns is not in the score. It is only in your afternoon.

The fix people use is a standard, not a setting: ASD-STE100

Nobody is solving this with a slider. The thing circulating right now is a single line in the rules file, and it names a published specification:

Use ASD-STE100 Simplified Technical English.

ASD-STE100 is Simplified Technical English, the controlled language the aerospace industry wrote so that a maintenance manual could not be misread in a hangar at three in the morning. Four properties matter here:

• A restricted vocabulary. Approved words only, and the rest are not available.
• One meaning per word. No word does double duty, which is precisely what "load-bearing" was doing.
• Short sentences, with a stated limit rather than a preference.
• One instruction per sentence, so a reader cannot half-perform a paragraph.

That list is close to the exact inverse of the failure mode above, which is why it works at all.

The evidence is thin but at least somebody measured:

There has been some talk about using ASD-STE100 as a shorthand way of improving text, but not many direct experiments, so I did one. I found that just mentioning ASD-STE100 can get you 72% of the way there.

johnrising, 3 August 2026

And a dissent worth keeping, because the simplest instruction sometimes wins:

I tried "use ASD-STE100 Simplified Technical English" but I've found "natural plain English" to work better.

FooBarWidget, 27 August 2026

I have had the standard in my own rules file for a while and I would put my experience nearer the first comment than the second. Naming a real specification beats an adjective, because "concise" is a taste and ASD-STE100 is a document, and a model that has read the document has something to comply with rather than something to interpret.

What it does not touch is the agreement loop. Prose style and completeness are different faults, and only one of them is a writing problem.

The part that should worry you

The vocabulary is spreading into the humans.

"I recently started using 'load bearing' in my languge to Claude… Obviously, we adapt habits of entities next to us."

marsxr360, 28 August 2026

I will not pretend to be outside this. I used the phrase in my own notes about this model before I sat down to write about it, and reviewing the draft I found it twice more. A style you spend eight hours a day reading is not something you evaluate from a distance. It is something you start writing.

The one-week problem

The best-known recent comparison of these two harnesses is a first-impressions post from 22 July, and it is honest about its own limits in a way most are not. The author says twice that he had used the new setup for a week. He flags that harnesses affect how a model feels. He names no version number for either tool.

His main criticism of Claude Code is sporadic rendering bugs, which he half-retracts in the same sentence: "I think it has gotten better." His praise for OpenCode on the same axis is that it "apparently doesn't have them as much and anecdotally I haven't seen them yet, but I have to caveat it's only been a week".

I went and counted. Across OpenCode's 867 releases, terminal and rendering fixes are the single largest category of changelog entries, running continuously into late August. The tool praised for having fewer rendering problems ships more rendering fixes than anything else it ships.

That is not a contradiction of him. It is what a week of not seeing something is worth. He also promised a verdict after a month, around 22 August. It has not appeared, which I read as the most honest outcome available: the month did not produce a clean answer either.

Next

If you are choosing today, the differences that survive scrutiny are not scores. Licence: OpenCode is MIT and the other two are not. Sandboxing: Claude Code and Codex enforce with OS primitives, and OpenCode's own documentation says its policies "do not address sandboxing, isolation, or restricting execution environments". Provenance: sst/opencode now 301s to anomalyco/opencode, which is worth knowing before you pin a dependency. And Claude Code's changelog contains 380 version headings and no dates at all, so every date anyone cites for it, including mine, came from npm publish timestamps.

The measurement I would want does not exist yet: turns per completed task, and the share of produced work discarded. I have argued elsewhere that a failure rate cannot see a request that succeeds and returns nothing usable, and this is the same blind spot one layer up.

If you take one thing: before you quote a benchmark at someone, open the leaderboard and read its methodology note. The one I opened turned a 1.13-point race into a 4.5-point one, and it took a single click that the toggle was already offering me.