Five real bugs, two agents, and the one nobody could fix

"Upgrading from 4.3.6 to 4.4.x causes issues with preprocess + catch. In v4.3.6 and below, the following schema works…"

That's a real zod user, reporting a real regression, on a real Tuesday. No fixture author wrote it. No benchmark team massaged it. And that's the point: for round two of Empryo's head-to-head benchmarks, I stopped inventing bugs and started harvesting them.

Disclosure, as always: I build Empryo, solo and part-time and unfunded, and I benchmark it against agents built by funded teams. Everything below reruns from a public harness. Verifying the tasks costs nothing and needs no API key.

The problem with benchmark fixtures

Synthetic bugs have two failure modes. They're small. Grep-and-read keeps up on a 30-file toy app, so code intelligence has nothing to prove. And they're *written by the vendor*, which means you're trusting the exam and the student to the same person.

Real bugs fix both. But they add a nastier problem, which is contamination. If the bug was fixed in 2024, every frontier model has read the patch. You're not benchmarking debugging. You're benchmarking recall.

So the rules for this round:

• Five real bugs from hono, zod and ky: shipping repos (61 to 580 files) with real test suites
• Every fix merged after January 2026, past the training cutoffs, so no model can regurgitate a memorized patch
• The agent gets the original issue text verbatim: user-written symptoms, zero file hints
• Git history scrubbed to a single baseline commit, so git log archaeology finds nothing
• Grading: the fix PR's own regression tests, held outside the workspace, dropped in after the run
• Before spending a token, a validator proves every task fails by assertion on the bug and passes on the real fix

The validator is free to run. Audit the exam before believing the grades.

The verdict

Two tiers: Claude Haiku 4.5 and Opus 4.8, both agents, all five bugs.

The zod bug was the whole story in miniature

The hardest target lived deep in zod's core parse pipeline, a 580-file monorepo.

On Haiku, both agents failed. But look at *how*:

zod-preprocess-catch · haiku 4.5
─────────────────────────────────────────
empryo   ✗  40 steps (capped) · $0.28 · 1m 53s
pi       ✗  87 steps          · $0.75 · 7m 14s

Empryo hit its step cap, reported failure, and stopped. pi has no cap, so it ground through 87 steps and seven minutes to reach the same wrong answer at 2.7× the price. Losing cheaply is an engineering feature. Somebody pays for optimism.

On Opus, Empryo cracked it in 33 steps, $2.10, first attempt. pi's first attempt hung until the harness killed it at eight minutes; its retry landed the fix in 69 steps and $2.42. Same destination, twice the steps.

Why the gap? Empryo starts every task with the Genome, a ranked structural map of the repo, so "where does this bug live" is a lookup rather than an expedition. On a 30-file fixture that barely matters. On 580 it's the difference between navigating and wandering.

The bug that beat everyone

ky's extend() drops a numeric retry limit when merging options. Four attempts across both agents and both tiers, and zero fixes. A subtle merge-semantics bug in a tiny file defeated $18 of frontier compute.

I could have quietly swapped that task for something winnable. It stayed, because a benchmark where every row is green is an ad.

Where I lose

• pi beat Empryo's per-cell cost on three of five Opus tasks, because its harness carries ~2.6k tokens of overhead to my ~21k, and Opus doesn't need my map to find its way.
• On one Haiku cell, Empryo left its debug prints in the diff and failed the hidden tests on cleanliness. Real agents fumble; the raw JSON keeps the receipts.
• n=1 per cell. Directions held across tiers and rounds; treat decimals with suspicion, including mine.

The part that matters for your wallet

The Haiku tier was a sweep: 3/5 vs 2/5 fixed, 19% cheaper, exactly half the wall clock. That's the practical finding. With a code-intelligence graph underneath it, the cheap model fixed more real bugs than the expensive-agent-on-cheap-model, and the expensive model only broke even against it.

Intelligence you rent per token. Understanding you index once.

Run it yourself

Everything (tasks, validator, runner, raw results including the voided cells) is in the bench repo. bun harness-real/validate.ts proves the tasks are honest without spending a cent. Two API keys rerun the whole round.

If your numbers come out different, publish them. I'll link them from the benchmarks page, next to mine.