Five real bugs, two agents, and the one nobody could fix

"Upgrading from 4.3.6 to 4.4.x causes issues with preprocess + catch. In v4.3.6 and below, the following schema works…"

That is a real zod user reporting a real regression. No benchmark author wrote it or cleaned it up. That's the point. For round two of Empryo's head-to-head benchmarks, I stopped inventing bugs and started collecting real ones.

A disclosure, as always: I build Empryo, solo, part-time and unfunded, and I benchmark it against agents from funded teams. Everything below can be re-run from a public benchmark repo. Checking the tasks costs nothing and needs no API key.

The problem with made-up bugsLink to this section

Made-up bugs have two problems. First, they're small. On a 30-file toy app, plain searching and reading keeps up fine, so an agent's understanding of code structure has nothing to prove. Second, the vendor writes them, so you're trusting the same person to write the exam and take it.

Real bugs fix both problems but add another: contamination. If a bug was fixed in 2024, every frontier model has probably read the patch. You'd be testing memory, not debugging.

So the rules for this round were:

• Five real bugs from hono, zod and ky: active projects (61 to 580 files) with real test suites.
• Every fix was merged after January 2026, after the models' training cutoffs, so it's unlikely any model had seen the patch.
• The agent gets the original issue text, word for word: the user's description of the symptoms, with no hints about which files to look at.
• Git history is reduced to a single starting commit, so digging through git log finds nothing.
• Grading uses the tests from the fix's own pull request. They're kept outside the workspace and added after the run.
• Before any tokens are spent, a validator checks that every task fails on the bug and passes with the real fix.

The validator is free to run. Check the exam before you trust the grades.

The resultLink to this section

Two model tiers, Claude Haiku 4.5 and Opus 4.8, for both agents on all five bugs.

The zod bug told the whole storyLink to this section

The hardest bug was deep in zod's core parsing code, in a 580-file monorepo.

On Haiku, both agents failed. But look at how:

zod-preprocess-catch · haiku 4.5
─────────────────────────────────────────
empryo   ✗  40 steps (capped) · $0.28 · 1m 53s
pi       ✗  87 steps          · $0.75 · 7m 14s

Empryo hit its step limit, reported that it had failed, and stopped. pi kept going for 87 steps and seven minutes and reached the same wrong result at 2.7× the price. Failing cheaply is a feature. Someone pays for an agent that won't give up.

On Opus, Empryo fixed it on the first attempt, in 33 steps, for $2.10. pi's first attempt hung until the runner killed it at eight minutes. Its retry fixed the bug in 69 steps for $2.42. Same result, twice the steps.

Why the difference? Empryo starts every task with the Genome, a ranked map of the repo's files and symbols that it keeps up to date. Finding where a bug lives becomes a lookup instead of a search. On a 30-file project that barely matters. On 580 files it's the difference between going straight there and wandering.

The bug that beat everyoneLink to this section

In ky, extend() drops a numeric retry limit when it merges options. Four attempts, across both agents and both tiers, produced zero fixes. A subtle bug in how options get merged, in a tiny file, beat $18 of frontier-model compute.

I could have quietly swapped that task for one we could win. It stayed in, because a benchmark where every row is green is an ad.

Where I loseLink to this section

• pi was cheaper than Empryo on three of five Opus tasks. Its harness adds about 2.6k tokens of overhead per model call against Empryo's 21k, and Opus doesn't need Empryo's map to find its way around.
• On one Haiku task, Empryo left debug prints in the diff and failed the hidden tests for it. Real agents make mistakes. The raw results keep the evidence.
• Each task ran once per agent and tier. The overall direction held across tiers and rounds, but don't read too much into the decimals, mine included.

What matters for your budgetLink to this section

On the Haiku tier, Empryo fixed 3 of 5 bugs against pi's 2 of 5, for 19% less money, in half the time. That is the practical finding. With a code map underneath it, the cheap model fixed more real bugs. On Opus, both agents fixed 4 of 5, so the expensive model mainly closed the gap.

You pay for model intelligence on every token. A map of your code is built once.

Run it yourselfLink to this section

Everything is in the benchmark repo: the tasks, the validator, the runner, and the raw results, including the runs we had to void. bun harness-real/validate.ts checks that the tasks are sound without spending anything. Two API keys re-run the whole round.

If your numbers come out different, publish them. I'll link them from the benchmarks page, next to mine.