No more fixtures. Real repos, real bugs, real fixes.
Five bugs that real users hit in real open-source projects — hono, zod and ky — each taken from a merged fix PR. The agents get the original issue text verbatim, the repo at the commit where the bug was live, and nothing else: git history is rewritten so the actual fix is unreachable, and every fix was merged after the models' training cutoffs — nobody can regurgitate a memorized patch. Grading: the fix PR's own regression tests, dropped in after the run.
- 7/10 vs 6/10
- bugs fixed Empryo vs pi
- −23%
- billed cost $7.08 vs $9.19 on the console
- −28%
- steps 274 vs 382
- −32%
- wall clock 22m 30s vs 32m 55s
- 3/5 vs 2/5
- haiku tier the cheap-model sweep
Where intelligence pays: Haiku on a 90k-LOC repo.
On small fixtures, grep-and-read keeps up. On real repos it doesn't: finding where the bug lives is the expensive part, and that's what Empryo's code graph is for. At the Haiku tier Empryo fixed one more bug than pi while spending 19% less and finishing in exactly half the wall clock. The hardest target — zod's parser regression, deep in a 580-file monorepo — took Opus on both sides: Empryo landed it in 33 steps and $2.10; pi's first attempt hung until the 8-minute kill, and its retry needed 69 steps and $2.42. On self-reported numbers pi looked ~11% cheaper across the round — the Anthropic console says otherwise (below).
Crash-invisible spend: the new under-reporting.
Credit where due: pi 0.80.7 fixed round 1's per-message undercounting — its Haiku tier matched the console exactly. The gap is entirely the crashed Opus cell: eight minutes of real model work, billed in full (~$3), self-reported as $0.00 because the process died before emitting its accounting event. So the billed truth is Empryo $7.08 vs pi $9.19 — cheaper in absolute terms, with more bugs fixed. If you benchmark agents on self-reported numbers, use separate keys.
| model | task | empryo | pi (0.80.7) |
|---|---|---|---|
| haiku 4.5 | hono · rpc-headers | ✓ $0.408 · 31 steps · 112s | ✗ $0.230 · 36 steps · 150s |
| haiku 4.5 | hono · trie-wildcard | ✓ $0.387 · 40 steps · 122s | ✓ $0.420 · 37 steps · 271s |
| haiku 4.5 | zod · preprocess-catch | ✗ $0.284 · 40 steps · 113s | ✗ $0.753 · 87 steps · 434s |
| haiku 4.5 | ky · hooks-request | ✓ $0.215 · 24 steps · 97s | ✓ $0.304 · 44 steps · 174s |
| haiku 4.5 | ky · extend-retry | ✗ $0.439 · 40 steps · 166s | ✗ $0.443 · 43 steps · 180s |
| opus 4.8 | hono · rpc-headers | ✓ $1.053 · 19 steps · 143s | ✓ $0.554 · 20 steps · 112s |
| opus 4.8 | hono · trie-wildcard | ✓ $1.056 · 19 steps · 155s | ✓ $0.302 · 12 steps · 60s |
| opus 4.8 | zod · preprocess-catch | ✓ $2.101 · 33 steps · 275s | ✓ $2.424 · 69 steps · 427s |
| opus 4.8 | ky · hooks-request | ✓ $0.549 · 13 steps · 75s | ✓ $0.275 · 15 steps · 62s |
| opus 4.8 | ky · extend-retry | ✗ $0.566 · 15 steps · 95s | ✗ $0.545 · 19 steps · 106s |
Empryo: headless, mode auto, max 40 steps, genome pre-warmed (indexing wall-clock reported separately, never hidden). pi 0.80.7: -p --mode json, defaults, no step cap; its Opus tier needs a custom model id with thinking off — the registry lacks Opus 4.8 and its default thinking payload is rejected. pi's Opus zod cell is a retry: the first attempt hung to the 8-minute hard kill with no output (tokens billed, reported as $0 by pi). ky's extend-retry bug defeated both agents at both tiers — this round's unsolved case. Single trial per cell; per-cell figures are each agent's own accounting — the console-verified totals are in the finding above.
Contamination-proof, by construction.
Every task is validated before a single token is spent: the fix PR's tests must fail — by assertion, not by crash — on the buggy commit, and pass on the real fix. Workspaces are scrubbed to a single baseline commit so no amount of git archaeology can surface the answer. And because every fix merged after January 2026, the models cannot have seen these patches in training. The task manifest, validator and runner are all public.
Don't trust our numbers. Run them.
Validate the tasks for free, then reproduce the whole round with two API keys.
bun harness-real/run-real.ts --tiers haiku,opus