Empryo vs piRound 22026-07-16Real world

No more fixtures. Real repos, real bugs, real fixes.

We compared Empryo and pi on five bugs that real users hit in three open-source TypeScript projects: hono, zod and ky. Each bug has a merged fix PR. The agents got the original issue text, word for word, and the repo at the commit where the bug was live. We rewrote git history so the fix can't be found, and every fix was merged after the models' training cutoffs, so no model can recall the patch. The fix PR's own regression tests, added after the run, graded each attempt. Empryo fixed 7 of 10 runs to pi's 6 and cost 23% less on Anthropic's bill. Caveat: there is one run per model and task.

Results at a glance

billed cost

−23%

$7.08 vs $9.19, per the Anthropic console

Empryo$7.08
pi$9.19

bugs fixed: 7/10 vs 6/10; Empryo vs pi
steps: −28%; 274 vs 382
time: −32%; 22m 30s vs 32m 55s
haiku tier: 3/5 vs 2/5; bugs fixed on the cheapest model

Empryo274 steps
pi382 steps

Empryo22m 30s
pi32m 55s

Where the code map pays off: Haiku on a 90k-line repo.

The cheap-model story.

On a small test app, searching text keeps up. On real repos it doesn't, because finding where the bug lives is the expensive part. That is what Empryo's Genome, its live map of how your code connects, is for. On Haiku, Empryo fixed one more bug than pi, spent 19% less, and finished in about half the time.

The hardest bug was a parser regression in zod, deep in a 580-file monorepo. Only Opus fixed it, for both agents. Empryo did it in 33 steps for $2.10. pi's first attempt hung until the 8-minute limit stopped it, and its retry took 69 steps and $2.42.

By each agent's own reports, pi looked about 11% cheaper over the round. The Anthropic console says otherwise, as the next section shows.

Opus 4.8 on zod preprocess-catch, one dot per step

Empryo33 steps$2.10, fixed
pi69 steps$2.42, fixed on the retry

Spend lost in a crash: a new kind of under-reporting.

Reported cost vs billed cost, round 2.

Empryo
Anthropic billed$7.08
It said$7.06
Matched, apart from ≤2¢ of display rounding
pi
Anthropic billed$9.19
It said$6.25
Left out $2.94, 32% of its own bill

Credit where due: pi 0.80.7 fixed the per-message undercounting we found in round 1. On Haiku, its reports matched the console exactly. The whole gap comes from one crashed Opus run. It did eight minutes of real model work, which Anthropic billed in full (about $3). pi reported it as $0.00 because the process died before it could record its cost. So the billed totals are Empryo $7.08 and pi $9.19: Empryo was cheaper and fixed more bugs. If you compare agents yourself, don't rely on the cost each one reports. Give each agent its own key.

All 20 runs

Cost as each agent reported it. A tick means the fix PR's own tests passed; the thin line under each cell is its time, on one shared scale.

All 20 runs, cost as each agent reported it
Model	Bug	Empryo	pi (0.80.7)
haiku 4.5	hono · rpc-headers	Empryo$0.40831 steps112s	pi (0.80.7)$0.23036 steps150s
haiku 4.5	hono · trie-wildcard	Empryo$0.38740 steps122s	pi (0.80.7)$0.42037 steps271s
haiku 4.5	zod · preprocess-catch	Empryo$0.28440 steps113s	pi (0.80.7)$0.75387 steps434s
haiku 4.5	ky · hooks-request	Empryo$0.21524 steps97s	pi (0.80.7)$0.30444 steps174s
haiku 4.5	ky · extend-retry	Empryo$0.43940 steps166s	pi (0.80.7)$0.44343 steps180s
opus 4.8	hono · rpc-headers	Empryo$1.05319 steps143s	pi (0.80.7)$0.55420 steps112s
opus 4.8	hono · trie-wildcard	Empryo$1.05619 steps155s	pi (0.80.7)$0.30212 steps60s
opus 4.8	zod · preprocess-catch	Empryo$2.10133 steps275s	pi (0.80.7)$2.42469 steps427s
opus 4.8	ky · hooks-request	Empryo$0.54913 steps75s	pi (0.80.7)$0.27515 steps62s
opus 4.8	ky · extend-retry	Empryo$0.56615 steps95s	pi (0.80.7)$0.54519 steps106s

Empryo ran headless in auto mode, capped at 40 steps, with its code map built before the run. Indexing time is reported separately, not hidden. pi 0.80.7 ran with -p --mode json, default settings and no step cap. For Opus, pi needed a custom model id with thinking turned off: its model list has no Opus 4.8, and that API rejects pi's default thinking settings. pi's Opus zod result is a retry. The first attempt hung until the 8-minute limit with no output; its tokens were billed, but pi reported $0. Neither agent fixed ky's extend-retry bug on either model. One run per cell. The costs here are each agent's own reports; the console-checked totals are in the section above.

Built so the answer can't leak in.

What the numbers mean

Plain words for every measure on this page.

A run: One agent, one model, one bug, one attempt. A cell in the tables below is one run.
Fixed: The bug only counts as fixed when tests the agent never saw pass on its code. We add those tests after the agent has finished, so it cannot aim at them.
Regression tests: The tests the real maintainers wrote alongside their merged fix. They fail on the buggy code and pass on the real fix, so they tell us whether the agent's fix really works.
Step: One round trip to the model: the agent sends the conversation so far and gets back its next move. Each step sends the whole conversation again, so every step you avoid is one you don't pay for.
Tokens: The pieces of text a model reads and writes, a word or part of a word each. Providers charge per token.
Reported cost: What the agent itself says the run cost, in its own logs or summary.
Billed cost: What the provider actually charged. Each agent had its own API key, so the provider's console shows each agent's spend on its own.
Time: Real time from start to finish, as on a stopwatch. Also called wall clock.
Training cutoff: The date a model's training data ends. A fix merged after it cannot be something the model memorised.
Genome: Empryo's live map of your code: which functions call which, and which files change together. The agent reads the map instead of guessing from text search.
Model tiers: Haiku is Anthropic's smallest and cheapest Claude model, Sonnet is the middle one, Opus is the largest and most expensive.

Don't trust our numbers. Run them.

Check the tasks for free, then rerun the whole round with two API keys.

bun harness-real/run-real.ts --tiers haiku,opus

Reproduce it on GitHub Round 1 results