Same bugs. Same models. The bill disagrees.
Three real bugs in a real TypeScript repo, reported the way humans report them — no file names, no hints. Empryo vs pi on Claude Haiku 4.5, Sonnet 4.6 and Opus 4.8, each agent on its own Anthropic API key, so the provider's bill — not anyone's telemetry — is the judge.
- 8/9 vs 7/9
- bugs fixed Empryo vs pi
- 5.7×
- fewer input tokens 1.09M vs 6.21M billed
- 6.3×
- fewer output tokens 14.4k vs 91.2k billed
- −28%
- billed cost $1.13 vs $1.58
- −57%
- wall clock 4m 16s vs 10m 0s
Empryo reported its costs 100% correctly.
The cost figure in Empryo's status bar is the number Anthropic bills you. If you benchmark agents on self-reported numbers, use separate keys.
Comments lie. Graphs don't.
One task hid a trap: a stale comment claiming event ids are ULIDs — “sorting by id is sorting by time” — while the id generator had quietly gone random two versions earlier. Grep-and-read finds the comment and believes it: pi burned 35 steps, 165 seconds and still shipped a wrong fix on Haiku. Empryo's Genome injects what the code actually does — symbols, calls, real relationships — and Haiku landed the correct fix in 10 steps and 20 seconds. Fewer steps isn't just faster: every avoided step is an entire context window you don't pay to resend.
| model | task | empryo | pi |
|---|---|---|---|
| haiku 4.5 | profile-race | ✓ $0.041 · 8 steps · 22s | ✓ $0.053 · 14 steps · 42s |
| haiku 4.5 | csv-quoting | ✓ $0.031 · 7 steps · 14s | ✓ $0.043 · 9 steps · 31s |
| haiku 4.5 | sort-order | ✓ $0.039 · 10 steps · 20s | ✗ $0.183 · 35 steps · 165s |
| sonnet 4.6 | profile-race | ✓ $0.110 · 7 steps · 42s | ✓ $0.137 · 10 steps · 63s |
| sonnet 4.6 | csv-quoting | ✓ $0.101 · 7 steps · 27s | ✓ $0.103 · 7 steps · 50s |
| sonnet 4.6 | sort-order | ✓ $0.077 · 5 steps · 19s | ✓ $0.084 · 7 steps · 32s |
| opus 4.8 | profile-race | ✓ $0.262 · 9 steps · 43s | ✓ $0.337 · 13 steps · 68s |
| opus 4.8 | csv-quoting | ✗ $0.158 · 5 steps · 17s | ✗ $0.236 · 10 steps · 99s |
| opus 4.8 | sort-order | ✓ $0.315 · 13 steps · 52s | ✓ $0.244 · 9 steps · 49s |
Single trial per cell — agent runs are stochastic; the aggregate direction held across all nine pairs. Per-cell figures are each agent's own accounting (pi's true per-task costs are proportionally higher — see the billing finding above). pi's Opus tier required a custom model id with thinking disabled; its registry has no Opus 4.8 and its default thinking payload is rejected by that API. Both agents failed csv-quoting on Opus with over-clever non-RFC escaping that Haiku and Sonnet got right.
Don't trust our numbers. Run them.
The fixture, the harness, the hidden acceptance tests, the raw results and the full report are public. Two API keys and one command reproduce the whole experiment.
bun harness/run-vs.ts --label my-run