Same bugs. Same models. The bill disagrees.
We compared Empryo and pi on three bugs planted in a realistic TypeScript webhook app. Each bug report reads the way a user would write it, with no file names and no hints. Both agents ran on Claude Haiku 4.5, Sonnet 4.6 and Opus 4.8. Each agent had its own Anthropic API key, so Anthropic's bill shows exactly what each one spent. Empryo fixed 8 of 9 runs to pi's 7 and cost 28% less on the bill. Caveat: there is one run per model and task, so single cells are noisy.
Results at a glance
fewer input tokens
5.7×1.09M vs 6.21M billed
- bugs fixed
- 8/9 vs 7/9
- Empryo vs pi
- fewer output tokens
- 6.3×
- 14.4k vs 91.2k billed
- billed cost
- −28%
- $1.13 vs $1.58
- wall clock
- −57%
- 4m 16s vs 10m 0s
Empryo reported its costs exactly.
Reported cost vs billed cost: what each agent said it used, next to what Anthropic billed.
Tokens
Empryo
Anthropic billedIt saidExact, to the token and the cent
pi
Anthropic billedIt saidUnder-reported its input tokens by 73%
Dollars
Empryo
Anthropic billedIt saidMatched the bill
pi
Anthropic billedIt saidBelow the bill
In this round, the cost Empryo showed matched Anthropic's bill to the token. If you compare agents yourself, don't rely on the cost each one reports. Give each agent its own key and read the provider's bill.
The comment was wrong. The code map wasn't.
Why the code map helped.
One task had a trap. A comment said event ids are ULIDs, so “sorting by id is sorting by time”. But two versions earlier the id generator had switched to random ids, and nobody updated the comment.
An agent that searches text finds that comment and believes it. On Haiku, pi spent 35 steps and 165 seconds and still shipped a wrong fix. Empryo's Genome is a live map of your code: its functions, what calls what, and how the pieces connect. It showed Haiku what the id generator really does, and Haiku landed the correct fix in 10 steps and 20 seconds.
Fewer steps also means a smaller bill. Each step sends the whole conversation so far back to the model, so every step you avoid is one you don't pay for.
Haiku 4.5 on sort-order, one dot per step
Empryo10 steps20 seconds, fixed
pi35 steps165 seconds, wrong fix
Four measures, drawn to scale
Each pair of strokes shares one ruler, so the shorter stroke really is that much smaller.
Input tokens billed5.7× fewer
Output tokens billed6.3× fewer
Billed cost−28%
Wall clock−57%
All 18 runs
Cost as each agent reported it. A tick means the hidden tests passed; the thin line under each cell is its time, on one shared scale.
| Model | Bug | Empryo | pi |
|---|---|---|---|
| haiku 4.5 | profile-race | Empryo$0.0418 steps22s | pi$0.05314 steps42s |
| haiku 4.5 | csv-quoting | Empryo$0.0317 steps14s | pi$0.0439 steps31s |
| haiku 4.5 | sort-order | Empryo$0.03910 steps20s | pi$0.18335 steps165s |
| sonnet 4.6 | profile-race | Empryo$0.1107 steps42s | pi$0.13710 steps63s |
| sonnet 4.6 | csv-quoting | Empryo$0.1017 steps27s | pi$0.1037 steps50s |
| sonnet 4.6 | sort-order | Empryo$0.0775 steps19s | pi$0.0847 steps32s |
| opus 4.8 | profile-race | Empryo$0.2629 steps43s | pi$0.33713 steps68s |
| opus 4.8 | csv-quoting | Empryo$0.1585 steps17s | pi$0.23610 steps99s |
| opus 4.8 | sort-order | Empryo$0.31513 steps52s | pi$0.2449 steps49s |
One run per cell. Agent runs vary from one try to the next, so read single cells with care. The costs in this table are what each agent reported itself. pi's real per-task costs are higher, as the billing check above shows. For Opus, pi needed a custom model id with thinking turned off: its model list has no Opus 4.8, and that API rejects pi's default thinking settings. Both agents failed csv-quoting on Opus with over-clever escaping that doesn't follow the CSV standard (RFC 4180). Haiku and Sonnet got it right.
What the numbers mean
Plain words for every measure on this page.
- A run
- One agent, one model, one bug, one attempt. A cell in the tables below is one run.
- Fixed
- The bug only counts as fixed when tests the agent never saw pass on its code. We add those tests after the agent has finished, so it cannot aim at them.
- Step
- One round trip to the model: the agent sends the conversation so far and gets back its next move. Each step sends the whole conversation again, so every step you avoid is one you don't pay for.
- Tokens
- The pieces of text a model reads and writes, a word or part of a word each. Providers charge per token.
- Input tokens
- Text sent to the model: the conversation, the code the agent looked at, its instructions and tool descriptions.
- Output tokens
- Text the model writes back: its reasoning, its tool calls and the code it changes.
- Reported cost
- What the agent itself says the run cost, in its own logs or summary.
- Billed cost
- What the provider actually charged. Each agent had its own API key, so the provider's console shows each agent's spend on its own.
- Time
- Real time from start to finish, as on a stopwatch. Also called wall clock.
- Genome
- Empryo's live map of your code: which functions call which, and which files change together. The agent reads the map instead of guessing from text search.
- Model tiers
- Haiku is Anthropic's smallest and cheapest Claude model, Sonnet is the middle one, Opus is the largest and most expensive.