I benchmarked two AI agents. Then I read the bill.

pi said it spent $6.25. Anthropic billed $9.19.

Empryo said it spent $7.06. Anthropic billed $7.08.

Same benchmark, same bugs, same models. Each agent had its own API key. That last detail is the only reason I know any of this.

Every number in this post can be reproduced from a public benchmark repo that includes the raw results.

Nobody checks the billLink to this section

Every coding agent shows a cost figure somewhere. Benchmark posts quote it. Very few people compare it with what the provider actually charged.

So I gave each agent its own Anthropic API key. The point was attribution. When each agent has its own key, the provider's console becomes an independent judge. It counts what was sent, whatever the agent reports.

The first audit, in round one, found this:

round 1 · anthropic console vs self-report
─────────────────────────────────────────────
empryo   said 1,089,262 tokens · $1.13
         billed 1,089,262 tokens · $1.13     ✓ exact
pi       said 1.70M tokens · $1.42
         billed 6.21M tokens · $1.58         ✗ −73% input

pi's own usage events missed 73% of the input tokens it actually sent. This wasn't deliberate. Its per-message accounting didn't count cache reads. But any comparison built on pi's self-reported cost at that time was off by that much.

pi fixed it, and a new problem appearedLink to this section

Credit where it is due: pi fixed the bug. By version 0.80.7 its per-message accounting was correct, and in round two its Haiku numbers matched the console exactly.

Then the audit caught something different.

One of pi's Opus runs hung. The benchmark runner killed it after eight minutes, as the rules say. pi reported that run as $0.00, because the process died before it recorded its usage. The console showed roughly $3 of Opus tokens spent during those eight minutes.

round 2 · anthropic console vs self-report
─────────────────────────────────────────────
empryo   said $7.06 · billed $7.08           ✓ within rounding
pi       said $6.25 · billed $9.19           ✗ −32%

In round one the problem was undercounting. In round two it was spend that disappears when an agent crashes mid-run. The cause changed, but the lesson didn't: the ways cost reporting breaks keep changing, so you have to keep checking.

Where I loseLink to this section

A benchmark where the vendor wins every row is an ad. Here is what went against me:

• pi's harness is about 8× lighter. Its fixed overhead per model call is about 2.6k tokens. Empryo sends about 21k, because the code map it gives the model isn't free. Opus is strong enough to find its way around a repo without that map, and on Opus pi was cheaper per task on three of five tasks.
• My agent made mistakes too. On one Haiku task, Empryo left debug prints in the diff and failed the hidden tests. It's in the raw results, unedited.
• One run per task. Agent runs vary from one attempt to the next. Read the results as a direction, not precise decimals. The per-task numbers are in the JSON.

What holds up after those caveats: more bugs fixed across both rounds (15/19 vs 13/19), 23% cheaper on the actual bill, 38% faster, and cost figures that match the invoice.

My opinionLink to this section

Treat any agent's self-reported cost as unconfirmed until someone checks it with a separate key. People quote pass rates to two decimal places while the dollar figures next to them can be off by 32 to 73%.

If you pick an agent based on published cost numbers that nobody checked against a bill, you may be picking on bad data.

Please try to prove me wrongLink to this section

The benchmark runner, the tasks, the hidden acceptance tests, and every raw result file are in one repo. With two API keys and one command you can re-run everything. If your results disagree with mine, publish them and I'll put your numbers next to mine on the benchmarks page.