I benchmarked two AI agents. Then I read the bill.

pi said it spent $6.25. Anthropic billed $9.19.

Empryo said it spent $7.06. Anthropic billed $7.08.

Same benchmark. Same bugs. Same models. Separate API keys, one per agent. That last part is the only reason I know any of this.

Full disclosure before we go further: I build Empryo. Alone, nights and weekends, no funding. The agents I benchmark against are backed teams. That's exactly why every number in this post is reproducible from a public harness with the raw results committed. A solo dev doesn't get the benefit of the doubt, and shouldn't ask for it.

Nobody audits the bill

Every agent shows you a cost figure in its status bar. Every benchmark post quotes it. Almost nobody checks it against what the provider actually charged.

So I gave each agent its own Anthropic API key. Not for convenience. For attribution. When each contestant has its own key, the provider's console becomes an independent judge that doesn't care about anyone's telemetry.

The first audit, round one, found this:

round 1 · anthropic console vs self-report
─────────────────────────────────────────────
empryo   said 1,089,262 tokens · $1.13
         billed 1,089,262 tokens · $1.13     ✓ exact
pi       said 1.70M tokens · $1.42
         billed 6.21M tokens · $1.58         ✗ −73% input

pi's own events missed 73% of the input tokens it actually sent. Not malice. Its per-message accounting simply didn't count cache reads. But every blog post ever written using pi's self-reported costs was off by that much.

Credit where due, and a new failure mode

Here's the part vendor posts usually skip: pi fixed it. By 0.80.7 its per-message accounting was clean, and in round two its Haiku tier matched the console *exactly*. Good engineering, honestly.

And then the audit caught something completely different.

One of pi's Opus runs hung. The harness killed it after eight minutes, per protocol. pi reported that cell as $0.00, because the process died before its accounting event fired. The console showed roughly $3 of very real Opus tokens for those eight minutes.

round 2 · anthropic console vs self-report
─────────────────────────────────────────────
empryo   said $7.06 · billed $7.08           ✓ exact
pi       said $6.25 · billed $9.19           ✗ −32%

Round one's bug was undercounting. Round two's was crash-invisible spend: money that vanishes from the books whenever an agent dies mid-run. Different mechanism, same lesson: the failure modes keep changing, so the audit can't stop.

Where I lose

A benchmark where the vendor wins every row is marketing. Here's my column of losses:

• pi's harness is ~8× lighter than mine. Its fixed overhead per model call is ~2.6k tokens; Empryo carries ~21k (the code-intelligence graph isn't free). On Opus — smart enough to navigate a repo without my map — pi beat my per-cell cost on three of five tasks.
• My agent fumbled too. On one Haiku cell Empryo left debug prints in the diff and failed the hidden tests. It's in the raw results, unedited.
• Single trial per cell. Agent runs are stochastic; I report direction, not decimals, and the per-cell spread is visible in the JSON.

The wins that survive those caveats: more bugs fixed (15/19 vs 13/19 across both rounds), 23% cheaper on the actual bill, 38% faster — and books that balance.

The opinion part

Treat every self-reported agent cost as fiction until someone with a separate key audits it. Berkeley's RDI group showed most agent benchmarks can be gamed outright; costs are even softer, because nobody's watching the meter. The industry quotes pass rates to two decimal places while getting *dollars* wrong by 32–73%.

If you're choosing an agent on published cost numbers, you're choosing on fiction.

Prove me wrong — please

The harness, the tasks, the hidden acceptance tests, and every raw result JSON are in one repo. Two API keys and one command re-run the whole thing. If your rerun disagrees with mine, publish it — I'll put your numbers next to mine on the benchmarks page.

That's an offer a bigger company's legal team wouldn't let them make.