Skip to content
Checked against the billSeparate keysReproducible

Every coding agent says it's fast and cheap. We check the bill.

Same bugs, same model, one API key per agent. Hidden tests decide what counts as fixed, and we publish what each agent reported next to what Anthropic billed.

See the roundsRun them yourself

Latest roundRound 3 on Opus 5, August 2026

Cost per round −27%

  • Empryo$2.27
  • pi$3.10

Wall clock −40%

  • Empryo5m 45s
  • pi9m 35s

Bars are to scale. Both agents fixed all 4 bugs. Read round 3

Where Empryo stands

Time and cost from every run we have published, added up for each rival. Shorter bars win. Claude Code on the newer Opus 4.7 finished one layout change 8 seconds sooner, and that run counts here too.

EmpryovsOpenCode

See the 3 runs

Time3 runs added up

Empryo10m 59s
OpenCode25m 43s

2.3× fasterFaster in 3 of 3 runs

Cost3 runs added up

Empryo$3.11
OpenCode$9.30

3.0× cheaperCheaper in 3 of 3 runs

EmpryovsClaude Code

See the 2 runs

Time2 runs added up

Empryo6m 06s
Claude Code11m 02s

1.8× fasterFaster in 1 of 2 runs

Costone run

Empryo$0.57
Claude Code$1.27

2.2× cheaperCheaper in 1 of 1 run

Empryovspi

See the 3 rounds

Time3 runs added up

Empryo32m 31s
pi52m 30s

1.6× fasterFaster in 3 of 3 runs

Cost3 runs added up

Empryo$10.48
pi$13.87

1.3× cheaperCheaper in 3 of 3 runs

Costs come from the Anthropic console with one API key per tool, except the cost audit (published in the Empryo README) and round 3 (every API call metered). One Claude Code run used a Claude Max plan, so it has no per-run bill and only its time counts.

Empryo against OpenCode and Claude Code

4 recorded tasks from April and May 2026, one run per tool, with the bill for each.

See the head-to-head runs

Empryo against pi

3 rounds from July and August 2026, graded by hidden tests.

See the rounds

Cost it reported vs what the provider billed

Each tool ran on its own API key, so the provider's bill shows what it really spent.

The full bar is what the provider billed each tool's own API key. The filled part is what the tool reported; the dashed part is spend it left out. Empryo's small gap in rounds 1 and 2 is display rounding.

Every round so far

Newest first. Each round has its own page with every run and how it was graded.

  1. Round 3Empryo vs pi, six modelsNew

    Code has a shape: six models, two providers

    Empryo, with all 37 tools and a live map of your code, against pi, a minimal agent with 4 tools. Both fixed four real bugs with merged fixes, on six models from Haiku to Opus 5, with every API call metered. On Opus 5 Empryo cost 27% less and finished 40% faster, and it never fixed fewer bugs than pi. Most numbers average two runs; a few are single runs.

    Task
    Question: does an agent need a big toolset and a code map?
    Test
    Graded by hidden tests. 4 real merged bugs, 6 models, 2 providers, 96 metered runs.
    Result
    Opus: −27% cost, −40% time. luna: 100% vs 88% fixed. Never fewer bugs fixed.

    Cost per round on Opus 5−27%

    • Empryo$2.27
    • pi$3.10
    Opus cost
    −27%
    Opus time
    −40%
    Opus cost vs our v1
    −56%
    See the full results

  2. Round 2Empryo vs pi, real world

    Five real bugs from hono, zod and ky

    Empryo vs pi on five bugs from real open-source projects, each with a merged fix. The agents got the original issue text and the repo as it was before the fix, with git history removed and every fix merged after the models' training cutoffs. The fix's own regression tests graded each run. Empryo fixed 7 of 10 runs to pi's 6 and cost 23% less on the bill. One run per model and task.

    Task
    "RPC headers not merging when function", hono #5089, word for word
    Test
    Graded by the fix PR's own regression tests, added after the run
    Result
    7/10 vs 6/10 fixed. Billed $7.08 vs $9.19. Time −32%.

    Billed cost, per the Anthropic console−23%

    • Empryo$7.08
    • pi$9.19
    Billed cost
    −23%
    Steps
    −28%
    Time
    −32%
    See the full results

  3. Round 1Empryo vs pi, hookboard

    Same bugs, same models, and the bill disagrees

    Empryo vs pi on three bugs planted in a realistic TypeScript webhook app, described the way a user would report them. One had a trap: an outdated comment claiming ids are time-sorted ULIDs. Text search finds the comment and trusts it; Empryo's code map shows what the id generator really does. Empryo fixed 8 of 9 runs to pi's 7 and cost 28% less. One run per model and task.

    Task
    "the order is all over the place, definitely not newest first"
    Test
    Trap: an outdated ULID comment. On Haiku, pi took 35 steps and missed; Empryo took 10.
    Result
    8/9 vs 7/9 fixed. 5.7× fewer input tokens billed. Cost −28%.

    Input tokens billed5.7× fewer

    • Empryo1.09M
    • pi6.21M
    Input tokens
    5.7× fewer
    Billed cost
    −28%
    Time
    −57%
    See the full results

How a round works

Four rules, the same in every round.

  1. Same bugs, same model

    Each agent gets the same bugs, described the way people report them, and runs on the same model.

  2. A key of its own

    Each agent runs on its own API key, so the provider's bill shows exactly what each one spent.

  3. Hidden tests decide

    Tests the agent never saw decide whether a fix works. Nobody grades by eye.

  4. Read the bill

    We compare the cost each agent reports with what the provider actually billed, and publish both.

What the numbers mean

Plain words for every measure on this page.

A run
One agent, one model, one bug, one attempt. A cell in the tables below is one run.
Fixed
The bug only counts as fixed when tests the agent never saw pass on its code. We add those tests after the agent has finished, so it cannot aim at them.
Step
One round trip to the model: the agent sends the conversation so far and gets back its next move. Each step sends the whole conversation again, so every step you avoid is one you don't pay for.
Tokens
The pieces of text a model reads and writes, a word or part of a word each. Providers charge per token.
Reported cost
What the agent itself says the run cost, in its own logs or summary.
Billed cost
What the provider actually charged. Each agent had its own API key, so the provider's console shows each agent's spend on its own.
Time
Real time from start to finish, as on a stopwatch. Also called wall clock.
Genome
Empryo's live map of your code: which functions call which, and which files change together. The agent reads the map instead of guessing from text search.

Don't trust our numbers. Run them.

The harness, tasks, hidden tests and raw results are public. With two API keys and one command you can rerun any round.

bun harness-real/run-real.ts --tiers haiku,opus
Benchmark harness on GitHubInstall Empryo