Skip to content
Empryo

One search cost $1.14, so I rebuilt the engine

86,000 tokens from a single search 😂 Rebuilt it and metered every dollar across six models.

One search cost $1.14, so I rebuilt the engine

One search result cost $1.14.

Not the reasoning, not the fix. One search dumped 86,000 tokens into the conversation when the model needed two lines of it. You pay for tool output once when it arrives, and again on every step until the session ends.

From the request log: tool output does not disappear after it is read. It is sent again, and billed again, on every later step.
From the request log: tool output does not disappear after it is read. It is sent again, and billed again, on every later step.

That receipt is why I rebuilt Forge, the engine that decides what context and tools Empryo sends the model on each step.

Every number below comes from recorded runs: Round 1, Round 2 and the new Round 3 results.

The testLink to this section

Four bugs, really reported and fixed in hono and opencode. The tests from each fix's pull request were hidden from both agents, and the git history was removed so the answer could not be looked up. Six models, two providers and 96 metered runs.

pi in its minimal setup and Empryo in its default setup: what each agent sends the model before any work starts.
pi in its minimal setup and Empryo in its default setup: what each agent sends the model before any work starts.

Partway through, pi's cost meter turned out wrong for GPT models, so I upgraded pi before comparing anything.

What the bill showedLink to this section

Forge v1 gave every model the same context on every step: the whole code map and every tool's full description. The request logs showed most of it was paid for and never used. The new engine changes the delivery by model.

Forge v2 in one picture: strong models get a short overview, small models get more guidance and a helper to scout the code, and every tool result has a size budget.
Forge v2 in one picture: strong models get a short overview, small models get more guidance and a helper to scout the code, and every tool result has a size budget.

Same bugs, same models, same keys. Switching engines alone cut cost by 56% on Opus and 46% on GPT-5.6 terra. GPT-5.6 luna went from fixing half the bugs to fixing all of them.

I also wrote some hints tuned to the benchmark, and they worked. 🫢 I deleted them, because an engine tuned to a test scores better and works worse.

One run that shows the ideaLink to this section

Two files with the same name and one bug. A text search has to guess which file matters. The code structure tells you.
Two files with the same name and one bug. A text search has to guess which file matters. The code structure tells you.

A text search has to guess. The code structure does not, because functions call other functions and some files always change together.

The resultsLink to this section

Round 3 wins. The full results, including every loss, are on the results page.
Round 3 wins. The full results, including every loss, are on the results page.

Some results did not go my way. On cheap and mid-priced models, pi's small prompt keeps the cost advantage. When a million tokens costs a quarter, a twenty-line prompt is hard to beat. pi's totals are consistent between runs, while mine vary: on Sonnet, $1.20 in one run and $1.47 in another. Making the cost more consistent is the next goal.

The stronger the model, the more a code map pays off. Frontier models with a map take fewer and better steps, and at frontier prices that is real money. Small models do better with the map in smaller pieces, so delivery matters as much as the map.

The tasks, hidden tests and metering are in the public benchmark repo. Current Empryo builds run Forge v2, and in the desktop app the feather icon next to the prompt box shows it is active.