A classifier inside the coding agent

We tried jev wherever a cheap, fast decision might save an expensive model call, then measured it against our own past sessions. One use clearly beat the code it would replace: deciding what to do when a model call fails. Two others, re-ranking grep results and reading review verdicts, were measured and removed. Behind every number here: 290 jev calls, $0.011 in total.

Re-ranking grep results

Live side-by-side runs told us nothing: the same prompt on the same model took 14 steps, then 6, then 9. So we replayed past sessions instead. We took every grep in 19,828 recorded tool calls, in Empryo’s own order, and used the files the turn went on to read as the answer key. We only scored searches where the answer was in the list, because the 32KB output limit decides that before any ranking happens.

21 searches fixed, 31 broken. The bar, set before the first call, was +15 points and three fixes for every break. This went the other way, so the code was written, measured and deleted. The useful finding was elsewhere: in 32.4% of searches the file the turn ended up reading was never in the output at all, which led to the bug fix below.

Reading review verdicts

A review agent’s report ends with VERDICT: PASS | FAIL | PARTIAL. When that line is missing, a model call works out the verdict, and jev looked like a $0.0001 replacement. We tested it on 70 real reports with the line removed, using the reviewer’s own verdict as the answer key.

Search output arrives in chunks, and each chunk was decoded as text on its own, so a character split across two chunks turned into garbage. Worse, the chunk that crossed the 32KB limit was thrown away whole instead of being trimmed.

55.6%1,742 of 3,134 capped searches, across 258,802 recorded greps, returned no matches at all. The model saw only “output capped” and treated that as the answer.

Both bugs are fixed: output is collected whole, trimmed to the limit and decoded once. The decoding bug never triggered in our data (0 garbled paths in 258,802 outputs, all ASCII), so the replay numbers above were not skewed by it.

When a model call fails, Empryo decides whether to retry, shrink the conversation or end the turn by matching the provider’s error message against patterns. Those messages change without notice, so the patterns need frequent fixes.

The one case jev catches and the patterns miss is a Codex error that reads Input exceeds the maximum length of 1048576 characters. It never says “context” or “token”, so recovery never started. jev labels it a context overflow with 0.96 confidence.

This shipped as error triage, off by default. The patterns still decide; jev is asked only when they have called an error permanent and the turn is about to end. It is never asked about sign-in or key errors, answers below 0.7 confidence are ignored, and a session with no failures makes no calls. Median time is 263ms.

Empryo gives the model a short, ranked map of your repository, most central files first. Across 87 local sessions, the top 120 entries included only 8.5% of the files those sessions went on to edit; the top 200 reached 13.1%, and only 2 sessions were fully covered.

In the pool test each session had 120 candidate files, the ones it edited plus random decoys, and jev saw only the first message. It found 80 of 278 against 59 for centrality, with one 348ms call per session. Better, but under our bar of +15 points and three fixes per break, so it is not shipped.

What did ship needs no classifier. Ranking the map by how often files change as well as how central they are raised coverage from 10.7% to 40.2% in the same 120 entries. Change frequency alone reaches 42.4% but leaves out the core types you need to find your way around. The history comes from git, which the Genome already reads.