Engineering note · 2026-09-16
A classifier inside the coding agent
TypeSafe’s jev answers typed questions. You send it a state and a list of questions, it sends back labels with probabilities. No text, nothing to parse. We wired it into the places in Empryo where a cheap, fast judgement could save an expensive one, then measured it against our own history. One recipe stayed. Two came back out.
What it costs, next to what it replaces
That ratio is the whole argument. A judgement costing a four-thousandth of a turn can be wrong sometimes and still pay, as long as it is wrong in a direction that costs nothing.
One turn, A to Z
Orange is jev. Blue is the model you pay real money for. The rest was already there.
The measurement that killed the grep recipe
Live A/B runs told us nothing. Three runs of one prompt on the same model took 14 steps, then 6, then 9 — its own variance is larger than anything a re-ranker does. So we replayed history instead: every grep in 19,828 recorded tool calls, its candidate files in the harness’s own order, and the files the turn went on to read as the answer key. Scored only where the answer was in the list, because the 32KB output cap decides membership before any ranker sees it.
21 searches fixed, 31 broken. Top-3 went 78.6% to 74.1%. The bar set before the first call was +15 points and three fixes per break. This is the other direction, so grep re-ranking is not wired into Empryo. The code was written, measured and deleted.
The useful finding came free, and it was not about ranking. In 32.4% of searches the file the turn ended up reading was never in the output. Chasing that led to the section below, which is the only thing on this page that changed the product.
Then the review recipe
An Empryo reviewer ends its report with VERDICT: PASS | FAIL | PARTIAL. When it forgets, we pay a model to restate it. jev looked like a $0.0001 replacement, so we tested it on 70 real review reports that do carry the line, with the line cut off. The reviewer’s own word is the answer key.
A wrong FAIL costs one extra coder iteration. A wrong PASS closes the loop on broken code. Two false passes in twenty is disqualifying at any price, so this one came out as well.
The bug the classifier found without being asked
Asking why a search missed the file meant reading how search output is collected. Two faults sat in ten lines of genome-grep.ts. Output arrives in chunks; each was decoded on its own and concatenated, so any character spanning a chunk boundary became two replacement characters — in a path, a file the model cannot open. And the chunk that crossed the 32KB cap was discarded whole instead of trimmed to the budget.
1,742 of 3,134 capped searches (55.6%) returned no matches at all — just the words “output capped” — across 258,802 recorded greps. The model read that as an answer. Both faults are fixed: bytes are kept as bytes, trimmed to the budget, decoded once.
The decode fault never fired here: 0 replacement characters in any parsed path field in those 258,802 outputs. Every path in this repo is ASCII. It was latent, not harmless — and it means the earlier replay numbers were not depressed by corruption, which is what we assumed for an afternoon.
Where it finally won: reading a failure
When a model call fails, something has to decide whether to retry it, shrink the conversation, or end the turn. That decision is made by regular expressions over the provider’s error prose. Ten commits in two months on that one file, every one a fix(), is what reading English with regex costs.
The regexes were written FROM these 24 strings, so they are the control, not a rival: scoring 23 of 24 on its own training set is the ceiling, not a win. jev had never seen any of them. The case it catches and they miss is a Codex overflow reported as Input exceeds the maximum length of 1048576 characters — no “context”, no “token”, so recovery never fired and that session died. jev calls it an overflow at 0.96.
Shipped as error triage, off by default. The regexes stay the mechanism; jev is asked at one point only — they have said “permanent”, the turn is about to end, and being wrong costs all of it. Never for an auth failure, never under 0.7 confidence, and a healthy session makes zero calls. 263ms median.
The map in the prompt is mostly not the code you touch
Every system prompt carries a ranked, truncated map of the repository, ordered by PageRank. Across 87 local sessions, its top 120 entries contained 8.5% of the files those sessions went on to edit; the top 200 reached 13.1%, and 2 sessions of 87 were fully covered. Centrality is not where the work happens. You pay for that map on every turn.
Controlled test: 30 sessions, a pool of 120 candidates per session (the files it edited plus random decoys), both rankers picking the same number, jev seeing only the session’s first message. jev found 80 of 278 (28.8%) against PageRank’s 59 (21.2%), in one call of 348ms per session.
Better, and still not shipped. The bar this project uses for a retrieval change is +15 points and three fixes per break; this is +7.6 points at roughly two to one, on 30 sessions with wide per-session swings. It is written down, not wired — the harness is in the repository and the next move is more sessions, not a switch.
What did ship came out of the same question, with no classifier in it. Rank the map by how often a file changes instead of how much the code leans on it, and coverage of the files a session edits goes from 10.7% to 40.2% in the same 120 entries (43 sessions, churn built only from commits predating each one; churn alone reaches 42.4%, and is not used because a map with no central types in it orients nobody). That is the Genome’s own git history, mined by the same walk that already builds co-change.
Three rows we did not build, and why
Each one was going to be next. Each died on its own data before a single call was spent.
Across 431,003 ranged reads, a turn had to come back for a span outside everything it had already asked for 1.3% of the time, and 85.2% of edits landed inside a range the turn had already read (n=52,565). A perfect oracle wins about one point. The calls cost more than the re-reads.
The session log keeps the memories that were injected, not the ones that lost. There is no candidate set to re-rank, and the scorer's own inputs are cumulative counters that already absorb every later follow-up, so replaying history against it scores the classifier on its own answer key. Prospective capture first, or nothing.
A wrong drop is invisible and unrecoverable: the parent never learns what it did not receive. Same objection that ruled out compaction.
What is actually wired
| where | state | evidence |
|---|---|---|
| find / symbol search | shipped, off by default, shadow mode | top-5 59% to 73% on 80 prompts — one point under its own bar |
| skill suggestion | shipped, off by default | no measured gain yet |
| error triage on a failed call | shipped, off by default | 24/24 against the regexes' 23/24 on real incidents |
| grep | not wired | top-3 78.6% to 74.1% on 220 replays |
| review verdicts | not wired | 60% agreement, 2 false passes in 20 |
| prefetching files for the model | not wired | right file first 35% vs 32% — two wrong files in three |
| read ranges | no headroom | 85.2% of edits already land in a range the turn read |
| memory recall | not measurable yet | the log keeps the winners, not the candidates |
| ranking the map in the prompt | measured, not shipped | 28.8% vs PageRank's 21.2% — real, under the bar |
| the 32KB search cap | fixed in our own code | 55.6% of capped searches returned nothing — no model call involved |
| compaction | ruled out | a wrong drop is unrecoverable and there is no baseline |
| tool pre-loading | ruled out | moving the tools array invalidates the cached prefix |
If you want to run it
It is off until you turn it on and give it a key. /typesafe in the terminal carries the key and every switch; on the desktop it is the TypeSafe section in settings. Every call shows up in chat as a side-call row saying what it decided and what it cost, and again in /usage afterwards. Setup is in the docs.
Behind this page: 290 jev calls, $0.011 total, 244ms median and 283ms at p90. The live A/B ran on Claude Haiku 4.5; the replayed reviews were written by Opus 5.