NVIDIA's SoL-Pi pruning cost me 30% more
A coding agent sends the whole conversation on every request. The provider caches the start, so the unchanged part costs a tenth of the price. So one habit beats any clever trimming: keep the start of the conversation exactly the same. That's a stable prefix.
NVIDIA's SoL-Pi is four cost-saving add-ons for the Pi coding agent. One, ObservationPack, does the opposite: it rewrites the middle of the conversation to shrink it. I ported it into Empryo line for line and fixed three real bugs 36 times to measure it.
How the cache bills youLink to this section
Anthropic bills a cached token at a tenth of the input price, and a token written to the one-hour cache at twenty times that. Everything after the first changed block counts as new and is written again.
How far back the removed result sits decides the size of that bill:
A removal only pays for itself after enough requests: tokens billed again × 19 ÷ tokens removed.
One extra request to fetch something back also cancels about 43 requests' worth of savings.
What ObservationPack doesLink to this section
Any tool result over 10 KB, such as a file the agent read or a test log, stays in full for two requests, then shrinks to a stub.
The testLink to this section
- Project: Hono, a web framework with a real test suite.
- Bugs: three recent fixes, each started from the commit just before the fix: form data read after text(), ETag changes with stream chunks and wildcard routes that match too much.
- Grading: the tests from the real fix, added after each run, unseen by the agent.
- Model: Claude Haiku 4.5 for every run.
- Runs: 6 per bug per side, side by side, 36 in total. Every number below is a median.
35 of 36 runs fixed their bug. The one miss, an ObservationPack run on the router bug, failed 1 of 644 tests.
The result: two losses and a tieLink to this section
Why it lostLink to this section
On the form data bug, ObservationPack stubbed src/request.ts, a 16.8 KB file, two requests after the agent read it. The agent was still editing it, so it read it again in small pieces, over and over 😅 The recall tool didn't help, because it returns the copy from before the agent's edits.
Each request cost about 1 cent on both sides. ObservationPack made more of them, and every prune rewrote the cache.
Other pruning rules I triedLink to this section
Before Hono I tried gentler rules on a synthetic project with 19 planted bugs. None beat the stable prefix.
The only rule that didn't lose removed results so new the cache hadn't moved past them. Each request got 16% cheaper, the agent made 17% more of them, and the total came out even. Every rule reaching further back cost more.
Their numbers and mineLink to this section
NVIDIA tested ObservationPack on its own before shipping it (write-up, paper). Same idea, opposite result:
| NVIDIA's test | My test | |
|---|---|---|
| Agent | Pi | Empryo |
| Tasks | 11 long EdgeBench tasks | 3 Hono bug fixes |
| Big results | Kept in full | Trimmed on arrival |
| Stub | 2 KB + 1.5 KB | 512 B + 512 B |
| Bill | 23.6% lower | 30% higher |
Across all four add-ons, they report 45 to 49% fewer tokens than plain Pi, about a third lower cost and about 94% of Pi's score. I'm not claiming SoL-Pi is wrong about Pi. The setups differ in three ways:
- Pi keeps large results in full. Their write-up says a large file "reappeared in every later request" in plain Pi. Empryo trims it the first time, so there is less to save.
- Their tasks are long. They note savings on shorter tasks "have little time to accumulate". A bug fix here takes 20 to 90 requests.
- Their stub was bigger. A 3.5 KB stub still hides most of a 16.8 KB file the agent is editing.
What I do insteadLink to this section
- Trim big output before it goes in. The cache never sees the full text, so there's nothing to rewrite later.
- Never edit what the model has already seen while the cache is warm.
- Shrink once, not every turn. When a conversation must get smaller, summarize it in one step instead of trimming a little on every request.
I'm not shipping ObservationPack. SoL-Pi's Action Fusion runs the test command in the same step as the edit and leaves the history alone, so I'll try that next.