Skip to content
Empryo

NVIDIA's SoL-Pi pruning cost me 30% more

Ran it on real Hono bugs and every pruning rule lost to one boring habit: don't touch what the model already saw.

NVIDIA's SoL-Pi pruning cost me 30% more

A coding agent sends the whole conversation on every request. The provider caches the start, so the unchanged part costs a tenth of the price. So one habit beats any clever trimming: keep the start of the conversation exactly the same. That's a stable prefix.

NVIDIA's SoL-Pi is four cost-saving add-ons for the Pi coding agent. One, ObservationPack, does the opposite: it rewrites the middle of the conversation to shrink it. I ported it into Empryo line for line and fixed three real bugs 36 times to measure it.

How the cache bills youLink to this section

Anthropic bills a cached token at a tenth of the input price, and a token written to the one-hour cache at twenty times that. Everything after the first changed block counts as new and is written again.

How far back the removed result sits decides the size of that bill:

A removal only pays for itself after enough requests: tokens billed again × 19 ÷ tokens removed.

One extra request to fetch something back also cancels about 43 requests' worth of savings.

What ObservationPack doesLink to this section

Any tool result over 10 KB, such as a file the agent read or a test log, stays in full for two requests, then shrinks to a stub.

The testLink to this section

35 of 36 runs fixed their bug. The one miss, an ObservationPack run on the router bug, failed 1 of 644 tests.

The result: two losses and a tieLink to this section

Why it lostLink to this section

On the form data bug, ObservationPack stubbed src/request.ts, a 16.8 KB file, two requests after the agent read it. The agent was still editing it, so it read it again in small pieces, over and over 😅 The recall tool didn't help, because it returns the copy from before the agent's edits.

Each request cost about 1 cent on both sides. ObservationPack made more of them, and every prune rewrote the cache.

Other pruning rules I triedLink to this section

Before Hono I tried gentler rules on a synthetic project with 19 planted bugs. None beat the stable prefix.

The only rule that didn't lose removed results so new the cache hadn't moved past them. Each request got 16% cheaper, the agent made 17% more of them, and the total came out even. Every rule reaching further back cost more.

Their numbers and mineLink to this section

NVIDIA tested ObservationPack on its own before shipping it (write-up, paper). Same idea, opposite result:

NVIDIA's testMy test
AgentPiEmpryo
Tasks11 long EdgeBench tasks3 Hono bug fixes
Big resultsKept in fullTrimmed on arrival
Stub2 KB + 1.5 KB512 B + 512 B
Bill23.6% lower30% higher

Across all four add-ons, they report 45 to 49% fewer tokens than plain Pi, about a third lower cost and about 94% of Pi's score. I'm not claiming SoL-Pi is wrong about Pi. The setups differ in three ways:

  • Pi keeps large results in full. Their write-up says a large file "reappeared in every later request" in plain Pi. Empryo trims it the first time, so there is less to save.
  • Their tasks are long. They note savings on shorter tasks "have little time to accumulate". A bug fix here takes 20 to 90 requests.
  • Their stub was bigger. A 3.5 KB stub still hides most of a 16.8 KB file the agent is editing.

What I do insteadLink to this section

  • Trim big output before it goes in. The cache never sees the full text, so there's nothing to rewrite later.
  • Never edit what the model has already seen while the cache is warm.
  • Shrink once, not every turn. When a conversation must get smaller, summarize it in one step instead of trimming a little on every request.

I'm not shipping ObservationPack. SoL-Pi's Action Fusion runs the test command in the same step as the edit and leaves the history alone, so I'll try that next.