There are 46 memories attached to one file in Empryo's own repo.
store.ts. The hot path. Every gotcha, every decision, every "never do X here again" eventually names it. So when the agent edits that file, which three of the 46 deserve the context window? Use count says: the same three as always. Similarity says: whichever ones share vocabulary with the prompt. Neither answer survives contact with real work.
That problem doesn't exist in a note-taking app. It only exists when your memory system is wired into a *codebase*. Which is the whole point of this post: Empryo's memory isn't a notes feature bolted onto an agent. It's wired into the Genome — the same file graph, git history, and edit stream the agent already navigates by.
Full disclosure as always: I build Empryo. Alone, nights and weekends, no funding. Every number below comes from the recall engine shipping in the current beta, measured against Empryo's own repo.
The markdown file taped to the prompt
The dominant memory system in coding agents today isn't a system at all. It's a markdown file — CLAUDE.md, AGENTS.md, "project rules" — injected wholesale into the system prompt on every single turn.
It works at ten lines. Then it compounds. Every rule you ever wrote rides along on every request, whether you're deep in auth or nudging a CSS margin. There is no relevance filter, so you pay tokens for all of it, always — and attention is the scarcer currency: fifty stale rules are fifty things competing with the actual task.
And markdown doesn't age. Nothing decays, nothing supersedes, nothing notices that the file a rule references was renamed two months ago. The file can't tell you which of its lines the agent has ever actually used, so it only grows. Staleness isn't an edge case of prompt-file memory; it's the steady state.
The other school is retrieval: memory MCP servers, vector-store plugins, ChatGPT-style memory. Embed the conversation, cosine against stored notes, inject top-k. Smarter — at least it filters. But retrieval is keyed entirely on what you said, which is the right design for a general assistant, because what you said is the only signal it has.
A coding agent has a better one: what you're touching. When the agent edits auth/routes.ts, a memory that names auth/routes.ts is relevant *by construction*. No embedding needed. No similarity guess. The file path is a foreign key, and general memory systems throw it away.
Worse, the query signal actively degrades in coding sessions. The classic failure: the user pastes a 3 KB stack trace with the actual question in the last line. Every token OR-matches something in full-text search, and the paste dominates the embedding vector. Empryo's recall sanitizes this before retrieval — strip code fences, hunt for the last question sentence, fall back to the last prose-looking line. Similarity search over raw coding conversations is retrieval over noise.
A memory is born tied to a file
Here's the loop from the user's chair.
You correct the agent once — "always use bun here, not npm" — and it writes the correction down itself: a one-line summary, the why in the details, a category (pref, decision, gotcha, or context), and the part that makes everything else in this post work: `file_paths`, the files the knowledge is actually about. You don't file anything. The agent does the remembering; the correction was the only work you did.
Next week, different session, cold context. The agent touches one of those files and the memory comes back on its own — before you asked, with a badge explaining *why* it surfaced:
[gotcha] ba35cc60 — Memory file-ref paths are normalized at the DB
boundary · via file, sem 0.59 ↳ has detailsThat via file, sem 0.59 is the receipt: this memory named the file being edited, and its meaning matched the task. Recall is never a black box — every surfaced memory shows which signals fired.
It works mid-session too. When the agent is about to edit a file that carries a known sharp edge, the warning lands inline on the tool result, worded as an instruction:
· gotcha "session save must go through safeRename" [ad63a294]
— review before editingAnd it's tuned to not nag: a surfaced memory goes quiet for 10 turns, hints stop entirely once the agent starts reading memory on its own, and there's a per-session budget so memory never becomes the loudest thing in the room.
When a memory matters enough that you never want to gamble on ranking, pin it. Pinned memories rank higher, pierce the hint budget, and are exempt from every decay mechanism described below. Pinning is the user's veto over the statistics.
Retrieval by what you touch
Empryo's recall scores every candidate memory with reciprocal-rank fusion over directional signals, plus bounded magnitude terms. The signals, in rough order of how much I trust them:
Direct file affinity. Memories carry file_paths. If one names a file you're editing right now, it enters the pool with a flat +0.25 magnitude — deliberately large. The comment in the source says it plainly: a memory attached to the exact file being edited must survive competition with weak semantic noise, not just contribute an RRF sliver.
Path rarity, as a premium. The answer to the 46-memory pile-up. Each file hit is weighted by how *rare* the path is: sole owner of an edited path earns a 0.40 bonus, a 46-way share earns 0.28. A memory that names the one file only it knows about outranks the crowd on the hot path — and rarity is strictly a premium on top of the floor, never a discount below it, so a file hit can never sink beneath text-match chatter. The reasoning is written into a comment block in recall.ts, because the next person to touch that constant will be tempted to flip it.
Git co-change expansion. Edit auth/routes.ts and recall also considers memories attached to auth/middleware.ts — files that historically change together, pulled from the Genome's git co-change graph. Capped at 2 neighbors per file, 6 total, and slotted at RRF rank 5, roughly 3× weaker than a direct hit. The memory arrives *before* you touch the second file. Similarity search cannot do this, because nothing in your query mentions the neighbor.
Blast radius. Memories tied to high-fan-in files get a small log-scaled boost. A gotcha on a file 60 others import matters more than one on a leaf.
Text still matters — two FTS5 indexes (unicode + trigram) with bm25 strength kept as a magnitude, plus a semantic channel. But text is *one voice in a committee*, not the whole ballot.
And the semantic channel is deliberately boring: a deterministic 384-dim hash-bag embedder. No model call, no API bill, no network, works offline. It's crude — cosine below 0.3 is treated as shared-topic noise and earns rank credit only, because ungated, that noise outscored file-affinity hits on vague queries. A provider embedder can slot in when configured. But the system was designed so the *strong* signals never depend on an embedding being good.
One more scoring choice worth naming: recency anchors on the last time the agent acted on a memory, not the last time it was shown. Anchor on showings and a memory can stay "fresh" forever just by being displayed — shown often because it scores well, scoring well because it's shown often. Anchoring on action breaks the loop. Being shown keeps nothing fresh; being useful does.
The repo moves. Memory moves with it.
This is what tying memories to files buys you that no prompt file or vector store can offer: the repo's own life becomes the staleness signal.
Delete or rename a file and every memory attached to it loses its strongest card *immediately* — the file-affinity boost fires on paths that exist in the Genome's graph, so a memory pointing at a gone file simply stops winning recall. No migration, no cleanup job in the hot path. The rank-down is by construction.
The dead reference doesn't linger invisibly either. Memory maintenance scans for records whose every named path has vanished from disk and flags them as dead-ref candidates for the next cleanup pass — the exact failure a rules file can't detect, the note about src/api/client.ts still asserting itself two months after the file became src/api/http.ts.
And because paths resolve through the Genome rather than string-matching, the connection is live in the other direction too: a memory on auth/middleware.ts rides the git co-change graph to surface when you edit auth/routes.ts, and a memory on a file with 60 importers carries more weight than one on a leaf. The repo's structure isn't metadata about memory. It's the ranking function.
Memory that notices it's being ignored
That distinction — surfaced vs. acted — runs through the whole system, and it's the part I haven't seen anywhere else.
Every memory tracks both counts. Surfaced 4+ times with zero follow-ups? Penalty grows to −0.12 and it steps back from rotation. Actually pulled by the agent? Small log-scaled bonus. Keep ignoring one long enough — six surfacings across 45 days with zero follow-ups — and it soft-archives itself at the next session start: hidden from recall, fully reversible, never silently destroyed. The system runs a feedback loop on its own suggestions. This is the staleness answer a prompt file structurally can't give — a markdown line has no counter for "read 400 times, followed zero." Here, that memory demotes itself, then retires itself.
Contradicted knowledge doesn't linger either: write the new rule and the old one can be superseded — it drops out of recall but stays in the record, so you can always see what changed and when.
Context is the budget that matters, so injection is stingy by design: top 3 results, roughly 2,400 characters, and only the one-line summary of each — full details are fetched on demand. Mid-session hints have a 10-turn cooldown per memory, a session budget, and go silent the moment the agent runs a memory action itself, because nagging someone already in memory-aware mode is noise.
You stay in charge
Everything above is automatic, but none of it is opaque. /memory opens a browser over both stores — every record listable, searchable, pinnable, deletable, restorable. /memory stats shows the feedback telemetry per category: how often memories get surfaced and how often the agent actually follows through. And when the store has genuinely aged — 20+ sessions since the last pass, 30+ active memories, 10+ stale candidates — a cleanup hint appears with the worst offenders pre-ranked by age and disuse. Pinned memories are excluded from every one of those lists. Curation is one keypress, not an archaeology project.
Memory is also scoped. Project memory lives with the repo; global memory follows you across repos for the preferences that are about *you*, not the codebase. Both are readable at recall time, deduped by content hash so the same knowledge saved twice never eats two of the three injection slots — project wins ties because it's the narrower claim.
Nothing gets written behind your back
Empryo's memory holds only what was explicitly written — by you, or by the agent in front of you. Four categories, every write visible in the transcript, no silent extraction of your conversations into a store you never audit.
There *is* an auto-distill pass, because explicit-write-only systematically under-collects: the session solves something hard, nobody says "remember this," the knowledge evaporates. But it's off by default, capped at 3 memories per session, allow-listed by category, deduped by content hash, and every distilled row is tagged auto-distilled so you can list or purge them wholesale. An accelerant, not a dependency.
The one-line version
General memory systems ask: *what does this conversation sound like?*
Code memory should ask: *what is the agent touching, what changes together with it in git, and did the last suggestion actually get used?*
The first question needs a good embedding model. The second needs your repo — the file graph, the git history, the edit stream. That's data a coding agent already has and a general memory system never will. Not using it is leaving the strongest retrieval signal on the table.
The whole engine is a few hundred lines of scoring over SQLite. No vector database, no embedding service, no memory SaaS. It just reads the diff.