Skip to content
Empryo

I let agents loose on my own app. They found 84 bugs

105 claimed, 84 survived two hostile reviewers. Plus a fixed suite that says the same thing every time.

I let agents loose on my own app. They found 84 bugs

Your tests are green and a user still finds a bug in five minutes. Two like that got past me, and no test I'd have written would catch either.

The first was in the installer, on a fresh Debian machine with no Empryo, no Bun and no TERM, since nothing sets one in a container shell.

Downloaded. Checksummed. Signature verified against the pinned RSA-4096 key. Then dead, with nothing installed.
Downloaded. Checksummed. Signature verified against the pinned RSA-4096 key. Then dead, with nothing installed.

The script did everything right until it ran clear before its welcome message. clear needs TERM, so it failed, set -e stopped the script, and the verified download was never unpacked.

The second was worse: nothing failed. The desktop app shipped with no right-click menu in text fields, so you couldn't paste with the mouse, and no test checked for it.

My tests import my code and never run the installer, and a test only repeats a question someone already thought to ask. I built one QA system for both: the Immune system.

One system, two ways to run itLink to this section

The same seeds, the same real machines, the same two reviewers. What changes is who picks the targets.
The same seeds, the same real machines, the same two reviewers. What changes is who picks the targets.

I run it as a fixed suite. 260 checks across three operating systems and the three ways to use Empryo: desktop, terminal UI and headless. Same answer every time, so it's the half I trust.

An agent runs it to explore. Same machines and reviewers, but the agent picks what to test, uses the product like a person and writes the check it wishes had existed. It finds new problems, so it's the half I double-check.

Neither side can add its own findings to the suite, and the design depends on that.

What the fixed suite runsLink to this section

Each check runs on one platform and one way of using Empryo. A general sweep inspects every result for crashes and garbled output, whatever the check was about.
Each check runs on one platform and one way of using Empryo. A general sweep inspects every result for crashes and garbled output, whatever the check was about.

Each check, a seed, states what to exercise, on which operating systems, in which way of using Empryo, and one sentence naming the bug it guards against. A run executes every seed on every platform it applies to, in parallel.

Only the model is faked. A scripted stand-in replaces the provider, so the agent loop, tool calls and network traffic run for real, offline and free, on a real terminal, desktop window, Linux machine and Windows x64 machine.

The Windows box has to be real x64 hardware, because an ARM64 Windows VM can't load x64 native modules and a pass there is a pass for a product nobody ships. I only use the VM for smoke tests.

What mattered most is a general sweep over every result. It flags stack traces, terminal escape codes printed as text, locked-database errors and processes that had to be killed, and catches most surprises. A check about resuming an unknown session led me to an unrelated crash in the runtime's HTTP client that was killing about 40% of runs.

How an agent picks what to testLink to this section

Empryo's own code map turned out to be a QA tool.

Grep ranks by what you already thought to type. The graph ranks by how many callers reach a function and how far a defect there would spread.
Grep ranks by what you already thought to type. The graph ranks by how many callers reach a function and how far a defect there would spread.

With grep, an agent picks whatever it can spell, usually a file whose name sounds like the feature and rarely the one that matters.

The Genome shows which files depend on a file, which change with it and how far a change would spread. It finds unused exports and lists every caller that can reach a function, with file and line. Those callers also name the platforms and ways of using Empryo a bug can affect, so the tool that finds the bug fills in half the new check.

One rule matters most: no agent can start another agent. At every handoff the agent writes down what it found and a person starts the next step. Otherwise the suite would fill up with checks that fail on correct behavior.

Every check goes through review firstLink to this section

Two reviewers per claim, both hostile, both defaulting to refuted. One argues the behaviour is documented intent. The other reproduces by hand on the shipped build.
Two reviewers per claim, both hostile, both defaulting to refuted. One argues the behaviour is documented intent. The other reproduces by hand on the shipped build.

Agents that hunt for bugs want to find bugs 😂 So every claim gets two reviewers before it becomes a check. One asks whether the product documents the behavior as intended. The other skips the test setup and reproduces it by hand on the released build, in a throwaway folder, like a user. Both assume the claim is wrong.

One run came back with 23 failing checks on Windows. Five were real. Sixteen came from one outdated binary the test machine was stuck on, and two from checks with Unix-only assumptions. Throwing out those eighteen false alarms is the review's most valuable job.

A QA report that never rejects anything only shows lenient reviewers. The record only grows and lists rejections next to confirmations: 105 claimed, 84 confirmed, 19 rejected and 2 disputed.

What it caughtLink to this section

Three of them, all fixed, each with a permanent check:

what a user was losing
edit_file wrote through a read-only fileYou chmod 444 a generated file. The agent edited it anyway, was told the edit succeeded, and kept reasoning about a file it should never have touched.
Editing a file lost its permissionsThe edit replaced the file with a new one, so its permissions reset to the default. Every edited script quietly stopped being executable.
--resume on an unreadable session answered with no historyThe session file couldn't be read, so the agent started empty, answered confidently, and billed you for a turn that had forgotten the conversation.

None threw an error or a failure code. They looked like success, which is why unit tests missed them.

The first-day checks, which install Empryo on a clean machine and watch what a new user sees, found four more: the clear failure above, a tar: warning printed once per file because macOS had packed extra file attributes into the archive, a warning that Empryo needs Bun when it ships its own, and curl's progress bar spilling into piped logs.

A release build also failed its own startup check. The binary was fine, but before answering --version it cleaned up leftover language servers, and on a machine with old data that triggered a migration slower than the check allowed. The fix was moving one call.

What is not coveredLink to this section

373 capabilities the product exposes. 96 have a check aimed at them. The gap is the honest number, and it is the one that moves.
373 capabilities the product exposes. 96 have a check aimed at them. The gap is the honest number, and it is the one that moves.

Of Empryo's 373 reachable capabilities, 96 have a check, so a run passes straight over 277. The coverage list is generated from the product, so I can't quietly stop counting the gaps.

Of the eight stages from install to uninstall, three are tested end to end on a machine that has never seen Empryo. A stage only counts from a clean machine, because setting things up first hides the bugs a new user meets.

What I have not builtLink to this section

This system doesn't fix itself. Agents find defects and write the checks, and they're good at it. However, every run is a command someone types, every finding goes to a reviewer by hand, and every fix is a person deciding the right behavior. So I don't call it self-healing.

What it does is close the loop: a defect is found, challenged by a reviewer that defaults to no, fixed, then guarded by a check written from that exact defect. 84 confirmed defects across three rounds is a good start, and says a lot about what was in there 😅

Next is scheduled runs, so I don't have to remember to test. Then agents that hand off to each other under clear rules instead of through me, and a fix step that proposes a patch and proves it against the failed check. It stops there, and a person makes the final call.

Trying the idea on your own appLink to this section

None of this is specific to Empryo. Test the build you ship instead of the code you import, pick targets from a map of your code, don't let a claim become a test until a skeptical reviewer fails to knock it down, and publish what's not covered next to what is. In Empryo the code map is built in, so the checker can read the code it checks.

Current numbers, gaps included, are at empryo.com/immunity and update on every deploy.