What is an AI agent harness? Follow one code change

Ask an AI coding agent to rename UserService to AccountService. The model can suggest the new name in one reply. Finishing the job takes more work. The agent has to find the declaration, distinguish it from unrelated names, update references, run checks, and tell you what changed.

An AI agent harness is the software around the model that runs that process. It supplies context, exposes tools, executes permitted actions, records results, and decides when to call the model again. In a coding agent, it also connects that loop to your repository and development environment.

I build Empryo. I'll use a rename to explain what to check when a coding agent changes your code. The example is a small invented project, and the harness concepts apply across tools.

Follow the rename

Consider this small repository. This is an illustrative example, not a benchmark result.

src/accounts/user-service.ts     exports UserService
src/accounts/routes.ts          imports UserService
src/tests/user-service.test.ts   tests UserService
src/legacy/user-service.ts       exports another UserService

A search for UserService finds both declarations. The requested change concerns the accounts service. Replacing the string across the repository would also change the legacy service.

The harness determines which operations the model can use to resolve that ambiguity. A text search exposes matching lines. Code-navigation tools can help distinguish a declaration from its references. The agent still needs to establish which service you mean before making changes.

Those tools answer different questions. A useful agent can combine them and inspect the source when a tool leaves uncertainty.

Context: put the relevant code within reach

Before the model chooses an action, the harness assembles its input. That can include the request, repository instructions, previous tool results, and a map of the code.

Give the agent the scope it needs: which service to change, which behavior to preserve, and which checks the project uses. Before accepting the result, compare the files it changed with that scope.

Search and navigation have limits. Dynamic imports, runtime configuration, and generated code can hide relationships. Inspect those cases when the application uses them.

Context also has a cost. A large search result may stay in the conversation and contribute input tokens on later calls. When comparing tools, record total usage as well as the final answer. A short final reply does not tell you how much work preceded it.

Tools: make the edit the repository needs

For the rename, the agent needs to identify the accounts declaration and its references. Check that the resulting edits follow that symbol and leave the unrelated legacy service alone.

Code relationships matter in this example: two declarations can have the same name while belonging to different parts of the program. A tool that follows the intended symbol gives the agent a more precise starting point than a list of matching strings. It still needs to account for references that the language tooling cannot resolve.

In Empryo, you can work on that change from the desktop app or terminal. State the intended scope, then inspect the diff and run your checks. Language support varies across tools, and a command named rename does not guarantee a complete refactor.

After the edit, search for remaining occurrences. Some strings may need a human decision. A public JSON field, migration file, or documentation example can share the old name without referring to the code symbol.

Permissions: decide which actions may run

A tool call can modify files, run a shell command, or contact an external service. The harness has to enforce the permissions for that action before execution.

For this rename, inspect the planned scope. Editing the accounts files fits the task. Pushing a release or changing production configuration needs its own authorization. Tool availability alone should not grant it.

When comparing agents, check where they enforce restrictions. A prompt that asks the model to avoid a command provides a different boundary from a runtime that refuses to execute it. Read the product's permissions documentation and test the behavior in a disposable repository.

Verification: ask the compiler and tests

The model's final message cannot establish that the rename worked. Run the repository's type checker and the tests that exercise the accounts service. Inspect the diff for unrelated edits.

For a TypeScript repository with these scripts, that might look like this:

npm run typecheck
npm test -- user-service
git diff --check
git diff

Use your project's actual commands. Passing a focused test only establishes what that test checks. A rename can compile while breaking a string-based dependency lookup, so inspect those boundaries when the application uses them.

Session state: leave enough evidence to continue

Long tasks can outgrow one context window. The next model call needs to know which files changed, which checks ran, and what remains unresolved.

Anthropic describes a long-running harness that uses an initialization step and progress artifacts to carry work across sessions. Its engineering write-up is useful because it discusses failures as well as the design. Compaction alone did not make the agent finish a complex application reliably.

Empryo lets you return to saved sessions. Resuming the conversation is useful, but the repository may have changed while you were away. Before continuing a refactor, have the agent inspect the current diff and rerun the relevant checks. A saved result describes the earlier checkout.

For a refactor, a practical handoff records the target symbol, modified files, exact check results, and unresolved callers. Save this beside the work so the next session can verify it against the repository.

Choose a harness with your own task

Run the same bounded change from a clean checkout in each candidate. Record the model, version, configuration, time limit, and permitted tools. Keep the acceptance checks outside the agent's instructions until grading if you want to test whether it solves the problem from the symptoms.

Compare these outcomes:

For a shortlist of coding tools, start with the alternatives guide. For a cost-focused trial, use this evaluation worksheet.

To try the rename in Empryo, follow the quickstart and ask for one change in a repository you can restore.