Compare AI coding agents on your own repository

You give two AI coding agents the same bug report. One finishes in five minutes and leaves a failing test. The other takes nine minutes and produces a patch you can merge. A speed chart would rank the first one higher.

If you are choosing between Claude Code, Codex, Cursor, OpenCode, or Empryo, run a small trial on code you maintain. Measure three things: would you accept the change, how much review did it need, and what did the whole attempt cost.

I build Empryo, so treat this as a method, not a verdict. It helps you compare agents on your own tasks and your own standards, and decide from your own results.

Pick three tasks you can gradeLink to this section

Use a bug fix, a refactor, and a small feature. Choose work with clear boundaries and an observable result.

For the bug, write a failing regression test before the trial. For the refactor, record the intended files and behavior that must stay the same. For the feature, write down the acceptance criteria before either agent starts.

Avoid tasks whose answer is already in the checkout. If it matters for your trial, remove the fix commits from the history the agent can see. A recent task lowers the chance that a model has memorized the fix, but a date can't prove it.

Keep the grading tests outside the agent's workspace during the run and add them afterward. That way you check the result against tests the agent couldn't change.

Choose what you are comparingLink to this section

There are two useful kinds of trial:

• A purchase decision. Use each product with the setup and plan you would actually use at work. Write down those differences, because they affect the result.
• The harness. The harness is everything around the model: the context it gets, its tools, and the loop that runs them. Keep the model and provider the same where the products allow it, so the harness is the only thing that differs.

An editor, a terminal agent, and a cloud agent fit into your day differently. Count setup and review time when that difference matters to you. And if the agents used different models, don't present the result as proof that one harness is better.

See the agent harness guide for the parts of that loop to inspect.

Freeze the starting pointLink to this section

Write down the baseline commit, lockfile, runtime versions, operating system, and test commands. Install dependencies and confirm the baseline fails for the reason you expect. A missing package is not the bug you meant to test.

Give each attempt its own checkout. Use the same prompt for every agent on a given task. Decide on a time or spending limit and a retry rule before you start. If you allow a second attempt, the failed first attempt still goes in the results.

This record is enough to start:

Task:
Baseline commit:
Agent and version:
Model and provider:
Configuration and permissions:
Prompt:
Time limit:
Retry rule:
Acceptance commands:

Keep secrets out of the record. Set up credentials the way each product normally expects, and use accounts with only the access the trial needs.

Grade the patch before reading the explanationLink to this section

Run the acceptance tests and read the diff. Check whether the agent disabled a test, edited a fixture to match its answer, or left debug output behind. Run your project's usual checks for the affected code.

Only then read the agent's explanation. Check claims like "all callers updated" against the code. Note any fix you had to make before you'd accept the patch.

Use a separate result for each attempt:

Repeat the tasks if you can afford to. One run can show a failure, but it tells you little about consistency. Keep every individual attempt, so an average can't hide timeouts.

Count the whole billLink to this section

Include input tokens, output tokens, cache charges, and any models used by subagents. Where the provider shows usage, compare it with the tool's own meter. If a run stops before it reports its final usage, mark the cost as incomplete until you've checked the bill.

Keep API costs separate from subscription usage. A flat monthly plan doesn't give you a reliable dollar price for one task. Record the plan, any quota use it reports, and whether a limit cut the run short.

For API trials, calculate:

Cost per accepted fix = total cost of all attempts / accepted fixes

If no attempt passes, report zero accepted fixes and the total spent, not a cost per fix of zero. Record your review time next to the token bill. Your time can easily cost more than the model.

Empryo shows token usage and cost estimates as it runs. Check them against your bill, especially with custom providers or subscriptions.

Read competitor comparisons as a shortlistLink to this section

Feature pages help you pick candidates. They can't tell you whether an agent will follow your repo's conventions or find your least obvious bug.

Our comparison pages cover Claude Code, Cursor, and OpenCode. Use them to find differences worth testing. Check current features and pricing with each vendor before you buy.

Keep your attempt records next to your final decision, including failures and how small the sample was. Let your accepted patches and review notes decide which tool you keep.