Compare AI coding agents on your own repository

Suppose you give two AI coding agents the same bug report. One finishes in five minutes and leaves a failing test. The other takes nine minutes and produces a patch you can merge. A speed chart alone would put the first agent ahead.

If you are choosing between Claude Code, Codex, Cursor, OpenCode, or Empryo, start with a small trial on code you maintain. Measure whether you would accept the change, how much review it needs, and what the complete attempt costs.

I build Empryo. Use this guide to compare coding agents with your own tasks and acceptance criteria, so you can decide from your own results.

Pick three tasks you can grade

Use a bug fix, a refactor, and a small feature. Choose work with clear boundaries and an observable result.

For the bug, write a failing regression test before the trial. For the refactor, record the intended files and behavior that must stay the same. For the feature, write down the acceptance criteria before either agent starts.

Avoid a task whose answer already appears in the checkout. Remove fix commits from the history the agent can access if that matters to your experiment. A recent task can reduce the chance of memorization, but its date cannot prove that no model has seen it.

Keep the grading tests outside the agent's workspace during the run. Drop them in afterward. This gives you a way to check the result against expectations the agent could not rewrite.

Choose what you are comparing

There are two useful trials:

• Your purchase decision. Use each product with the setup and plan you would use at work. Record those differences because they affect the result.
• The harness. Hold the model and provider settings constant where the products allow it. Then compare the surrounding context, tools, and execution loop.

An editor, a terminal agent, and a cloud agent have different workflows. Count setup and review time when those differences affect your day. Do not describe a test of different models as proof that one harness is better.

See the agent harness guide for the parts of that loop to inspect.

Freeze the starting point

Record the baseline commit, dependency lockfile, runtime versions, operating system, and test commands. Install dependencies and confirm the baseline fails for the intended reason. A missing package is not the bug you meant to test.

Give each attempt a separate checkout. Keep the prompt identical within each task. Set a time or spending limit in advance, along with a rule for retries. A failed first attempt still belongs in the results if you allow a second attempt.

This record is enough to start:

Task:
Baseline commit:
Agent and version:
Model and provider:
Configuration and permissions:
Prompt:
Time limit:
Retry rule:
Acceptance commands:

Keep secrets out of the record. Store credentials through the product's normal configuration and use accounts with appropriate access for the trial.

Grade the patch before reading the explanation

Run the acceptance tests and inspect the diff. Check whether the agent disabled a test, changed a fixture to match its answer, or added debug output. Run the project's normal checks for the affected area.

Then read the explanation. Verify claims such as “all callers updated” against the code. Record any correction you had to make before accepting the patch.

Use a separate result for each attempt:

Repeat the tasks if you can afford it. One run can expose a failure but gives you little information about consistency. Publish the individual attempts so an average cannot hide timeouts.

Count the whole bill

Include input, output, cache charges, and any models used by subagents. Reconcile the tool's meter with provider usage when the provider makes that possible. If a run stops before it reports its final usage, mark the cost as incomplete until you can check billing.

Keep API costs separate from subscription allowances. A flat monthly plan does not give you a reliable dollar price for one task. Record the plan, any reported quota use, and whether a limit interrupted the run.

For API trials, calculate:

Cost per accepted fix = total cost of all attempts / accepted fixes

If no attempt passes, report zero accepted fixes and the total spent. Do not report a zero cost per fix. Keep human review time beside the token bill because your time can exceed the model cost.

Empryo shows usage and model cost estimates. Check estimates against billing, especially for custom providers or subscription routes.

Read competitor comparisons as a shortlist

Feature pages can help you choose candidates. They cannot tell you whether an agent will handle your repository's conventions or your least obvious bug.

The Empryo comparisons cover Claude Code, Cursor, and OpenCode. Use them to identify differences worth testing. Verify current product behavior and pricing with the vendor before making a purchase.

Keep your attempt records beside the final comparison. Include failures and the limits of the sample. Your accepted patches and review notes should decide which tool you keep.