How to Compare AI Coding Agents: Speed, Cost and Working Code
A coding agent that produces tokens quickly is not necessarily the one that finishes a task quickly. It can spend less time generating its first patch and more time correcting mistakes, repeating tool calls or waiting for a human to repair the result.
For a practical comparison, measure the completed job: did the requested behaviour work, how long did it take, and what did all the attempts cost?
That does not make token speed irrelevant. It makes token speed one component of a larger workflow rather than a substitute for its outcome.
Start with the same task and a clear definition of success
Write the acceptance criteria before running either agent. “Build a working page” leaves too much room to reinterpret a failure as a successful demonstration.
A more useful task might require a form to submit with a keyboard, display validation errors, remain usable on a narrow screen and preserve an existing URL. The evaluator should exercise those behaviours, not merely confirm that the agent produced files.
For a repository bug, reproduce the defect and define the expected corrected behaviour. Where appropriate, include regression checks for unaffected behaviour. Do not use the agent's final statement as the pass condition.
This outcome-based approach is also visible in established benchmarks. The SWE-bench evaluation guide describes applying generated patches to repositories and running tests in a Docker-based environment. That gives a concrete meaning to success on those tasks. It does not establish that the same score predicts every browser workflow or every private codebase.
Compare configurations, not just model names
A coding agent includes more than its language model. Its tools, instructions, context management, permission policy and retry strategy can change the result.
Record at least:
| Item | Why it matters |
|---|---|
| Model identifier and available revision | A shared product label can hide a changed model |
| Agent and tool versions | The surrounding execution system affects what the model can do |
| Starting repository snapshot | Different inputs create different tasks |
| Instructions and context | One agent may otherwise receive additional guidance |
| Attempt and time budgets | Unlimited retries are not comparable with a single attempt |
| Cache and execution conditions | Reused context, dependencies or results can change time and cost |
| Human interventions | A repaired result is different from an independently completed task |
An identical setup is not always possible across products. Report the differences instead of describing an unequal comparison as controlled.
Measure end-to-end time, not just generation speed
Keep distinct measurements for:
- Time to first response: how long the user waits before anything arrives.
- Generation speed: tokens produced per second under a stated measurement method.
- End-to-end task time: from the start of the attempt until its result is verified or the budget expires.
- Human review and repair time: effort required after the agent's own work.
Tool execution, builds, tests, retries and clarification requests can dominate the elapsed time. For interactive work, a short first response may improve the experience without reducing the time needed to reach a correct result.
Report what your timer includes. If human review is tracked separately, do not quietly omit it when describing the total effort needed to use the result.
Count failed attempts in the cost
For a fixed evaluation batch, a useful descriptive measure is:
Cost per successful task = total measured batch cost ÷ number of successful tasks.
Include unsuccessful attempts in the numerator. State which costs are included: model charges, tool fees, execution infrastructure and human effort are different categories and should not be silently mixed.
Here is a fictional example, not a model result:
| Configuration | Tasks attempted | Tasks passed | Total model charges | Model charges per passed task |
|---|---|---|---|---|
| Agent A | 5 | 3 | $1.20 | $0.40 |
| Agent B | 5 | 5 | $1.50 | $0.30 |
Agent A costs less for the batch but more per successful task under this accounting. That does not make Agent B universally better: the sample is tiny, and it says nothing about other tasks or unmeasured costs. If no tasks pass, report zero successes and the expenditure rather than presenting a finite cost per success.
Preserve failures and distinguish their causes
Reset the task state between independent attempts and record every attempt under the declared budget. Do not keep only the best-looking run or quietly grant one configuration additional hints.
Retain patches, tool traces, test outputs and relevant runtime evidence. A failed infrastructure setup and a patch that breaks the requested behaviour are different observations. Report both; neither should disappear from the results simply because it is inconvenient.
The SWE-bench guide makes this distinction explicit in its evaluation reporting, separating unresolved tasks from instances with evaluation errors and documenting the logs needed to investigate them.
For interactive features, add direct interaction checks. A passing unit test does not establish that a real user can complete the workflow in the browser.
Use public scores to form a shortlist
A leaderboard is useful evidence about the tasks and conditions it evaluated. Before applying its result to your work, check the dataset, agent configuration, scoring method, attempt policy and available reproduction material.
Then test a small set of representative tasks from your own workflow. The site's tool-use evaluation protocol gives an example of specifying such tasks. It is a prepared protocol, not a set of measured model results.
The practical question is not which agent has the largest number next to its name. It is which configuration completes your work reliably, within an acceptable budget, with enough evidence that someone else can check the result.
