Data page · coding agents

AI coding agent benchmark: 50 tasks, 10 agents, four metrics

A reproducible 50-task protocol for coding agents, with published Terminal-Bench reference points for 10 configurations. Pass rate and cost are reported where public; time and interventions are labelled when they are not.

Snapshot. Baseline established 20 November 2026. The 50-task suite and scoring protocol are published here; no Plugsky-run results exist yet. The reference table reports published Terminal-Bench 4.1 results for 10 agent configurations, not results on this suite. Time and intervention counts are marked “Not reported” where the source does not publish them.

Published reference points for 10 agent configurations

These rows come from the Terminal-Bench 4.1 leaderboard, retrieved on 20 November 2026. They are useful for calibrating what a current agent can do on a terminal-heavy task suite. They are not results on the 50-task suite below, and the cost column is the total cost reported for the full Terminal-Bench run, not a cost per task.

Terminal-Bench 4.1 published results, retrieved 20 November 2026. Resolution rate is the share of tasks solved. Cost and tokens are totals for the full run. Time and interventions are not published by the leaderboard.
AgentModel (effort)ResolutionCostTokensReleaseTimeInterventions
GitHub CopilotCopilot Fusion (max)71.2%$2.0k3.3BNov 17, 2026Not reportedNot reported
Claude CodeOpus 5.5 (max)65.8%$4.7k8.0BSep 22, 2026Not reportedNot reported
Claude CodeFable 5.2 (max)65.8%$16.5k15.7BNov 9, 2026Not reportedNot reported
Antigravity CLIGemini 4 Carbon (high)64.5%$1.4k3.8BOct 28, 2026Not reportedNot reported
CodexGPT-6.2 Astra (max)62.1%$1.4k1.1BNov 17, 2026Not reportedNot reported
CodexGPT-6.2 Sol (max)61.5%$0.3k1.1BNov 17, 2026Not reportedNot reported
Muse CodeMuse Wonder (xhigh)47.0%$2.5k5.5BOct 12, 2026Not reportedNot reported
Grok BuildGrok 4.7 (xhigh)38.8%$3.7k5.5BSep 21, 2026Not reportedNot reported
mini-swe-agentMiMo V2.6 Pro (none)27.0%$0.3k7.8BSep 22, 2026Not reportedNot reported
mini-SWE-agentGemini 3.8 Flash (high)18.2%$1.8k17.2BSep 2, 2026Not reportedNot reported

Source: Terminal-Bench leaderboard, checked 20 November 2026. Two configurations are shown for Claude Code, Codex and mini-swe-agent because the model and effort setting change the result. A single “agent” score hides that choice.

50-task suite results

The table below is the results schema for this suite, not a set of measurements. The baseline date is 20 November 2026 and no Plugsky-run result has been recorded yet. The columns are the four metrics plus the run status, so the first completed run can be read without changing the format.

50-task suite, baseline 20 November 2026. “Not run” means no result has been recorded, not a zero score.
Agent configurationPass rateCostp50 / p95 timeInterventionsStatus
GitHub Copilot — Copilot Fusion (max)————Not run
Claude Code — Opus 5.5 (max)————Not run
Claude Code — Fable 5.2 (max)————Not run
Antigravity CLI — Gemini 4 Carbon (high)————Not run
Codex — GPT-6.2 Astra (max)————Not run
Codex — GPT-6.2 Sol (max)————Not run
Muse Code — Muse Wonder (xhigh)————Not run
Grok Build — Grok 4.7 (xhigh)————Not run
mini-swe-agent — MiMo V2.6 Pro (none)————Not run
mini-SWE-agent — Gemini 3.8 Flash (high)————Not run

The reference table above and this results table are intentionally separate. The reference table is published evidence from a different suite; this table is the protocol's own baseline, which is currently empty.

The 50-task suite

The suite is deliberately small enough to run in a day and broad enough to expose different failure modes. It has five tasks in each of ten categories: bug fixing, feature work, refactoring, tests, performance, security, data, frontend, DevOps and documentation. Every task has a deterministic acceptance test, a stack and a difficulty label.

50 tasks, five per category. The acceptance test is the pass criterion; a patch that passes by weakening the test fails.
IDCategoryStackTaskAcceptance testDifficulty
BF-01Bug fixPython / FastAPIFix an off-by-one in cursor pagination that skips the last item on a full page.New pagination test passes; existing suite green.Easy
BF-02Bug fixTypeScript / NodeFix a null dereference in an async request handler when the auth header is absent.Handler returns 401; no unhandled rejection in logs.Easy
BF-03Bug fixGoFix a data race in a worker pool that drops jobs under load.Race detector clean; 10k-job test drains exactly once.Hard
BF-04Bug fixRustFix a borrow-checker error in a streaming parser without cloning the buffer.Cargo test passes; no new allocations per chunk.Medium
BF-05Bug fixJava / SpringFix a timezone bug that shifts scheduled jobs by one hour across DST.DST boundary test passes in two zones.Medium
FE-01FeaturePython / FastAPIAdd cursor pagination to a list endpoint with a stable sort key.Page 1 + page 2 contain every row exactly once.Medium
FE-02FeatureTypeScript / NodeAdd idempotency keys to a payment-adjacent POST endpoint.Duplicate request returns the first response; one row written.Hard
FE-03FeaturePythonAdd streaming CSV export for a large table without loading it all.Memory stays under 100 MB for 1M rows; CSV parses.Medium
FE-04FeatureGoAdd per-key token-bucket rate limiting middleware.Burst test returns 429 after the configured limit.Medium
FE-05FeatureTypeScriptAdd webhook retries with exponential backoff and a dead-letter queue.Transient failures retry; permanent failures dead-letter.Hard
RF-01RefactorRuby / RailsExtract a service object from a 400-line controller without changing behaviour.Request specs unchanged and green.Medium
RF-02RefactorJavaScriptReplace a nested callback chain with async/await and preserve error semantics.All error-path tests pass; no unhandled rejections.Easy
RF-03RefactorPythonSplit a 900-line module into three cohesive modules with no circular imports.Import graph acyclic; tests green.Medium
RF-04RefactorJavaIntroduce constructor injection for a hard-coded dependency.Unit tests use a fake; production wiring unchanged.Medium
RF-05RefactorAnyRemove a dead feature flag and all branches that can no longer execute.No references remain; coverage does not drop.Easy
TS-01TestsTypeScriptAdd tests for auth middleware, including expired and malformed tokens.Branch coverage for the middleware exceeds 90%.Easy
TS-02TestsPythonAdd property-based tests for a parser's round-trip invariant.Hypothesis finds no counterexample in 1,000 cases.Medium
TS-03TestsGoAdd an integration test with a disposable Postgres container.Test runs in CI without a shared database.Medium
TS-04TestsRustAdd snapshot tests for CLI output, including the error path.Snapshots are reviewed and stable across runs.Easy
TS-05TestsPythonRaise coverage on a payment calculator without testing implementation details.Coverage +15 points; mutation score improves.Hard
PF-01PerformancePython / SQLAlchemyFix an N+1 query in an order list endpoint.Query count is constant in the number of orders.Medium
PF-02PerformancePostgresAdd the right index for a slow filtered query and prove the plan changes.EXPLAIN shows an index scan; p95 drops.Medium
PF-03PerformanceGoCache a hot configuration read with correct invalidation.Stale read window is bounded; race detector clean.Medium
PF-04PerformanceReactReduce the initial bundle by code-splitting a heavy route.Bundle budget passes; route still renders.Medium
PF-05PerformancePythonCut memory in an image pipeline that loads full-resolution files.Peak RSS halves on a 200-image test.Hard
SEC-01SecurityPython / SQLFix a SQL injection in a search endpoint without breaking filters.Injection payload is parameterized; tests green.Medium
SEC-02SecurityTypeScriptAdd CSRF protection to cookie-authenticated state-changing routes.Cross-site POST is rejected; same-site POST works.Medium
SEC-03SecurityJavaScriptSanitize user HTML before rendering it in a rich-text viewer.XSS payload is neutralized; formatting preserved.Medium
SEC-04SecurityAnyRemove a leaked secret from the repo and rotate it without breaking CI.Secret is absent from history; CI uses the new value.Hard
SEC-05SecurityJavaAdd an object-level authorization check to a read endpoint.User A cannot read user B's record; admin can.Medium
DATA-01DataPostgresWrite a migration with a backfill that is safe to run online.Migration is idempotent; backfill batched; rollback works.Hard
DATA-02DataPython / pandasFix duplicate rows in an ETL join caused by a many-to-many key.Row count matches the source; no duplicate IDs.Medium
DATA-03DataSQLAdd partition pruning to a date-filtered reporting query.Plan scans one partition; result unchanged.Medium
DATA-04DataPythonReconcile ledger totals against a bank export and explain the difference.Every unmatched row has a reason code.Hard
DATA-05DataTypeScriptAdd schema validation at an ingestion boundary.Invalid payload returns 422 with a field path.Easy
WEB-01FrontendCSSFix a responsive nav that traps focus on mobile.Keyboard focus escapes; menu closes on route change.Easy
WEB-02FrontendReactAdd an accessible modal with focus trapping and restore.Screen-reader labels and focus order pass the audit.Medium
WEB-03FrontendNext.jsFix a hydration mismatch caused by locale-dependent date formatting.No hydration warning; dates match the server.Medium
WEB-04FrontendReactAdd an optimistic update with rollback on failure.UI reverts on error; no duplicate rows on retry.Medium
WEB-05FrontendTypeScriptFix form validation that submits on Enter before async validation finishes.Invalid submit is blocked; valid submit works.Easy
OPS-01DevOpsDockerAdd a health check that does not report ready before migrations finish.Container is unhealthy during migration; healthy after.Medium
OPS-02DevOpsDockerFix a Docker build that reinstalls dependencies on every code change.Layer cache is reused; image builds in under 2 minutes.Easy
OPS-03DevOpsGoAdd graceful shutdown that drains in-flight requests.SIGTERM finishes requests; no 502s in the test.Medium
OPS-04DevOpsPythonAdd structured JSON logging with request IDs and no PII.Logs parse; PII fields are redacted.Medium
OPS-05DevOpsGitHub ActionsAdd a CI matrix for three language versions and cache dependencies.All matrix jobs pass; cache hit on the second run.Easy
DOC-01Docs/toolingOpenAPIUpdate API docs from an OpenAPI spec and fix broken examples.Spec validates; every example returns 2xx.Easy
DOC-02Docs/toolingMarkdownWrite a runbook for a failed nightly job, including rollback.A new operator can follow it in a tabletop test.Easy
DOC-03Docs/toolingMarkdownAdd a changelog entry that matches the actual diff.Entry names the user-visible change and the version.Easy
DOC-04Docs/toolingMarkdownWrite an ADR for a database choice with alternatives and consequences.ADR follows the repo template and links the issue.Easy
DOC-05Docs/toolingCLIAdd help text and a usage example for a new flag.`--help` is accurate; example runs.Easy

The four metrics

A pass rate alone is not a procurement decision. The suite records four metrics for every agent and task, and reports them together. Missing metrics are labelled, not estimated.

Metric definitions. Cost uses the provider's list price on the run date; time is wall-clock time inside the container.
MetricDefinitionWhy it matters
Pass rateTasks whose acceptance test passes in a clean container, with existing tests green and no hand edits, divided by 50.Capability. A high pass rate on easy tasks can hide failures on the hard ones; report the per-category breakdown.
CostToken usage × list price on the run date, plus any tool or sandbox fees. Report the total and the cost per solved task.Economics. A 70% agent at $20 per task can be worse than a 60% agent at $2.
TimeWall-clock time from task start to accepted patch, including retries, tool calls and test runs. Report p50 and p95.Developer experience. An agent that solves a task in 40 minutes is not replacing a 10-minute fix.
InterventionsHuman messages, file edits, command retries or approvals after the task starts. Report the count and the share of tasks with at least one.Autonomy. An agent that needs a human every three steps is a pair-programming tool, not an autonomous one.

What published benchmarks measure — and what they don't

The benchmark landscape is fragmented. SWE-bench measures patches against real GitHub issues, mostly Python. Terminal-Bench measures terminal tasks in containerized environments. Aider's polyglot leaderboard measures edit-format compliance across languages. None of them publishes a complete cost, time and intervention record for every agent, which is why the four-metric protocol exists.

Metric coverage by public source. “Sometimes” means some entries publish it; “No” means the leaderboard does not.
SourcePass rateCostTimeInterventionsWhat it measures
Terminal-Bench 4.1YesYes (total)NoNoTerminal tasks in containers
SWE-bench VerifiedYesSometimesNoNoPatches for real GitHub issues
Aider PolyglotYesYesNoNoEdit-format compliance across languages
This 50-task suitePlannedPlannedPlannedPlannedFour-metric agent comparison on a fixed suite

That gap is the point. A leaderboard that reports resolution rate and total cost but not time or interventions answers “can it solve it?” and “what did the tokens cost?”. It does not answer “how long did it take?” or “how often did a human have to step in?”. Those are the questions a team adopting an agent actually asks.

Methodology

Environment. Each task runs in a clean container built from a pinned image. The repository is checked out at a fixed commit. Network access is disabled unless the task explicitly requires a package install, in which case the mirror is pinned. The agent receives the task text, the repository and the test command; it does not receive the acceptance test source.

Run. The agent works until it declares done or the 30-minute wall-clock limit expires. The harness then runs the acceptance test and the existing test suite. A patch is accepted only if both pass and no test file was modified to weaken an assertion. The patch is stored with the run log.

Scoring. Pass rate is accepted tasks divided by 50. Cost is the sum of token usage at list price plus sandbox fees. Time is wall-clock seconds from start to acceptance, reported as p50 and p95. Interventions are counted from the harness event log: any human message, edit, command or approval after start. A task with zero interventions and a passing test is “autonomous”; a task with a passing test and three interventions is still a pass, but it is reported with the intervention count.

Baseline. The baseline date is 20 November 2026. The suite has no Plugsky-run results on that date. When results exist, the table will list the agent, model, version, effort setting, run date, pass rate, cost, p50/p95 time, intervention count and a link to the raw run. The task IDs and acceptance tests will not change between runs; if a task is retired, it is replaced with a new ID and the change is recorded.

How to run it

  1. Pin the harness. Use a container image with the language runtime and test dependencies. Record the image digest.
  2. Pin the agent. Record the agent version, model ID, effort setting, temperature and tool permissions. An agent is a configuration, not just a product name.
  3. Run the 50 tasks. Use a fresh container per task. Do not reuse a working tree between tasks.
  4. Score mechanically. Run the acceptance test and the existing suite. Reject patches that modify tests, skip tests or add network access to the test command.
  5. Record the four metrics. Export pass/fail, token usage, wall-clock time and intervention events. Keep the raw logs.
  6. Report the distribution. Publish per-category pass rates and p50/p95 time, not just the mean. A single aggregate hides the failures that matter.

If you publish a run, include the exact command and the image digest. A score without a reproducible command is a claim, not a benchmark. Related: the best AI model for coding, building a coding agent, and the coding model selector.

FAQ

Are the pass rates on this page from the 50-task suite?

No. The pass rates, costs and token counts in the reference table are published results from Terminal-Bench 4.1, not from the 50-task suite on this page. The 50-task suite is a protocol with a baseline date of 20 November 2026; it has no Plugsky-run results yet. Mixing the two would be misleading, so they are labelled separately.

Why publish a benchmark with no results?

Because inventing results is worse than an empty column. The page publishes the task suite, the acceptance tests, the metric definitions and the reference points so the first run can be reproduced and compared. When runs are complete, the results table will carry the same task IDs and the same scoring rules.

What counts as a pass?

A task passes when its acceptance test passes in a clean container, with no human edits to the agent's patch, within the time limit, and with the repository's existing tests still green. A plausible explanation without a working patch is a fail. A patch that passes by deleting the failing test is a fail.

What is an intervention?

An intervention is any human message, file edit, command or approval that changes the agent's trajectory after the task starts. Clarifying the original prompt before the run does not count. Retrying a failed command on the agent's behalf, editing its patch or telling it which file to open does. Interventions are counted per task and reported with the pass rate.

Can I submit a run?

Yes, in principle: publish the agent name and version, model and version, the exact command, the container image, the per-task pass/fail, the token usage, the wall-clock time, the intervention count and the raw logs. The page's protocol defines the fields. A submission without logs or with hand-edited patches is not comparable.

Sources and verification

Primary sources were opened and checked on 20 November 2026. Product terms and datasets change; the linked source is authoritative.

Baseline 20 November 2026. Terminal-Bench reference results retrieved on that date. The 50-task suite has no Plugsky-run results yet; the page will report them with the same task IDs and scoring rules.