Snapshot. Baseline established 20 November 2026. The 50-task suite and scoring protocol are published here; no Plugsky-run results exist yet. The reference table reports published Terminal-Bench 4.1 results for 10 agent configurations, not results on this suite. Time and intervention counts are marked “Not reported” where the source does not publish them.
Published reference points for 10 agent configurations
These rows come from the Terminal-Bench 4.1 leaderboard, retrieved on 20 November 2026. They are useful for calibrating what a current agent can do on a terminal-heavy task suite. They are not results on the 50-task suite below, and the cost column is the total cost reported for the full Terminal-Bench run, not a cost per task.
| Agent | Model (effort) | Resolution | Cost | Tokens | Release | Time | Interventions |
|---|---|---|---|---|---|---|---|
| GitHub Copilot | Copilot Fusion (max) | 71.2% | $2.0k | 3.3B | Nov 17, 2026 | Not reported | Not reported |
| Claude Code | Opus 5.5 (max) | 65.8% | $4.7k | 8.0B | Sep 22, 2026 | Not reported | Not reported |
| Claude Code | Fable 5.2 (max) | 65.8% | $16.5k | 15.7B | Nov 9, 2026 | Not reported | Not reported |
| Antigravity CLI | Gemini 4 Carbon (high) | 64.5% | $1.4k | 3.8B | Oct 28, 2026 | Not reported | Not reported |
| Codex | GPT-6.2 Astra (max) | 62.1% | $1.4k | 1.1B | Nov 17, 2026 | Not reported | Not reported |
| Codex | GPT-6.2 Sol (max) | 61.5% | $0.3k | 1.1B | Nov 17, 2026 | Not reported | Not reported |
| Muse Code | Muse Wonder (xhigh) | 47.0% | $2.5k | 5.5B | Oct 12, 2026 | Not reported | Not reported |
| Grok Build | Grok 4.7 (xhigh) | 38.8% | $3.7k | 5.5B | Sep 21, 2026 | Not reported | Not reported |
| mini-swe-agent | MiMo V2.6 Pro (none) | 27.0% | $0.3k | 7.8B | Sep 22, 2026 | Not reported | Not reported |
| mini-SWE-agent | Gemini 3.8 Flash (high) | 18.2% | $1.8k | 17.2B | Sep 2, 2026 | Not reported | Not reported |
Source: Terminal-Bench leaderboard, checked 20 November 2026. Two configurations are shown for Claude Code, Codex and mini-swe-agent because the model and effort setting change the result. A single “agent” score hides that choice.
50-task suite results
The table below is the results schema for this suite, not a set of measurements. The baseline date is 20 November 2026 and no Plugsky-run result has been recorded yet. The columns are the four metrics plus the run status, so the first completed run can be read without changing the format.
| Agent configuration | Pass rate | Cost | p50 / p95 time | Interventions | Status |
|---|---|---|---|---|---|
| GitHub Copilot — Copilot Fusion (max) | — | — | — | — | Not run |
| Claude Code — Opus 5.5 (max) | — | — | — | — | Not run |
| Claude Code — Fable 5.2 (max) | — | — | — | — | Not run |
| Antigravity CLI — Gemini 4 Carbon (high) | — | — | — | — | Not run |
| Codex — GPT-6.2 Astra (max) | — | — | — | — | Not run |
| Codex — GPT-6.2 Sol (max) | — | — | — | — | Not run |
| Muse Code — Muse Wonder (xhigh) | — | — | — | — | Not run |
| Grok Build — Grok 4.7 (xhigh) | — | — | — | — | Not run |
| mini-swe-agent — MiMo V2.6 Pro (none) | — | — | — | — | Not run |
| mini-SWE-agent — Gemini 3.8 Flash (high) | — | — | — | — | Not run |
The reference table above and this results table are intentionally separate. The reference table is published evidence from a different suite; this table is the protocol's own baseline, which is currently empty.
The 50-task suite
The suite is deliberately small enough to run in a day and broad enough to expose different failure modes. It has five tasks in each of ten categories: bug fixing, feature work, refactoring, tests, performance, security, data, frontend, DevOps and documentation. Every task has a deterministic acceptance test, a stack and a difficulty label.
| ID | Category | Stack | Task | Acceptance test | Difficulty |
|---|---|---|---|---|---|
BF-01 | Bug fix | Python / FastAPI | Fix an off-by-one in cursor pagination that skips the last item on a full page. | New pagination test passes; existing suite green. | Easy |
BF-02 | Bug fix | TypeScript / Node | Fix a null dereference in an async request handler when the auth header is absent. | Handler returns 401; no unhandled rejection in logs. | Easy |
BF-03 | Bug fix | Go | Fix a data race in a worker pool that drops jobs under load. | Race detector clean; 10k-job test drains exactly once. | Hard |
BF-04 | Bug fix | Rust | Fix a borrow-checker error in a streaming parser without cloning the buffer. | Cargo test passes; no new allocations per chunk. | Medium |
BF-05 | Bug fix | Java / Spring | Fix a timezone bug that shifts scheduled jobs by one hour across DST. | DST boundary test passes in two zones. | Medium |
FE-01 | Feature | Python / FastAPI | Add cursor pagination to a list endpoint with a stable sort key. | Page 1 + page 2 contain every row exactly once. | Medium |
FE-02 | Feature | TypeScript / Node | Add idempotency keys to a payment-adjacent POST endpoint. | Duplicate request returns the first response; one row written. | Hard |
FE-03 | Feature | Python | Add streaming CSV export for a large table without loading it all. | Memory stays under 100 MB for 1M rows; CSV parses. | Medium |
FE-04 | Feature | Go | Add per-key token-bucket rate limiting middleware. | Burst test returns 429 after the configured limit. | Medium |
FE-05 | Feature | TypeScript | Add webhook retries with exponential backoff and a dead-letter queue. | Transient failures retry; permanent failures dead-letter. | Hard |
RF-01 | Refactor | Ruby / Rails | Extract a service object from a 400-line controller without changing behaviour. | Request specs unchanged and green. | Medium |
RF-02 | Refactor | JavaScript | Replace a nested callback chain with async/await and preserve error semantics. | All error-path tests pass; no unhandled rejections. | Easy |
RF-03 | Refactor | Python | Split a 900-line module into three cohesive modules with no circular imports. | Import graph acyclic; tests green. | Medium |
RF-04 | Refactor | Java | Introduce constructor injection for a hard-coded dependency. | Unit tests use a fake; production wiring unchanged. | Medium |
RF-05 | Refactor | Any | Remove a dead feature flag and all branches that can no longer execute. | No references remain; coverage does not drop. | Easy |
TS-01 | Tests | TypeScript | Add tests for auth middleware, including expired and malformed tokens. | Branch coverage for the middleware exceeds 90%. | Easy |
TS-02 | Tests | Python | Add property-based tests for a parser's round-trip invariant. | Hypothesis finds no counterexample in 1,000 cases. | Medium |
TS-03 | Tests | Go | Add an integration test with a disposable Postgres container. | Test runs in CI without a shared database. | Medium |
TS-04 | Tests | Rust | Add snapshot tests for CLI output, including the error path. | Snapshots are reviewed and stable across runs. | Easy |
TS-05 | Tests | Python | Raise coverage on a payment calculator without testing implementation details. | Coverage +15 points; mutation score improves. | Hard |
PF-01 | Performance | Python / SQLAlchemy | Fix an N+1 query in an order list endpoint. | Query count is constant in the number of orders. | Medium |
PF-02 | Performance | Postgres | Add the right index for a slow filtered query and prove the plan changes. | EXPLAIN shows an index scan; p95 drops. | Medium |
PF-03 | Performance | Go | Cache a hot configuration read with correct invalidation. | Stale read window is bounded; race detector clean. | Medium |
PF-04 | Performance | React | Reduce the initial bundle by code-splitting a heavy route. | Bundle budget passes; route still renders. | Medium |
PF-05 | Performance | Python | Cut memory in an image pipeline that loads full-resolution files. | Peak RSS halves on a 200-image test. | Hard |
SEC-01 | Security | Python / SQL | Fix a SQL injection in a search endpoint without breaking filters. | Injection payload is parameterized; tests green. | Medium |
SEC-02 | Security | TypeScript | Add CSRF protection to cookie-authenticated state-changing routes. | Cross-site POST is rejected; same-site POST works. | Medium |
SEC-03 | Security | JavaScript | Sanitize user HTML before rendering it in a rich-text viewer. | XSS payload is neutralized; formatting preserved. | Medium |
SEC-04 | Security | Any | Remove a leaked secret from the repo and rotate it without breaking CI. | Secret is absent from history; CI uses the new value. | Hard |
SEC-05 | Security | Java | Add an object-level authorization check to a read endpoint. | User A cannot read user B's record; admin can. | Medium |
DATA-01 | Data | Postgres | Write a migration with a backfill that is safe to run online. | Migration is idempotent; backfill batched; rollback works. | Hard |
DATA-02 | Data | Python / pandas | Fix duplicate rows in an ETL join caused by a many-to-many key. | Row count matches the source; no duplicate IDs. | Medium |
DATA-03 | Data | SQL | Add partition pruning to a date-filtered reporting query. | Plan scans one partition; result unchanged. | Medium |
DATA-04 | Data | Python | Reconcile ledger totals against a bank export and explain the difference. | Every unmatched row has a reason code. | Hard |
DATA-05 | Data | TypeScript | Add schema validation at an ingestion boundary. | Invalid payload returns 422 with a field path. | Easy |
WEB-01 | Frontend | CSS | Fix a responsive nav that traps focus on mobile. | Keyboard focus escapes; menu closes on route change. | Easy |
WEB-02 | Frontend | React | Add an accessible modal with focus trapping and restore. | Screen-reader labels and focus order pass the audit. | Medium |
WEB-03 | Frontend | Next.js | Fix a hydration mismatch caused by locale-dependent date formatting. | No hydration warning; dates match the server. | Medium |
WEB-04 | Frontend | React | Add an optimistic update with rollback on failure. | UI reverts on error; no duplicate rows on retry. | Medium |
WEB-05 | Frontend | TypeScript | Fix form validation that submits on Enter before async validation finishes. | Invalid submit is blocked; valid submit works. | Easy |
OPS-01 | DevOps | Docker | Add a health check that does not report ready before migrations finish. | Container is unhealthy during migration; healthy after. | Medium |
OPS-02 | DevOps | Docker | Fix a Docker build that reinstalls dependencies on every code change. | Layer cache is reused; image builds in under 2 minutes. | Easy |
OPS-03 | DevOps | Go | Add graceful shutdown that drains in-flight requests. | SIGTERM finishes requests; no 502s in the test. | Medium |
OPS-04 | DevOps | Python | Add structured JSON logging with request IDs and no PII. | Logs parse; PII fields are redacted. | Medium |
OPS-05 | DevOps | GitHub Actions | Add a CI matrix for three language versions and cache dependencies. | All matrix jobs pass; cache hit on the second run. | Easy |
DOC-01 | Docs/tooling | OpenAPI | Update API docs from an OpenAPI spec and fix broken examples. | Spec validates; every example returns 2xx. | Easy |
DOC-02 | Docs/tooling | Markdown | Write a runbook for a failed nightly job, including rollback. | A new operator can follow it in a tabletop test. | Easy |
DOC-03 | Docs/tooling | Markdown | Add a changelog entry that matches the actual diff. | Entry names the user-visible change and the version. | Easy |
DOC-04 | Docs/tooling | Markdown | Write an ADR for a database choice with alternatives and consequences. | ADR follows the repo template and links the issue. | Easy |
DOC-05 | Docs/tooling | CLI | Add help text and a usage example for a new flag. | `--help` is accurate; example runs. | Easy |
The four metrics
A pass rate alone is not a procurement decision. The suite records four metrics for every agent and task, and reports them together. Missing metrics are labelled, not estimated.
| Metric | Definition | Why it matters |
|---|---|---|
| Pass rate | Tasks whose acceptance test passes in a clean container, with existing tests green and no hand edits, divided by 50. | Capability. A high pass rate on easy tasks can hide failures on the hard ones; report the per-category breakdown. |
| Cost | Token usage × list price on the run date, plus any tool or sandbox fees. Report the total and the cost per solved task. | Economics. A 70% agent at $20 per task can be worse than a 60% agent at $2. |
| Time | Wall-clock time from task start to accepted patch, including retries, tool calls and test runs. Report p50 and p95. | Developer experience. An agent that solves a task in 40 minutes is not replacing a 10-minute fix. |
| Interventions | Human messages, file edits, command retries or approvals after the task starts. Report the count and the share of tasks with at least one. | Autonomy. An agent that needs a human every three steps is a pair-programming tool, not an autonomous one. |
What published benchmarks measure — and what they don't
The benchmark landscape is fragmented. SWE-bench measures patches against real GitHub issues, mostly Python. Terminal-Bench measures terminal tasks in containerized environments. Aider's polyglot leaderboard measures edit-format compliance across languages. None of them publishes a complete cost, time and intervention record for every agent, which is why the four-metric protocol exists.
| Source | Pass rate | Cost | Time | Interventions | What it measures |
|---|---|---|---|---|---|
| Terminal-Bench 4.1 | Yes | Yes (total) | No | No | Terminal tasks in containers |
| SWE-bench Verified | Yes | Sometimes | No | No | Patches for real GitHub issues |
| Aider Polyglot | Yes | Yes | No | No | Edit-format compliance across languages |
| This 50-task suite | Planned | Planned | Planned | Planned | Four-metric agent comparison on a fixed suite |
That gap is the point. A leaderboard that reports resolution rate and total cost but not time or interventions answers “can it solve it?” and “what did the tokens cost?”. It does not answer “how long did it take?” or “how often did a human have to step in?”. Those are the questions a team adopting an agent actually asks.
Methodology
Environment. Each task runs in a clean container built from a pinned image. The repository is checked out at a fixed commit. Network access is disabled unless the task explicitly requires a package install, in which case the mirror is pinned. The agent receives the task text, the repository and the test command; it does not receive the acceptance test source.
Run. The agent works until it declares done or the 30-minute wall-clock limit expires. The harness then runs the acceptance test and the existing test suite. A patch is accepted only if both pass and no test file was modified to weaken an assertion. The patch is stored with the run log.
Scoring. Pass rate is accepted tasks divided by 50. Cost is the sum of token usage at list price plus sandbox fees. Time is wall-clock seconds from start to acceptance, reported as p50 and p95. Interventions are counted from the harness event log: any human message, edit, command or approval after start. A task with zero interventions and a passing test is “autonomous”; a task with a passing test and three interventions is still a pass, but it is reported with the intervention count.
Baseline. The baseline date is 20 November 2026. The suite has no Plugsky-run results on that date. When results exist, the table will list the agent, model, version, effort setting, run date, pass rate, cost, p50/p95 time, intervention count and a link to the raw run. The task IDs and acceptance tests will not change between runs; if a task is retired, it is replaced with a new ID and the change is recorded.
How to run it
- Pin the harness. Use a container image with the language runtime and test dependencies. Record the image digest.
- Pin the agent. Record the agent version, model ID, effort setting, temperature and tool permissions. An agent is a configuration, not just a product name.
- Run the 50 tasks. Use a fresh container per task. Do not reuse a working tree between tasks.
- Score mechanically. Run the acceptance test and the existing suite. Reject patches that modify tests, skip tests or add network access to the test command.
- Record the four metrics. Export pass/fail, token usage, wall-clock time and intervention events. Keep the raw logs.
- Report the distribution. Publish per-category pass rates and p50/p95 time, not just the mean. A single aggregate hides the failures that matter.
If you publish a run, include the exact command and the image digest. A score without a reproducible command is a claim, not a benchmark. Related: the best AI model for coding, building a coding agent, and the coding model selector.
FAQ
Are the pass rates on this page from the 50-task suite?
No. The pass rates, costs and token counts in the reference table are published results from Terminal-Bench 4.1, not from the 50-task suite on this page. The 50-task suite is a protocol with a baseline date of 20 November 2026; it has no Plugsky-run results yet. Mixing the two would be misleading, so they are labelled separately.
Why publish a benchmark with no results?
Because inventing results is worse than an empty column. The page publishes the task suite, the acceptance tests, the metric definitions and the reference points so the first run can be reproduced and compared. When runs are complete, the results table will carry the same task IDs and the same scoring rules.
What counts as a pass?
A task passes when its acceptance test passes in a clean container, with no human edits to the agent's patch, within the time limit, and with the repository's existing tests still green. A plausible explanation without a working patch is a fail. A patch that passes by deleting the failing test is a fail.
What is an intervention?
An intervention is any human message, file edit, command or approval that changes the agent's trajectory after the task starts. Clarifying the original prompt before the run does not count. Retrying a failed command on the agent's behalf, editing its patch or telling it which file to open does. Interventions are counted per task and reported with the pass rate.
Can I submit a run?
Yes, in principle: publish the agent name and version, model and version, the exact command, the container image, the per-task pass/fail, the token usage, the wall-clock time, the intervention count and the raw logs. The page's protocol defines the fields. A submission without logs or with hand-edited patches is not comparable.
Sources and verification
Primary sources were opened and checked on 20 November 2026. Product terms and datasets change; the linked source is authoritative.
- Terminal-Bench leaderboard — resolution rate, cost and token totals for 10 agent configurations checked 20 November 2026
- SWE-bench — real-issue patch benchmark and leaderboards checked 20 November 2026
- Aider Polyglot leaderboard — edit-format compliance and cost across languages checked 20 November 2026
- SWE-bench repository — harness, task format and evaluation code checked 20 November 2026
Baseline 20 November 2026. Terminal-Bench reference results retrieved on that date. The 50-task suite has no Plugsky-run results yet; the page will report them with the same task IDs and scoring rules.