Terminal-Bench 4.0 datasets

Long-horizon tasks in real containers, from payment systems to power grids. Every task has a recorded passing run.

Dataset
Terminal-Bench 4.0
Tasks
2,400+
Format
Harbor task environments
Languages
Python, Go, SQL, R and more
Samples
Within 24 hours
37%
first-try pass rate of the best agent we tested
100%
Every task is verified for full solvability.

Example tasks

Six tasks from the sample set. Each one is a complete Harbor task with held-out tests, a reference solution and a recorded passing run.

SecurityPython

Certify files in a damaged archive

Find which files and byte ranges in a corrupted 7z archive can still be trusted, with 100% precision and at least 98% recall.

SystemsGo

Catch duplicate card payments at scale

Repair an ISO 8583 retry detector so it tracks about 76 million live payment identities within strict memory and speed limits.

Data engineeringPython

Audit releases without hindsight

Fix an audit library so each release review uses only the evidence that existed at the time, across 60,000 measurement rows.

LogisticsSQL

Rebuild a shipping-container ledger

Reconcile four exports that disagree into one stock ledger, and split movements across bookings without double counting.

LogisticsR

Plan refrigerated delivery routes

Recover the rules behind old route plans, then produce routes that pass an independent cost and timing check on new cases.

PhysicsPython

Keep a power grid's day in one model

Fix a scheduling builder so one full-day safe region keeps the links between time slices, within the memory limit.

What every task ships with

Samples use the same folders as the full dataset. Run them in Harbor with the agent harness you already use. Senior engineers write and review every task, and a fairness check removes tasks that are hard only because the spec is unclear.

task/one Harbor task
├── instruction.mdwhat the agent must do
├── task.tomlmetadata and timeouts
├── environment/the Docker sandbox and its inputs
├── tests/held-out verifier, hidden from the agent
└── solution/expert reference solution
+ passing runa recorded run that scores 1
+ QA reportreview notes for the task

Hard on the first try, solvable by the third

The best agent we tested fails most tasks on the first try but solves most within three. That gap gives a model room to learn.

First try (pass@1)36.7%
Best of three tries (pass@3)81.7%

GPT-5.6 Sol with Codex, 5 runs on each of the 6 sample tasks (30 runs).

License it, then extend it.

Any lab can license Terminal-Bench 4.0. We size the full set to your complexity requirements, and we can extend it with new domains or harder tasks.