Terminal-Bench 4.0 datasets

Long-horizon tasks in real containers, from payment systems to power grids. Every task has a recorded passing run.

Dataset
Terminal-Bench 4.0
Tasks
2,400+
Format
Harbor task environments
Languages
Python, Go, SQL, R and more
Samples
Within 24 hours
37%
first-try pass rate of the best agent we tested
100%
Every task is verified for full solvability.

Example tasks

Six tasks from the sample set. Each one is a complete Harbor task with held-out tests, a reference solution and a recorded passing run.

Security
Python

Certify files in a damaged archive

Find which files and byte ranges in a corrupted 7z archive can still be trusted, with 100% precision and at least 98% recall.

Systems
Go

Catch duplicate card payments at scale

Repair an ISO 8583 retry detector so it tracks about 76 million live payment identities within strict memory and speed limits.

Data engineering
Python

Audit releases without hindsight

Fix an audit library so each release review uses only the evidence that existed at the time, across 60,000 measurement rows.

Logistics
SQL

Rebuild a shipping-container ledger

Reconcile four exports that disagree into one stock ledger, and split movements across bookings without double counting.

Logistics
R

Plan refrigerated delivery routes

Recover the rules behind old route plans, then produce routes that pass an independent cost and timing check on new cases.

Physics
Python

Keep a power grid's day in one model

Fix a scheduling builder so one full-day safe region keeps the links between time slices, within the memory limit.

Shipped with every task

Each task is a standard Harbor task, so it runs with the agent harness you already use. A senior engineer authors every task, two independent reviewers check it, and a fairness check removes tasks that are hard only because the spec is unclear.

task/one Harbor task
├── instruction.mdwhat the agent must do
├── task.tomlmetadata and timeouts
├── environment/the Docker sandbox and its inputs
├── tests/held-out verifier, hidden from the agent
└── solution/expert reference solution
+ passing runa recorded run that scores 1

Hard on the first try, solvable by the third

The best agent we tested fails most tasks on the first try but solves most within three. That gap gives a model room to learn.

First try (pass@1)
36.7%
Best of three tries (pass@3)
81.7%

GPT-5.6 Sol with Codex, 5 runs on each of the 6 sample tasks (30 runs).

Ready to license now, or extend it to fit your needs

Any lab can license Terminal-Bench 4.0. We size the full set to your complexity requirements, and we can extend it with new domains or harder tasks.