Expert-built, containerized environments to train and evaluate agents on long-horizon work across domains. Each comes with tasks, verifiers, proof of solvability and pass rates.
Each environment is a self-contained folder that runs the same way in your harness every time.
The goal and the end state, written the way an engineer would receive it. The config sets the time and resource limits and records difficulty and category.
my-task/ ├── instruction.md ├── task.toml ├── environment/ ├── solution/ └── tests/
Shown in the Harbor task format. We also deliver SWE-bench style task sets and match other formats on request.
The agent gets the instruction, works in the container, and the verifier grades the result. Labs use the same tasks to train with the reward and to evaluate with the pass rate.
The task is handed to the agent.
It reads, edits and runs code in the container.
The verifier checks the final state of the environment.
The tests compute the reward for the attempt.
Every task is reproducible, properly specified and solvable. We run frontier models on our tasks and record the pass rate. Reviews catch reward hacking, shortcuts and non-determinism.
The reference solution passes every test inside the container. A fairness review confirms tasks are hard, not unclear.
Tests check the end state, not the steps the agent took, and accept any correct solution.
We check tasks against public benchmarks and check that they haven’t leaked into pretraining.
A peer checks the tests and the code. A senior reviewer then checks complexity, reproducibility and test coverage, with consensus on disagreements.
Versioned environments with deterministic resets, so every rollout starts from the same state and results compare across runs and models.
We run frontier models on our tasks and record the pass rate, so you can choose tasks by difficulty. Terminal-Bench style tasks pass the same CI checks and rubrics as the original benchmark.
We build custom environments to your spec, for your domains and failure surfaces.
A researcher sets the domain, the task format and the target difficulty with your team.
Engineers who do this work in their own jobs author the tasks, from new problems, open source pull requests or real freelance jobs.
Each task gets its tests and a reference solution, then a solvability review.
We record frontier pass rates per task and deliver in Harbor or your own format, with the trajectory traces from every run.
Containerized environments authored by experts from our network, each with its tools, a task and a verifier, in formats you already run.
Very hard coding tasks on real codebases, set at expert level.
Real freelance coding jobs, with tests.
More collections on request.
See all datasetsBuild, debug, refactor and migrate real codebases.
Configure systems, fix builds, run services.
Infrastructure as code, networks, pipelines.
Find and fix vulnerabilities, harden systems.
Data, training, evaluation and serving.
Scientific computing and more, scoped with you.
Most collections ship in the Harbor format, so they run in the harness you may already use. We also deliver SWE-bench style task sets and match other formats on request.
Our research team runs frontier models on private task sets and publishes the results.
A live leaderboard on private collections that stay unpublished: SWE-Lancer, Terminal Bench 2.1 and 3.0.
Request dataset samples before you license a collection.