Compatibility

Evolve imports Harbor-format task packages and exports Harbor-format results. Managed jobs use Evolve’s own CLI and API contract. A local runner’s configuration is not passed through automatically. Use the task field reference for exact accepted fields and defaults. Before publishing an existing task package, validate it:

evolve dataset check ./my-dataset

Validation checks declarations. Image builds, provider compatibility, and task execution still need their own checks.

Harbor versions

Evolve reads tasks written for the newest Harbor release it is verified against and the six months of Harbor releases before it. Tasks written for older Harbor releases are a subset of that format, so they are read too. Harbor does not change behavior based on a task’s schema_version, so Evolve checks two things: the declared schema_version, and the Harbor release that introduced each field the task uses.

Harbor release	Released	`schema_version` it writes	Evolve
`main`, after v0.25.0	Unreleased	`"1.4"`	Not yet. A field that exists only on Harbor main is refused, and the refusal says so.
v0.25.0	2026-10-25	`"1.4"`	Supported. The newest release Evolve is verified against. See the v0.25.0 notes below.
v0.21.0 to v0.24.0	2026-08-10 to 2026-10-04	`"1.4"`	Supported
v0.13.0 to v0.20.0	2026-05-30 to 2026-07-18	`"1.3"`	Supported
v0.6.0 to v0.9.0	2026-04-29 to 2026-05-28	`"1.2"`	Supported. v0.6.0 opens the six-month window.
v0.3.0 to v0.5.0	2026-03-27 to 2026-04-23	`"1.1"`	Read, older than the window
0.1.x to 0.2.x	2025-11 to 2026-03	`"1.0"`, as root `version`	Read (Terminal-Bench 2.0 era)
Terminal-Bench 1 (`task.yaml`)	Before Harbor	None	Not supported. Convert with `harbor tasks migrate -i <tasks> -o <out>`, review the result, then publish. Both `evolve dataset check` and the import name the converter.

Accepted schema_version values are "1.0" through "1.4", the corpus spellings "1" and "2.0", or none. A patch spelling such as "1.0.0" reads as "1.0". A newer schema, such as "1.5", is refused with the newest supported schema named.

What each release added

evolve dataset check reports the oldest Harbor release that understands every field a corpus uses, when that release is newer than the window start: For example, a corpus using verifier.collect requires Harbor v0.14.0 or newer. The output also names Evolve’s supported release window and schema. A corpus that uses no newer field gets the accepted tasks use no field newer than Harbor v0.6.0 instead.

Release	Task format change	Evolve
v0.3.0	`[task]` package section (`name`, `description`, `authors`, `keywords`). Root `version` renamed `schema_version`.	Accepted. Either spelling is read; do not declare both.
v0.4.0	`[environment] healthcheck` and `workdir`	Honored
v0.5.0	`[[steps]]`, `multi_step_reward_strategy`, `artifacts`	Honored. See multi-step tasks for the step limits.
v0.6.0	`[environment] os`	Only `"linux"`. Windows tasks are refused.
v0.7.0	Separate verifier: `[verifier] environment_mode`, `[verifier.environment]`	Honored. `[verifier.environment.env]` and verifier MCP servers are refused.
v0.8.0	String `memory` / `storage` removed in favor of `memory_mb` / `storage_mb`	Both spellings accepted
v0.9.0	`[environment] tpu`	Refused: no TPU sandboxes
v0.13.0	`network_mode` and `allowed_hosts` on `[environment]`, `[agent]` and `[verifier]`, replacing `allow_internet`	Honored. `allow_internet` is still read.
v0.14.0	`[[verifier.collect]]` hooks, artifact `service`	Honored
v0.21.0	`[task] version` and `[dataset] version`. Schema `"1.4"`.	Honored. They become the task and dataset version.
v0.25.0	MCP server `headers`	Refused: a v0.25.0 field Evolve does not support yet
v0.25.0	dataset.toml `[metadata]` table	Accepted, descriptive only
v0.25.0	Fractional `cpus`	Accepted. E2B runs only 1 or an even number of vCPUs, so such a task runs on another provider there; see the provider verdicts in `evolve dataset check`.

Harbor v0.25.0 behavior that is not a field

Change	Evolve
`.harborignore` filters a task’s uploaded and published files	Ignored. Excluded files are still uploaded with the task, and a dataset.toml digest Harbor computed with exclusions does not match Evolve’s.
`harbor tasks init` defaults to a separate verifier with `tests/Dockerfile`	Single-step tasks import, and Evolve builds the verifier image from `tests/Dockerfile`, a second image build per task. A multi-step scaffold is refused (multi-step tasks need shared verification): pass `--verifier-mode shared`.
`harbor tasks init --steps` writes `collect = []` under each `[steps.verifier]`	Imports. An empty list declares no hook.
The default skills directory moved from `/harbor/skills` to `/runtime/skills`	Not a task field. Set `[environment] skills_dir` explicitly when a task or verifier reads the path.

Within the supported releases, a few features are not provided on managed sandboxes: Windows, TPU, MCP headers, [solution.env], [verifier.environment.env], metric.py, and multi-step tasks combined with Compose or separate verification. See the tables below and the task field reference.

Refusals name the release

An unknown field is refused, and the refusal says which case applies:

The field exists only on Harbor main: Harbor added ”…” on main after v0.25.0 and has not released it yet.
The field arrived in a Harbor release newer than the supported one.
Harbor removed the field.
The field belongs to a supported Harbor release that Evolve does not carry: “headers” is a Harbor v0.25.0 (2026-10-25) field Evolve does not support yet.
No Harbor release has the field. Harbor ignores unknown keys, but Evolve refuses them: check the spelling, or move the key under [metadata].

Release-specific errors apply to field names. An unsupported type or value gets a field-specific error without a release number. When a new Harbor release changes the task format, Evolve moves the supported release forward and the window moves with it.

Tasks and execution

Area	Managed Evolve support
Task packages	Instructions, `task.toml`, Dockerfiles or public pinned images, tests, optional solutions, skills, MCP, healthchecks, and supported Compose declarations. Unknown task fields are rejected. Windows, Apptainer, TPU configuration, and managed GUI desktop sessions are not provided.
Dataset sources	Publish local files, pinned Git sources, public HTTPS archives, or public Harbor Hub packages, then select the published dataset in a job. Arbitrary JSON task registries and direct local/repository task paths are not managed run sources.
Job configuration	Use the documented run flags or SDK job fields. Host plugin classes, install-only runs, runtime mounts, extra Compose overlays, run-level artifact declarations, and per-arm concurrency groups are not accepted job options.
Agent harnesses	Use a built-in harness or registered agent. Registration takes an install script or uploaded files plus a headless run command. Python `BaseAgent` classes, `acp:<registry-id>` lookup, and `harbor-agent.json` manifests are not registration formats. Managed ACP simulated-user/persona configuration is not provided.
Sandbox providers	E2B, Daytona, and Modal, subject to each task’s resource and network requirements. A local sandbox runner or custom provider plugin cannot be selected for a managed job.
Saved conversations	A task-level ATIF file can seed Claude Code or Codex. See saved conversations. Run-level trajectory loading and a managed trial-to-local-CLI handoff operation are not provided.
Multi-step tasks	Ordered steps share a filesystem, with per-step instructions, setup, checks, timeouts, and reward thresholds. Multi-step tasks use shared verification; Compose, separate verification, between-step conversation resume, and per-step artifact snapshots are not supported.

Verification and records

Area	Managed Evolve support
Verifiers	Task `tests/test.sh` runs in a shared or separate sandbox and writes reward files. Verifier scripts and supported judges can implement custom grading; a host-side `BaseVerifier` class is not a managed input.
Regrade	Eligible separate-mode trials can rerun their recorded verifier against retained inputs. Regrade does not select a replacement verifier and excludes shared-mode, multi-step, and judge-enabled trials. See jobs.
Rewards and metrics	Tasks can produce named rewards, and jobs expose platform aggregates and pass@k. Dataset-level `metric.py` scripts are rejected. Compute additional aggregates from exported results.
Artifacts	Root task declarations transfer files for separate verification and export retained payloads in the job archive. Shared verification uses the existing filesystem; inspect live or captured files through the filesystem API. See trial outputs.
Credentials and images	Job-attached secrets supply supported agent variables; managed judge credentials use the verifier contract. Task images must be publicly readable. User-managed private image-registry credential storage or selection is not exposed.
Sharing and leaderboards	Share jobs, task checks, and sessions, or use team ownership. Public dataset visibility is platform-managed. Job comparison and exported results are available; a user-managed leaderboard object or ranking API is not provided.

These boundaries describe the managed evaluation service. Custom agent and verifier code still runs within the documented sandbox and credential contracts.

Start here

Run evaluations

Task format

Validate your work

Read results

Work together