Tasks

my-task/
├── instruction.md    what the agent is asked to do
├── task.toml         timeouts, network, resources, verifier mode
├── pre_artifacts.sh  optional: collects the agent's work after the run
├── environment/
│   └── Dockerfile    the box the agent works in
├── tests/
│   └── test.sh       the verifier: writes the reward
└── solution/
    └── solve.sh      optional reference solution

​

instruction.md

​

environment/

​

tests/

• /logs/verifier/reward.txt: one number in [0, 1], usually 1 or 0.
• /logs/verifier/reward.json: a JSON object of named numbers, for a score with several parts.

#!/bin/bash
set -uo pipefail

if uvx pytest /tests/test_outputs.py; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

​

pre_artifacts.sh

#!/bin/bash
set -uo pipefail
mkdir -p /logs/artifacts
git diff --binary "$(git rev-list --max-parents=0 HEAD)" HEAD > /logs/artifacts/model.patch

​

solution/

​

task.toml

[agent]
timeout_sec = 3600

[verifier]
timeout_sec = 600
environment_mode = "shared"

[environment]
network_mode = "allowlist"
allowed_hosts = ["pypi.org"]
cpus = 2
memory_mb = 4096
storage_mb = 10240

• Timeouts. agent.timeout_sec and verifier.timeout_sec, in seconds. The defaults are 3600 s for the agent and 600 s for the verifier; a declared value always wins. A job can stretch them with --timeout-multiplier.
• Network access. environment.network_mode is no-network (the agent reaches nothing but its model), allowlist (only the hosts in allowed_hosts, accepted with this mode only), or public (the open internet, and the default). The older allow_internet = false means no-network, true means public. Only no-network makes the spend cap airtight; see models.
• Verifier network. A shared verifier sees the agent box’s network; a [verifier] network_mode that differs from it is refused at import. A separate verifier box uses the mode of [verifier.environment], else a copy of [environment]’s, with [verifier] network_mode overriding either. The platform never seals a verifier on its own.
• Verifier mode. verifier.environment_mode is shared, the verifier runs inside the agent’s sandbox after the agent finishes and its credentials are revoked, or separate, the verifier boots a fresh copy of the environment and judges only what the task lists under artifacts. Without that list a separate verifier sees none of the agent’s work, and even the reference solution scores 0. A [verifier.environment] table implies separate. Only settled separate-mode trials can be regraded.
• Resources. environment.cpus, memory_mb, storage_mb, and gpus with gpu_types. Omitted, the provider’s default sizing applies. A size above a provider’s ceiling refuses on that provider; above every ceiling it is refused at import. The ceilings are on sandboxes.
• Artifacts. A top-level artifacts = ["/app/out.json"] list names the paths carried out of the agent’s sandbox: into a separate verifier, and, on a separate-mode trial, into the trial’s stored artifacts.

​

Environment variables

[environment.env]
APP_MODE = "ci"                        # literal: lands in the box as written
GITHUB_TOKEN = "${GITHUB_TOKEN}"       # template: a secret the job must attach
LOG_LEVEL = "${LOG_LEVEL:-info}"       # template with a fallback

​

LLM judges

[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"

​

Multi-step tasks

migrate-then-prove/
├── task.toml                [[steps]] entries, in order
├── environment/
│   └── Dockerfile
├── tests/                   optional shared helpers for every step's verifier
└── steps/
    ├── 01-migrate/
    │   ├── instruction.md
    │   ├── tests/           merged over the shared tests/
    │   └── workdir/         optional files landed before the step; its setup.sh runs first
    └── 02-prove/
        ├── instruction.md
        └── tests/

multi_step_reward_strategy = "mean"    # "mean" (default) or "final"

[[steps]]
name = "01-migrate"
min_reward = 0.5                       # below this the trial stops here

[[steps]]
name = "02-prove"

[steps.agent]
timeout_sec = 600                      # this step's override of the task-level timeout

​

MCP servers

[[environment.mcp_servers]]
name = "docs-search"
transport = "streamable-http"          # or "sse" (the default); "http" is read as streamable-http
url = "http://mcp-server:8000/mcp"     # a compose service, reachable with no egress

[[environment.mcp_servers]]
name = "sqlite"
transport = "stdio"                    # launched inside the box by the harness
command = "uvx"
args = ["mcp-server-sqlite", "--db-path", "/app/data.db"]

​

Healthcheck and the agent user

[environment.healthcheck]
command = "curl -fsS http://localhost:8000/health"
interval_sec = 5           # every field below command is optional; these are the defaults
timeout_sec = 30
start_period_sec = 0
start_interval_sec = 5
retries = 3

​

Not supported

• Computer-use and desktop tasks. [environment].os other than Linux is refused at import. A task whose instruction assumes a desktop is not refused; it runs on a Linux box without one.
• Run-level trajectory seeding. No job flag names a session to resume from. Task-level seeding works: a trajectory.json beside instruction.md (or in the first step’s directory) is seeded into the agent’s session on claude and codex; other harnesses refuse before any spend. [agent] load_trajectory and resume_trajectory keys are refused.
• Verifier scripts whose PEP-723 header needs a package index. Under no-network the verifier box has no index (only rewardkit and litellm resolve, on a judge task), so such a task is refused at import. Under allowlist or public, or without uv run or uvx in tests/, nothing changes.

​

From tasks to a dataset