Tasks

A task is one problem for an agent to solve. It packages an instruction, a Linux environment, and a verifier that scores the result. Evolve accepts the Harbor task format, with the managed capabilities and limits described in these guides.

New to task authoring? Build your first task from five small files.

my-task

instruction.md

task.toml

environment

Dockerfile

tests

test.sh

solution

solve.sh

File	Purpose
`instruction.md`	What the agent must do
`task.toml`	Timeouts, resources, and policies
`environment/Dockerfile`	Tools and files available to the agent
`tests/test.sh`	Checks the work and writes a reward
`solution/solve.sh`	Optional reference solution

For a single-step task, instruction.md, task.toml, and a nonempty tests/test.sh are required. Define the environment with a Dockerfile, a pinned image, or supported Docker Compose configuration.

What happens in a trial

The agent receives the instruction and environment. The verifier and reference solution are not handed to it as task inputs.

Write a clear instruction

State the requested change, the output path, and the conditions for success. The verifier should check those same conditions.

instruction.md

Create /app/answer.txt containing exactly:

Hello, world!

End the file with a newline.

Use absolute paths when the location matters. A benchmark canary marker is removed before the instruction reaches the agent.

Add a reference solution

solution/solve.sh demonstrates a correct answer. For the instruction above:

solution/solve.sh

#!/bin/bash
printf 'Hello, world!\n' > /app/answer.txt

Keep it separate from the environment image. Task checks can use it to test whether the verifier accepts a valid solution.

Build the rest

Environment

Images, working directories, services, and MCP tools.

Verifier

Write a reward and choose where grading runs.

Configuration

Resources, network access, variables, and timeouts.

Multiple steps

Continue work across ordered instructions in one environment.

When the task is ready, publish it as a dataset.

Start here

Run evaluations

Task format

Validate your work

Read results

Work together

What happens in a trial

Write a clear instruction

Add a reference solution

Build the rest

Environment

Verifier

Configuration

Multiple steps