How evaluations fit together

Concept	What it means	Example
Task	An instruction, environment, and grading logic.	Fix a bug and pass the tests.
Dataset	A named, versioned collection of tasks.	`deepswe@1.1`.
Agent	A harness running a selected model.	Codex with `gpt-6-luna`.
Sandbox	The isolated environment where a trial runs.	A task image on Daytona, E2B, or Modal.
Job	A group of trials with shared run settings.	10 tasks × 2 configurations × 3 attempts = 60 trials.
Trial	One attempt at one task.	One score and its supporting evidence.
Verifier	The task’s code that evaluates the output.	A script writing a reward of `0` or `1`.
Trace	The recorded agent conversation and actions.	Messages, tool calls, and results.

Concept

What it means

Example

An instruction, environment, and grading logic.

Fix a bug and pass the tests.

A named, versioned collection of tasks.

deepswe@1.1.

A harness running a selected model.

Codex with gpt-6-luna.

The isolated environment where a trial runs.

A task image on Daytona, E2B, or Modal.

A group of trials with shared run settings.

10 tasks × 2 configurations × 3 attempts = 60 trials.

One attempt at one task.

One score and its supporting evidence.

The task’s code that evaluates the output.

A script writing a reward of 0 or 1.

The recorded agent conversation and actions.

Messages, tool calls, and results.

Two ways to examine quality

Check the task

Review the instruction, environment, solution, and verifier before scaling the evaluation.

Review a recorded trial against a rubric to understand the agent’s behavior.

Checks and analyses are optional model-powered operations with their own results and costs. The task’s verifier produces the evaluation reward.

Read the result at the right level

Level	What to read
Job configuration	What was selected and resolved
Job summary	Counts, scores, and costs
Each trial: status	Whether execution and grading completed
Each trial: reward	How the verifier scored the work
Each trial: evidence	Trace, logs, and recorded files