Analyze trials

An analysis reviews a trial’s trace, result, and available task files. It produces explanations and evidence; it does not replace the trial’s reward.

1. Select settled trials
           ↓
2. Run one analyzer per trial
           ↓
3. Validate every criterion result
           ↓
4. Read findings + evidence + summary

Start an analysis

Use the ID returned by your evaluation as $JOB_ID:

evolve analyze "$JOB_ID" --watch

The manual command requires a finished job. Cancelled trials are excluded.

Choose trials
Analyze automatically

evolve analyze "$JOB_ID" \
  --failing \
  -l 20 \
  --watch
evolve analyze "$JOB_ID" \
  -t "$TRIAL_ID" \
  --watch

--passing selects scored trials with primary reward 1. --failing selects every other eligible outcome, including errors and missing rewards. They cannot be combined.Repeat -t with full trial IDs to select specific trials; selection happens before the -l cap.

evolve run \
  -d deepswe@1.1 \
  -i anko-default-function-arguments \
  -a codex \
  -m gpt-6-luna \
  --max-trial-spend 1 \
  -r 0 \
  --analyze \
  --watch

--analyze schedules analysis after each eligible trial settles. Analysis has its own lifecycle and cost.

When using embedded analysis through an SDK, wait for the job first, then its analyses. A temporary zero-pending analysis count does not mean a running job has finished producing trials.

Use the SDK setup from the quick start. Before creating its job, add analyze: {} to the TypeScript request or analyze={} to the Python call. Then use that job response below:

import { jobs } from "@evolvingmachines/evolve";

// job was created with analyze: {}.
const finished = await jobs().watch(job.id);
const reviewed = await jobs().watchAnalysis(finished.id);

from evolve import jobs

# job was created with analyze={}.
finished = await jobs().watch(job.id)
reviewed = await jobs().watch_analysis(finished.id)

Rubrics and prompts

Read the current rubric, prompt, model, effort, and provider before customizing:

evolve analyze --show-defaults

evolve analyze \
  --show-defaults \
  -a codex
evolve analyze "$JOB_ID" \
  -a codex \
  --watch

The default analyzer is codex with gpt-6-luna at medium effort. Read --show-defaults for your server’s resolved settings. Use -a to choose another built-in harness. Omit -m to use its default model for analyses, or choose one of its models. The default rubric reviews eight criteria:

Criterion	Reviews
`specification_completeness`	Whether the instructions, environment, and grading agree.
`solution_discoverability`	Whether an expert could find a solution from the available information and resources.
`verifier_correctness`	Whether grading accepts correct answers and rejects incorrect ones.
`boundary_fairness`	Whether the pass threshold measures meaningful quality.
`policy_refusal`	Whether a content or usage policy stopped the agent.
`execution_blocked`	Whether a limit or external fault stopped useful work.
`unearned_credit`	Whether the result came from work, exploitation, or prior knowledge of the answer.
`meaningful_difficulty`	Whether the outcome reflects the capability the task intends to test.

Use your own rubric or prompt

A rubric is a list of uniquely named criteria. Supply TOML, YAML, or JSON:

rubric.toml

[[criteria]]
name = "changes_are_explained"
description = "The final response explains the changes made."
guidance = "Compare the final response with the recorded file edits."

evolve analyze "$JOB_ID" \
  -r rubric.toml \
  -p prompt.txt \
  --watch

An analyzer prompt can use {trial_path}, {task_section}, and {criteria_guidance}. Evolve appends the required output format.Use --effort, -e, and -n to set reasoning effort, sandbox provider, and concurrency. On job creation, use the corresponding --analyze-* flags. See the full reference.

The result

Each criterion has an outcome, an explanation, and an evidence list. A failed criterion requires evidence; a pass or not_applicable may have an empty list.

Outcome	Meaning
`pass`	The record supports the criterion.
`fail`	The record contradicts it.
`not_applicable`	No subject applies, or essential evidence is unavailable. Read the explanation.

The result also includes a written summary and failure_modes: independently identified findings that can overlap the rubric. An empty list means no failure mode was supported. The API does not return an overall verdict. Older results stored before October 4, 2026 can contain unknown. New results use the three outcomes above. Analyses imported from Harbor have evidence: null because their source files do not contain evidence entries. The dashboard computes a short summary label from those outcomes. Its rules depend on the result’s criterion names. Apply the first matching rule for the current defaults:

Rule	Summary label
A criterion other than `execution_blocked` fails.	Flagged
`execution_blocked` fails.	Env fault
No criterion passes or fails.	Unclear
At least one criterion passes and `failure_modes` is nonempty.	Failure modes
At least one criterion passes and no failure mode is recorded.	Clean

Earlier default rubric: read historical results

Results with the earlier seven-criterion rubric keep their original summary rules. The two score criteria are score_is_earned and score_is_correct. Apply the first matching rule:

Rule	Summary label
A score criterion, `task_was_fair`, or `report_is_truthful` fails.	Flagged
`environment_worked` fails.	Env fault
Any of these five criteria is unknown, or a score criterion is not applicable.	Unclear
None of the above.	Clean

ended_by_its_own_decision and worked_as_for_a_real_user do not change this label. Read their findings even when the summary is Clean. If the otherwise clean result has recorded failure modes, its label becomes Failure modes.

Other rubrics: read the dashboard summary

Apply the first matching rule across all criteria:

Rule	Summary label
Any criterion fails.	Flagged
Any criterion is unknown.	Unclear
Neither of the above, but failure modes are recorded.	Failure modes
None of the above.	Clean

A missing or extra criterion makes the analysis result invalid. A run without a valid result can retry once. The analysis’s estimated_cost_usd and the job’s stats.analysis.cost_usd are separate from evaluation spend.

Read it back

evolve analysis list --job "$JOB_ID"
evolve analysis show "$ANALYSIS_ID"
evolve analysis trace "$ANALYSIS_ID"
evolve analysis download "$ANALYSIS_ID" -o analyses/

show, trace, and download also accept a trial ID to read its latest analysis. Filesystem commands require the analysis ID itself. Run evolve analyze again to apply a new rubric or model. Only one analysis wave runs per job at a time. Earlier analyses remain accessible by their own IDs.

Analyze reference

Selection, customization, and downloads.

Check tasks

Review the benchmark before running agents.

Start here

Run evaluations

Task format

Validate your work

Read results

Work together

Analyze trials

Start an analysis

Rubrics and prompts

The result

Read it back

Analyze reference

Check tasks