Check tasks

A check reviews task quality. An analysis reviews an agent’s completed trial. Both use rubrics; they answer different questions.

evolve check ./tasks --watch

evolve check \
  -d deepswe@1.1 \
  -i anko-default-function-arguments \
  --watch

Local checks upload the selected task material. Dataset checks use the retained package; they do not publish a new version.

Choose the tasks

Read the source

Source	Uses
Task directory	One task.
Task collection	Top-level task directories.
Published dataset	Retained task package.

Filter task names

Apply include (-i) and exclude (-x) globs.

Limit the selection

Apply the task cap (-l).

Run the checks

Run one checker per selected task.

evolve check ./tasks \
  -i 'auth-*' \
  -x 'auth-legacy' \
  -l 5 \
  --watch

A check is a hosted model run and incurs cost. evolve dataset check is a separate metadata preflight; it does not perform this rubric review.

What it reviews

The default rubric has 26 criteria, grouped here by purpose:

Review area	What it checks
Instructions	Clear requirements, paths, output schemas, and alignment with tests.
Solution and limits	A valid solution that performs the work within suitable resources.
Grading	Correct, reproducible behavior checks, justified reference values, and meaningful pass conditions.
Security	Resistance to cheating, malicious code, and changes to protected files or grading state.
Environment and artifacts	Separation of private material, sufficient verifier inputs, dependencies, and efficient artifact collection.
Reviewability	Useful explanations and per-test reporting.

Read the complete rubric guidance and the current agent, model, effort, provider, and prompt:

evolve check --show-defaults

evolve check \
  --show-defaults \
  -a codex
evolve check ./tasks \
  -a codex \
  --watch

The default checker is codex with gpt-6-luna at medium effort. Read --show-defaults for your server’s resolved settings. Use -a to choose another built-in harness. Omit -m to use its default model for checks, or choose one of its models.

Use your own rubric or prompt

Use -r rubric.toml or -p prompt.txt to customize the review. A rubric accepts TOML, YAML, or JSON with uniquely named criteria:

rubric.toml

[[criteria]]
name = "inputs_are_documented"
description = "The instructions describe every required input."
guidance = "Compare the input files with the instructions."

Check prompt tokens are {task_path}, {file_tree}, and {criteria_guidance}. The required result format is appended automatically. See the check reference for all options.

The result

One parent check contains one task check per selected task:

check
├── task check: anko-default-function-arguments
│   ├── criteria
│   │   ├── outcomes
│   │   ├── explanations
│   │   └── evidence
│   └── checker trace + files
└── task check: another-task
    └── ...

The result includes a written summary and named criterion findings:

Outcome	Meaning
`pass`	The evidence supports the criterion.
`fail`	The evidence contradicts it. At least one evidence entry is required.
`not_applicable`	No subject applies, or essential evidence is unavailable. Read the explanation.

Every criterion explains its judgment. Pass and not_applicable findings may have an empty evidence list. Older results stored before October 4, 2026 can contain unknown; new results do not. The API does not return an overall verdict or a separate execution flag. The dashboard summarizes the current default rubric using the first matching rule:

Rule	Summary label
Any criterion fails.	Has a problem
At least one criterion passes.	No problem found
Every criterion is `not_applicable`.	Unclear

No problem found does not prove the task ran. Read the evidence and any not_applicable explanations. Current checks have no Executed or Not executed label.

Earlier default rubric: read historical results

Results with the earlier thirteen-criterion rubric keep these summary rules. Apply the first matching rule:

Rule	Summary label
Any criterion fails.	Has a problem
A criterion is unknown, except an execution criterion or `attempt_isolation`.	Unclear
Neither of the above.	No problem found

attempt_isolation changes the summary only when it fails.The execution group contains these five criteria:

reference_solution_is_valid
verifier_rejects_non_solutions
environment_builds_and_runs
verification_is_stable
limits_allow_the_task

If any is unknown, the dashboard adds Not executed. Otherwise it adds Executed. This note summarizes the outcomes; it is not a separate execution audit.Read the evidence before relying on the note. An unknown means decisive execution evidence was unavailable, such as when the checker could not run the environment.

Other rubrics: read the dashboard summary

Apply the first matching rule across all criteria:

Rule	Summary label
Any criterion fails.	Has a problem
Any criterion is unknown.	Unclear
Neither of the above.	No problem found

Other rubrics have no Executed or Not executed note.

The parent lifecycle is queued → running → completed. A completed check can contain failed task checks; inspect each task’s status and failure.

Read it back

evolve check list --status running
evolve check show "$CHECK_ID"
evolve check trace "$TASK_CHECK_ID"
evolve check download "$CHECK_ID" -o checks/

Use the parent check ID for the whole report. Use a task-check ID for one checker’s transcript, filesystem, or individual result:

evolve check download "$TASK_CHECK_ID" --stream task-check

evolve check share "$CHECK_ID" --link

The creator can share by link or email. Recipients can read the report and available files. Sharing does not give them permission to stop task checks.

Check reference

Flags, custom rubrics, and task-check commands.

Start here

Run evaluations

Task format

Validate your work

Read results

Work together

Choose the tasks

What it reviews

The result

Read it back

Check reference

Choose the tasks

What it reviews

The result

Read it back

Share

Check reference