Skip to main content
An analysis reads a settled trial’s whole record, the trace and the task, and rules on every criterion of a rubric. It runs on the platform, one analyzer run per trial. Ask for it on a finished job:
Or arm it when you start the job, so each trial is analyzed the moment it settles:

​
Rubrics and prompts

The default rubric has seven criteria, score_is_earned first. Bring your own as TOML, YAML or JSON in the shape below and, if you want, your own prompt for the analyzer.
rubric.toml
-r is the rubric file, -p the prompt file. --passing and --failing restrict the batch of analyses to trials with reward 1.0, or below it; both together is refused. -l caps the number of trials, -n how many analyses run at once, beneath the organization’s max_concurrent_analyses (fleet default 60). -m picks the analyzer’s model, --effort its reasoning effort, -e its sandbox provider. A result that misses or invents a criterion is a failed analysis, never a partial pass. A rubric with unknown keys, empty or duplicate criteria, or a length bound exceeded is refused with invalid_rubric; an off-roster model or an unknown effort or provider with invalid_input; a job with no analyzable trial, or a --passing or --failing filter that selects none, with no_analyzable_trials; a job still running with job_not_terminal. On evolve run the same knobs are --analyze-model, --analyze-rubric, --analyze-prompt, --analyze-effort and --analyze-provider. Each of them implies --analyze. Running evolve analyze again on the same job, with another rubric or model, is the re-analysis path: one wave at a time (analysis_already_running meanwhile), and each trial then serves its newest analysis while earlier ones stay under their own ids. Cancelled trials are never analyzed.

​
The result

Each analysis produces a short summary of what happened in the trial, and one entry per criterion:
  • outcome: pass, fail, not_applicable or unknown
  • explanation: the analyzer’s reasoning, in plain words
  • evidence: what in the trial record supports it
Under the default rubric, the platform derives one label from the outcomes:
  • flagged: a fail on score_is_earned, score_is_correct, task_was_fair or report_is_truthful
  • env_fault: otherwise, a fail on environment_worked
  • unclear: otherwise, an unknown on any of those five, or not_applicable on score_is_earned or score_is_correct
  • clean: everything else
A custom rubric carries its per-criterion outcomes and no label. The analyzer’s spend is shown apart from the trial’s: estimated_cost_usd on the analysis, stats.analysis.cost_usd on the job. A failed analysis carries failure with a phase and a message; a run that produces no valid result is retried once.

​
Read it back

analysis show prints the verdict document. Every analysis verb also accepts a trial id, which names that trial’s latest analysis. analysis trace prints the analyzer’s own transcript. analysis download saves the run as a trial folder, or streams one artifact: analysis, trace-parsed, trace-stdout, trace-stderr or agent-home.

analyze reference

Every flag of evolve analyze and evolve analysis.