Analyses

Analysis asks an agent to read a trial’s evidence against a rubric. It produces criterion outcomes and explanations. It does not replace the verifier’s reward.

For every method’s inputs, return values, and language differences, see the analyses method reference.

Trial trace + task + rubric
             │
             ▼
        Analyzer run
             │
             ▼
 Summary + criterion outcomes + evidence

Examples reuse the SDK setup. Use IDs returned by the relevant list or create call. Run Python snippets inside the async function shown in the Python guide.

Start an analysis

Start work through the jobs client. Read analyzer runs through the analyses client.

import { jobs, analyses } from "@evolvingmachines/evolve";

await jobs().analyze(jobId, {
  agent: "codex",
  model_name: "gpt-6-luna",
  failing: true,
  n_trials: 1,
});
const job = await jobs().watchAnalysis(jobId);
console.log(job.stats.analysis);

from evolve import jobs, analyses

await jobs().analyze(
    job_id,
    agent="codex",
    model_name="gpt-6-luna",
    failing=True,
    n_trials=1,
)
job = await jobs().watch_analysis(job_id)
print(job.stats.get("analysis"))

Manual analysis requires a terminal job. It returns immediately after queuing work. Only one analysis wave can run for that job at a time. To analyze trials as they finish, pass analyze: {} / analyze={} when starting the job. Wait for job completion before calling the analysis watch.

Configuration

The field names are the same in both languages. TypeScript passes an object; Python uses keyword arguments on analyze() or a dictionary inside job creation.

Field	Purpose
`agent`	Agent the analyzer runs on; omitted uses the current platform default from `analyses().defaults()`
`model_name`	Analyzer model, one of that agent’s models
`reasoning_effort`	Effort that agent accepts
`sandbox_provider`	Analyzer’s sandbox provider
`rubric`	`{criteria: [{name, description, guidance}, ...]}`
`prompt`	Replacement prompt template text
`n_concurrent`	Requested parallel analyses; integer 1–150, also bounded by team quota
`passing`	Only scored trials with primary reward exactly 1
`failing`	Other analyzable trials
`n_trials`	Positive cap after selection
`trial_ids`	Explicit trial ids from this job; manual analysis only

passing and failing cannot both be true. Cancelled trials are excluded. Explicit ids are filtered before the cap; unknown, duplicate, or empty ids are refused.

Analysis watch options

TypeScript	Python	Purpose
`onStats`	`on_stats`	Called with the job when analysis tally changes
`pollIntervalMs`	`poll_interval_s`	Initial interval; default 2000 ms / 2 s
`signal`	`timeout_s`	Stop waiting / bound the Python wait

Polling slows to at most 30 seconds while the tally stays unchanged. Call the watch after starting analysis; a job never analyzed can have a null tally indefinitely.

Read current defaults

const defaults = await analyses().defaults();
console.log(defaults.model_name, defaults.rubric, defaults.prompt);

const codex = await analyses().defaults({ agent: "codex" });
console.log(codex.model_name, codex.reasoning_effort);

defaults = await analyses().defaults()
print(defaults["model_name"], defaults["rubric"], defaults["prompt"])

codex = await analyses().defaults(agent="codex")
print(codex["model_name"], codex["reasoning_effort"])

The response includes agent, model_name, rubric, prompt, reasoning_effort, and sandbox_provider. The prompt is the editable template text. Omit agent to read the current platform defaults. Pass it to read a specific harness’s analysis model default and that model’s default effort. See supported models and efforts. A custom analyze prompt may use {trial_path}, {task_section}, and {criteria_guidance}. The required result schema is appended by the platform. See analysis concepts for rubric design.

Read results

for await (const result of analyses().list({ job: jobId })) {
  console.log(result.id, result.status, result.checks);
}

async for result in analyses().list(job=job_id):
    print(result["id"], result["status"], result["checks"])

List options: scope, job, status list, limit, cursor. Status values are queued, running, completed, failed. trial.analysis prefers the latest completed Evolve result, then an imported Harbor result, then the newest Evolve result in any state. A newer pending or failed run can appear in trial.newer_analysis while the earlier completed result stays available. New criterion results use pass, fail, or not_applicable; older records can contain unknown. not_applicable can mean the criterion does not apply or essential evidence is unavailable; read its explanation. failure_modes lists separate findings supported by evidence. See result fields for imported results and nullable evidence. The API returns no overall verdict. Use checks for your own result handling; the dashboard’s summary label is computed from those outcomes.

Inspect the analyzer run

Method	TypeScript	Python
List runs	`list(options)`	`list(...)`
Read defaults	`defaults({ agent })`	`defaults(agent=...)`
Download run	`download(analysisId, { to })`	`download(analysis_id, to=...)`
Browse sandbox files	`filesystem(analysisId)`	`filesystem(analysis_id)`
Read one analysis directly	`get(analysisId)`	Use list results or `trial.analysis`
Read analyzer transcript	`transcript(analysisId, { since })`	Not exposed
Read raw stored artifact	`artifact(analysisId, stream)`	Not exposed

analysisId identifies the analyzer run, not the trial it read. since is an inclusive nonnegative event index; the transcript returns events, total, run metadata, and separate gateway_calls. TypeScript artifact selectors are trace-stdout, trace-stderr, and agent-home. The first two return text or null; agent-home returns a file map or null. Downloads return a wrapper-trial archive. See filesystem and download options.

HTTP API

Python and TypeScript

Run agents

Shared behavior

SDK workflows

SDK resources

SDK methods

Start an analysis

Configuration

Read current defaults

Read results

Inspect the analyzer run