Checks

A check asks an agent to inspect tasks against a quality rubric. It can read a local directory or an existing dataset version.

For every method’s inputs, return values, and language differences, see the checks method reference.

Examples reuse the SDK setup. Run Python snippets inside the async function shown in the Python guide.

Create and wait

import { checks } from "@evolvingmachines/evolve";

const client = checks();
const check = await client.create({
  source: { directory: "./tasks" },
  agent: "codex",
  model_name: "gpt-6-luna",
  n_tasks: 1,
});
const finished = await client.watch(check.id);
for (const task of finished.results) {
  console.log(task.task_name, task.status, task.checks);
}

from evolve import checks

client = checks()
check = await client.create(
    "./tasks",
    agent="codex",
    model_name="gpt-6-luna",
    n_tasks=1,
)
finished = await client.watch(check["id"])
for task in finished["results"]:
    print(task["task_name"], task["status"], task["checks"])

For a published dataset:

const check = await client.create({
  source: { dataset: "deepswe@1.1" },
  agent: "codex",
  model_name: "gpt-6-luna",
  include_task_names: ["anko-default-function-arguments"],
});

check = await client.create(
    dataset="deepswe@1.1",
    agent="codex",
    model_name="gpt-6-luna",
    include_task_names=["anko-default-function-arguments"],
)

Choose one source. A local check does not first publish a dataset.

Configuration

Option	Purpose
`name`	Check label; defaults to an acceptance timestamp
`agent`	Agent the checker runs on; omitted uses the current platform default from `checks().defaults()`
`model_name`	Checker model, one of that agent’s models
`reasoning_effort`	Effort that agent accepts
`sandbox_provider`	Checker sandbox provider
`rubric`	`{criteria: [{name, description, guidance}, ...]}`
`prompt`	Replacement prompt template
`n_concurrent`	Requested parallel task checks; 1–150, bounded by team quota
`include_task_names`	Task-name include globs
`exclude_task_names`	Task-name exclude globs
`n_tasks`	Positive cap after filters
`onUploadProgress` / `on_upload_progress`	Local archive progress callback: sent/total bytes

TypeScript places these alongside source in the input object. Python takes keyword arguments.

Read current defaults

const defaults = await client.defaults();
console.log(defaults.rubric, defaults.prompt);

const codex = await client.defaults({ agent: "codex" });
console.log(codex.model_name, codex.reasoning_effort);

defaults = await client.defaults()
print(defaults["rubric"], defaults["prompt"])

codex = await client.defaults(agent="codex")
print(codex["model_name"], codex["reasoning_effort"])

Defaults include agent, model, rubric, prompt, effort, and provider. Omit agent to read the current platform defaults. Pass it to read a specific harness’s check model default and that model’s default effort. See supported models and efforts. Check prompt tokens are {task_path}, {file_tree}, and {criteria_guidance}. They differ from analysis prompt tokens.

Understand the result

Check
├── status
│   queued → running → completed
└── results[]
    ├── task_name, status, summary
    ├── checks
    │   └── criterion
    │       ├── outcome
    │       ├── explanation
    │       └── evidence
    └── failure, cost_usd

Each task can be queued, running, completed, or failed. The check becomes completed once every task settles, even if a task failed.

Field	Read it as
`summary`	Main findings and evidence gaps; null before completion or on older results
`checks`	Criterion outcomes: `pass`, `fail`, `not_applicable`; older records can contain `unknown`
`failure`	Why that task check could not produce a result

The API returns no overall verdict or separate execution flag. Read summary and checks for findings and evidence. not_applicable can mean the criterion does not apply or essential evidence is unavailable; its explanation distinguishes the two. Missing evidence leaves that part of the task’s behavior unresolved. Older records may use unknown for that case; see how to read a check result.

Read and follow checks

Method	Options/result
`get(id)`	Full check and its task results
`list(...)`	`scope`, `status` list, `dataset`, `limit`, `cursor`; paginated handle
`watch(id, ...)`	Wait until completed; callback on changes
`defaults({ agent })`	Rubric, editable prompt, and settings for the selected harness

Watch options are onProgress, pollIntervalMs, signal in TypeScript; on_progress, poll_interval_s, timeout_s in Python. Polling starts at 2 seconds and slows to 30 seconds while unchanged.

Inspect files and transcripts

Need	TypeScript	Python
Download whole check or one task check	`download(id, { to })`	`download(id, to=...)`
Browse a task check’s sandbox	`taskFilesystem(checkId, taskCheckId)`	`task_filesystem(check_id, task_check_id)`
Read one task check	`task(taskCheckId)`	Read the entry in `check["results"]`
Read checker transcript	`transcript(taskCheckId, { since })`	Not exposed
Read checker artifact	`artifact(taskCheckId, stream)`	Not exposed

For TypeScript transcripts, since is an inclusive nonnegative event index. Artifact selectors are trace-stdout, trace-stderr, and agent-home; the last returns a file map, the others text. Missing content is null. Filesystem methods and download options are shared with trials and analyses.

Action	TypeScript	Python
Share by link/email	`share(id, { link, emails })`	`share(id, link=..., emails=...)`
Revoke link/email access	`unshare(id, { link, emails })`	`unshare(id, link=..., emails=...)`
Read current shares	`shares(id)`	`shares(id)`

Only the creator can manage shares. Sharing by email can send invitations. See sharing.

HTTP API

Python and TypeScript

Run agents

Shared behavior

SDK workflows

SDK resources

SDK methods

Create and wait

Configuration

Read current defaults

Understand the result

Read and follow checks

Inspect files and transcripts

Create and wait

Configuration

Read current defaults

Understand the result

Read and follow checks

Inspect files and transcripts

Share a check