checks - Evolve

checks() returns the checks client.

create

TypeScript
Python

create(input: { source: { directory: string } | { dataset: string }; model_name?: string; rubric?: Rubric; prompt?: string; reasoning_effort?: string; sandbox_provider?: EvalSandboxProvider; n_concurrent?: number; include_task_names?: string[]; exclude_task_names?: string[]; n_tasks?: number; onUploadProgress?: (sentBytes: number, totalBytes: number) => void }): Promise<Check>

Check a local task directory or a directory of them, uploaded as one archive (upload_too_large past the published ceiling), or a published dataset, nothing uploaded. Returns the accepted check with one queued result per task; an empty selection is refused with no_checkable_tasks, and a dataset version resolves as for a job (dataset_not_found, no_active_version, version_not_ready). A check on a dataset version also appears on every job spanning it, as check on jobs().tasks() rows.

const check = await checks().create({ source: { dataset: "my-swe@1.0" } });

async def create(
    directory: Optional[str] = None,
    *,
    dataset: Optional[str] = None,
    model_name=None,
    rubric: Optional[Rubric] = None,
    prompt=None,
    reasoning_effort=None,
    sandbox_provider=None,
    n_concurrent=None,
    include_task_names=None,
    exclude_task_names=None,
    n_tasks=None,
    on_upload_progress=None,
) -> Check

Check a local task directory or a directory of them, uploaded as one archive (upload_too_large past the published ceiling), or a published dataset, nothing uploaded; one source, not both. Returns the accepted check with one queued result per task; an empty selection is refused with no_checkable_tasks, and a dataset version resolves as for a job. A check on a dataset version also appears on every job spanning it, as check on jobs().tasks() rows.

check = await checks().create(dataset="my-swe@1.0")

get, list, watch

TypeScript
Python

get(checkId: string): Promise<Check>
list(options?: { scope?: "my" | "shared"; status?: CheckStatus[]; limit?: number; cursor?: string }): CheckList
watch(checkId: string, options?: { onProgress?: (check: Check) => void; signal?: AbortSignal; pollIntervalMs?: number }): Promise<Check>

One check with its per-task results, your checks, or a poll until every task settled.

const done = await checks().watch(check.id);

async def get(check_id: str) -> Check
def list(
    *,
    scope: Optional[JobListScope] = None,
    status: Optional[List[CheckStatus]] = None,
    limit: Optional[int] = None,
    cursor: Optional[str] = None,
)
async def watch(
    check_id: str,
    *,
    on_progress: Optional[Callable[[Check], None]] = None,
    poll_interval_s: float = 2.0,
    timeout_s: Optional[float] = None,
) -> Check

One check with its per-task results, your checks, or a poll until every task settled.

done = await checks().watch(check.id)

task, transcript, artifact

TypeScript
Python

task(taskCheckId: string): Promise<TaskCheck>
transcript(
  taskCheckId: string,
  options?: { since?: number },
): Promise<TaskCheckTranscript>
artifact(
  taskCheckId: string,
  stream: "trace-stdout" | "trace-stderr",
): Promise<string | null>
artifact(
  taskCheckId: string,
  stream: "agent-home",
): Promise<Record<string, string> | null>

One task’s result by its task check id, the checker’s own transcript (everything after since in one read), or one stored artifact; a task check has no verifier log and no trajectory. An id you may not read answers trial_not_found.

const result = await checks().task(done.results[0].id);

The Python client has no per-task reads. A task’s result is in Check.results, returned by get and watch.

download

TypeScript
Python

download(id: string): Promise<Buffer>
download(
  id: string,
  options: { to: string },
): Promise<string>
download(
  id: string,
  options: { stream: true },
): Promise<ReadableStream<Uint8Array>>

The whole check by its check id (check-<id>/ with check_report.json and one folder per task), or one task’s folder by its task check id (check-<task>__<7 chars>/), as one .tar.gz. Each task folder is a trial tree of the checker’s own run, with artifacts/check-result.json on a completed run. An id that is neither answers check_not_found; an unsettled check or task check answers check_not_terminal.

const path = await checks().download(
  check.id,
  { to: "./checks" },
);

async def download(
    id: str,
    *,
    to: Optional[str] = None,
)

The whole check by its check id (check-<id>/ with check_report.json and one folder per task), or one task’s folder by its task check id (check-<task>__<7 chars>/), as one .tar.gz. An id that is neither answers check_not_found; an unsettled check or task check answers check_not_terminal.

path = await checks().download(
    check.id,
    to="./checks",
)

​create

​get, list, watch

​task, transcript, artifact

​download

create

get, list, watch

task, transcript, artifact

download