evolve run. It is the short form of evolve job start, and both take the same flags.
-d names a dataset and may repeat: one job can span several. -a names the agent. -m names the model and may repeat too; each model is one arm, so two models against one agent run every task twice.
-k is the number of attempts per task and arm (default 1). -n is how many trials run in parallel (default 4, at most 150). --job-name gives the job a label; omitted, the server generates one.
Narrow the task set
-i and -x are include and exclude globs over task names, applied to every dataset. -l caps the task count of each dataset after the filters.
Watch it
--watch streams the job’s events until it finishes: each trial as it starts, scores and settles, then the final block. -q keeps only the final block. Without --watch the command prints the job id and returns; read the job later with evolve job show.
Spend and retries
Every trial runs under a model-spend cap,--max-trial-spend <usd>; omitted, the server’s default applies, $200. It is the only spend limit: there is no job-wide budget. The job states the cap it applied as max_trial_spend_usd and the most it can cost as worst_case_spend_usd, the cap times the trials times the attempts the retry policy allows. A trial that exhausts its cap is not a failure; it is scored on what it produced.
Your credit balance sits under every cap. A job started at a zero balance is refused with insufficient_credits, and a balance that runs out stops spending mid-job.
Trials that fail on infrastructure errors are retried automatically. -r <n> sets how many times; the default is 2, and 0 turns it off. No other status re-runs on its own. --retry-include and --retry-exclude choose by exception type, exclude wins, and the names are on the run reference.
A retried trial keeps its history: n_retries and the retries list on the trial, stats.n_retries on the job, and every retried attempt’s spend stays in the job total. Two infrastructure failures of the same kind in a row stop the retries early; the trial stays INFRASTRUCTURE_ERROR, its message says why, and the event stream carries trial.retry_circuit_broken.
Read the results
job show prints the whole job, with pass@k once every attempt has settled. job trials lists the trials, --status and --dataset narrow the list. job tasks is one row per task.
job compare puts two to ten jobs side by side. job list is your jobs, newest first; --scope shared lists your organizations’ jobs that teammates created.
A job’s spend prints as at least $X while it holds settled trials whose spend has not been measured yet.
pass@k is the chance that k attempts contain at least one success, averaged over the tasks of an arm. The k values are the powers of two and the multiples of five up to the smallest attempt count, so a single-attempt job has none. An attempt that produced no reward counts as a failure, and a group with non-binary rewards or attempts still in flight shows nothing. The archive’s result.json carries the same numbers.
Statuses
A job isQUEUED (accepted, waiting for dispatch), RUNNING, CANCELLING (cancel requested, in-flight trials winding down), COMPLETED (every trial settled), CANCELLED, or FAILED. FAILED is reserved: nothing sets it today, so failure is null on every job. A job that goes wrong does so one trial at a time and still reaches COMPLETED; read trials.byStatus for where. Trial statuses are on trials.
Derive a new job
Every derived run is a new job linked to its source throughsource_jobs. The source is never changed.
resume takes a finished job’s failed or stopped trials and runs them: ScoringError, InfrastructureError, ApiUsageLimitError and IncompleteTrialError, plus stopped trials and the still-queued trials of a cancelled job. -f <type> narrows it by exception type. Scored trials are never re-run. A live source is refused with job_not_terminal, an empty selection with no_failed_trials.
retry runs selected trials again: every trial, only the failed ones with --failed-only (SCORING_ERROR, INFRASTRUCTURE_ERROR, BUDGET, INDETERMINATE), or exactly the ones named with -t. A scored trial is a legitimate target. The whole-job and --failed-only forms need a finished source; the named form works on a running job once every named trial has settled, else trial_not_settled, or trial_not_found for an id the job does not own. Naming trials and --failed-only together is refused.
regrade re-runs only the verifier of a finished job, under the network policy the source trial ran with; --status and --task narrow the set. Only settled separate-mode trials are eligible; shared-mode trials and judge tasks are not. A single-trial regrade of an ineligible source is refused with regrade_source_ineligible; a whole-job regrade skips the ineligible trials and answers no_regradable_trials when none remain. The result is a job with is_regrade true.
Stop, cancel, delete
cancel asks the whole job to stop; a finished job is a no-op. stop ends one dataset’s live trials and leaves the job running. delete removes a job you created permanently: its trials, traces, analyses and stored files. Without --yes it asks first.
Only the job’s creator may delete it; another member of the organization is refused with org_forbidden. The job must be finished: a live job, a running analysis wave or a live regrade derived from it refuses, naming what to wait for. Regrade jobs and the source_jobs entries of derived jobs stay. Deleting an uploaded job is how it is replaced; see upload.
Download
results/job-<id>/: the job’s config.json, lock.json, result.json and job.log, and one directory per trial. See trial outputs for what each trial directory holds.
Config files
Put the job in a file and run it with-c. The file is YAML or JSON with the same field names the SDK’s jobs().start() takes. Flags on the command line override the file’s fields, one field at a time. --print-config prints the resolved job body as JSON and exits without running anything.
nightly.yaml
run reference
Every flag of
evolve run.