evolve run

Start a hosted job. evolve job start accepts exactly the same options.

evolve run \
  --dataset deepswe@1.1 \
  --include-task-name anko-default-function-arguments \
  --agent codex \
  --model gpt-6-luna \
  --max-trial-spend 1 \
  --max-retries 0 \
  --watch

This selects one task and one agent/model combination. Each selected task runs once per combination unless you increase --n-attempts. Without --watch, the command returns when the job is accepted. Acceptance does not mean the trials have finished. Keep the returned job ID to inspect results.

evolve run --help

Select tasks and agents

Option	Meaning	Default
`-d`, `--dataset <name[@version]>`	Dataset to evaluate. Repeat for more datasets.	Required, unless in config.
`-a`, `--agent <name[@version]>`	Built-in or registered agent harness.	Required, unless in config.
`-m`, `--model <name>`	Model for the harness. Repeat for one combination per model.	Required, unless in config.
`-i`, `--include-task-name <glob>`	Include matching tasks in every dataset. Repeatable.	All tasks.
`-x`, `--exclude-task-name <glob>`	Exclude matching tasks after inclusion. Repeatable.	None.
`-l`, `--n-tasks <n>`	Cap tasks per dataset, after filters. Integer of at least `1`.	No cap.
`-k`, `--n-attempts <n>`	Attempts per task and agent/model combination. Integer of at least `1`.	`1`
`-n`, `--n-concurrent <n>`	Parallel trials. Integer from `1` to `150`.	`4`

A bare dataset name selects its active version. Pin name@version to select a specific version. Quote globs, such as 'auth-*', so your shell does not expand them.

Use a config file

Use YAML or JSON for several agent harnesses or different filters per dataset. Save this as job.yaml:

job.yaml

job_name: deepswe-check
datasets:
  - name: deepswe
    version: "1.1"
    task_names:
      - anko-default-function-arguments
agents:
  - name: codex
    model_name: gpt-6-luna
n_attempts: 1
n_concurrent_trials: 4
max_trial_spend_usd: 1
retry:
  max_retries: 0

Inspect the request, then submit it:

evolve run --config job.yaml --print-config
evolve run --config job.yaml --watch

--print-config creates no job and uploads no local skills. It does not resolve catalog names or fill every server default. It masks sandbox credentials, but prints inline agent secret values.

How flags override the file

Flag	What it replaces
`-d`	The entire dataset list.
`-a` and `-m`	The entire agent list. Supply both.
`-i`, `-x`, `-l`	That field on every dataset selector.
`--effort`, `--preset`	That field on every agent combination.
`--ak`	Matching keys inside each agent’s `kwargs`.
`--skill`	Every agent’s skills list.
`--verifier-env`	The complete `verifier_env` object.
`--secret` / `--secret-inline`	The complete secret attachment list.
`--own-infra`	The file’s `sandbox_credentials`.
Retry or analysis options	Only the named field inside `retry` or `analyze`.

Other flags override their matching field. Select an organization with --org or evolve auth org use; the CLI does not retain org from the job file.Quote version strings in YAML, for example version: "1.10". Config validation rejects unknown fields, duplicate keys, and invalid field types before submission.

Set spend and retries

Option	Meaning	Default
`--max-trial-spend <usd>`	Agent model-spend cap per trial attempt.	Platform default, normally `$200`.
`-r`, `--max-retries <n>`	Additional attempts after eligible infrastructure errors. Integer of at least `0`; `0` disables them.	Platform default, normally `2`.
`--retry-include <exception>`	Retry only these exception types. Repeatable.	Any type not excluded.
`--retry-exclude <exception>`	Never retry these exception types. Repeatable; takes priority over include.	Built-in exclusions.

The spend cap must be from $0.000001 to $999999.999999; values are stored to six decimal places. Each retry receives the full trial cap. A low score does not trigger an infrastructure retry. Capacity waits are separate and can still occur with --max-retries 0. The cap stops new model calls when reached, but a call already in progress can finish above it. There is no job-wide spend cap. Verifier judge spend, analysis, and GPU compute are separate from the agent cap. See Jobs for estimates and retry rules.

Additional options

Job, organization, and sandbox

Option	Meaning
`--job-name <name>`	Job label. Generated when omitted.
`--org <name>`	Organization to use. Otherwise use the saved CLI default, then your personal organization. `--org personal` overrides a saved team default.
`-e`, `--env <provider>`	Requested sandbox provider: `daytona` (default), `e2b`, or `modal`.
`--own-infra`	Run the job’s sandboxes in your own account on the selected provider. See your own provider account.
`--system-log`	Record the sandbox’s system log. Off by default.

For --own-infra, keys come from your shell: E2B_API_KEY, DAYTONA_API_KEY, or MODAL_TOKEN_ID and MODAL_TOKEN_SECRET. Modal also reads its active profile in ~/.modal.toml. Missing values fall back to your stored secrets. --print-config masks these values.On Evolve’s account, a task may run on another compatible provider. Inspect its trial for the provider actually used. With --own-infra, an incompatible task is rejected instead of moved. See Sandboxes for restrictions.

Agent settings and skills

These options apply to every agent combination.

Option	Meaning
`--effort <value>`	Reasoning effort. Defaults to the agent’s own setting. See the model reference.
`--preset <name>`	Supported settings preset: `no-internet` or `pinned-context`.
`--ak <key=value>`, `--agent-kwarg <key=value>`	Agent setting. Repeatable. `config` accepts a local JSON/TOML file or inline JSON object.
`--skill <ref>`, `--skills <ref>`	Skill reference or local directory. Repeatable.

evolve run --config job.yaml \
  --effort high \
  --ak config=./codex.toml

The example reads native agent settings from codex.toml. Other --ak values parse as JSON, then True, False, or None, then plain text.Skills accept skills.sh/owner/repo[/skill], org/repo[@ref], a Git URL, upload:<id>, or name:<skill-name>. Use a path such as ./my-skill for a local directory; it uploads before the job starts.Unsupported effort, presets, or native configuration settings are refused. See Agents and Skills.

Secrets and verifier environment

Option	Meaning
`--secret <NAME[@LABEL][=ENVNAME]>`	Attach a stored secret to agent runs. Repeatable. `@LABEL` selects a row; `=ENVNAME` renames the environment variable.
`--secret-inline <NAME[@LABEL]:DELIVERY=VALUE>`	Store a secret and attach it. Evaluation jobs require `direct` delivery. Repeatable.
`--verifier-env <KEY=VALUE>`, `--ve <KEY=VALUE>`	Verifier overrides: `REWARDKIT_JUDGE` and `REWARDKIT_MODEL`. Repeatable.

Store sensitive values with evolve secrets set, then attach them by name:

evolve run --config job.yaml --secret GITHUB_TOKEN

Analyze traces as trials finish

Option	Meaning
`--analyze`	Enable analysis with the platform defaults.
`--analyze-agent <name>`	Agent the analyzer runs on. Uses the platform analysis default when omitted.
`--analyze-model <name>`	Analyzer model, from that agent’s models.
`--analyze-rubric <path>`	TOML, YAML, or JSON rubric. Defaults to the built-in rubric.
`--analyze-prompt <path>`	Non-empty prompt file that replaces the built-in prompt.
`--analyze-provider <provider>`	Analyzer sandbox provider. Defaults to the platform analysis provider.
`--analyze-effort <value>`	Analyzer reasoning effort. Uses the resolved analysis default when omitted.

Any --analyze-* option enables analysis. Cancelled trials are skipped. Read the current defaults with evolve analyze --show-defaults.To analyze an existing job, use evolve analyze.

Timeouts

Every multiplier must be finite and greater than zero. A phase-specific value overrides the general multiplier for that phase.

Option	Timeout affected
`--timeout-multiplier <x>`	All task timeouts. Default `1.0`.
`--agent-timeout-multiplier <x>`	Agent execution.
`--verifier-timeout-multiplier <x>`	Verifier execution.
`--agent-setup-timeout-multiplier <x>`	Agent installation/setup.
`--environment-build-timeout-multiplier <x>`	Environment preparation.

Values below 1 shorten a timeout. The task definition is unchanged. A resulting phase timeout above 2,147,483,647 milliseconds (about 24.86 days) is rejected.

Configuration and output

Option	Meaning
`-c`, `--config <path>`	Load YAML or JSON job configuration.
`--print-config`	Print the constructed request and exit.
`--watch`	Follow events until the job settles.
`-q`, `--quiet`	With watch, suppress intermediate events.
`-y`, `--yes`	Accepted for compatibility; no effect.

With --json alone, the command prints the accepted job object. With --watch --json, it prints one JSON record per line: job.created, event records, then job.final. Quiet mode keeps the initial and final records.A watched job exits with code 0 for COMPLETED, or 1 for FAILED or CANCELLED. A completed job can contain failed trials; inspect the trial results before treating it as a successful evaluation.

Global options also apply. Next, inspect the job.

Start work

Inspect results

Manage resources

Select tasks and agents

Use a config file

Set spend and retries

Additional options