> ## Documentation Index > Fetch the complete documentation index at: https://docs.evolvingmachines.ai/llms.txt > Use this file to discover all available pages before exploring further. # Jobs > Start a job, watch it, read its results, and derive new jobs from it. A job runs every task of its datasets with every agent arm, a fixed number of attempts each. Start one with `evolve run`. It is the short form of `evolve job start`, and both take the same flags. ```bash theme={null} evolve run \ -d terminal-bench-4@4.0 \ -a codex \ -m gpt-6-astra \ -k 2 \ -l 5 \ --watch ``` `-d` names a dataset and may repeat: one job can span several. `-a` names the agent. `-m` names the model and may repeat too; each model is one arm, so two models against one agent run every task twice. `-k` is the number of attempts per task and arm (default 1). `-n` is how many trials run in parallel (default 4, at most 150). `--job-name` gives the job a label; omitted, the server generates one. ## Narrow the task set ```bash theme={null} evolve run \ -d terminal-bench-4@4.0 \ -a codex \ -m gpt-6-astra \ -i 'auth-*' \ -x 'auth-legacy' \ -l 20 \ --watch ``` `-i` and `-x` are include and exclude globs over task names, applied to every dataset. `-l` caps the task count of each dataset after the filters. ## Watch it `--watch` streams the job's events until it finishes: each trial as it starts, scores and settles, then the final block. `-q` keeps only the final block. Without `--watch` the command prints the job id and returns; read the job later with `evolve job show`. ## Spend and retries Every trial runs under a model-spend cap, `--max-trial-spend `; omitted, the server's default applies, \$200. It is the only spend limit: there is no job-wide budget. The job states the cap it applied as `max_trial_spend_usd` and the most it can cost as `worst_case_spend_usd`, the cap times the trials times the attempts the retry policy allows. A trial that exhausts its cap is not a failure; it is scored on what it produced. Your credit balance sits under every cap. A job started at a zero balance is refused with `insufficient_credits`, and a balance that runs out stops spending mid-job. Trials that fail on infrastructure errors are retried automatically. `-r ` sets how many times; the default is 2, and `0` turns it off. No other status re-runs on its own. `--retry-include` and `--retry-exclude` choose by exception type, exclude wins, and the names are on the [run reference](/cli-reference/run#spend-and-retries). A retried trial keeps its history: `n_retries` and the `retries` list on the trial, `stats.n_retries` on the job, and every retried attempt's spend stays in the job total. Two infrastructure failures of the same kind in a row stop the retries early; the trial stays `INFRASTRUCTURE_ERROR`, its message says why, and the event stream carries `trial.retry_circuit_broken`. ## Read the results ```bash theme={null} evolve job show evolve job trials --status INFRASTRUCTURE_ERROR,SCORING_ERROR evolve job tasks evolve job compare evolve job list --search nightly ``` `job show` prints the whole job, with pass\@k once every attempt has settled. `job trials` lists the trials, `--status` and `--dataset` narrow the list. `job tasks` is one row per task. `job compare` puts two to ten jobs side by side. `job list` is your jobs, newest first; `--scope shared` lists your organizations' jobs that teammates created. A job's spend prints as `at least $X` while it holds settled trials whose spend has not been measured yet. pass\@k is the chance that k attempts contain at least one success, averaged over the tasks of an arm. The k values are the powers of two and the multiples of five up to the smallest attempt count, so a single-attempt job has none. An attempt that produced no reward counts as a failure, and a group with non-binary rewards or attempts still in flight shows nothing. The archive's `result.json` carries the same numbers. ## Statuses A job is `QUEUED` (accepted, waiting for dispatch), `RUNNING`, `CANCELLING` (cancel requested, in-flight trials winding down), `COMPLETED` (every trial settled), `CANCELLED`, or `FAILED`. `FAILED` is reserved: nothing sets it today, so `failure` is null on every job. A job that goes wrong does so one trial at a time and still reaches `COMPLETED`; read `trials.byStatus` for where. Trial statuses are on [trials](/core-concepts/trials#statuses). ## Derive a new job Every derived run is a new job linked to its source through `source_jobs`. The source is never changed. ```bash theme={null} evolve job retry --failed-only evolve job retry -t -t evolve job resume -f InfrastructureError evolve job regrade --task tricky-task ``` `resume` takes a finished job's failed or stopped trials and runs them: `ScoringError`, `InfrastructureError`, `ApiUsageLimitError` and `IncompleteTrialError`, plus stopped trials and the still-queued trials of a cancelled job. `-f ` narrows it by exception type. Scored trials are never re-run. A live source is refused with `job_not_terminal`, an empty selection with `no_failed_trials`. `retry` runs selected trials again: every trial, only the failed ones with `--failed-only` (`SCORING_ERROR`, `INFRASTRUCTURE_ERROR`, `BUDGET`, `INDETERMINATE`), or exactly the ones named with `-t`. A scored trial is a legitimate target. The whole-job and `--failed-only` forms need a finished source; the named form works on a running job once every named trial has settled, else `trial_not_settled`, or `trial_not_found` for an id the job does not own. Naming trials and `--failed-only` together is refused. `regrade` re-runs only the verifier of a finished job, under the network policy the source trial ran with; `--status` and `--task` narrow the set. Only settled separate-mode trials are eligible; shared-mode trials and judge tasks are not. A single-trial regrade of an ineligible source is refused with `regrade_source_ineligible`; a whole-job regrade skips the ineligible trials and answers `no_regradable_trials` when none remain. The result is a job with `is_regrade` true. ## Stop, cancel, delete ```bash theme={null} evolve job cancel evolve job stop --dataset terminal-bench-4 evolve job delete --yes ``` `cancel` asks the whole job to stop; a finished job is a no-op. `stop` ends one dataset's live trials and leaves the job running. `delete` removes a job you created permanently: its trials, traces, analyses and stored files. Without `--yes` it asks first. Only the job's creator may delete it; another member of the organization is refused with `org_forbidden`. The job must be finished: a live job, a running analysis wave or a live regrade derived from it refuses, naming what to wait for. Regrade jobs and the `source_jobs` entries of derived jobs stay. Deleting an uploaded job is how it is replaced; see [upload](/core-concepts/upload). ## Download ```bash theme={null} evolve job download -o results/ ``` The results unpack as the standard job directory, `results/job-/`: the job's `config.json`, `lock.json`, `result.json` and `job.log`, and one directory per trial. See [trial outputs](/core-concepts/trial-outputs) for what each trial directory holds. ## Config files Put the job in a file and run it with `-c`. The file is YAML or JSON with the same field names the SDK's `jobs().start()` takes. Flags on the command line override the file's fields, one field at a time. `--print-config` prints the resolved job body as JSON and exits without running anything. ```yaml nightly.yaml theme={null} datasets: - name: terminal-bench-4 version: "4.0" - name: harbor-examples version: "1.0" agents: - name: codex model_name: gpt-6-astra - name: claude model_name: fable n_attempts: 2 max_trial_spend_usd: 25 ``` ```bash theme={null} evolve run -c nightly.yaml --print-config evolve run -c nightly.yaml --watch ``` Every flag of `evolve run`.