Quick start

A trial is one agent’s attempt at one task. A job groups trials. Start with a one-trial job.

Install and authenticate

You need Node.js, a Full access key from the dashboard’s API keys page, and credits on your account.

npm install -g @evolvingmachines/evolve
export EVOLVE_API_KEY="your-api-key"
evolve auth status

auth status prints your user, email, and key ID. Python and TypeScript setup is on the installation page.

Start the job

Run Codex with GPT-6 Luna on one DeepSWE task. Pick the tab you work in; all three start the same job.

evolve run \
  -d deepswe@1.1 \
  -i anko-default-function-arguments \
  -a codex \
  -m gpt-6-luna \
  --max-trial-spend 1 \
  --max-retries 0 \
  --watch

# Save as eval.py; run: python eval.py
import asyncio
from evolve import hosted

async def main():
    async with hosted() as evolve:
        job = await evolve.jobs.start(
            datasets=[{
                "name": "deepswe",
                "version": "1.1",
                "task_names": [
                    "anko-default-function-arguments",
                ],
            }],
            agents=[{
                "name": "codex",
                "model_name": "gpt-6-luna",
            }],
            max_trial_spend_usd=1,
            retry={"max_retries": 0},
        )
        print("Job:", job.id)
        finished = await evolve.jobs.watch(job.id)
        print("Status:", finished.status)

asyncio.run(main())

// Save as eval.mts; run: npx tsx eval.mts
import { hosted } from "@evolvingmachines/evolve";

const evolve = hosted();
const job = await evolve.jobs.start({
  datasets: [{
    name: "deepswe",
    version: "1.1",
    task_names: ["anko-default-function-arguments"],
  }],
  agents: [{
    name: "codex",
    model_name: "gpt-6-luna",
  }],
  max_trial_spend_usd: 1,
  retry: { max_retries: 0 },
});
console.log("Job:", job.id);
const finished = await evolve.jobs.watch(job.id);
console.log("Status:", finished.status);

Choice	Meaning
`-d` and `-i`	The dataset version and one task name.
`-a` and `-m`	Codex with the selected model.
`--max-trial-spend 1`	A $1 model-spend cap for this trial.
`--max-retries 0`	No automatic retries after an infrastructure error. Capacity waits can still occur.
`--watch`	Follow progress until the job settles.

The output includes the job ID. The job keeps running on Evolve if you close the terminal. The spend cap stops new model calls once reached; a call already in progress can finish above it.

Read the result

JOB_ID="paste-job-id-here"
evolve job show "$JOB_ID"
evolve job trials "$JOB_ID"

Each trial row shows its task, agent, status, reward, spend, and trial ID. Add --json for structured output.A finished job is not a passed task: read the trial’s status and reward. A missing reward is not zero.

Inspect and save the evidence

Use the trial ID from job trials:

TRIAL_ID="paste-trial-id-here"
evolve trial trace "$TRIAL_ID"
evolve trial download "$TRIAL_ID" \
  --stream verifier
evolve trial download "$TRIAL_ID" \
  -o ./results/

The trace shows what the agent did. The verifier log shows how the task was graded. The last command saves the trial’s files under ./results/; see trial outputs.

Next steps

Compare agents and models

Add a second model, more tasks, or repeated attempts.

Create your own task

Write a task, publish it, and run it.

Browse datasets

evolve dataset show deepswe@1.1 lists its 113 tasks.

Use your coding agent

Install the Evolve skill and read the CLI’s manual.

Authentication failed

Export EVOLVE_API_KEY in the same shell that runs the command, then repeat evolve auth status. Creating jobs needs a Full access key; a read-only key can only inspect them. See authentication.

Insufficient credits

A job refused with HTTP 402 and insufficient_credits means the account has no credits left. Add credits in the dashboard, then run the command again.

Start here

Run evaluations

Task format

Validate your work

Read results

Work together

Next steps

Compare agents and models

Create your own task

Browse datasets

Use your coding agent