Python

Use Evolve’s Python client to run Codex on one DeepSWE task and save the results. The managed evaluation client makes HTTP requests directly; it does not need Node.js.

Install and authenticate

You need Python 3.10 or newer and a Full access key from the dashboard’s API keys page. Replace your-api-key with that key. See installation for account setup.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install evolvingmachines-evolve
export EVOLVE_API_KEY="your-api-key"

Run one task

Save this as eval.py. It selects only anko-default-function-arguments from deepswe@1.1 and runs one trial with Codex and GPT-6 Luna.

eval.py

import asyncio
from evolve import hosted


async def main():
    async with hosted() as evolve:
        job = await evolve.jobs.start(
            datasets=[{
                "name": "deepswe",
                "version": "1.1",
                "task_names": [
                    "anko-default-function-arguments",
                ],
            }],
            agents=[{
                "name": "codex",
                "model_name": "gpt-6-luna",
            }],
            max_trial_spend_usd=1,
            retry={"max_retries": 0},
        )
        print("Job:", job.id)

        finished = await evolve.jobs.watch(job.id)
        print("Status:", finished.status)
        print(
            "Model cost:", finished.stats.get("cost_usd")
        )

        async for trial in evolve.jobs.trials(job.id):
            print(
                trial.task_name,
                trial.status,
                trial.reward,
            )

        archive = await evolve.jobs.download(
            job.id, to="./results"
        )
        print("Saved:", archive)


asyncio.run(main())

Run it in the shell where you set the API key:

python eval.py

The program prints the job ID, final status, model cost, trial result, and saved archive path under ./results/.

Call	Result
`jobs.start()`	Returns the accepted job before the trial finishes.
`jobs.watch()`	Waits for the final job status.
`async for trial in jobs.trials()`	Reads trials across all pages.
`jobs.download(..., to="./results")`	Saves the job’s `.tar.gz` archive and returns its path.

Use attributes for jobs and trials, and dictionary keys for stats. A completed job can contain failed trials. Read the trial’s status and reward; a missing value is not zero.

The example sets a $1 model-spend cap for this trial attempt and disables configured infrastructure retries. A model call already in progress can finish above the cap. Provider capacity waits can still occur. Closing the program does not cancel the job.

Show progress

Replace the program’s watch() call with one of these snippets inside the same async with block, after creating job.

finished = await evolve.jobs.watch(
    job.id,
    on_event=lambda event: print(event.seq, event.type),
)

Or consume the events directly:

async for event in evolve.jobs.watch(job.id):
    print(event.type, event.data)
finished = await evolve.jobs.get(job.id)

Choose one consumption style for each watch handle.

Read a trial

Place this inside the same async with block, after the job finishes. It reads the trial’s normalized trace and its stored-file listing.

page = await evolve.jobs.trials(job.id, limit=1)
if page.items:
    trial = page.items[0]
    trajectory = await evolve.trials.artifact(
        trial.id, "trace-atif"
    )
    files = await evolve.trials.files(trial.id)
    print(trajectory, files.items)

Awaiting jobs.trials() reads one page; async for reads every page. The stored-file listing is also one page. Follow its next_cursor when present; see trial files.

HTTP API

Python and TypeScript

Run agents

Shared behavior

SDK workflows

SDK resources

SDK methods

Install and authenticate

Run one task

Show progress

Read a trial

Next step

Configure a job

Understand returned objects