evolve dataset

A dataset version is a named collection of tasks. Jobs select name@version, or the active version when only the name is given.

evolve dataset list --search deepswe
evolve dataset show deepswe@1.1

Browse the catalog

Command	Options
`dataset list`	`--search <text>` matches names and descriptions; all shared list options.
`dataset show <name[@version]>`	`-l, --limit <n>` and `--cursor <cursor>` page the task list.

The detail view includes versions, tasks, build failures, and provider information.

Check before publishing

evolve dataset check ./tasks

This reads local task metadata and sends it to Evolve for validation. It does not publish the corpus or run the tasks. The report includes a Harbor: line naming the supported task-format versions. When accepted tasks use newer fields, it also names the oldest release they require. A Terminal-Bench 1 task (task.yaml, no task.toml) fails the check and names harbor tasks migrate. The command returns exit code 1 if any task or the dataset manifest is refused. With --json, it prints one report containing tasks_refused and the manifest result. For an agent-based quality check, use evolve check.

Publish a version

Choose one source.

Local directory
Git repository
Public archive
Harbor package

evolve dataset publish \
  --dir ./tasks \
  --name my-benchmark \
  --version 1.0 \
  --watch

A dataset.toml manifest can supply the name and version. Local publishing runs metadata preflight before uploading.

evolve dataset publish \
  --git https://github.com/acme/tasks.git \
  --ref v1.0.0 \
  --path benchmark \
  --name my-benchmark \
  --version 1.0 \
  --watch

Pin a tag or full commit SHA. Branch names are refused. Omit --path to use the repository root.

evolve dataset publish \
  --from https://example.com/tasks.tar.gz \
  --name my-benchmark \
  --version 1.0 \
  --watch

Evolve fetches the HTTPS archive. Supply both name and version.

evolve dataset publish --from hub:cookbook/hello-world --watch

Import a public Harbor Hub package using hub:org/name[@ref]. Name and version may come from the package.

Publish options

Option	Meaning
`--dir <path>`	Local corpus directory.
`--git <url>`	Git repository URL. Requires `--ref`.
`--ref <tag\|sha>`	Pinned Git revision.
`--path <subfolder>`	Corpus folder within the Git repository.
`--from <source>`	Public HTTPS tarball or `hub:org/name[@ref]`.
`--name <name>`	Dataset name. Required for Git and HTTPS sources.
`--version <version>`	Version label. Required for Git and HTTPS sources.
`--org <name>`	Organization for a new dataset; otherwise the saved CLI default, then personal.
`--skip-preflight`	Skip metadata preflight for a local directory.
`--watch`	Follow the import and version build to completion.

Do not combine source types. For a local directory, refused metadata stops publication before upload. --skip-preflight skips that early check; import validation still applies.

Follow the build

1. Upload or fetch the corpus
                 ↓
2. Validate tasks and build environments
                 ↓
3. Finish the version build
   ├── CASE A: At least one task built → READY
   └── CASE B: No task built          → FAILED

A READY version can contain failed tasks. The final report shows how many built and which failed. On your own dataset, the new ready version becomes active. To reconnect while an import is running:

evolve dataset watch my-benchmark

To inspect a specific import, including one that has already finished, set $IMPORT_ID to the full ID printed during publishing:

evolve dataset watch "$IMPORT_ID"

Watching by dataset name selects an active import; when none exists, the command returns nothing_to_watch and exit code 1. With --json, publication without --watch returns one import record. Watching prints one JSON record per line, ending with import.final. A failed import or a build that cannot be confirmed returns exit code 1; a READY version with some failed tasks returns 0. Use a new version label for changed tasks. Publishing an existing label replaces its contents, so name@version does not guarantee unchanged content. If watching returns import_not_found because the upload was abandoned or refused before it arrived, repeat the publish command.

Change the active version

evolve dataset activate my-benchmark 1.0

The chosen version must be READY. This changes what a bare my-benchmark selects; version-pinned jobs still use their selected version.

Download the corpus

evolve dataset download my-benchmark@1.0 -o corpora/

-o, --output-dir <dir> sets the destination; the default is the current directory. This downloads the original stored corpus package. Access is limited to the dataset owner.

Read one task’s files

DATASET="deepswe@1.1"
TASK="anko-default-function-arguments"
evolve dataset files list "$DATASET" "$TASK" /
evolve dataset files cat "$DATASET" "$TASK" /instruction.md

Use dataset files status/list/cat for package metadata, folder pagination, and byte ranges. These commands require an explicit dataset version. Global options apply. Next, run a job against the published version.

Start work

Inspect results

Manage resources