Environment lint

Environment lint reads task files during publication and flags potential grading problems. It does not run the task or verifier.

evolve dataset publish \
  --dir ./tasks \
  --name my-bench \
  --version 1.0 \
  --watch

Findings are advisory, including those marked error. They do not block publication or execution. Review the suggested fixes, update affected tasks, and publish a new version.

Where you see it

The report appears after task validation, before image builds:

Surface	What to read
`publish --watch`	Finding counts, affected rules, and suggested fixes
`publish --watch --json`	The `import.lint` event
SDK `watchImport` / `watch_import`	The `onLint` / `on_lint` callback
`dataset show`	Summary plus task findings, with file and line
Dataset detail	The `lint` summary and each task’s `lint` findings

evolve dataset show my-bench@1.0

The import includes an environment_lint_findings warning when it has errors or warnings. Rejected tasks are not linted; a publication with no accepted tasks has no report.

Report value	Meaning
Template-wide	A rule affects at least half the tasks and at least five tasks. Check for a shared template or verifier. The CLI prints it once with its task count.
`lint: null` / not linted	No lint report is available, including for older publications. It does not mean the tasks passed lint.
`truncated` / `?` in the LINT column	Error or warning findings were omitted to fit the report limit. A task with no listed findings may still have findings.
`infos_truncated`	Informational findings were omitted. When `truncated` is false, all errors and warnings remain listed.

Summaries group findings by rule and severity, with the most severe first. Lint can detect only problems visible in the files. Use a task check to review behavior that requires execution, such as whether a numeric check rejects wrong answers.

Rule origins

The rules cover static checks from the defect classes reported by Tokenless for Terminal-Bench 4, DeepSWE 1.1, and BFCL v4. See envcheck-findings.

Rules

Rule	Level	What it catches
`answer-in-agent-image`	error	The agent’s Dockerfile copies `solution/` or an oracle tree into the image and no later `RUN` deletes it.
`reward-constant`	error	Every reward the verifier writes is a constant 1, with no `set -e` or `&&` that makes the write depend on the tests.
`instruction-missing`	error, warning or info	`instruction.md` is missing or has fewer than 20 characters (error), is not UTF-8 text (warning), or is over 512 KiB, too large for the linter to read (info).
`answer-material-in-tests`	warning or info	`tests/` holds copies of solution files the agent does not already have, or files named like answers: `oracle`, `golden`, `answer(s)`, `ground_truth`, `solution(s)`, `reference` (for code files only `oracle`, `golden`, `solution(s)`, `reference`). A warning when the verifier visibly runs the submission’s code while `/tests` is readable, info otherwise.
`verifier-runs-submission-test-config`	warning	Tests run inside the submission’s tree, so its `conftest.py`, `pytest.ini`, `TestMain`, runner config and test helpers run with the graded tests.
`verifier-installs-submission-deps`	warning	The verifier installs a requirements file, `package.json` or lock file from the agent’s workspace with its own privileges (an install run as an unprivileged user is the fix, and is not flagged).
`test-without-assertion`	warning	A test function, in `tests/` or added by `tests/test.patch`, asserts nothing.
`test-trivial-assertion`	warning	A test asserts only a constant, such as `assert True`.
`reward-not-written`	warning	Nothing under `tests/` writes `reward.txt` or `reward.json`.
`verifier-network-install`	warning	The verifier fetches from the network (`pip`/`uv`/`npm` installs, `curl`, `wget`, `git clone`, `apt`) while its network mode is `no-network`. Requests to localhost and offline installs are not counted.
`verifier-unpinned-install`	warning	The verifier installs packages without pinned versions at grading time.
`verifier-remote-script`	warning	The verifier pipes a downloaded script into a shell.
`solution-missing`	warning	There is no `solution/solve.sh`, so nobody can show a correct submission scores 1.
`duplicate-instruction`	warning	Another task in the version has the same instruction and the same tests.
`test-exit-code-only`	info	A test checks only an exit code or status code, never the state or output it should produce.
`instruction-output-untested`	info	The instruction asks for an output file that no file under `tests/` mentions.
`shared-verifier`	info	The verifier runs inside the agent’s box, where anything the agent leaves behind runs during grading.
`base-image-unpinned`	info	A base image has no fixed tag, so a rebuild can change the environment.

Start here

Run evaluations

Task format

Validate your work

Read results

Work together

Environment lint

Where you see it

Rules