feat: remove the eval CLI - #2610
Open
mikasenghaas wants to merge 2 commits into
Open
mikasenghaas wants to merge 2 commits into
mikasenghaas wants to merge 2 commits into
Conversation
prime-rl's `uv run eval` replaces it: the `eval` console script, its runner, resume, rich dashboard, platform upload and example configs go, along with the `prime-runs` dependency. `RunConfig` moves to `configs/cli/run.py` for the run-directory CLIs that stay (validate, gepa), and the e2e suite runs tasksets in-process.
This was referenced Sep 15, 2026
mikasenghaas
marked this pull request as ready for review
September 15, 2026 18:39
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
Contributor
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — This PR removes the standalone eval CLI, platform run integration, dashboard, resume machinery, and bundled evaluation workflow, shifting evaluation to prime-rl and changing related production configuration/output code. The cross-repository migration and unresolved scaffold onboarding concern create substantial user-facing risk requiring human review. Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
The scaffold installs only verifiers, which has no eval executable since the eval CLI moved to prime-rl. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mikasenghaas
added a commit
that referenced
this pull request
Sep 16, 2026
## Summary - Served episodes stream turn by turn over the env-serve wire (`verifiers/v1/serve/delta.py`). `Trace.notify` fires at every phase change and after every recorded turn; the worker's `DeltaStreamer` diffs the run's live traces against what it already sent and ships the new part (header once, appended nodes/calls/errors, semantic links, changed scalars), so every byte crosses the wire once. A cursor advances only after its delta is on the wire. `EnvClient.run` assembles the episode from the deltas (`EpisodeAssembly`, which copies the nodes it keeps so a later link never mutates a delta the caller holds), validates it once, and takes `on_delta` so a caller can relay or persist the stream; the client's receive loop survives a malformed frame. - `pending` preview: the interception server hands each model request's uncommitted tail (tool results, user turns) to `Trace.preview`, so a watcher sees a tool result before the model has answered; the committed turn replaces it and the finished record never carries it. - The eval CLI is `uv run vf-eval` and runs in-process only. Serving an env to many consumers is prime-rl's job (its `uv run eval` runs env servers and consumes this stream), so the CLI keeps the quick local path and never assembles deltas; its TUI footer and console log point at prime-rl's `eval`. The env-server e2e fixture drives a worker pool through `EnvClient` directly. Based on `main`; the eval CLI stays for a transition period (#2610 removes it later). prime-rl PrimeIntellect-ai/prime-rl#3471 consumes the stream for its live traces. ## Breaking - Env-serve wire: replies are 4 frames (`client_id, request_id, kind, data`) with `kind` = `delta` | `reply`; `RunResponse` carries the episode head and per-trace counts instead of the traces. A client and server must be on the same side of this change. - Every console script is prefixed: `vf-eval`, `vf-validate`, `vf-debug`, `vf-replay`, `vf-init`, `vf-gepa`. Generic names collide with the scripts of a workspace that installs verifiers as a dependency (prime-rl's `eval`), and the winner is whichever package was installed last. `vf-eval` has no `[serve]` / `--serve.*` / `--no-serve`: rollouts always run in-process; use prime-rl's `uv run eval` for env-server runs. Existing venvs: `uv sync --reinstall-package verifiers --reinstall-package prime-rl` regenerates the scripts. - `Env.run_slot` takes `on_trace`; `Trace` gains `watch`/`notify`/`preview`/`clear_preview`. ## Verification - `tests/v1/test_serve_delta.py`: turns, a link onto an earlier node, a retried attempt and the final reward round-trip through `DeltaStreamer` → `EpisodeAssembly` → `WireEpisode`; a refused send is diffed again; the pending preview streams, is replaced by the committed turn and never reaches the record; assembling never mutates a delta; the field lists cover every serialized `Trace` field. - `uv run pytest tests/v1 -m "not e2e"` green apart from the config-parse cases of tasksets not installed in the venv; ruff clean. - End to end through prime-rl: gsm8k single turn, terminal-bench-2 fix-git with bash and with the rlm harness delegating to a sub-agent (semantic links `subagent_call`/`subagent_return`), kuhn-poker two-seat episodes, 5-step reverse-text RL runs with token ids and logprobs, including router replay plus sampling replay on a MoE; no assembly mismatch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Stream served episodes turn by turn with delta frames > - Adds `DeltaStreamer` in [delta.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-106a565d15f6f5c86f7357854d512768b3f8a4775026a0eae32c69b828f5da31) that watches trace changes during `run_slot` and sends incremental deltas; the client `EpisodeAssembly` reconstructs a `WireEpisode` from those deltas plus a final episode head and `TraceSummary` counts. > - Changes the wire protocol to three-frame messages (request ID, kind, data). Delta frames are relayed by the pool and client without closing the request; reply frames close it. Updated in [server.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-64422f6a8749fcc024415b2756129bc98000e830e74b3b5245de50a555480a22), [pool.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-0c9c59578214be2580d340bc524321e352f48038207a68945637d00fc399600c), and [client.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-ef98a21052536967b8f8723e36022f69e539f4e8b13794791e8254434ac4696e). > - Adds a trace watcher/preview API in [trace.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-834820c3aa80d87ee1c1c57fb54c74f09c66ac727036d964e6005c8613c00b5e). `Env.run_slot` gains an `on_trace` callback; `Rollout.open`/`close` and `InterceptionServer` notify the trace on phase changes and pending turn previews. > - `PendingTurn.commit`/`commit_prompt` now clear the trace's pending preview after committing. > - Risk: `RunResponse` in [types.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2512/files#diff-8e290c0dfc583c92b20ee9f20fa2ecd0df82459ad63f8e819e87aa856de97cf4) removes the serialized `WireEpisode` field and replaces it with an episode head and `TraceSummary` list — any out-of-tree consumer expecting the full episode in the run response will break. > > <!-- Macroscope's changelog starts here --> > #### Changes since #2512 opened > > - Renamed console script entry points in `pyproject.toml` and updated all documentation and usage strings to use `vf-` prefixed command names [02a6eff] > - Refactored trace preview system from unkeyed to keyed architecture [55500a3] > - Updated all trace preview and clear operations to use keyed API with PendingTurn instances as keys [55500a3] > - Added abandon mechanism for cleaning up previews when turns fail or are cancelled [55500a3] > - Changed `verifiers.v1.trace.Trace` to store preview entries by `id()` of the key object rather than by the key object itself [73d602d] > - Updated `PRIME_RL_HINT` constant in the `hint.py` module [fc7e45b] > <!-- Macroscope's changelog ends here --> > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized fe771d8.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here --> <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **High Risk** > Breaking env-serve wire protocol and `RunResponse` shape require matching client/server versions; CLI renames and removal of eval’s serve path change how users run evaluations. > > **Overview** > This PR **streams env-server episodes incrementally** instead of returning one fat `RunResponse`. Workers use **`DeltaStreamer`** to diff live traces on `Trace.notify()` and send **`delta`** frames; clients **`EpisodeAssembly`** rebuilds traces and validate once against a final **head + `TraceSummary` counts**. The wire format is now **`[request_id, kind, data]`** with `kind` = `delta` | `reply`; the pool relays deltas without closing the request. > > **`Trace`** gains **`watch` / `notify` / keyed `preview` / `clear_preview`**; rollouts and the interception server fire notifications (including **pending tool/user tails** before commit). **`Env.run_slot`** adds **`on_trace`** so the server can subscribe traces to the streamer. > > **`vf-eval` is renamed and narrowed**: console scripts are **`vf-*`** to avoid clashing with prime-rl’s **`eval`**, docs/config comments follow suit, and **`vf-eval` always runs in-process**—**`[serve]` / `--no-serve` are removed** from eval config—with dashboard/log hints to use **prime-rl** for env-server scale and live dashboards. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit fc7e45b. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
mikasenghaas
added a commit
to PrimeIntellect-ai/prime-rl
that referenced
this pull request
Sep 16, 2026
## Summary Migrates the eval CLI from verifiers into the prime-rl ecosystem: `uv run eval` is prime-rl's entrypoint for evaluating any model on any environment, standalone or as the online evals of a training run. Why: - consolidation of features: adaptive concurrency, the local dashboard, resume from the trace stream, live rollouts - easier code sharing between the CLI entrypoints (`rl`, `sft`, `eval`) - direct access to the pre-installed environments (`prime-envs` submodule) - one workspace for post-training: training and evals Important: - the verifiers eval CLI stays for a transition period as the quick in-process path, renamed `uv run vf-eval` so prime-rl owns `eval` (PrimeIntellect-ai/verifiers#2610 removes it later); env servers and their streamed traces (PrimeIntellect-ai/verifiers#2512, merged; the submodule points at verifiers `main`) are consumed here only - evals run through env servers, for scalability; the launcher (`eval`, `sft`, `rl`) owns them - `--resume` continues an interrupted run from its trace stream: landed episodes rejoin the epoch, only the owed rollouts run; `num_examples`/`group_size` may change, the model, sampling and env config may not - multi-env capable (`[[source]]` blocks); the single-env command shorthands are preserved - adaptive concurrency only against local deployments (it reads vLLM `/metrics`); against an API, concurrency is pinned via `min_inflight = max_inflight` (the default pins 128) - prime-rl's core dependencies are plain CPU; install them and the environments with `uv sync --all-packages` - Prime runs and traces integrate through the single monitor abstraction (`--monitors.prime`, `--monitors.wandb`, the file monitor) Examples: ```bash uv run eval gsm8k # default model on Prime Inference uv run eval gsm8k -n 32 -r 4 -m openai/gpt-5.6-luna # 32 tasks × 4 rollouts, another model uv run eval terminal-bench-2 --env.taskset.tasks '["fix-git"]' --env.agent.harness.id bash -n 1 -r 2 uv run eval @ configs/debug/eval/single-turn.toml # gsm8k, null harness uv run eval @ configs/debug/eval/multi-turn.toml # 16 terminal-bench-2 tasks, bash harness uv run eval @ configs/debug/eval/multi-env.toml # terminal-bench-2 under bash and rlm side by side uv run eval @ configs/debug/eval/aime2026.toml # AIME 2026, avg@16 uv run eval @ configs/debug/eval/tb2.toml # Terminal-Bench 2, avg@4 uv run eval @ configs/debug/eval/resume.toml --run.name x # then: --resume uv run eval gsm8k --monitors.prime # stream the evaluation to the platform ``` ## Breaking - `uv run evals` is `uv run eval` (`EvalsConfig` → `EvalConfig`); config shape: `[eval.client]` → `[client]`, `[eval.concurrency]` → `[concurrency]`, `[[eval.source]]` → `[[source]]`, `eval.num_examples`/`eval.group_size` → top level. `[online]` is gone: online evals are spawned by `sft`. - `eval` has no `[ckpt]` block; `--resume` is a switch that reads the run's trace stream. - SFT online evals: no public entrypoint (`uv run sft` with an `[eval]` block spawns it); `logs/attempt_N/evals.log` → `eval.log`, `evals.json` → `eval.json`, W&B label `online-eval`; `EvalsEvalConfig` → `SFTOnlineEvalConfig`. - `ratio` is accepted on training sources only, `interval` on online eval sources only; a standalone eval's `[[source]]` takes neither. - `orchestrator.env_server_base_port` and the eval entrypoint's `env_server_base_port` are removed: launcher-managed env servers bind an OS-assigned port and publish it (`configs/attempt_N/resolved/envs/<split>/<name>.address`); `serve.address` pins one. `uv run env-server` without `serve.address` binds an OS-assigned port (`address_file` publishes it). - `configs/evals/swe.toml` is removed (`uv run eval swebench-verified --env.agent.harness.id bash`). ## Verification - Unit: `uv run pytest tests/unit` green (configs, orchestrator, eval, monitors incl. the live reader). - Debug configs: single-turn, multi-turn, multi-env and aime2026 run to completion against Prime Inference. Resume: single-env and multi-env runs interrupted with SIGTERM and resumed hold exactly examples × group_size unique episodes with the plan and epoch metrics over the full set; resuming with a larger `group_size` runs only the extra rollouts; resuming with another model is refused. - Core install (`uv sync` without extras) runs `uv run eval`; an SFT run with online evals starts one env server per source. - Live streaming, on the final code: gsm8k 16 single-turn episodes, terminal-bench-2 fix-git with bash (13-14 turns, the pending preview alternating with committed turns in the live files), kuhn-poker 4 two-seat episodes (8 traces), a 5-step reverse-text RL run (768 train episodes with token ids, logprobs and masks); no assembly mismatch. - Platform: `--monitors.prime` on a two-rollout fix-git eval opened the evaluation 9 s after launch, streamed both rollouts and closed it; the dashboard linked to it throughout. - Dashboard behaviour checked in headless Chromium: live rows and viewer follow, handover in place, metrics pane per env, filter menus, platform button states. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Breaking CLI and config renames affect existing scripts; env-server address discovery replaces fixed ports across `rl`, `sft`, and `eval` launch paths. > > **Overview** > **Replaces `uv run evals` with `uv run eval`** as the standalone evaluation entrypoint, sharing the orchestrator eval pipeline (per-source env servers, adaptive or pinned concurrency, file/W&B/Prime monitors). Config is **flattened** (`[[source]]`, `[client]`, `[concurrency]` at top level); **`--resume`** continues from the trace stream instead of a `[ckpt]` cursor; SFT **online eval** is renamed and typed as `SFTOnlineEvalConfig` (no public `[online]` block). > > **Env server wiring changes:** fixed `env_server_base_port` ranges are removed—launcher-managed servers bind **OS-assigned loopback ports** and publish addresses via **`address_file`** / `configs/.../envs/<split>/<name>.address`. Training **`ratio`** and online-eval **`interval`** are scoped to the right source types only. > > **Docs and examples** add `docs/eval.md`, per-example `eval.toml` files, and `configs/debug/eval/*` smokes; **`configs/evals/swe.toml` is deleted**. CI adds a **gsm8k eval** integration test on VM runners. > > **Dashboard** gains live rollout APIs and eval-centric metrics (plan-driven progress, platform links, eval run status), renames **`evals.log` → `eval.log`**, and tightens static asset revalidation. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 82f2f0e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evalconsole script and everything only it used:verifiers/v1/cli/eval/(main, runner, resume), the rich eval dashboard,configs/cli/eval.py,utils/platform.py, theconfigs/*.tomlexamples and theevaluate-environmentsskill. Evaluation moves to prime-rl'suv run eval(feat: consolidate eval cli prime-rl#3471), which brings adaptive concurrency, the local dashboard, pre-installed environments and a shared workspace with training.RunConfig+default_run_namemove toconfigs/cli/run.pyfor the CLIs that stay (validate,gepa);cli/output.pytakes any config that owns a run directory.tests/v1/runner.pyinstead of the CLI.prime-runsis no longer a dependency; docs and scaffold hints point atuv run validateand prime-rl'suv run eval.On hold. The eval CLI stays for a transition period after prime-rl's
uv run evallands; #2512 (streamed traces) no longer stacks on this branch and lands first. Rebase ontomainonce the transition period ends.Breaking
uv run evalis gone, with--rich,--push,--resume,--dry-runand theconfigs/*.tomlexamples. Use prime-rl'suv run eval <taskset-id>.verifiers.v1.configs.cli.eval(EvalConfig,RichConfig),verifiers.v1.cli.eval,verifiers.v1.cli.dashboard.eval,verifiers.v1.utils.platform.RunConfig/default_run_namenow import fromverifiers.v1.configs.cli.run.prime-runsis no longer a transitive dependency; install it directly if you relied on it.uv run eval --run.attach $EVALUATION_IDinside the sandbox (feat(v1):run.attachstreams the eval into a run the platform already created #2569); that path needs prime-rl'suv run evalto grow--run.attachbefore this lands.Verification
uv run pytest tests/v1 -m "not e2e": 81 passed;uv run ruff check . && uv run ruff format .clean.🤖 Generated with Claude Code
Note
High Risk
Breaking public API and CLI (
uv run eval, push/resume/rich flags,EvalConfig); hosted eval attach paths need prime-rl parity before upgrade.Overview
Removes the in-repo
uv run evalproduct and hands full model evaluations to prime-rl (uv run eval). Theevalconsole script,EvalConfig, the eval runner/resume path, the Rich live dashboard, platform push (prime-runs/utils/platform.py), and allconfigs/*.tomleval examples and theevaluate-environmentsskill are deleted.What stays wired for local dev:
RunConfiganddefault_run_namemove toconfigs/cli/run.py;validate,gepa,debug, andreplaystill write run dirs via generalizedcli/output.py. E2E tests exercise envs through newtests/v1/runner.py(RunnerConfig+run_episodes) instead of spawning the eval CLI. Docs,AGENTS.md, andinitscaffolding now emphasizeuv run validateand prime-rl for evals.Reviewed by Cursor Bugbot for commit 067fe68. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Remove the
evalCLI and replace with sharedrun_episodesrunnerevalconsole-script registration and theprime-runsdependency from pyproject.toml; thevalidate,debug,replay, andgepascripts remain.RunnerConfigandrun_episodesin tests/v1/runner.py to execute task rollouts through either an in-process environment backend or a spawned env-server pool, replacing the oldrun_evalpath.RunConfig,default_run_name) into verifiers/v1/configs/cli/run.py and updatesvalidate,gepa, and output helpers to use them.run_v1/run_v1_serverfixtures backed byrun_episodesinstead of the eval CLI.initscaffold command to direct users touv run validatefor checks and to prime-rl for evaluations.verifiers.v1.cli.dashboardno longer exposes the evaluationdashboardattribute; any out-of-tree consumers of that symbol will break. Installing the project no longer provides anevalcommand.Macroscope summarized 067fe68.