diff --git a/README.md b/README.md index 820e761..66639f4 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ A new systematic diagnostic tool, **gmp analyse**, automatically produces a **fi | Date | Highlight | | :--- | :--- | +| **2026-09** | 🔧 **Minor EBench fixes and improvements** — Adjusted the step limits for `bottle` and `shop` from 3,000 to 5,000, and refined the web interface of the online evaluation platform. | | **2026-09** | 🛠️ **OpenPI baseline update** — Updated π0 / π0.5 configs. Please use the updated baseline code for evaluation and reproducing results. See the [update notes](baselines/openpi/README.md#update-notes). | | **2026-07** | 🛠️ **Evaluation toolkit** — [`genmanip-client`](https://github.com/InternRobotics/genmanip-client) brings submission, monitoring, action/state plots, and interactive episode visualization together in the `gmp` CLI. | | **2026-06** | 🚀 **Public release** — EBench, reference baselines, training data, and held-out online evaluation are now available. | @@ -115,6 +116,12 @@ gmp status A full validation pass takes roughly **30 minutes on 8× RTX 4090**. Detailed setup, asset download, and the complete `gmp` reference are in the [docs site](https://internrobotics.github.io/EBench-doc/). +## Evaluate with a coding agent + +The repository includes five reusable [agent skills](skills/README.md) for environment setup, policy integration, evaluation runs, debugging, and result analysis. They guide agents through the checked-out baseline and client implementations, including real-policy launch commands and result coverage checks. + +To start, ask your agent: **“Read `skills/ebench-setup/SKILL.md` and check whether this environment is ready to evaluate my model.”** Then use [ebench-evaluate](skills/ebench-evaluate/SKILL.md) to run it and [ebench-analyze](skills/ebench-analyze/SKILL.md) to interpret the results. See the [skill catalog and usage guide](skills/README.md) for all workflows; automatic discovery depends on your agent's skill installation mechanism. + ## Tasks **26 task types** across *Long-Horizon*, *Pick-and-Place*, and *Dexterous & Precise*, expanded with the four generalization axes and three splits into **794 evaluation task instances**. Browse the video gallery at [Task Showcase](https://internrobotics.github.io/EBench-doc/evaluation/task-showcase/). diff --git a/README.zh-CN.md b/README.zh-CN.md index d7d85fa..a13b54c 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -45,6 +45,7 @@ | 日期 | 重要更新 | | :--- | :--- | +| **2026-09** | 🔧 **小修复与优化** — 将 `bottle` 和 `shop` 的步数上限从 3,000 调整为 5000，并优化了在线评测平台的网页界面。 | | **2026-09** | 🛠️ **OpenPI Baseline 更新** — 更新 π0 / π0.5 配置。评测与复现结果时，请使用更新后的 Baseline 代码。详见[更新说明](baselines/openpi/README.md#update-notes)。 | | **2026-07** | 🛠️ **评测工具链** — [`genmanip-client`](https://github.com/InternRobotics/genmanip-client) 通过 `gmp` CLI 统一提供评测提交与监控、动作/状态曲线绘制以及 episode 交互式查看。 | | **2026-06** | 🚀 **正式发布** — EBench、参考 Baseline、训练数据及训测隔离的在线评测现已公开。 | @@ -117,6 +118,12 @@ gmp status 8 卡 RTX 4090 上跑完整套验证集大约需要 **30 分钟**。完整的环境配置、资产下载与 `gmp` 命令参考请见[文档站点](https://internrobotics.github.io/EBench-doc/zh-cn/)。 +## 使用 coding agent 评测 + +仓库提供五个可复用的 [agent skills](skills/README.md)，覆盖环境准备、模型接入、评测运行、故障排查和结果分析。它们引导 agent 按当前仓库的 baseline 与客户端实现完成操作，并核对真实模型入口及结果覆盖情况。 + +可以先告诉 agent：**“请读取 `skills/ebench-setup/SKILL.md`，检查当前环境是否能评测我的模型。”** 随后使用 [ebench-evaluate](skills/ebench-evaluate/SKILL.md) 运行评测、[ebench-analyze](skills/ebench-analyze/SKILL.md) 解读结果。完整流程和调用示例见 [skill 目录与使用说明](skills/README.md)；是否自动发现取决于所用 agent 的技能安装机制。 + ## 任务总览 **26 种任务**分布在*长程任务（Long-Horizon）*、*抓取放置（Pick-and-Place）*、*灵巧精细操作（Dexterous & Precise）*三个任务族；与四维泛化、三层划分组合后，共构成 **794 条评测任务**。完整视频演示见[任务展示](https://internrobotics.github.io/EBench-doc/zh-cn/evaluation/task-showcase/)。 diff --git a/skills/README.md b/skills/README.md new file mode 100644 index 0000000..e10192e --- /dev/null +++ b/skills/README.md @@ -0,0 +1,60 @@ +# EBench coding agent skills + +These skills help coding agents evaluate **robot policies on EBench**: prepare environments, integrate models, run experiments, diagnose failures, and interpret results. They are not a benchmark for evaluating coding agents themselves. + +## Where to start + +| Skill | Priority | When to use it | Deliverables | +| --- | --- | --- | --- | +| [ebench-setup](ebench-setup/SKILL.md) | P0 | First-time setup, environment checks, baseline reproduction | Environment findings, missing prerequisites, executable launch commands | +| [ebench-evaluate](ebench-evaluate/SKILL.md) | P0 | Local or online evaluation, launching multiple workers | Traceable runs, logs, and completion status | +| [ebench-analyze](ebench-analyze/SKILL.md) | P0 | Report generation, model comparison, capability analysis | HTML reports, data coverage, and evidence-backed conclusions | +| [ebench-integrate-policy](ebench-integrate-policy/SKILL.md) | P1 | Connecting a custom VLA to EBench | Observation/action adapter and contract validation | +| [ebench-debug](ebench-debug/SKILL.md) | P1 | Stalled runs, errors, unexpected actions or success rates | Diagnosis, minimal fixes, and verification evidence | + +P0 skills cover the complete workflow for an existing model, from preparation to reporting. For custom models, start with policy integration; use debugging when failures occur. Training, data downloads, and leaderboard publication do not have separate skills yet because they are not required for every evaluation. Follow the baseline documentation for training and data preparation. + +## Usage + +Skills are versioned under `skills/`, with a standard `SKILL.md` containing `name` and `description` in each directory. **Not every agent automatically discovers this directory.** The simplest approach is to ask your agent to read a specific file from the EBench repository: + +```text +Read skills/ebench-setup/SKILL.md, check whether this environment can run X-VLA online evaluation, and list any missing prerequisites. +Read skills/ebench-evaluate/SKILL.md, submit an online evaluation using the configured token, wait for the endpoint and task_id, then launch my model. +Read skills/ebench-evaluate/SKILL.md and launch my model using the existing endpoint and run_id, with worker 0 only. +Read skills/ebench-integrate-policy/SKILL.md and connect my policy to EvalClient, validating the input/output contract first. +Read skills/ebench-debug/SKILL.md and investigate this run's reset timeout while preserving existing results. +Read skills/ebench-analyze/SKILL.md, compare these two result directories, generate a report, and explain whether their evaluation coverage matches. +``` + +You can also import these directories through your agent's skill installation mechanism. Once installed, agents that support `$skill-name` can invoke them by names such as `$ebench-evaluate`. Copying the entire collection into a global user directory is not a prerequisite for evaluation. + +Repository paths in all skills are relative to the **EBench root**, not the skill directory. Skill instructions are in English for international reuse; agents can respond in the user's language. For a first run, provide the model/checkpoint, online or local mode, split/track, and GPU/worker budget. For an existing task, also provide its run ID and endpoint. Supply tokens through the local environment or an existing credential mechanism, never through committed files or reports. + +## Online evaluation workflow + +[ebench-evaluate](ebench-evaluate/SKILL.md#online-evaluation-queue-obtain-endpoint-then-evaluate) includes complete command examples. The agent follows this sequence: + +1. Check the local model environment, checkpoint, platform URL, and token. +2. Run `gmp online submit --print_endpoint` to join the queue and wait for resources. +3. Parse `endpoint` and `task_id` from the returned JSON, using them as the evaluation URL and `run_id`, respectively. +4. Launch the baseline or custom policy client for real model evaluation, then monitor progress and save results. + +A new task does not require the user to provide an evaluation URL or task ID in advance. After a queue timeout, query the existing task before retrying to avoid duplicate submissions. Receiving a URL does not mean evaluation is complete. The setup and policy integration skills also explain how to connect to this workflow. + +## Maintenance and acceptance checks + +Treat the baseline code and `third_party/genmanip-client` source in the current checkout as the source of truth. After updating submodules, review CLI arguments, action conversions, and result parsing. Do not assume one baseline's normalization or action layout applies to every model. + +When adding or changing a skill, check its frontmatter and file references, then review its behavior against these scenarios: + +| Scenario | Expected behavior | +| --- | --- | +| A fresh clone is missing submodules | Identify missing dependencies rather than treating import errors as model failures | +| The user requests real model evaluation | Use a baseline/custom policy entry point; do not report `gmp eval` fake actions as model performance | +| An existing online task is waiting for resources | Query the existing task instead of creating another one | +| A step times out with unknown execution state | Discard the old chunk, use bounded recovery, and record the interruption rather than blindly resending actions | +| Analysis inputs are empty | Report missing local results rather than presenting bundled reference data as the user's results | +| Two runs have different splits or task coverage | Flag them as not directly comparable and explain denominators and missing data | + +These scenarios are maintenance acceptance criteria, not evidence that GPU or online platform end-to-end validation has been performed. diff --git a/skills/ebench-analyze/SKILL.md b/skills/ebench-analyze/SKILL.md new file mode 100644 index 0000000..77def48 --- /dev/null +++ b/skills/ebench-analyze/SKILL.md @@ -0,0 +1,40 @@ +--- +name: ebench-analyze +description: Generate and interpret EBench evaluation reports, compare runs and baselines, and diagnose capability or generalization gaps with explicit data coverage and aggregation semantics. +--- + +# Analyze EBench evaluation results + +Work from the EBench root. Read `third_party/genmanip-client/src/genmanip_client/extensions/analyse_cli.py` and `analyse.py` for the pinned behavior; inspect `default_cluster_map.json` only when interpreting taxonomy or checking task coverage. + +## Verify inputs before rendering + +Identify each run's model/checkpoint, benchmark revision, track, split, task set, seed/episode coverage, and completion state from its manifest and result files. Keep incomplete runs labeled. Missing episodes are not automatic successes or failures; state the coverage and denominator instead of inventing outcomes. + +Default discovery scans `/saved/eval_results//`. Client results commonly live under `client_results//` instead: pass the concrete run directory explicitly. Do not point to a single seed directory or assume EBench's root contains server outputs. + +The loader prefers per-episode `result_info.json`, then run-level `result.json`, then task-level `episode_result.json`. These formats retain different detail. Raw `result_info.json` can carry `metric_score` needed for atomic-skill aggregation; absent metrics cannot be recovered from a total success rate. Check loaded records and parse failures before drawing conclusions. + +## Generate the report + +With actual run paths already verified: + +```bash +gmp analyse "$RUN_A_DIR" "$RUN_B_DIR" --no-reference -o "$REPORT_PATH" +``` + +Omit `--no-reference` when bundled baseline comparison is desired. The default includes bundled reference models; `--reference` renders only those reference data and skips local runs. Label their bundled version rather than claiming they are freshly measured or current leaderboard standings. + +**An HTML file is not proof that local results loaded.** The pinned CLI falls back to bundled reference data when no runs/records load, even when `--no-reference` was supplied. Check the CLI's loaded-record messages and report payload/run IDs against the requested inputs. If empty, report missing data; never describe the fallback as the user's model performance. + +Use `--group 'Label=pattern'` only to combine intended compatible runs. Grouping different checkpoints, splits or overlapping retries can conceal variation or double-count evidence. Explicitly identify grouping members. Choose a fresh output path so prior reports remain available. + +## Interpret with the right denominator + +- Report SR and score separately, per split, with coverage. In the current aggregator, top-line means are over loaded records; cluster summaries first average within each task and then across tasks. Run-level aggregated input is not equivalent to raw per-episode input for weighting or uncertainty. +- Explain capability dimensions (Scene, Atomic Skill, Horizon, Precision, Mobility) and generalization dimensions (Object, Background, Instruction, Mixed) only where task labels and metrics support them. Mark absent axes as unavailable, not zero. +- Compare models on aligned benchmark revisions, split/task coverage, and evaluation settings. If coverage differs, present that difference and, when raw data permits, a clearly labeled matched subset; do not silently compare partial scores as a full benchmark. +- Do not present standard deviation across task/episode records as a confidence interval across independent runs. Separate repeated-seed variation from variation between tasks. +- Ground failure hypotheses in task-level metrics and representative episode traces. Aggregate scores alone cannot establish a camera bug, planning failure, or causal explanation. Use validation splits for suggested tuning; keep held-out results for final assessment. + +Deliver a linked HTML report, input run paths/IDs, actual loaded coverage, aggregation/reference settings, key supported findings, and limitations. If only reference data or partial results exist, make that the main conclusion. Do not publish results to a leaderboard as a side effect of analysis. diff --git a/skills/ebench-debug/SKILL.md b/skills/ebench-debug/SKILL.md new file mode 100644 index 0000000..bc4ec70 --- /dev/null +++ b/skills/ebench-debug/SKILL.md @@ -0,0 +1,33 @@ +--- +name: ebench-debug +description: Diagnose EBench evaluation failures, stalled workers, transport errors, invalid actions, and unexpectedly low scores using logs and episode artifacts. +--- + +# Diagnose an EBench run + +Work from the EBench root. Obtain the failing command with credentials redacted, run ID, worker ID, first relevant traceback, checkpoint/config, and whether the failure occurs during loading, readiness, reset, inference, step, or result saving. Read existing logs before rerunning anything. + +## Isolate the failing layer + +| Evidence | Next useful check | +| --- | --- | +| Import/config/weight failure | Active interpreter, installed package path, pinned submodules, baseline overlays, checkpoint files/statistics; server restart will not fix a local import. | +| Online task pending | Query the existing task with `gmp online ready`; distinguish scheduling from failure and retain its task ID. | +| Authentication or route failure | Distinguish platform URL from evaluation endpoint; check run ID and credential presence without printing credentials. | +| Reset timeout | Compare readiness, status and server logs if available. Asset loading and worker failure are hypotheses until supported. `EvalClient.reset()` kills/recreates its workers, so it is not a read-only probe. | +| Step timeout/disconnect | Correlate worker/server timestamps. Server execution may have completed; inspect progress before recovery and discard stale observations/chunks. | +| GPU OOM | Inspect model size, dtype and workers per GPU; reduce concurrency within scope before changing the model or evaluation protocol. | +| Implausible motion/near-zero SR | Inspect camera order, normalization, joint/gripper order, relative/absolute base semantics, replan horizon, and history reset before attributing failure to model capability. | +| Empty/missing report | Locate the actual result root, check completed episode files and writer errors; distinguish client traces from server result artifacts. | + +Read `baselines/X-VLA/run.py` for bounded recovery and `third_party/genmanip-client/src/genmanip_client/eval_client.py` for current reset/step/storage behavior. Do not turn every exception into an endless reconnect loop. Stop retries when the same failure persists without new evidence, and report the unresolved cause and retained artifacts. + +## Inspect one episode + +For client traces, `gmp plot "$EPISODE_DIR"` reads `steps.jsonl` and `merged*.mp4`, producing action/state plots and optional merged video. Inspect these against the adapter's transformations. + +For server-side results, `gmp visualize --project_root "$GENMANIP_ROOT"` uses the project's result tree. Its optional `visualize` dependencies and Rerun viewer cache are separate from model inference; the pinned README calls for Python 3.11+ for that viewer path. A viewer dependency failure does not establish that inference failed. + +Preserve logs, partial results, and task/seed identifiers. `gmp clean` deletes server evaluation results and logs; do not use it as a generic repair. Avoid resetting all workers to investigate one worker. Make the smallest justified fix and rerun only the needed verification within the requested evaluation scope. + +Deliver the observed symptom, evidence-backed cause (or remaining hypotheses), changed files/settings, validation outcome, and whether interrupted/partial episodes affect reported scores. Keep policy failures separate from infrastructure failures. diff --git a/skills/ebench-evaluate/SKILL.md b/skills/ebench-evaluate/SKILL.md new file mode 100644 index 0000000..3c6289a --- /dev/null +++ b/skills/ebench-evaluate/SKILL.md @@ -0,0 +1,80 @@ +--- +name: ebench-evaluate +description: Run and monitor an EBench policy evaluation against a local GenManip server or the online service, including baseline launch commands, worker allocation, and reproducible run records. +--- + +# Run an EBench policy evaluation + +Work from the EBench root. Read the selected baseline entry point and the relevant CLI implementation under `third_party/genmanip-client/src/genmanip_client/`. Verify installed `gmp ... --help` before relying on flags from a different revision. + +## Define the run + +Resolve model/checkpoint, track, split, server mode, GPU/worker budget, and output directory from the request. Use `val_train` / `val_unseen` for tuning. Run held-out `test` when requested for final evaluation; do not silently substitute a split or tune on held-out results. + +Record a small manifest beside the run logs: EBench and submodule commits, local code changes, checkpoint revision/path, config and normalization source, track/split/task selection, run/task ID, worker-to-GPU mapping, action/replan horizon, start time, sanitized command, and result locations. Mark unavailable fields as unknown. Exclude tokens and signed credentials. + +## Select one submission path + +- **Local GenManip:** verify the actual server config path or benchmark alias, then use `gmp submit "$CONFIG_PATH" --run_id "$RUN_ID" --host "$SERVER_HOST" --port "$SERVER_PORT"`. The pinned submit CLI takes host/port, not the online platform's `--base_url`. Submission schedules jobs; it does not load the user's policy. +- **New online task:** inspect `extensions/online_cli.py`; `gmp online submit --base_url "$PLATFORM_URL" --token "$TOKEN" --model_name "$MODEL_NAME" --benchmark_set EBench --timeout 600 --print_endpoint` creates a task and waits for readiness. Choose metadata, visibility and wait budget appropriate to the user's request; 600 seconds is an example budget, not a service guarantee. Capture returned `task_id` and `endpoint`; use that task ID as the client run ID. +- **Existing online task:** reuse it. Query `gmp online ready --base_url "$PLATFORM_URL" --token "$TOKEN" --task_id "$RUN_ID"`; use its returned evaluation endpoint. A wait timeout does not prove creation failed. Inspect existing task state before retrying creation; do not create duplicate tasks while resources are pending. + +The platform URL, returned evaluation endpoint, and an OpenPI model server address serve different purposes. Do not interchange them. Preserve credentials through local environment/configuration, omit them from reports, and avoid shell tracing of authenticated commands. + +## Online evaluation: queue, obtain endpoint, then evaluate + +For a new online evaluation, the agent should complete this sequence rather than require the user to obtain an endpoint manually. First verify the local model environment/checkpoint and obtain the platform URL and locally configured token. A new task does not need a pre-existing evaluation URL or task ID. + +1. **Submit and wait in the queue.** `gmp online submit` creates the task and polls until evaluation resources are ready. Run it as a monitored process; while pending, report that the task is waiting, not evaluating. This Bash example requires `jq` and uses a configurable wait budget: + + ```bash + set -euo pipefail + : "${PLATFORM_URL:?Set the online platform URL}" + : "${TOKEN:?Set the API token locally}" + : "${MODEL_NAME:?Set the model name}" + READY_JSON=$(gmp online submit \ + --base_url "$PLATFORM_URL" \ + --token "$TOKEN" \ + --model_name "$MODEL_NAME" \ + --model_type VLA \ + --benchmark_set EBench \ + --timeout "${QUEUE_TIMEOUT_SECONDS:-600}" \ + --print_endpoint) + + # Parse only a successful ready response; never launch with empty values. + EVAL_URL=$(printf '%s' "$READY_JSON" | jq -er '.endpoint | strings | select(length > 0)') + RUN_ID=$(printf '%s' "$READY_JSON" | jq -er '.task_id | strings | select(length > 0)') + export EVAL_URL RUN_ID + ``` + + `--print_endpoint` returns a JSON object containing both `endpoint` and `task_id`, not a plain URL. If submission fails, times out, or either field is missing, stop before launching the client. A timeout may leave a task queued: recover its ID from available logs/platform state and query `gmp online ready` instead of submitting again. If its ID cannot be determined, report that uncertainty rather than create a duplicate. + +2. **Save the assignment.** Record the returned task ID and endpoint in the local run metadata, excluding credentials. Use the returned task ID unchanged as `RUN_ID`; do not substitute a friendly experiment name. Keep any credential-bearing endpoint out of shared reports. +3. **Start actual model inference against the assigned endpoint.** Use the baseline commands below or the custom adapter. Pass `EVAL_URL` as the evaluation server address and `RUN_ID` as the run ID. Do not run a second `gmp submit` against the online endpoint: the online task already schedules the evaluation. For OpenPI, ensure its separate local model server is ready before launching the eval client. +4. **Monitor until evaluation completes.** Use the assigned endpoint/task ID for status and preserve the resulting logs and episode artifacts. Queue readiness only means resources are available; it does not mean the model has been evaluated. + +An existing ready task starts at step 2; an existing queued task uses `gmp online ready` until ready within the chosen wait budget. Once both fields are valid, continue to evaluation within the user's request without asking them to copy the values back manually. + +## Launch the real policy + +`gmp eval` supplies fake actions. Use it only for an explicitly scoped connectivity smoke test, never as evidence of a checkpoint's performance. + +For X-VLA, the existing wrapper accepts environment variables (here `EVAL_URL` is the evaluation endpoint): + +```bash +MODEL_PATH="$CHECKPOINT" BASE_URL="$EVAL_URL" RUN_ID="$RUN_ID" \ +TOKEN="$TOKEN" WORKER_IDS=0 GPU_IDS=0 LOG_DIR="$RUN_LOG_DIR" \ + bash scripts/run_xvla_eval.sh +``` + +For OpenPI, use the baseline README plus the actual `serve_policy.py` and `pi_eval_client_online.py` arguments; resolve the launch template's placeholders first. The current client constructs its policy adapter for `worker_ids[0]`, so launch one client process per worker rather than assuming one process drives all listed workers. + +For InternVLA-A1, run `inference.py` from its baseline directory with its supported `--ckpt_path`, `--url`, `--run_id`, `--token`, and `--worker_ids` arguments. Check the implementation before relying on a wrapper mentioned only in documentation. + +Across hosts, share the run ID and assign disjoint worker IDs. A GPU index is not a worker ID. Start with a small validation smoke run for a newly integrated policy before expanding to the requested budget; do not submit an extra held-out smoke run by default. + +## Monitor and finish + +Use `gmp status --url "$EVAL_URL" --run_id "$RUN_ID" --token "$TOKEN"` with worker logs. Distinguish waiting for resources, model loading, active steps, completed episodes, and transport failures. Bound polling/recovery to the user's time budget. Do not overwrite runs, clean results, or restart unrelated workers as routine recovery. + +At completion, verify server status and expected versus saved episode coverage, not just process exit code. Capture failed/missing episodes and interruptions. Locate actual client outputs (default `client_results`, overridable by `GENMANIP_RESULT_DIR`) and server outputs separately. Report completion or partial completion, run ID, manifest/log/result paths, and any blocker. A request to run evaluation does not by itself request separate leaderboard publication. diff --git a/skills/ebench-integrate-policy/SKILL.md b/skills/ebench-integrate-policy/SKILL.md new file mode 100644 index 0000000..c50862e --- /dev/null +++ b/skills/ebench-integrate-policy/SKILL.md @@ -0,0 +1,60 @@ +--- +name: ebench-integrate-policy +description: Implement or review a custom VLA policy adapter for EBench EvalClient, including observation preprocessing, action semantics, chunking, and episode resets. +--- + +# Integrate a policy with EBench + +Resolve paths from the EBench root. Read `third_party/genmanip-client/src/genmanip_client/eval_client.py` and the closest adapter: `baselines/X-VLA/run.py`, `baselines/openpi/scripts/pi_eval_client_online.py`, or `baselines/InternVLA-A1/inference.py`. Inspect the user's policy inference API and training transforms before choosing a mapping. + +## Establish the contract + +Observations are keyed by string worker IDs; a worker's model observation is `obs[wid]["obs"]`. Existing adapters use: + +- `video.overlook_camera_view`, `video.left_camera_view`, `video.right_camera_view`; +- `state.joints`, `state.gripper`, `state.base`, and optionally `state.ee_pose`; +- `instruction`, and `reset` for episode-boundary handling. + +Verify image type, RGB ordering, shape, resize/padding, proprioception ordering, units, and normalization against training. Do not invent missing cameras or silently replace missing input with zeros. Keep checkpoint normalization and model transforms paired. + +For the current r5a/lift2 joint-position adapters, the dispatched `action` orders left arm (6), left gripper (2), right arm (6), right gripper (2); `base_motion` has 3 components. The payload explicitly declares `control_type`, `is_rel`, and `base_is_rel`. Verify a different robot/control mode against its server contract rather than generalizing these dimensions. + +Model outputs are **not** interchangeable: + +- X-VLA rescales base/gripper channels, reorders joints/grippers, and sends absolute base motion. +- OpenPI reorders joints/grippers and differences chunk-relative base predictions into per-step deltas with `base_is_rel=True`. +- InternVLA-A1 has checkpoint statistics and an `action_mode` setting; inspect its conversion before selecting delta or absolute behavior. + +Document the chosen model-to-server channel mapping, normalization, frame/units, gripper interpretation, and relative/absolute semantics. Match the checkpoint, not whichever baseline is easiest to copy. + +## Implement lifecycle and chunking + +Use `EvalClient.reset()` for initial observations and `step()` for execution; inspect supported single-action/chunk forms in the pinned client. Keep string worker IDs consistent. Limit deployed horizon to available predictions and reset model history, cached actions, and temporal state at each episode boundary. `done` indicates evaluation completion, not task success; use saved result metrics for success. + +Never execute the remainder of an old chunk after an episode reset. For multiple workers, keep history/chunks independent and handle worker-specific resets. If a step times out, execution may already have occurred: do not blindly resend actions. Use bounded client recovery and fresh observations, discard stale actions, and record the interruption. Close the client in `finally` so recordings/results are flushed. + +## Connect the adapter to online evaluation + +Expose evaluation URL, run ID, token, and worker IDs as runtime settings rather than hardcoding them. For a new online run, follow the queue-and-launch sequence in [ebench-evaluate](../ebench-evaluate/SKILL.md): call `gmp online submit --print_endpoint`, wait for readiness, and parse its returned `endpoint` and `task_id`. No evaluation endpoint is required from the user before this submission. + +Only after both values are available, construct the client using the returned assignment: + +```python +import os +from genmanip_client import EvalClient + +client = EvalClient( + base_url=os.environ["EVAL_URL"], # online submit response: endpoint + run_id=os.environ["RUN_ID"], # online submit response: task_id + token=os.environ["TOKEN"], + worker_ids=["0"], +) +``` + +Connect this client to the policy's reset/inference/step loop and close it in `finally`. The platform base URL is used for queue submission; `EvalClient.base_url` is the returned evaluation endpoint. Do not create/reset workers while the task is still queued, or resubmit an online task each time the adapter reconnects. Adapter implementation alone does not require submitting a live task; use this flow when running the requested online evaluation. + +## Validate the adapter + +Use a representative local observation or fixture to check preprocessing, finite action values, dimensions, channel order, inverse normalization, chunk-length boundaries, and reset behavior without a simulator. Include a known-value action conversion example that would expose swapped channels or incorrect delta semantics; shape-only assertions are insufficient. + +Then run a small validation rollout when a server/checkpoint is available and evaluation is in scope. Inspect actual state/action traces before scaling. Deliver the adapter, concrete launch command, mapping description, and evidence distinguishing offline contract checks from live rollout validation. Keep model-specific code in its baseline/adapter directory rather than changing unrelated upstream submodules. diff --git a/skills/ebench-setup/SKILL.md b/skills/ebench-setup/SKILL.md new file mode 100644 index 0000000..3fbad97 --- /dev/null +++ b/skills/ebench-setup/SKILL.md @@ -0,0 +1,33 @@ +--- +name: ebench-setup +description: Prepare or check an EBench evaluation environment for OpenPI, X-VLA, InternVLA-A1, or a custom policy. Use for first-run setup and baseline reproduction prerequisites. +--- + +# Prepare EBench evaluation + +Resolve paths from the EBench repository root. Read `README.md`, the selected `baselines//README.md`, and `third_party/genmanip-client/pyproject.toml`. Prefer the checked-out implementation when examples disagree with it. + +## Establish the execution path + +- Identify the model/checkpoint, online endpoint versus locally hosted GenManip, intended track/split, and available GPUs. Reuse information already supplied; ask only for missing inputs needed for the next action. +- EBench contains adapters, not the Isaac Sim server. An online client does not need a local Isaac Sim installation. A local server requires a separate GenManip checkout and its environment/assets; inspect that checkout's instructions before proposing server commands. +- Inspect `git submodule status --recursive`, the active Python interpreter, installed package locations, GPU availability, and checkpoint existence. Initialize missing pinned submodules with `git submodule update --init --recursive` when setup is requested; do not advance them to arbitrary upstream heads. +- Install the pinned client in the selected model/client environment with `python -m pip install -e third_party/genmanip-client`. Choose `full_numpy1` or `full_numpy2` extras to match the baseline environment; do not install both. Keep incompatible baseline dependencies in separate environments. + +## Baseline-specific checks + +| Baseline | Source of truth | Checks that change the launch plan | +| --- | --- | --- | +| X-VLA | `baselines/X-VLA/run.py`, `scripts/run_xvla_eval.sh` | Check matching processor/checkpoint and requirements; worker IDs and GPU IDs are separate settings. | +| OpenPI | `baselines/openpi/README.md`, `baselines/openpi/src/openpi/training/config.py`, `baselines/openpi/src/openpi/policies/ebench_policy.py`, `baselines/openpi/scripts/pi_eval_client_online.py` | Apply EBench overlays together; verify the selected config actually exists and uses checkpoint-compatible normalization. Model WebSocket server and eval client have separate endpoints/environments. | +| InternVLA-A1 | `baselines/InternVLA-A1/inference.py`, `baselines/InternVLA-A1/eval_pjsim.sh` | Check upstream dependencies, checkpoint config/weights/`stats.json`, and the statistics key. Verify a documented wrapper actually exists before using it. | + +`scripts/launch_pi_onlineeval.sh` contains placeholder paths, activation commands, and settings; it is a template, not a ready-to-run launcher. Inspect shell wrappers before execution. Create a concrete local launch command using the user's paths rather than running placeholders or copying another machine's paths. + +## Verify and hand off + +For a new online evaluation, collect the platform URL and locally configured API token; the user does not need an evaluation endpoint or task ID yet. Check `gmp online submit --help` and availability of `jq` if using the shell example in [ebench-evaluate](../ebench-evaluate/SKILL.md). After local prerequisites pass, an evaluation request proceeds through `gmp online submit` → queue/readiness → returned `endpoint` and `task_id` → model launch using that endpoint and `run_id=task_id`. A setup-only request stops at reporting readiness. Reuse an existing task when supplied. + +Check imports and `gmp --help` in the selected environment before allocating a full run. Inspect CLI source if imports are unavailable. A connectivity test with `gmp eval` uses fake actions and is not a model evaluation; label it accordingly and keep its run separate from reported model results. + +Report the interpreter/environment, pinned revisions, model path/config, dependency gaps, and launch command with credential placeholders. Distinguish checks actually run from inferred compatibility. Do not claim readiness while weights, server access, or imports remain unverified. If asked to proceed with evaluation, continue within the existing request once prerequisites are satisfied.