You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This proposal introduces fine-grained and unified evaluation parallelism for RoboTwin and LIBERO using Labtasker.
It follows up on PR #28 and addresses its central evaluation-granularity concern by evolving the original whole-benchmark-task scheduling design into deterministic episode/trial-minibatch sharding.
The main features are:
Fine-grained evaluation: partition each benchmark task into independently schedulable episode/trial minibatches, directly addressing the requested RoboTwin slow-tail problem.
Unified evaluation interface: use the same submission, Worker, retry, monitoring, and result-aggregation abstractions across benchmarks.
Dynamic parallelism: run the same evaluation on one GPU, multiple local GPUs, multiple nodes, or independent jobs without changing its definition.
Elastic Worker capacity: add, remove, or replace Workers while an evaluation is running.
Built-in operations: persistent Task state, retries, failure recovery, progress reporting, Web UI, CLI/API access, and agent-based operation.
Reusable infrastructure: future benchmark adapters only need to define how to prepare evaluation instances and execute one minibatch, rather than designing another scheduler.
Deliberately small dependency footprint: Environments with strict dependency requirements can install only labtasker-client, which has just four direct runtime dependencies—httpx, packaging, pydantic, and typer. Its entire transitive dependency closure is already covered by OpenWAM’s core runtime dependencies. Adding it therefore has virtually no additional dependency side effect or compatibility impact on OpenWAM’s existing environment.
A normal RoboTwin benchmark task contains 100 episodes. Assigning the complete task to one Worker creates a long-tail problem: Workers finishing early cannot help with the remaining slow tasks, and a transient failure may repeat all 100 episodes.
The proposed default partition is five episodes or trials per Labtasker Task:
Workers dynamically claim the next available minibatch. This improves utilization and limits retries to the failed minibatch instead of the complete benchmark task.
The batch size remains configurable.
Unified evaluation parallelism
OpenWAM’s current benchmark launchers provide inconsistent orchestration. LIBERO has a relatively complete local multi-GPU launcher, but its queue is owned by one parent process and cannot naturally coordinate Workers across machines or independent jobs. Other benchmarks either lack comparable parallel evaluation or implement their own partial retry and resume behavior.
Extending these launchers separately would require every benchmark to redesign:
task partitioning and dynamic scheduling;
retries and crashed-Worker recovery;
task-state persistence and resume;
progress monitoring;
result discovery and aggregation;
local and multi-machine execution.
Labtasker provides these as shared, tested infrastructure. Each benchmark keeps its native evaluator and result semantics while using one common workflow:
prepare the evaluation
→ submit episode/trial minibatches
→ start or stop Workers as needed
→ monitor Tasks
→ aggregate results
This keeps benchmark adapters smaller and makes additional integrations easier to implement and maintain.
Design
1. Prepare a sealed evaluation manifest
Before evaluation is partitioned, OpenWAM prepares one small manifest for each benchmark task. The manifest fixes the ordered evaluation population and assigns every logical episode or trial a stable identity.
Important
FYI, individual episodes cannot always be reconstructed from an integer index. This is the inherent design flaw in LIBERO and Robotwin benchmarks.
In RoboTwin, candidate seeds are checked by an expert planner and infeasible scenes are skipped. Logical episode i is therefore the i-th accepted seed, not simply base_seed + i.
In LIBERO, trials consume a continuous environment RNG stream. Earlier resets advance that stream, so trial i cannot generally be reconstructed using seed + i.
The manifest records the accepted RoboTwin seeds and instructions, or the LIBERO reset RNG states and fixture identities. A minibatch then selects an exact range from this fixed population.
The manifest is immutable and content-addressed. Evaluation Tasks reference its concrete path and hash, ensuring that arbitrary Worker order, minibatch boundaries, and retries do not change which episodes are evaluated.
The overhead is small:
Manifests contain only compact evaluation metadata—not images, trajectories, or model artifacts.
The estimated complete cache is approximately 7 MiB for LIBERO and 2–3 MiB for RoboTwin across both modes.
Manifest construction is parallelizable by benchmark task.
The same manifest can be reused across checkpoints.
For LIBERO, this is a lightweight preprocessing step. For RoboTwin, it extracts the existing expert-feasibility scan from policy evaluation: the scan is performed once and reused rather than repeated for every checkpoint. Repeated evaluations can therefore save time compared with the original workflow while also comparing policies over exactly the same accepted episode population.
The manifest stores evaluation inputs only. Task scheduling, attempts, progress, and results remain in Labtasker.
2. Partition evaluation at the episode-minibatch level
After the manifests are available, evaluation is partitioned into contiguous minibatches.
a submission ID used to group the complete evaluation.
Checkpoint resolution happens once during submission, so all Workers use the same concrete model rather than independently resolving a potentially changing “latest” checkpoint.
Task IDs are deterministically derived from the submission ID and stable Task index. Repeating an interrupted submission with the same definition creates only missing Tasks without duplicating or resetting existing work.
3. Run reusable and elastic Workers
Users start Workers on resources they already control. Each Worker repeatedly:
claims a compatible minibatch;
verifies and loads its manifest range;
loads or reuses an OpenWAM policy endpoint;
runs the existing benchmark evaluator;
reports progress and a structured result;
claims another minibatch.
Each Worker owns one exclusive policy endpoint because the server maintains action-queue and episode state. The model remains loaded across minibatches with the same checkpoint and effective configuration; a model change causes a safe reload.
Worker capacity is elastic:
start additional Workers to accelerate an active submission;
stop Workers without cancelling the remaining Queue;
replace failed or preempted Workers;
connect Workers from additional nodes or SLURM jobs;
reuse running Workers for later compatible submissions.
No evaluation repartitioning or resubmission is required when capacity changes.
Eleven Workers serving two concurrent RoboTwin routes. Ten Workers are busy processing independently claimed Tasks, while one remains idle and available for new work.
4. Monitor Tasks and aggregate results from Labtasker
Task progress and results are queried directly from the Labtasker database.
The Web UI provides a unified view of evaluation Tasks and Workers across benchmarks, routes, and submissions.
One Labtasker Queue coordinating 540 fine-grained Tasks across concurrent LIBERO evaluation and RoboTwin manifest-building routes. Task status, progress, attempts, priority, results, and route-level Worker activity are visible without parsing launcher logs.
Users can:
monitor pending, running, succeeded, failed, and cancelled Tasks;
inspect progress, errors, attempts, arguments, and structured results;
filter Tasks by route, status, submission, priority, or result fields;
inspect active and idle Workers;
cancel, reprioritize, or requeue selected work;
save views for recurring evaluation workflows.
The same operations are available through the CLI and Python API, enabling scripted automation and agent-based operation. An agent can inspect failures, requeue exhausted Tasks, adjust priorities, cancel work, or generate summaries through documented Labtasker interfaces.
Successful Tasks store compact per-episode/trial results and references to detailed artifact directories. Videos and logs remain in OpenWAM’s normal output storage.
Summaries query all Tasks associated with a submission ID and aggregate their results into:
Labtasker contributes functionality that would otherwise need to be implemented and tested separately for every benchmark:
dynamic load balancing across Workers;
configurable priorities and retry budgets;
persistent Task status and structured results;
Worker lease recovery and stale-attempt fencing;
cancellation and explicit requeue;
real-time progress snapshots;
deterministic, idempotent submission;
CLI, Python API, and optional Web UI;
automatic local mode for one machine;
shared HTTP mode for multiple machines or independent SLURM jobs;
elastic Worker capacity without resubmission;
documented interfaces suitable for automation and coding agents.
Labtasker does not allocate GPUs or launch SLURM jobs. Users or existing schedulers start Worker processes; Labtasker coordinates them after they are running.
Common evaluation requirements
The coordinator lifecycle is reflected directly in each approach:
Labtasker state follows the Server and its database.
A parent-process queue follows the launcher process.
A file-lock queue follows its queue files and shared POSIX filesystem.
Legend: ◎ native fit; △ workable with additional integration or operational constraints; × unsupported or impractical with the current design.
File-lock queue (FastWAM LIBERO manager; also used by OpenWAM VLABench, implemented with flock)
Dynamic load balancing on one machine
◎ Native Worker claiming
◎ Native while the parent scheduler remains alive
◎ Native while queue files remain available
Multiple nodes in one allocation
◎ Workers connect to one internal Server
△ Requires remote child launching such as srun
△ Requires a shared POSIX filesystem
Multiple independent jobs
◎ Shared HTTP Server provides unified state
× Workers cannot share the parent’s memory queue
△ Possible only through shared queue files
Elastic Worker scale-up and scale-down
◎ Workers may join or leave an active Queue
△ Capacity is normally owned by the original parent
△ Possible, but Worker and file lifecycle handling is custom
Worker failure or preemption
◎ Native leases, retries, and stale-attempt fencing
△ Depends on the parent surviving and implementing recovery
△ Requires claimed-state tracking, timeout recovery, and fencing
Resume after coordinator restart
◎ Task state persists with the Server database
△ Usually reconstructed from result directories
△ Depends on the consistency and lifetime of queue files
No shared filesystem requirement
◎ Internal HTTP transport is sufficient
◎ Within one parent process or allocation
× Coordination depends on shared POSIX files
UI-based observability
◎ Web UI plus CLI/API
△ Requires a custom monitor
△ Requires inspecting files or building a custom monitor
Agent-based operation
◎ Documented Task API, filters, CLI, and agent skill
△ Requires benchmark-specific tooling
△ Requires benchmark-specific file and lock handling
Live priority, cancellation, and requeue
◎ Built-in Task operations
△ Must be implemented in the launcher
△ Must be encoded into the file protocol
The simpler alternatives remain reasonable within their original scope. The benefit of Labtasker is that the same benchmark interface continues to work when evaluation grows from one local launcher to elastic Workers across nodes or independent jobs.
Validation
The implementation has been extensively validated, covering fine-grained sharding correctness, determinism, operational behavior, real-policy execution, and compatibility with the original evaluation paths.
Fine-grained sharding and determinism: RoboTwin and LIBERO were evaluated with batch sizes 10, 5, 2, and 1. Single-instance Tasks were deliberately executed in reverse logical order, and selected Tasks were retried from the same manifest entries. Neither reordered execution nor policy-side Python, NumPy, Torch CPU, and Torch CUDA RNG mutation changed benchmark-owned traces.
Real-checkpoint evaluation: the sharding matrix was exercised with released OpenWAM checkpoints, not only mocked policies. RoboTwin preserved accepted seeds, final instructions, and success results across all granularities. LIBERO preserved trial identity, fixture selection, reward, and success.
Original entry-point compatibility: the ordinary non-Labtasker RoboTwin and LIBERO entry points remain manifest-free. Pre-adaptation and adapted versions were run over the same representative instances and produced matching selections and results, confirming that the new infrastructure does not break the existing evaluation workflow.
Task lifecycle and recovery: tests cover deterministic Task IDs, interrupted and idempotent resubmission, minibatch-level retries, cancellation handling, progress forwarding, result validation, and incomplete-submission reporting.
Worker and model lifecycle: Workers were verified to reuse one policy endpoint across minibatches with the same model configuration and to reload it when the checkpoint or effective deployment settings change.
Database-backed aggregation: summaries were verified to query Task status and structured results from Labtasker. They remained correct after the local manifest cache was made unavailable, confirming that result aggregation does not depend on scanning evaluator output files.
The final CPU contract suite contains 107 passing tests, supplemented by strict benchmark-trace comparisons and real released-checkpoint runs. Together, these tests validate the proposal’s main guarantees: minibatches may execute out of order, retry independently, and move between Workers without changing the manifest-defined evaluation.
Limitation: RoboTwin manifest identity
Independent RoboTwin expert-feasibility scans may produce different valid accepted-seed populations because of upstream planner and physics behavior.
This does not affect reproducibility within the proposed workflow: once a manifest is created, every Worker, retry, and checkpoint evaluation uses the same sealed manifest. Changing the policy / model / checkpoint will not affect the manifest, nor would they require manifest update.
The recommended practice is:
build one canonical manifest for an evaluation setup;
preserve and reuse it across policy comparisons;
if strict reproduction across machines or organizations is required, release the manifest alongside the checkpoint or evaluation artifacts.
Because the complete manifest is only a few MiB, publishing it is inexpensive and provides a precise, portable definition of the evaluated population.
Summary
This proposal introduces fine-grained and unified evaluation parallelism for RoboTwin and LIBERO using Labtasker.
It follows up on PR #28 and addresses its central evaluation-granularity concern by evolving the original whole-benchmark-task scheduling design into deterministic episode/trial-minibatch sharding.
The main features are:
labtasker-client, which has just four direct runtime dependencies—httpx,packaging,pydantic, andtyper. Its entire transitive dependency closure is already covered by OpenWAM’s core runtime dependencies. Adding it therefore has virtually no additional dependency side effect or compatibility impact on OpenWAM’s existing environment.Existing non-Labtasker evaluation entry points remain supported.
Motivation
Fine-grained evaluation
A normal RoboTwin benchmark task contains 100 episodes. Assigning the complete task to one Worker creates a long-tail problem: Workers finishing early cannot help with the remaining slow tasks, and a transient failure may repeat all 100 episodes.
The proposed default partition is five episodes or trials per Labtasker Task:
Workers dynamically claim the next available minibatch. This improves utilization and limits retries to the failed minibatch instead of the complete benchmark task.
The batch size remains configurable.
Unified evaluation parallelism
OpenWAM’s current benchmark launchers provide inconsistent orchestration. LIBERO has a relatively complete local multi-GPU launcher, but its queue is owned by one parent process and cannot naturally coordinate Workers across machines or independent jobs. Other benchmarks either lack comparable parallel evaluation or implement their own partial retry and resume behavior.
Extending these launchers separately would require every benchmark to redesign:
Labtasker provides these as shared, tested infrastructure. Each benchmark keeps its native evaluator and result semantics while using one common workflow:
This keeps benchmark adapters smaller and makes additional integrations easier to implement and maintain.
Design
1. Prepare a sealed evaluation manifest
Before evaluation is partitioned, OpenWAM prepares one small manifest for each benchmark task. The manifest fixes the ordered evaluation population and assigns every logical episode or trial a stable identity.
Important
FYI, individual episodes cannot always be reconstructed from an integer index. This is the inherent design flaw in LIBERO and Robotwin benchmarks.
In RoboTwin, candidate seeds are checked by an expert planner and infeasible scenes are skipped. Logical episode
iis therefore thei-th accepted seed, not simplybase_seed + i.In LIBERO, trials consume a continuous environment RNG stream. Earlier resets advance that stream, so trial
icannot generally be reconstructed usingseed + i.The manifest records the accepted RoboTwin seeds and instructions, or the LIBERO reset RNG states and fixture identities. A minibatch then selects an exact range from this fixed population.
The manifest is immutable and content-addressed. Evaluation Tasks reference its concrete path and hash, ensuring that arbitrary Worker order, minibatch boundaries, and retries do not change which episodes are evaluated.
The overhead is small:
For LIBERO, this is a lightweight preprocessing step. For RoboTwin, it extracts the existing expert-feasibility scan from policy evaluation: the scan is performed once and reused rather than repeated for every checkpoint. Repeated evaluations can therefore save time compared with the original workflow while also comparing policies over exactly the same accepted episode population.
The manifest stores evaluation inputs only. Task scheduling, attempts, progress, and results remain in Labtasker.
2. Partition evaluation at the episode-minibatch level
After the manifests are available, evaluation is partitioned into contiguous minibatches.
A RoboTwin Task identifies:
A LIBERO Task identifies:
Each Task also contains:
Checkpoint resolution happens once during submission, so all Workers use the same concrete model rather than independently resolving a potentially changing “latest” checkpoint.
Task IDs are deterministically derived from the submission ID and stable Task index. Repeating an interrupted submission with the same definition creates only missing Tasks without duplicating or resetting existing work.
3. Run reusable and elastic Workers
Users start Workers on resources they already control. Each Worker repeatedly:
Each Worker owns one exclusive policy endpoint because the server maintains action-queue and episode state. The model remains loaded across minibatches with the same checkpoint and effective configuration; a model change causes a safe reload.
Worker capacity is elastic:
No evaluation repartitioning or resubmission is required when capacity changes.
Eleven Workers serving two concurrent RoboTwin routes. Ten Workers are busy processing independently claimed Tasks, while one remains idle and available for new work.
4. Monitor Tasks and aggregate results from Labtasker
Task progress and results are queried directly from the Labtasker database.
The Web UI provides a unified view of evaluation Tasks and Workers across benchmarks, routes, and submissions.
One Labtasker Queue coordinating 540 fine-grained Tasks across concurrent LIBERO evaluation and RoboTwin manifest-building routes. Task status, progress, attempts, priority, results, and route-level Worker activity are visible without parsing launcher logs.
Users can:
The same operations are available through the CLI and Python API, enabling scripted automation and agent-based operation. An agent can inspect failures, requeue exhausted Tasks, adjust priorities, cancel work, or generate summaries through documented Labtasker interfaces.
Successful Tasks store compact per-episode/trial results and references to detailed artifact directories. Videos and logs remain in OpenWAM’s normal output storage.
Summaries query all Tasks associated with a submission ID and aggregate their results into:
Incomplete evaluations explicitly expose succeeded, pending, running, failed, and expected counts.
What Labtasker provides
Labtasker contributes functionality that would otherwise need to be implemented and tested separately for every benchmark:
Labtasker does not allocate GPUs or launch SLURM jobs. Users or existing schedulers start Worker processes; Labtasker coordinates them after they are running.
Common evaluation requirements
The coordinator lifecycle is reflected directly in each approach:
Legend: ◎ native fit; △ workable with additional integration or operational constraints; × unsupported or impractical with the current design.
srunThe simpler alternatives remain reasonable within their original scope. The benefit of Labtasker is that the same benchmark interface continues to work when evaluation grows from one local launcher to elastic Workers across nodes or independent jobs.
Validation
The implementation has been extensively validated, covering fine-grained sharding correctness, determinism, operational behavior, real-policy execution, and compatibility with the original evaluation paths.
Fine-grained sharding and determinism: RoboTwin and LIBERO were evaluated with batch sizes 10, 5, 2, and 1. Single-instance Tasks were deliberately executed in reverse logical order, and selected Tasks were retried from the same manifest entries. Neither reordered execution nor policy-side Python, NumPy, Torch CPU, and Torch CUDA RNG mutation changed benchmark-owned traces.
Real-checkpoint evaluation: the sharding matrix was exercised with released OpenWAM checkpoints, not only mocked policies. RoboTwin preserved accepted seeds, final instructions, and success results across all granularities. LIBERO preserved trial identity, fixture selection, reward, and success.
Original entry-point compatibility: the ordinary non-Labtasker RoboTwin and LIBERO entry points remain manifest-free. Pre-adaptation and adapted versions were run over the same representative instances and produced matching selections and results, confirming that the new infrastructure does not break the existing evaluation workflow.
Task lifecycle and recovery: tests cover deterministic Task IDs, interrupted and idempotent resubmission, minibatch-level retries, cancellation handling, progress forwarding, result validation, and incomplete-submission reporting.
Worker and model lifecycle: Workers were verified to reuse one policy endpoint across minibatches with the same model configuration and to reload it when the checkpoint or effective deployment settings change.
Database-backed aggregation: summaries were verified to query Task status and structured results from Labtasker. They remained correct after the local manifest cache was made unavailable, confirming that result aggregation does not depend on scanning evaluator output files.
The final CPU contract suite contains 107 passing tests, supplemented by strict benchmark-trace comparisons and real released-checkpoint runs. Together, these tests validate the proposal’s main guarantees: minibatches may execute out of order, retry independently, and move between Workers without changing the manifest-defined evaluation.
Limitation: RoboTwin manifest identity
Independent RoboTwin expert-feasibility scans may produce different valid accepted-seed populations because of upstream planner and physics behavior.
This does not affect reproducibility within the proposed workflow: once a manifest is created, every Worker, retry, and checkpoint evaluation uses the same sealed manifest. Changing the policy / model / checkpoint will not affect the manifest, nor would they require manifest update.
The recommended practice is:
Because the complete manifest is only a few MiB, publishing it is inexpensive and provides a precise, portable definition of the evaluated population.