[Proposal] Shared sim.scene services and procedural scene construction #760

Proposal

Introduce embodichain.lab.sim.scene as the shared layer for scene composition, procedural construction, spatial layout, deterministic operations, physical state restoration, validation, and expansion. Design interfaces for general scenes and explicit parallel-environment selection; implement the tabletop/B=1 integration first.

Motivation

GenSim Scene Engine already implements support detection, group alignment/clamping, constrained XY layout and gravity settling, but those algorithms are coupled to image-generation artifacts, a mandatory table, and the Y-up editing model. PR #749 introduces reusable scene proposals, while PR #750 supplies physical preparation and initial-state restoration. Sharing their numerical and scene services avoids maintaining separate layout, state and validation implementations.

This is the broader module roadmap. #748 continues to track the Task Program scene-expansion use case. The phase-one replacement PR will absorb #749, preserve its public compatibility imports and proposal identities, and adapt #750's host lifecycle. Close #749 only after its replacement is published and the dependent work is accounted for.

Architecture and ownership

• sim.scene: explicit asset/config composition, spatial proposals and queries, deterministic changes, physical state services, scene checks and candidate provenance/budgets.
• compute: pure tensor geometry/arithmetic kernels with explicit batch dimensions and dtype/device preservation; no scene configuration, search policies, logging, simulation/Gym/GenSim dependencies. Scene-specific CPU optimization and preprocessing belong to sim.scene.
• GenSim: image/text understanding, VLM decisions, asset retrieval/generation, editing models, pipeline scheduling and format adapters.
• Spawn/SimulationManager: authoritative entity registries, native handles, topology preparation, replication, backend configuration, physics stepping and teardown.
• Gym or standalone hosts: reset/step timing, temporary simulation lifecycle, observation/recording refresh and persistence.
• Task Program/motion: semantic bindings, execution, task acceptance, IK/workspace and planning.
• Scene/task components: domain recipes and physical environment configuration. No second runtime scene graph or executable scene DSL.

Public capability groups

composition (fragments and namespaced config assembly), procedural (explicit recipe execution and seeded sampling), geometry/query (support regions and existing-entity views), layout/placement (pure proposals), operations (preflight and apply), state (owned physical initial states), validation (measured checks), expansion (existing candidate contracts). Format adapters are introduced when needed, without empty placeholder modules or a new persistence format in phase one.

Contracts use physical UIDs, metres, Z-up arena-local transforms and xyzw quaternions. GenSim adapters own Y-up/Euler conversion and preserve asset-scale semantics. Candidate identity excludes physical slots. Queries and layout solvers never advance physics. Native writes do not promise transactional rollback. Stable, geometrically valid, reachable and task-successful are separate outcomes; required unverified checks cannot accept a candidate. State restoration is fixed-topology episode restoration, not a complete solver checkpoint.

Implementation roadmap

Phase 1 — Shared tabletop foundation (current implementation)

• Extract single-owner support geometry, rigid group placement and constrained XY/AABB layout algorithms into compute and scene services.
• Preserve separate generation group-placement and editing local-optimization policies through GenSim adapters.
• Extract physical state/query/pose operations from Integrate scene workspace checks and add a UR5 showcase #750 while retaining its reset ordering, metadata, round-trip tolerances and recording boundaries.
• Move Add scene candidates and measured Task Program execution #749 candidates to scene.expansion, preserving old imports, IDs and permissions.
• Add explicit opt-in convergence checks while retaining fixed-step settling defaults.
• Add a model-service-free tabletop recipe: work surface, two containers, existing parts and a keep-out region, using the same layout and validation interfaces.
• Publish API documentation and routed project context; verify GenSim and Task Program regressions and native tabletop/execution/restoration smokes.

Runtime integration remains B=1; unqualified batch mutation is rejected before writes. Interfaces carry environment selection and retain state batch dimensions. Supported layout is horizontal support with fixed-orientation XY proxies and hierarchical placement, not arbitrary 6D exact collision solving.

Phase 2 — Parallelism and general regions

• Qualify batched state writes, selective environments, independent logical random streams and untouched-row isolation.
• Add multiple support regions, containment volumes, orientation sampling, layout repair and exact collision adapters.
• Extend composable recipes, layout/style separation and spatial distributions.
• Specify shared-World stepping effects for settling instead of assuming partial physics isolation.

Phase 3 — Structural editing and large scenes

• Add asset insertion/removal/replacement, attachment constraints, scene fragments and portable scene packages through existing Spawn/Manager owners.
• Add room/shelf layout, visibility, navigability and robot deployment queries.
• Keep domain semantics and complex asset generation in upstream consumers/providers.

Validation and acceptance

Test undersized regions, contradictory constraints, fixed objects, group preservation, stacking and bounded solver failure; asymmetric rotations and nonuniform scale; unsupported/unknown entities rejected before mutation; topology/owner guards and state deep copies; control targets and quaternion-sign-equivalent restoration; settling timeout/escape and post-settle checks; GenSim export compatibility; legacy candidate IDs/imports and rejected-recording persistence gates. Test production services rather than tutorial scripts. Run one native tabletop preparation and one Task Program execution/restoration smoke; document unqualified backend/GPU paths.

Phase-one delivery

Implemented in PR #761, with the adapted concrete host in PR #750, forming native stack #762. PR #749 is superseded and closed.

The numerical boundary has been refined: compute.geometry now contains five pure, broadcastable Torch CPU/CUDA functions (AABB intersection, containment violation, masked group-height reduction, affine box transforms, triangle geometry). It owns no scene configuration, logging, object IDs, mesh/polygon preprocessing or search policy. The CPU support detector, placement/search strategies and SLSQP implementation live under sim.scene; no general backend/solver framework was added. The former draft compute.layout namespace and compute-side authoring classes were removed, while established scene/GenSim entry points remain compatible.

Latest validation: 452 CPU/integration/context/docs checks and 37 explicit CUDA checks passed in the foundation; 241 upper-layer scene/workspace/host checks and native UR5 execution/restoration passed. Native tabletop preparation, measured footprints/keep-out checks and restoration also passed. API/Black/context gates and the foundation's fresh Sphinx build passed.

Reproducible benchmark: python -m scripts.benchmark.scene.geometry_kernels --device cuda:0 --iterations 100. For 1024 independent layouts/candidates with 32 objects each on an RTX PRO 5000 72GB Blackwell, the float32 geometry workload measured 20.214 ms with Torch CPU (4 threads), 1.203 ms with CUDA-resident inputs, and 2.310 ms including H2D/D2H. All accuracy checks passed (max absolute numerical error 3.83e-6); peak allocated VRAM was 67.35 MiB. Single-layout CPU execution was faster. These shared-machine measurements cover geometry functions, not the SLSQP solver or full scene execution.

Batched kernels do not change the B=1 physical-host qualification. Phase-two parallel physical state mutation and phase-three capabilities remain open.

Additional context

Research references: Infinigen Indoors, Infinigen-Articulated, ProcTHOR, Holodeck, RoboCasa, Scene Synthesizer, BlenderProc.

Checklist

• I have checked that there is no similar issue in the repo; [Proposal] Task Program Scene Expansion and Expert Trajectory Generation #748 covers the narrower expansion workflow and is linked above.