Skip to content

Add shared scene services and batched geometry kernels - #761

Merged
yuecideng merged 11 commits into
mainfrom
codex/scene-foundation
Oct 13, 2026
Merged

yuecideng merged 11 commits into
mainfrom
codex/scene-foundation

Conversation

@yuecideng

@yuecideng yuecideng commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Description

GenSim generation/editing, procedural recipes and Task Program scene expansion now share embodichain.lab.sim.scene services for configuration composition, placement, layout, physical state, deterministic operations and validation. A model-service-free tabletop recipe builds two containers, parts and a keep-out region, then prepares, settles, re-measures and restores the scene.

The numerical boundary is deliberately small: compute.geometry exports five pure Torch functions for AABB intersection, containment violation, masked group-height reduction, affine box transforms and triangle normals/areas. They support broadcast batch dimensions, float32/float64 and CPU/CUDA without reading tensor values on the host. Scene configuration, UID mappings, polygon/mesh preprocessing, logging, candidate search and SLSQP remain in scene-owned CPU modules. No general backend/solver framework is introduced. The earlier draft's compute-side authoring classes and compute.layout namespace were removed; established scene and GenSim entry points remain compatible.

This PR absorbs #749, retaining its compatibility imports, proposal identities, movable permissions and rejected-recording gates. Main's current dynamic Task Program language behavior is preserved. The concrete workspace/Gym host remains in #750 and consumes the shared services. Numerical batches do not imply batched physical mutation: the runtime integration remains B=1.

Dependencies: the standard EmbodiChain installation uses SciPy and trimesh supplied by DexSim and directly requires shapely>=2.0 for its placement API. No separate scene extra or additional scene installation step is needed. The tensor kernels require Torch only. No new physics backend or model provider.

Refs #760 (module roadmap), #748 (Task Program expansion workflow). Supersedes closed #749; later scene capabilities remain tracked in #760.

Stack

Type of change

  • New feature
  • Enhancement
  • Documentation update

Validation

  • Final dependency/review fixes: 506 focused scene, GenSim, Task Program/demo and context tests passed. Wheel metadata verifies unconditional shapely>=2.0 and no scene extra; Black, API coverage (2589/2589), context and whitespace gates passed.

  • Rectangular projection now computes boundary offsets directly and rounds inward before exact containment validation; decimal-coordinate regressions and 150 boundary probes passed while oversized AABBs remain rejected.

  • Snapshot restoration rejects saved qpos/position targets outside current limits before any entity writes. Regression coverage includes excursions, limits changed after capture and a round trip through the production articulation setter. Native/GPU qualification below is from the earlier implementation; these final changes were validated with CPU tests.

  • Example/tutorial visualization coverage and API-doc tooling: 9 checks passed. The tabletop recipe explicitly disables browser visualization and rendering, and its native proposal/convergence/layout/restoration checks passed again on 9c78251f.

  • The scene reference and Scene Engine guide now list the shared adapters, distinguish the explicit recipe path, and document the B=1 physical runtime versus batched tensor-kernel scope. Existing routed context remains accurate; no context change is needed for this example/documentation fix.

  • 452 CPU/integration/context/docs checks passed; 37 CUDA cases were skipped in that CPU run and then all 37 passed in an explicit, serial CUDA run.

  • Coverage includes GenSim Scene Engine, scene proposals and physical services, padding/empty batches, masks, dtype/device preservation, transformed-box corner oracles, and numerical degeneracy without implicit host synchronization.

  • Independent review preserved contained/zero-sized AABB solver semantics and kept single-pair collision queries constant-sized instead of building an N-by-N tensor.

  • Benchmark reference checking exposed and fixed degenerate-triangle cancellation residuals being normalized into false unit normals; no tolerance was loosened.

  • Native B=1 CPU/no-render tabletop recipe passed proposal, convergence, measured XY/keep-out checks and state restoration with the existing proposal identity.

  • 101 architecture/context/docs tests passed. Black 26.3.1 black ., whitespace and context gates passed; API coverage is 2589/2589.

  • Standard fresh Sphinx dummy build passed with 830 existing warnings; new scene/compute/benchmark pages and architecture have no new warnings.

Geometry benchmark

Reproduce with python -m scripts.benchmark.scene.geometry_kernels --device cuda:0 --iterations 100. The report includes CPU/GPU memory, an independent NumPy float64 accuracy reference and all measured implementations. Measurements below use float32, 4 Torch CPU threads and an RTX PRO 5000 72GB Blackwell on a shared machine; B is independent layouts/candidates and N is objects per layout.

B × N Torch CPU CUDA resident CUDA including H2D/D2H
1 × 32 0.476 ms 1.184 ms 2.187 ms
64 × 32 2.151 ms 1.209 ms 1.969 ms
1024 × 32 20.214 ms 1.203 ms 2.310 ms

All accuracy checks passed (maximum absolute numerical error 3.83e-6 across outputs); largest measured CUDA peak was 67.35 MiB. Small CPU workloads remain appropriate for CPU execution. These timings cover the geometry workload, not the SLSQP solver or an entire scene pipeline.

Horizontal/fixed-orientation proposals and proxy checks remain explicit limits. Changed orientations require measured footprints. Full solver checkpoints, batched physical mutation, exact mesh collision and general room generation are not qualified in this phase.

Screenshots

No UI change.

Checklist

  • I have run black . with Black 26.3.1.
  • I reviewed and updated API docs, routed context and architecture evidence for the final compute/scene ownership.
  • Public API changes are reflected in the API docs.
  • Focused production tests, CPU/CUDA comparisons and native smoke validation cover the behavior.
  • Dependency declarations and installation instructions reflect standard installation and the required Shapely version.

@yuecideng yuecideng added enhancement New feature or request physics Things related to physics refactor assets Related to simulation assets (robot, CAD, material, etc) labels Oct 11, 2026
@yuecideng
yuecideng added this pull request to stack #762 October 11, 2026 16:42
@yuecideng yuecideng changed the title Add shared scene services and tabletop generation Add shared scene services and batched geometry kernels Oct 12, 2026
@yuecideng
yuecideng requested a review from MuziWong October 12, 2026 10:35
@yuecideng
yuecideng marked this pull request as ready for review October 12, 2026 12:18
@greptile-apps

greptile-apps Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium impact] The PR appears safe to merge, with no new actionable issues established.

Summary

Adds shared scene services and five batched Torch geometry functions. GenSim adapters reuse the scene-owned CPU strategies, while execution hosts retain physics and recording ownership.

  • Scene recipes build and check layouts while hosts keep control of physics.
  • Torch geometry kernels compare and transform batched boxes and triangles.
  • Task Program episodes can evaluate a scene proposal and keep results separate.
  • GenSim scene helpers now use shared support, placement, layout, and settling strategies.
  • Project guides and API pages explain the new scene and geometry boundaries.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  G[GenSim adapters] --> S[Shared scene services]
  R[Explicit recipes] --> S
  S --> C[CPU placement and layout strategies]
  C --> T[Batched Torch geometry]
  S --> P[B=1 physical state services]
  H[Caller-owned execution host] --> P
  H --> V[Measured checks]
  V --> A[Accept or reject]
Loading

Reviews (3) · Last reviewed commit: "fix(scene): simplify dependencies and en..." · Reviewed by Greptile

Comment thread pyproject.toml Outdated
@MuziWong

Copy link
Copy Markdown
Collaborator

Scene Randomization Design: Persistence and Policy

Hi Yueci! Thanks for your work on the shared scene services in #761. I've drafted the proposal below to build on those services, and I'd appreciate your feedback on whether it aligns with the intended scope.

Status: design draft. Base-data caching, randomization adapters, and layout I/O remain to be implemented. Date: 2026-10-13.
Based on PR #761: Add shared scene services and batched geometry kernels
and commit 9c78251f0 on codex/scene-foundation.

The proposed approach groups each base object directly supported by the table with all objects supported above it, and randomizes the group using its combined 2D AABB. The source scene and assets are shared; each result stores only the group layout.

Persistent Data

What to Store

Data File Contents and purpose
Source scene scene_export/ Existing configurations, support graph, and final assets that define the common baseline
Base data scene_randomization_base.yaml Table region, automatic groups, baseline poses, internal group structure, and whole-group AABBs
Randomization settings scene_randomization_policy.yaml Participating groups, displacement limits, clearance, seed, budgets, and diversity settings
Layout results layouts/candidate_<ordinal>.json Absolute XY center and yaw delta for each group, plus source identity and configuration references
Run records policy_resolved.yaml, run_result.json, and optional selection_result.json Effective configuration, generation outcomes, and selection results for replay and inspection

The base YAML and layout JSON files are new GenSim-derived data defined by this design.
PR #761 provides shared services; these files do not introduce a new sim.scene scene format
or become required inputs to the existing importer.

The proposed directory structure is below. Entries marked "new" belong to this design:

<workspace>/
├── existing_scene/
│   └── scene_export/
│       ├── scene_config.json
│       ├── scene_graph.json
│       ├── scene.json
│       ├── mesh_assets/<id>/<id>.glb
│       ├── articulated_assets/                  Existing exports may include these; the first version does not consume articulations
│       └── scene_randomization_base.yaml        New, one shared copy
├── scene_randomization_policy.yaml              New, user settings
└── scene_layouts/                               New, a fresh directory for each run
    ├── policy_resolved.yaml
    ├── run_result.json
    ├── selection_result.json                    When diversity is enabled
    └── layouts/
        ├── candidate_000000.json
        ├── candidate_000001.json
        └── ...

The source format remains embodichain.scene-export/v1; PR #761 does not change it.
scene_config.json provides entity UIDs, relative asset paths, baseline poses, scales, and physics
configurations. scene_graph.json provides the parent / on hierarchy; scene.json is an editable
debug snapshot. Reusing an existing scene requires at least the first two JSON files and all
referenced final assets. Intermediate directories from image understanding, generation, or editing
are not required. USD/USDZ alone does not provide the support graph and safe rectangle required
by this design.

Base Data

The first version consumes rigid-body support subtrees on a single horizontal, connected table
surface without holes. The existing Scene Engine importer and SceneGraph require the table UID
to be table; the shared LayoutProblem.support_uid has no such naming restriction.
All values and digests below are illustrative.

schema_version: 1
coordinate_frame: local_arena_z_up
source_scene:
  scene_id: scene-engine-example
  export_dir: .
  content_sha256: "0000000000000000000000000000000000000000000000000000000000000000"

table:
  entity_id: table
  center_xy: [0.0, 0.0]
  size_xy: [1.2, 0.8]
  surface_z: 0.75

groups:
  - root_id: bottle_001
    root_pose:
      position: [-0.30, 0.10, 0.85]
      quaternion_xyzw: [0.0, 0.0, 0.0, 1.0]
    members: []
    aabb_2d:
      center_xy: [-0.30, 0.10]
      size_xy: [0.10, 0.10]

  - root_id: tray_001
    root_pose:
      position: [0.15, 0.10, 0.77]
      quaternion_xyzw: [0.0, 0.0, 0.0, 1.0]
    members:
      - entity_id: block_001
        parent_id: cup_001
        pose_in_root:
          position: [0.03, 0.02, 0.15]
          quaternion_xyzw: [0.0, 0.0, 0.0, 1.0]
      - entity_id: cup_001
        parent_id: tray_001
        pose_in_root:
          position: [0.03, 0.02, 0.08]
          quaternion_xyzw: [0.0, 0.0, 0.0, 1.0]
    aabb_2d:
      center_xy: [0.16, 0.10]
      size_xy: [0.30, 0.20]
  • Grouping: Each object directly on table.entity_id is a root, and all its on descendants
    belong to the same group. Every entity other than the table belongs to exactly one group.
    A group containing only its root is valid. Entities that cannot be covered make the input unsupported.
  • Internal structure: The root stores its baseline pose. Each member stores its direct parent
    and its pose relative to the same root, including members deeper in the hierarchy.
    Reconstruction uses T_member = T_root × T_member_in_root.
  • Table: Its center and size describe a safe rectangle inside the support surface;
    surface_z is the top-surface height. These are not the table's overall AABB.
  • Group AABB: Combine the final geometry of the root and all members after applying their
    existing scales and baseline poses. The center may differ from the root origin;
    an individual object's center_xy cannot substitute for the whole-group center.
  • Coordinates: All base data uses arena-local Z-up coordinates, meters, and normalized xyzw
    quaternions, without environment replication offsets. The editable scene remains Y-up.
    Exported rigid-body init_pos and init_rot are already Z-up; init_rot uses XYZ Euler angles
    in degrees. GLB assets and body_scale retain their original Y-up convention. The adapter must
    avoid repeating the axis conversion or incorrectly swapping scale components.

AABB minimum and maximum corners are derived from center_xy ± size_xy/2 and are not stored again.
Groups and members are sorted by ID. Sizes must be positive, values finite, and entities unique;
relative poses must reconstruct the source baseline.
The base file contains layout and geometry facts only. It stores neither velocities nor control
targets and cannot replace a physical SceneState.

Layouts and Run Records

Each candidate layout has one JSON file containing all groups; a run stores multiple candidate
files. Every layout references the same base file and stores no scene copy, assets, or duplicate
member poses. For example:

{
  "schema_version": 1,
  "candidate_id": "candidate_000001",
  "proposal_ordinal": 1,
  "seed": 42,
  "algorithm_version": "group_xy_slsqp/v1",
  "coordinate_frame": "local_arena_z_up",
  "geometry_model": "fixed_orientation_aabb",
  "source_scene": {
    "scene_id": "scene-engine-example",
    "content_sha256": "0000000000000000000000000000000000000000000000000000000000000000",
    "base_data": "../../existing_scene/scene_export/scene_randomization_base.yaml",
    "base_data_sha256": "0000000000000000000000000000000000000000000000000000000000000000"
  },
  "policy_file": "../policy_resolved.yaml",
  "policy_sha256": "0000000000000000000000000000000000000000000000000000000000000000",
  "groups": [
    {"root_id": "bottle_001", "center_xy": [-0.30, 0.10], "yaw_delta_deg": 0.0},
    {"root_id": "tray_001", "center_xy": [0.36, 0.10], "yaw_delta_deg": 0.0}
  ]
}

center_xy is the absolute center of the whole-group AABB; yaw_delta_deg must be 0 in the first
version. Every group appears exactly once, including fixed groups at their baseline centers.
Candidate IDs correspond to zero-based proposal ordinals. Failed attempts produce no layout
file, so numbering may contain gaps. This example is not the actual sample produced by seed=42.

policy_resolved.yaml stores the effective parameters and defaults. run_result.json stores the
source digest, seed, algorithm version, actual attempt count, valid candidate count and references,
and solver status counts. Sampling status is completed, budget_exhausted, invalid_input,
or unsupported. When selection is enabled, selection_result.json stores selection order,
references to the original layouts, scores, and target/actual counts. Its status is completed,
insufficient_candidates, or insufficient_diversity. Sampling and selection statuses are
recorded separately.

Every valid candidate is saved as a layout JSON; the number of files equals the actual number
of valid candidates. Enabling diversity preserves the entire candidate pool.
selection_result references the selected layouts without copying files, and consumers read
them in selection order. With selection disabled, consumers read all valid candidates in the
proposal order recorded by run_result. For example, generating 100 candidates and selecting
20 produces 100 layout files and a selection file referencing 20 of them.

File references are resolved relative to the file that contains them. The source content_sha256
covers the configuration, graph, and asset paths and contents in a stable order, using relative
paths and excluding machine-specific absolute paths and the debug scene.json.
base_data_sha256 verifies the base file. policy_sha256 hashes the effective behavioral
parameters, including defaults, while excluding input/output location paths.
Moving the complete package while preserving its relative references does not change source
or candidate identity because of machine paths.

Preparing and Reusing Base Data

  1. Read the source configuration, graph, and assets without modifying them. Validate the format,
    entity set, and first-version scope.
  2. If an existing base file has a valid schema, coordinate frame, source digest, and structure,
    reuse its surface, groups, poses, and AABBs. Digest verification still reads source file contents
    but does not parse meshes again to compute geometry.
  3. If the base file is missing or invalid, group entities from the support graph, extract relative
    poses, and read final meshes to compute whole-group AABBs. Use shared table detection results,
    verify the safe rectangle, and save the base YAML without implicitly rewriting the source
    configuration or exporting it again.

Table detection currently retains only the largest outer contour, which alone does not prove
coverage by the actual support surface. During initial extraction or recomputation, project
the support triangles from the detector's vertices/faces onto XY and compute their union.
Require a single valid Polygon without holes and union.covers(rectangle). Reuse existing
support fields when they agree with the current geometry; otherwise verify the detected height
and rectangle. Report failure for unsupported inputs; this design adds no rectangle-search
algorithm. Ordinary loading of a valid cache does not repeat this calculation.

Changes to entities, support relations, member poses, geometry, scale, or the table region require
rebuilding the base file. Moving the package alone does not. Sampling modifies neither the source
scene nor the base file. The existing importer overwrites the debug scene.json; the new
preparation entry point must read source data without inheriting this write side effect.
Restoring a scene still requires the source assets alongside the base YAML.

Randomization Policy

First-Version Scope Aligned with the PR

PR #761 initially supports fixed-orientation XY/AABB layouts on horizontal supports.
This policy preserves each group's orientation, Z height, and internal structure, keeps its entire
AABB inside the table's safe rectangle, and enforces clearance between both fixed and sampled groups.

GenSim owns grouping, coordinate conversion, and format adaptation, and directly reuses
sim.scene sampling, layout solving, and candidate contracts. Compute handles pure tensor geometry
only; this design adds no separate SLSQP implementation, compute.layout, or backend framework.
Solving currently runs on CPU, and physical application remains B=1. CUDA batch support in the
geometry kernels does not mean these workflows run in parallel.
The PR already provides seeded XY sampling and solving through SceneRecipe → generate,
pose application, and layout checks. Our additions are automatic grouping and base-data caching,
transform expansion to every group member, lightweight layout files, and diversity selection.

Configuration

This example generates 100 valid candidates and then selects 20:

schema_version: 3
input:
  base_data: ./existing_scene/scene_export/scene_randomization_base.yaml

groups:
  tray_001:
    mode: sample
    translation_xy_m:
      x: null
      y: null
  bottle_001:
    mode: fixed

layout:
  collision_margin_m: 0.02
  max_slsqp_iterations: 500

sampling:
  seed: 42
  candidate_count: 100
  max_attempts: 1000

diversity:
  enabled: true
  count: 20

output:
  directory: ./scene_layouts
  • groups accepts only root IDs from the base data. Unlisted groups are fixed, and at least one
    sampled group is required. Fixed groups have no motion parameters.
  • XY displacement limits in translation_xy_m default to null. [min,max] defines a displacement
    window in meters relative to the baseline AABB center; [0,0] locks that axis. The two layout
    fields map directly to LayoutCfg; other numerical controls retain shared defaults.
  • seed defaults to 0. candidate_count is the valid candidate-pool target P and defaults to 10.
    max_attempts is the total budget for complete attempts, including successes and failures;
    it defaults to 1000 and must be ≥P.
  • Diversity is disabled by default. When enabled, count is required and defines the final target M,
    with 1≤M≤P.

The seed must be a nonnegative integer; counts and iteration limits must be positive integers.
Clearance must be finite and nonnegative; displacement windows must be finite with min≤max.
Reject duplicate keys, unknown fields, unknown groups, and rotation/overhang/batch parameters
from earlier drafts. The output directory must not overwrite the source or existing run results.
Before sampling, check fixed-group boundaries and clearances, and each sampled group's feasible
center interval.

Geometric Constraints and Solver Inputs

Let c0 be the baseline group center, h its half-extents, and Smin/Smax the table's safe boundaries.
The target center interval is:

center_min = Smin + h
center_max = Smax - h

If a displacement window is configured, intersect this interval with
[c0+delta_min,c0+delta_max] on each axis. Equal endpoints give a deterministic coordinate;
an empty interval makes the input unsupported. The final solver result must also respect this window.

LayoutProblem field Mapping
support_uid / frame Table entity UID / arena; the table is excluded from pairwise group collisions
seed_xy_by_uid root_id → c0 in the baseline recipe; generate internally replaces sampled-group centers for each attempt
half_extents_xy_by_uid root_id → whole-group half-extents h
bounds_xy_by_uid root_id → (center_min-h, center_max+h); fixed groups also have bounds
fixed_xy_by_uid Fixed-group root_id → c0
relations / preferred_uids () / None
skip_fixed_collisions False; LayoutCfg.enforce_all_clearances is always True

Bounds describe the allowed region for the complete AABB, not its center interval.
Without a displacement window, they are the table boundaries Smin/Smax.
Here, root_id identifies a whole-group proxy, not the root origin.
Clearance reuses collision_margin_m=m:

gap_x = abs(cx_i-cx_j) - hx_i - hx_j
gap_y = abs(cy_i-cy_j) - hy_i - hy_j
Require gap_x >= m or gap_y >= m

Use the shared feasibility_tolerance; m=0 allows edge or corner contact.
This is axis-wise clearance, not the shortest Euclidean distance.
Strict randomization constraints do not alter compatibility behavior in the legacy GenSim editing
adapter. Cross-group directional relations and table_region in the source graph are not constraints
in the first version. A candidate editing graph preserves the on hierarchy; invalidated directional
descriptions must be recomputed or cleared.

Sampling

  1. Prepare feasible center intervals, fixed groups, and half-extents once. Build a SceneFragment
    from baseline entity configurations, then a SceneRecipe. The layout lists only each group's
    root_id and uses whole-group AABBs as geometry proxies; the fragment contains the table,
    roots, and members. Meshes are not loaded for each candidate.
  2. Assign a zero-based proposal_ordinal to each complete attempt and call the existing
    generate(recipe, seed=seed, ordinal=proposal_ordinal, cfg=layout_cfg).
    It uses SeedSequence([seed, ordinal]) and samples centers independently and uniformly inside
    each sampled group's feasible XY rectangle, in root-ID order. Fixed groups consume no random
    draws. Reuse this sampler rather than implementing it again.
  3. Generate calls solve_layout once, allowing the shared CPU solver to adjust centers to satisfy
    constraints. The final feasible layout distribution is not guaranteed to be uniform.
    Extract group centers from the returned layout.xy_by_uid.
  4. Only layout.status=solved enters the pool. Record no_solution or budget_exhausted and
    attempt another complete layout; never apply a partial result. Stop on unsupported or an
    input exception. Search failure does not prove mathematical infeasibility.
  5. Stop when P is reached or the total attempt budget is exhausted, then select candidates if enabled.

The shared solver uses the current random centers c_sample as its reference and minimizes weighted
squared displacement, sum_g w_g * ||c_g - c_sample_g||². This policy retains the same default
weight for every group. The baseline c0 is used for displacement windows and layout reconstruction.
The solver first handles boundaries and fixed positions, then discovers group pairs with insufficient
clearance, tries adding X- or Y-axis separation constraints, and solves again in successive rounds.
One generate call can include multiple SLSQP calls; it returns solved only after the final checks pass.

Diversity Selection

Use incremental max-min selection after generating the pool, comparing only the XY positions
of sampled groups. The selection approach draws on
SceneMosaic §3.4, with a project-specific distance:

D(a,b) = mean_g[((Δx/W)^2 + (Δy/H)^2) / 2]

W/H are the table dimensions, and sampled groups have equal weight. Remove identical layouts first,
retaining the earliest proposal. Initialize d_min using the baseline layout as a reference.
At each step, select the candidate with the largest d_min, break ties by the earliest proposal,
and update the remaining candidates with
d_min = min(d_min, D(candidate, newly_selected_layout)).
The baseline is not output, and selection does not modify layouts.

Stop when M is reached, the pool is exhausted, or the maximum difference is 0.
Record any shortfall accurately. When selection is disabled, output all valid candidates.
Time complexity is O(PMG), where G is the number of sampled groups; distance-cache space is O(P).
No P×P matrix is constructed, and global optimality is not guaranteed.

Layout Reconstruction and Application

When loading a layout, verify source/base/policy digests, complete group coverage, finite centers,
yaw=0, fixed groups, and geometric constraints. For each group, compute
delta_xy = center_new - center_base and reconstruct from the common baseline:

T_root_new = Translate(delta_x, delta_y, 0) × T_root_base
T_member_new = T_root_new × T_member_in_root

For example, moving the tray AABB center from [0.16,0.10] to [0.36,0.10] moves the root origin
from [0.15,0.10,0.77] to [0.35,0.10,0.77]; the cup and block also translate by 0.20 m.
Do not use the AABB center as the root origin or accumulate transforms from the previous candidate.

Currently, procedural.generate updates only entities listed in the layout; it does not automatically
move their on descendants. Its returned fragment/variant is therefore an intermediate result
containing root transforms and cannot be applied directly as a complete group layout.
Using the solved group centers and common baseline, the adapter expands ScenePoseChange entries
and the actual-UID movable allowlist for the root and every member of each sampled group.
Fixed groups are not written. SceneVariant.seed comes from the layout seed, and ordinal is assigned
in consumption order. The original proposal_ordinal, base/policy digests, and algorithm version
are stored as strings in metadata; absolute paths are excluded. parent_scene_id uses source
content identity, and candidate identity is independent of physical slots.

The host owns physical baseline restoration, B=1 / (0,) application, geometry/planning/observation
refresh, and stepping. A native write failure may leave partial changes; the host must restore
or rebuild the state and cannot treat it as a complete candidate.
After application and optional settling, measure every member and combine their actual AABBs
into whole-group AABBs. Pass them to check_layout keyed by root_id, using the baseline recipe
and the same constraints. Additionally check the table, every member of fixed groups, and the
internal group structure that must be preserved. Root displacement alone cannot establish
the actual state of the whole group.

solved means only that the proxy geometry satisfies the layout constraints. Actual support,
stability, reachability, and task success require separate validation. Missing required evidence
is unverified and cannot justify acceptance or a successful execution record.
Record physical outcomes separately, without overwriting the layout proposal or common baseline.
The source USD also does not represent the candidate's current poses.

Future Scope

Retain the earlier requirements: yaw defaults to 12 candidates evenly spaced over 360°, with
a configurable count or explicit angle list; allow up to x% of a whole-group AABB's area outside
the support boundary; and optionally support shortest Euclidean clearance.
The current shared solver does not provide these policies, so the first version exposes no such parameters.
Batch sampling, GPU search, adding distractors, multiple support surfaces, and regions with holes
remain future design work. Rotation requires a geometry proxy for the new orientation;
after physical application with a changed orientation, existing validation requires measured AABBs.

@yuecideng
yuecideng merged commit 0ec0d35 into main Oct 13, 2026
11 checks passed
@yuecideng
yuecideng deleted the codex/scene-foundation branch October 13, 2026 14:20
@yuecideng

Copy link
Copy Markdown
Contributor Author

Scene Engine follow-up: grouped scene randomization

PR #761 is merged as 0ec0d35b6449c72c39b4fa1854bfc568cf102b48. The checklist below turns the persistence and policy proposal into follow-up implementation work for the Scene Engine owner. Track the broader roadmap in #760 and coordinate physical execution with #750.

First-version scope: rigid-body support subtrees on one horizontal, connected table surface without holes; fixed orientation and Z height; XY translation using whole-group AABBs. Place export/graph/coordinate and file-format adaptation under embodichain/gen_sim/scene_engine/, reusing embodichain.lab.sim.scene services.

1. Freeze the data and interface contracts

  • Define strict schemas for scene_randomization_base.yaml, the randomization policy, candidate layout JSON, policy_resolved.yaml, run_result.json, and optional selection_result.json. Reject duplicate keys, unknown fields and unsupported options.
  • Agree on coordinate conventions, tolerances, source/base/policy digests, extraction and sampling algorithm versions, and cache invalidation rules.
  • Separate the saved layout identity and original proposal_ordinal from consumption order and execution-attempt identity. SceneVariant.variant_id includes its ordinal and metadata; preserve original provenance when selection changes consumption order.

Acceptance: one minimal fixture can be serialized, loaded and mapped to the shared contracts, with invalid inputs rejected explicitly.

2. Prepare and cache the common baseline

  • Add a public read-only scene/graph loading path that reuses importer validation without writing scene.json or changing source assets/configuration.
  • Group every object directly on table with all its on descendants. Require unique, complete entity coverage and reject unsupported entities or invalid hierarchies.
  • Store root baseline poses, each member's direct parent and pose relative to the common root, and whole-group AABBs computed from final scaled/oriented geometry. Keep AABB centers distinct from root origins.
  • Validate the safe rectangle against the union of the detector's actual support triangles; reject holes, disconnected surfaces and uncovered rectangles in this version.
  • Reuse valid cached geometry without reparsing meshes. Rebuild when source content, extraction version or relevant parameters change. Use relative references so package relocation preserves identity.

Acceptance: cover nested groups, offset root origins, nonuniform scale, coordinate conversion, cache hits/invalidation, unsupported surfaces and unchanged source files.

3. Generate and persist the candidate pool

  • Map group policy to SceneFragment, SceneRecipe, LayoutProblem and LayoutCfg. Unlisted groups remain fixed. Apply displacement windows, full-AABB bounds and clearance to fixed and sampled groups.
  • Reuse generate(recipe, seed=..., ordinal=...) and the existing solver. Count every complete attempt against the budget; only solved layouts enter the pool. Preserve solver status counts and distinguish search failure from proven infeasibility.
  • Save one lightweight layout file per valid candidate, including every group. Preserve original proposal ordinals, including gaps from failed attempts. Record resolved defaults, actual attempts and all candidate references in the run manifest.
  • Verify digests, complete group coverage, fixed positions, finite values, zero yaw and geometric constraints on load. Protect source/existing run directories and ensure interrupted writes cannot appear as completed runs.

Acceptance: deterministic replay for the same inputs/seed/version; correct fixed-group and displacement behavior; accurate budget exhaustion; portable references and detectable tampering/incomplete output.

4. Select diverse layouts

  • Implement the proposal's incremental max-min selection over sampled-group XY positions, normalized by table dimensions, with baseline initialization and deterministic tie-breaking by original proposal ordinal.
  • Define duplicate equivalence explicitly. Keep the full candidate pool; selection stores ordered references and scores. Report candidate/diversity shortfalls separately from sampling status. Avoid a P-by-P distance matrix.

Acceptance: selection leaves layout files unchanged, is reproducible, and reports actual selected counts. This can follow the generation/application workflow.

5. Adapt layouts for physical consumption

  • Reconstruct each candidate from the common baseline and expand ScenePoseChange entries plus movable permissions for the root and every member of each sampled group. Fixed groups are not written. The root-only result returned by generate() is an intermediate group proposal.
  • Provide the host with baseline/identity information and the same constraints used in generation. After application/settling, aggregate every member's measured AABB under its root UID and supply it to check_layout; separately validate fixed members and internal relative poses using agreed tolerances.
  • Preserve the on hierarchy in candidate editing artifacts and recompute or clear invalidated directional descriptions. Record physical outcomes separately from immutable layout proposals.

Acceptance: a root → child → grandchild fixture moves together; an offset AABB center is handled correctly; switching A → B → A causes no accumulated displacement; a member left behind is rejected by measured validation.

Integration ownership

Owner Deliverable
Scene Engine colleague Steps 1–5: source loading, grouping/cache, policy adaptation, layout I/O, diversity, full-member proposal expansion and group measurement adaptation
Shared scene/workspace integration owner #750 host integration: baseline restoration, B=1 application, settling, geometry/planner/observation refresh and recovery after partial writes
Task Engine colleague Task-specific initial semantic requirements, grounding/planning, measured task acceptance and execution evidence

Geometric solved status is insufficient for expert-data acceptance. Cross-group task relations still need explicit checks; changing a description does not preserve the original task requirement. Missing physical evidence remains unverified. Snapshot restoration now rejects saved joint positions/targets outside current limits before writing, so hosts must handle that preparation failure explicitly.

Suggested delivery order: baseline/contracts → generation + layout I/O → full-member application and measured integration → diversity. Yaw, overhang, distractors, multiple support surfaces and batched physical mutation remain later capabilities.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

assets Related to simulation assets (robot, CAD, material, etc) enhancement New feature or request physics Things related to physics refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants