SpaceCode-Bench

Construct, reconstruct and edit 3D environments with executable Blender code.

Project website · Dataset · Native protocol · Worked example

SpaceCode-Bench evaluates the native scene produced by a coding model. Each task publishes its geometric requirements; the evaluator reopens the submitted .blend file, extracts geometry and controlled views, and checks those requirements. Construction starts from text, reconstruction from one or four calibrated images, and editing from an existing scene.

The procedural development collection contains 1,200 materialized scenes, 3,600 task bundles and 4,800 execution conditions across 40 space profiles and 240 profile/layout lineages. All five variants and all conditions of a lineage share a split.

This release measures native geometry, observed-view agreement and geometric edit preservation. Semantic identity (C), visual finish (V) and full_score are unmeasured. The website contains author controls and no formal model rankings. The shared procedural grammar and public reference assets make this a development benchmark; the splits do not establish contamination resistance or real-world generalization.

Dataset

The Hugging Face repository contains a browsable task catalogue, complete native data archives, integrity manifests and a worked example. The catalogue has one row per execution condition, including the full prompt and contract. Its reference previews are for inspection; solvers receive only their task's specified inputs.

git clone https://github.com/KurtDu/SpaceCode-Bench.git
cd SpaceCode-Bench
python -m venv .venv
source .venv/bin/activate
python -m pip install -c requirements-tested.txt .
python scripts/download_dataset.py
export DATASET="$PWD/data/SpaceCode-Bench-1.0.0-dev-data"
python scripts/verify_release.py --root "$DATASET"

The downloader selects an immutable Hub revision, checks all 12 archives, and verifies the assembled data before exposing the destination. Allow about 9 GB for the compressed cache and extracted data. See the data card for splits, units, provenance and limitations.

Install and evaluate

Requires Python 3.10 or newer. Candidate execution requires Linux amd64 and Docker.

python -m venv .venv
source .venv/bin/activate
python -m pip install -c requirements-tested.txt '.[test]'
python scripts/hydrate_web_assets.py
python -m pytest -q
python scripts/build_runtime.py

The runtime builder verifies Blender 4.2.23 LTS before building the image. A prepared runtime archive is also available through GitHub Releases; see deployment for loading and verifying it.

A provider adapter receives one JSON request and returns Blender Python plus model and usage metadata. Credentials remain in the controller environment. Use an exact model snapshot and explicit public sampling settings.

python -m spacecode batch --blender image-runtime \
  --index "$DATASET/index.json" --out runs/my-model \
  --model-id provider/exact-model-snapshot \
  --adapter-config sampling.json \
  --adapter python /path/to/provider_adapter.py

python -m spacecode export-run --run-dir runs/my-model --out runs/my-model.web.json

Import the exported JSON in the website's Compare view. Results retain the planned task set, lineage, coverage, failures, dimensions, runtime, usage and recorded cost. Comparisons require compatible dataset, evaluator, adapter, budget and sampling identities. Infrastructure failures and missing work remain visible; they do not become model scores. Resume instructions and the adapter contract are in deployment and run records.

Scoring

Dimension	Measurement
A	Published architectural envelope, regions and floor requirements
G	Geometric extents, component regions and negative space
F	Prescribed route clearance and floor support
R	Controlled image agreement at supplied reconstruction cameras
P	Geometry preservation outside the allowed edit region
C, V	Semantic identity and visual finish: unmeasured

The native score is 100 times the geometric mean of the applicable native dimensions. A measured critical failure caps it at 49. Required reconstruction or preservation evidence must be complete: unresolved R/P leaves the score and overall judgment null, including when another requirement has already failed. Partial diagnostics never enter the aggregate. The scorer uses a separate process with fixed CPU, wall-time, memory and output limits and verifies its actual numerical dependency identity.

The native protocol specifies thresholds, numerical limits, uncertainty handling and aggregation. Geometric measurements do not establish semantic correctness, arbitrary navigation, physical stability or material quality.

Website

Requires Python 3 and Node.js 22.12 or newer.

cd web
npm ci
npm run dev

Public assets are fetched and SHA256-verified on first use. npm run build produces web/dist/ for static hosting. The site supports full-prompt search, native 3D previews, edit comparisons, a complete author-control walkthrough, and local result imports. GitHub Pages uses the same build.

Development and citation

See CONTRIBUTING.md, release validation, SECURITY.md and CITATION.cff. Historical review reports retain their original version and evidence scope.

Code: MIT. Authored procedural data: CC BY 4.0. Cite Yuhao Du, SpaceCode-Bench, version 1.0-procedural (2026), and record the dataset and evaluator fingerprints used in an experiment.