One machine, thousands of sandboxes. Sandstorm pauses idle sandboxes and snapshots them to disk, then restores them quickly when a request arrives, so one machine can hold far more sandboxes than it could run at once. It supports two sandbox backends, gVisor containers and Firecracker microVMs. Every sandbox has its own network namespace, and both backends can run in one fleet, each forming a resource pool of its own. The master serves a web console for viewing and managing sandboxes and scheduling across the fleet.
The 64 MiB app, p50 of 20 started one at a time, log scales. Each panel is ranked on its own, best first, in gold; Sandstorm's other rows are in blue. FC: Firecracker. Cube: CubeSandbox. Also shown: microsandbox, Kata, Docker, Podman, bubblewrap. Warm: restored from an image of the app already started. Cloud VM: six 2-core, 2 GB nodes for gVisor and containers; one 2-core, 3.6 GB VM under PVM for Firecracker and CubeSandbox. KVM machine: 2 × Xeon E5-2637 v2, 31 GB. Clones are of a running app, until all 20 answer (microsandbox: until running).
Agents spend much of their time waiting. Sandstorm is designed around this behavior, with four sandbox tiers that progressively suspend idle sandboxes and quickly wake them when requests arrive.
| Tier | State | Holds Memory | Back to hot |
|---|---|---|---|
| hot | Running | Yes | — |
| frozen | Paused in place (cgroup.freeze); cold pages compressed into zswap |
Some | ~10 ms |
| snapshot | Checkpointed to node disk; network namespace released | No | ~0.2 s |
| detached | Only the volume remains | No | Cold start |
-
One master, many nodes. The master keeps desired state in SQLite and places sandboxes. Nodes join by address and pairing code, or dial out from behind NAT.
-
Wake on traffic. A gateway on the sandbox's node wakes it and forwards the request.
-
Node self-protection. Nodes refuse new work when memory runs low, so SSH stays reachable.
-
One binary.
sandstormdserves as the master, node, and web console.
The master's web console shows sandbox states, tier transitions, and node headroom.
Six 2-core, 2 GB cloud nodes on gVisor, no KVM, 2026-10-30. Requests from each node to its own gateway, so no network is in the path.
| Wake from frozen, 64 MiB app | 100-170 ms |
| Wake from disk, 16 / 64 / 256 MiB app | 130 / 150 / 750 ms |
| Wake from disk, 512 MiB / 1 GiB app | 1.1 / 3.3 s |
| Kept in memory per node: web / 64 MiB / 256 MiB app | 50 / 13 / 6 |
Under traffic, with 40 sandboxes per node where about 12 fit awake and a few getting most requests: at 1, 2, 4 and 8 requests a second per node, 1.3%, 0.2%, 0.1% and 1.3% were refused, of 40,523. p50 is 6-8 ms; p99 is about 2 s, the requests that had to wait for a restore. A wake that cannot start within 2 s is refused rather than queued: a node restores two at a time and queues four, so of 10 or more woken at the same instant, six are answered and the rest get a 503 at once.
The figure at the top: every runtime started one at a time, polled every 5 ms, p50 of 20 (compare.py). Sandstorm's time includes the master placing the sandbox and the node setting up its network. The cloud column is the median of six 2-core, 2 GB nodes, and for Firecracker and CubeSandbox one 2-core, 3.6 GB VM with CubeSandbox's PVM host kernel, which gives a VM without KVM a /dev/kvm.
A warm template restores an image of the app already started, as CubeSandbox always does; it is opt-in here, because every sandbox starts from the same memory and the template has to say how to reseed. CubeSandbox's API always restores a snapshot, so its cold figure comes from its own cubemastercli multirun, with a template's create request minus the snapshot annotations and a 5 ms probe: from the client's call until the program answers, as for the rest.
Under PVM the two warm starts are level: Sandstorm 52 / 83 ms, CubeSandbox 53 / 87 ms for web and the 64 MiB app. On the KVM machine Sandstorm's is ahead: 67 / 100 ms against 78 / 129 ms. Cold, Sandstorm leads on both: 545 / 921 ms against 828 / 999 ms on the KVM machine, 667 / 869 ms against 845 / 1,094 ms under PVM, where a guest boots without a serial console unless --fc-serial-console says otherwise. The 64 MiB app was 74 ms and 78 ms on the same machine a day earlier, with the same binary, so take that column as relative. A snapshot of a running sandbox took Sandstorm 30 ms and CubeSandbox 304 ms. Clones into 1, 10 and 20 took Sandstorm 138, 507 and 938 ms, and CubeSandbox 494, 1,046 and 1,064 ms; Sandstorm creates them one after another on the node, about 45 ms each, where CubeSandbox creates them at once, so a larger n would close that gap. E2B self-hosted was not set up, and memory under PVM was not measured. Raw rows: bench/baremetal/results/2026-10-30, bench/fleet/results/2026-10-30.
You need Linux with cgroup v2, runsc on the PATH, and Rust to build.
git clone https://github.com/deeplethe/sandstorm.git && cd sandstorm
cargo build --release -p sandstorm-core
sudo ./target/release/sandstormd serve --master --nodeThat runs a master and a node in one process, with the console at https://127.0.0.1:8080. The API token is in /var/lib/sandstorm/pki/api-token. A sandbox starts from a template, a rootfs plus an OCI spec registered with POST /v1/templates; scripts/fleet.sh seed shows the smallest one.
A fleet of machines you can ssh to:
scripts/fleet.sh up --master m1 --nodes n1,n2,n3 --listen 80
scripts/fleet.sh seed # put a template on each node
scripts/fleet.sh verify # start one sandbox and check it runsA node that is already installed listens with SANDSTORM_LISTEN=0.0.0.0:80. On first start it writes a pairing code to /var/lib/sandstorm/pki/node/pairing-code. Give the master its address and that code, with the master's CA and API token from its /var/lib/sandstorm/pki:
curl --cacert ca.pem -H "Authorization: Bearer $(cat api-token)" -H 'Content-Type: application/json' \
https://master:8080/v1/nodes -d '{"address": "10.0.0.5", "pairing_code": "<code>"}'The code stays the same across restarts, and whoever holds it can run anything on that node. Keep it like a root password.
The HTTP API is documented in docs/API.md. To load-test a fleet, use bench/fleet.
v0.6 is not released yet, and it changes a lot: the backend moves from Firecracker microVMs to gVisor, and one host becomes a fleet. The last release, v0.5.3, is the Firecracker fork runtime; its README is at that tag. The design decisions are recorded in docs/adr.
Next, from what the fleet runs found:
- Make room on wake: put the sandbox least in demand to disk instead of refusing the request.
- Retry a create the node held back once memory frees, instead of parking it; and clear its error once it runs.
- Clear away a sandbox that dies, network included, and restart it if it is still wanted.
- Reserve capacity on the master as it places, so a burst cannot overshoot a node's last report.
- Restore several snapshots in parallel on one node.