On self-hosted Embed, a failed pause or checkpoint can stop the original sandbox and leave no snapshot the SDK can restore. Its files and process state become inaccessible.
Reproduce
Pause: disk sync failure
- Create a sandbox and write a file inside it.
- Call
sandbox.pause(keep_memory=True).
- Use
strace to make fsync return EIO for the rootfs diff file during the pause.
Pause returns 500: Error pausing sandbox; the orchestrator reports failed to sync file: input/output error. Reproduced 2/2 times. Without the injected error, pause/resume preserves files and process memory.
Fault injection command
Attach after the diff file appears and before it is synced (about 1–2 seconds). Replace the path with the actual diff file path; a watcher was used to catch this window.
strace -f -p "$(pgrep -x orchestrator)" -qq -yy \
-P "/orchestrator/build/<build-id>-rootfs.ext4-<random-suffix>" \
-e trace=fsync,fdatasync \
-e inject=fsync:error=EIO:when=1 \
-o /tmp/pause-fsync.strace
The original incident followed repeated host No space left on device errors. Those logs do not establish the cause of the later fsync error.
Checkpoint: memory allocation failure
Exhaust the host's reserved hugepage memory pool, then call create_snapshot(). Starting the replacement sandbox fails with mmap memfd: cannot allocate memory. Reproduced 1/1 time with real memory exhaustion on the same release binary. Normal checkpoint/restore works with memory available.
Actual result
In both cases:
- The original sandbox's Firecracker process stops;
get_info returns “not found.”
- The snapshot build is marked
failed; restoring it returns 404: template not found.
The pause failure deletes the local snapshot artifacts. The checkpoint failure leaves local files, but they cannot be restored through the SDK.
Expected result
A failed pause or checkpoint should leave the sandbox recoverable: either the original sandbox remains available or a usable snapshot can be restored.
Environment
- Single-node Embed, local storage (
file:///var/lib/e2b/storage/templates)
- Ubuntu 26.04 ARM64 VM with nested virtualization
- Orchestrator
v0.16.202609130627-59497eb9134 (unmodified release binary)
- API
v0.14.202609170000-908833e4c12; Python SDK 2.51.0
- Compose revision:
a065a4ddb3f2c6a4149634d9acb14b62f65839ac
Related: #3659 covers a snapshot marked successful that crashes on restore.
On self-hosted Embed, a failed pause or checkpoint can stop the original sandbox and leave no snapshot the SDK can restore. Its files and process state become inaccessible.
Reproduce
Pause: disk sync failure
sandbox.pause(keep_memory=True).straceto makefsyncreturnEIOfor the rootfs diff file during the pause.Pause returns
500: Error pausing sandbox; the orchestrator reportsfailed to sync file: input/output error. Reproduced 2/2 times. Without the injected error, pause/resume preserves files and process memory.Fault injection command
Attach after the diff file appears and before it is synced (about 1–2 seconds). Replace the path with the actual diff file path; a watcher was used to catch this window.
The original incident followed repeated host
No space left on deviceerrors. Those logs do not establish the cause of the laterfsyncerror.Checkpoint: memory allocation failure
Exhaust the host's reserved hugepage memory pool, then call
create_snapshot(). Starting the replacement sandbox fails withmmap memfd: cannot allocate memory. Reproduced 1/1 time with real memory exhaustion on the same release binary. Normal checkpoint/restore works with memory available.Actual result
In both cases:
get_inforeturns “not found.”failed; restoring it returns404: template not found.The pause failure deletes the local snapshot artifacts. The checkpoint failure leaves local files, but they cannot be restored through the SDK.
Expected result
A failed pause or checkpoint should leave the sandbox recoverable: either the original sandbox remains available or a usable snapshot can be restored.
Environment
file:///var/lib/e2b/storage/templates)v0.16.202609130627-59497eb9134(unmodified release binary)v0.14.202609170000-908833e4c12; Python SDK2.51.0a065a4ddb3f2c6a4149634d9acb14b62f65839acRelated: #3659 covers a snapshot marked successful that crashes on restore.