Skip to content

ephemeral: Drop bwrap, just mount and chroot into the hybrid root #395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cgwalters merged 1 commit into from
Oct 2, 2026
Merged

ephemeral: Drop bwrap, just mount and chroot into the hybrid root #395

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 25 additions & 37 deletions crates/kit/scripts/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,49 +2,52 @@
set -euo pipefail

SELFEXE=/run/selfexe
TMPROOT=/run/tmproot

# Check for required binaries early
if ! command -v bwrap &>/dev/null; then
echo "Error: bwrap (bubblewrap) is currently required in the target container image" >&2
exit 1
fi
for bin in mount chroot; do
if ! command -v "$bin" &>/dev/null; then
echo "Error: $bin (util-linux or coreutils) is required in the target container image" >&2
exit 1
fi
done

# Shell script library
init_tmproot() {
if test -d /run/inner-shared; then return 0; fi
# Should have been created by podman when initializing
# the bind mount
cd /run/tmproot
cd "$TMPROOT"

# Create essential symlinks
ln -sf usr/bin bin
ln -sf usr/lib lib
ln -sf usr/lib64 lib64
ln -sf usr/sbin sbin
mkdir -p {etc,var,dev,proc,run,sys,tmp}
mkdir -p {etc,var/tmp,dev,proc,run,sys,tmp}
# Ensure we have /etc/passwd as ssh-keygen wants it for bad reasons
systemd-sysusers --root $(pwd) &>/dev/null

# Copy DNS configuration from container's /etc/resolv.conf (configured by podman --dns)
# into the bwrap namespace so QEMU's slirp can use it for DNS resolution
# into the new root so QEMU's slirp can use it for DNS resolution
if [ -f /etc/resolv.conf ]; then
cp /etc/resolv.conf /run/tmproot/etc/resolv.conf
cp /etc/resolv.conf "$TMPROOT/etc/resolv.conf"
fi

# Shared directory between containers
# We're already privileged in the container's own mount namespace,
# which is torn down with it, so set up the hybrid root's mounts
# here; later `podman exec` invocations share them. Bind /run first,
# so its copy under the new root doesn't also pick up the mounts
# made below. /tmp needs nothing: /run is already a tmpfs.
mount --rbind /run "$TMPROOT/run"
mount -t proc proc "$TMPROOT/proc"
mount --rbind /dev "$TMPROOT/dev"
mount --rbind /var/tmp "$TMPROOT/var/tmp"

# Shared directory between containers; also signals that the root is ready
mkdir /run/inner-shared
}

BWRAP_ARGS=(
--bind /run/tmproot /
--proc /proc
--dev-bind /dev /dev
--bind /var/tmp /var/tmp
--tmpfs /run
--tmpfs /tmp
--bind /run/inner-shared /run/inner-shared
)

# Pass ALL arguments to container-entrypoint
# Default to "run-ephemeral" if no args
if [[ $# -eq 0 ]]; then
Expand All @@ -60,21 +63,6 @@ fi
# Check systemd version from the container image (not host)
export SYSTEMD_VERSION=$(systemctl --version 2>/dev/null)

# Execute with proper environment passing
# Set up signal handlers that will cleanly exit on INT or TERM
trap 'kill -TERM $BWRAP_PID 2>/dev/null; exit 0' INT TERM

# Run bwrap in background so we can handle signals; xref
# https://github.com/containers/bubblewrap/pull/586

@cgwalters cgwalters Oct 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now the questions is do we still need this? I bet we don't and we could just exec?

@cgwalters-bot cgwalters-bot Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, we do not need it: the container already has its own mount and PID namespaces, so the entrypoint now does the mounts into /run/tmproot once in the container namespace and just exec chroots bcvk (the chroot is still needed for the host /usr). Squashed into 0b07a7e (retitled, your sign-off kept); run-ssh on c9s/c10s/f43 and the ephemeral+mount integration tests pass.

# But probably really we should switch to systemd
# Bash normally gives a background command /dev/null on stdin, so
# keep stdin attached so QEMU can receive input.
bwrap --as-pid-1 --unshare-pid "${BWRAP_ARGS[@]}" --bind /run /run -- ${SELFEXE} container-entrypoint "$@" <&0 &
BWRAP_PID=$!

# Wait for bwrap to complete
wait $BWRAP_PID
EXIT_CODE=$?

# Exit with the same code as bwrap
exit $EXIT_CODE
# container-entrypoint handles SIGTERM/SIGINT itself, so it can just
# replace us (as PID 1 for the main invocation).
exec chroot "$TMPROOT" "$SELFEXE" container-entrypoint "$@"
18 changes: 9 additions & 9 deletions crates/kit/src/run_ephemeral.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
//! The execution follows this chain:
//! 1. **Host Process**: `bcvk run-ephemeral` invoked on host
//! 2. **Container Launch**: Podman privileged container with KVM and host mounts
//! 3. **Namespace Setup**: bwrap creates isolated namespace with hybrid rootfs
//! 3. **Root Setup**: `mount` and `chroot` into the hybrid rootfs
//! 4. **Binary Re-execution**: Same binary re-executes with `container-entrypoint`
//! 5. **VM Launch**: QEMU starts with VirtioFS root and additional mounts
//!
Expand All @@ -41,12 +41,12 @@
//! └── [other dirs created empty for container compatibility]
//! ```
//!
//! ### Phase 3: Namespace Isolation (bwrap)
//! Uses bubblewrap to create isolated namespace:
//! - New mount namespace with `/run/tmproot` as root
//! ### Phase 3: Root Setup (mount + chroot)
//! The container is already privileged with its own mount namespace, so no
//! further namespaces are needed:
//! - `/run`, `/proc`, `/dev` and `/var/tmp` mounted into `/run/tmproot`
//! - Shared `/run/inner-shared` for virtiofsd socket communication
//! - Proper `/proc`, `/dev`, `/tmp` mounts
//! - Re-executes binary: `bwrap ... -- /run/selfexe container-entrypoint`
//! - Re-executes binary: `exec chroot /run/tmproot /run/selfexe container-entrypoint`
//!
//! ### Phase 4: VM Execution (`run_impl`)
//! - Runs inside the container after namespace setup
Expand Down Expand Up @@ -1219,8 +1219,8 @@ fn parse_service_exit_code(status_content: &str) -> Result<i32> {
fn check_required_container_binaries() -> Result<()> {
// systemctl: used for checking cloud-init and other systemd operations
// objcopy: for UKI kernel extraction (when using UKI images)
// NOTE: bwrap is checked earlier in entrypoint.sh, not here, because by the
// time run_impl() executes we're already inside the bwrap namespace
// NOTE: mount and chroot are checked earlier in entrypoint.sh, not here, because by the
// time run_impl() executes we're already inside the hybrid root
let required_binaries = ["systemctl", "objcopy"];

let mut missing = Vec::new();
Expand Down Expand Up @@ -1934,7 +1934,7 @@ Options=
// Check if disk file exists and is accessible
if !Utf8Path::new(&disk_file).exists() {
return Err(eyre!(
"Disk file does not exist in bwrap namespace: {} (serial: {})",
"Disk file does not exist in the hybrid root: {} (serial: {})",
disk_file,
serial
));
Expand Down
2 changes: 1 addition & 1 deletion docs/src/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Optional:
For `bcvk ephemeral` operations, the bootc container images you run must contain:
- systemctl (systemd)
- objcopy (binutils)
- bwrap (bubblewrap)
- mount (util-linux), chroot (coreutils)
- ssh, ssh-keygen (openssh-clients)

## Development Binaries
Expand Down
Loading