[jmarrero-bot]

bootc install to-filesystem --defer-var installs without seeding the deployment's state directory for /var (ostree/deploy/STATEROOT/var, or state/os/default/var on composefs). This is the first of the two PRs proposed in bootc-dev#2599. On its own it is for callers that fill /var themselves, before the first boot. Combined with --initialize-var-mounts (bootc-dev#2521, which will build on this PR), each file under /var is stored only once.

Both backends use the same mechanism. Before the first deployment, bootc writes an empty .bootc-defer-var marker into the state directory, and labels the marker and the directory as /var. bootc's composefs deployment skips its copy when the marker is there. libostree's prepare_deployment_var() only seeds an empty stateroot var, so on ostree the marker alone is enough. This doesn't rely on libostree's .ostree-selabeled stamp, which libostree considers slated for removal: libostree still relabels /var and writes its stamp on the first deployment, as it does for any install. Later upgrades, switches and rollbacks leave /var alone, as they do for any populated /var. The marker stays in place as a record.

On composefs, setup-root now leaves /var to systemd on a system installed with --defer-var (its stateroot var has the marker) when the fstab that /etc shows has a mount entry for /var, as ostree's generator does. Without this, the initramfs bind-mounts the state directory over /var, and systemd considers an fstab /var volume already mounted, so a volume the caller filled would be hidden behind the empty state directory. Other systems keep the bind mount. That includes existing composefs systems with a /var fstab entry, and new installs with a /var volume but without --defer-var, whose volume may be empty or unlabeled. A mount entry is what systemd would mount: as in libmount, source, mountpoint and type, and not swap. As in ostree's fstab_has_mount_entry() (through glibc's getmntent, which by itself would also count two-field and swap lines), fields are separated by spaces or tabs, and noauto counts. The fstab is read through the mounted /etc, which also covers an overlay /etc, and symlinks resolve inside the deployment root. It is read as bytes, opening it is retried on EAGAIN, and any error keeps the bind mount, so the fstab can't fail a boot. An explicit [var] setting in setup-root-conf.toml still wins.

bootc --version now prints a Features: list after the version, with defer-var, so image builders can check for it. The first line is still the bare version. bootc-dev#2521 adds its own entry to the list.

With --defer-var alone, /var must be filled before the first boot, and the man page says so. On composefs an empty /var is visible in the initramfs before the SELinux policy is loaded, so directories created there then are unlabeled. On CentOS Stream 10 that's /var/lib and /var/lib/nfs, from dracut's nfs cleanup hook (dracut 107; dracut-ng guards the hook on an NFS root since bf29f4b79a, in 110). A volume at /var/lib/appdb then can't get its mountpoint, and the boot goes to emergency mode. This is a composefs issue rather than this PR's (draft upstream issue to follow).

Tested at ff41dcc on a 16-core RHEL 10 devspace (jmarrero-devspace-40867227224): static checks and unit tests. Since 9506ed3 the setup-root wiring only moved into var_config(), and the fstab is opened with an EAGAIN retry, so the tmt and VM results below are from 9506ed3 (jmarrero-devspace-40852491972, images built without reusing other branches' build caches). Logs: https://gist.github.com/jmarrero-bot/cb0ed128e38cab8cc220d7b21e63de0b (rounds 6 and 7).

• cargo fmt --check, clippy (Makefile flags), cargo doc with -D warnings, cargo check --no-default-features (lib), cargo xtask update-generated direct --check, and update-generated from-code with no diff.
• cargo test -p bootc-lib -p bootc-initramfs-setup: 336 + 16 passed. The tests cover the setup-root decision table and var_config() with and without the marker, fstab records (comments, noauto, swap, two fields, tabs, Unicode whitespace), 0xff and Latin-1 bytes, symlinks resolved in the root, a socket and other non-regular files, an fstab that can't be read (also as root), and the marker reached through the state's var symlink.
• Reverting each fix in turn makes its test fail: the comment filter, the field count, swap, whitespace, slashes, the lossy decode, the read-error fallback, the regular-file check, RESOLVE_IN_ROOT, the marker gate, explicit [var] and transient precedence, the marker path, the 0755 mode, var_config() ignoring the decision, treating ENXIO as an error, and the alongside check. cargo check --tests passes at each commit. Not covered by a test: passing the wrong root to var_config() in setup_root() (the VM cases cover that), and the EAGAIN retry, which can't be induced.
• tmt plan-32-install-to-filesystem-var-mount: a plain install seeds a known file from the test image into the state directory; with --defer-var the state directory holds only the marker, plus libostree's stamp on ostree. PASSED on ostree/grub and composefs grub ext4 BLS (unsealed).
• VM cases (bcvk install onto LVM, qemu boot, SELinux enforcing; composefs on UEFI with OVMF):
  • ostree, --defer-var with nested volumes: the state root and the marker are var_t, and the directories created at boot are labeled as the policy says. Through bootc switch to v2, reboot, bootc rollback and reboot, the data on a volume stays, no image content appears, and the marker stays.
  • composefs with a /var volume and nested volumes, --defer-var, filled by the caller: /var is the volume, and setup-root says so in the journal. The same checks hold through switch and rollback. After a Latin-1 byte is added to fstab and the system reboots, /var is still the volume.
  • The gate: composefs installed from main without --defer-var, with a /var fstab entry, then switched to this build. The state var is bind-mounted before and after the update.
  • composefs, --defer-var with nothing filled and a volume at /var/lib/appdb: emergency mode, as described above.

Open questions:

• Gating the composefs fstab check on the marker answers install: Defer the /var copy and initialize separately mounted /var filesystems from the image bootc-dev/bootc#2599's open question 2: nothing changes for existing systems. PR 2 widens the gate for --initialize-var-mounts, which fills the /var volume itself: it writes a second marker, .bootc-var-mounts, in the stateroot var, and setup-root accepts either. Reusing .bootc-defer-var doesn't work, because composefs seeding would then skip the state directory that --initialize-var-mounts alone still seeds.
• Should --defer-var also apply to install to-disk? It only makes sense when the caller provides /var, so it is limited to to-filesystem for now.
• Should the marker stay after installation? Here it does, as a record that /var was deferred.

Generated-by: https://github.com/jmarrero-bot#llms

---

Review draft in jmarrero-forge, not upstream yet. This section is removed when the PR is opened upstream.

• Upstream: bootc-dev/bootc, base main
• Board item: PVTI_lADOFBIZ7c4BlZPrzhEPhno
• Fork CI: off; the devspace testing described above is this PR's CI, and upstream CI runs once it is opened there

To review:

• Approve, or comment /promote on a line of its own, to open it upstream, ready for review. Either covers only the commits pushed so far.
• If upstream requires DCO, approving also signs off: promote adds Signed-off-by: Joseph Marrero Corchado <jmarrero@redhat.com> to the commits lacking it (the bot's and yours; anyone else's only if you ask), with you as committer.
• Add a /draft line (in the same comment or before) to open it upstream as a draft (/ready undoes that).
• Close to drop it.
• Edit the title and description freely: they become the upstream PR's. Review comments are squashed into the commits they concern, with a reply here.

_{Bot run: session e7bb5c51, agents a819c411 +2 · claude-opus-5-5 102.5M in (97% cached) / 249k out · ~$41.99 inference + ~$4.99 compute (est., list prices) · 2h 5m wall · 31.7 core-h, Actions run 40714907801}

_{Bot run: session e7bb5c51, agents a819c411 +2 · claude-opus-5-5 223.6M in (95% cached) / 378k out · ~$108.13 inference + ~$11.52 compute (est., list prices) · 17h 57m wall · 73.1 core-h, Actions runs 40832841988, 40714907801}

_{Bot run: session e7bb5c51, agents a6341f71 +1 · claude-opus-5-5 30.0M in (97% cached) / 137k out · ~$12.66 inference + ~$2.58 compute (est., list prices) · 15h 54m wall · 16.4 core-h, Actions runs 40848700131, 40739864348}

[jmarrero-bot]

Changes needed at a20a4617.

1. P1 — composefs can fail its first boot. crates/lib/src/install/defer_var.rs:27–38 labels the empty state directory correctly at install, but crates/initramfs/src/lib.rs:657 exposes it as /var before SELinux policy loads. The older dracut 95nfs/nfsroot-cleanup.sh can mkdir -p "$NEWROOT/var/lib/nfs/rpc_pipefs" when its initramfs RPC filesystem is mounted, creating an unlabeled /var/lib. The published layout-D AVC then explains the failure to create /var/lib/appdb. This is consistent with the proposed cause, not a bad label on the stateroot root or a libostree relabel problem. Ostree normally defers the /var bind until the host's var.mount, avoiding this exposure. Fix the boot-time contract: preferably defer composefs's persistent /var mount until policy is loaded, or provide correctly ordered, policy-aware labeling of initramfs-created state before host mounts/services need it. Creating caller mountpoints or a hard-coded /var/lib skeleton only masks this instance. Current dracut-ng also guards NFS cleanup on an NFS netroot; check/backport that fix for the tested distro, but retain an enforcing-SELinux regression test for empty composefs /var.

2. P2 — the committed integration test misses the failing behavior. tmt/tests/booted/test-install-to-filesystem-var-mount.sh:198,240–265 disables SELinux and never boots the installed target. Add an enforcing boot case with the nested /var/lib/appdb volume, assert logical-path labels for the state root/marker and boot-created parents, then verify caller data and absent image sentinels through upgrade, switch and rollback on both backends. Use a known image payload for the plain-install assertion: on ostree .ostree-selabeled alone satisfies the current nonempty check. Published PR1 evidence covers ostree switch/reboot and composefs boot in the layout without appdb; it does not demonstrate PR1 rollback or a successful composefs switch/upgrade. PR2/prototype results cannot fill that gap.

3. P2 — document the composefs separate-/var limitation. docs/src/man/bootc-install-to-filesystem.8.md:19–29 suggests filling a separate filesystem, but this head still unconditionally binds the state /var by default (crates/initramfs/src/lib.rs:652–657); an fstab /var entry alone is ineffective. Document that callers currently need the image's setup-root configuration [var] mount = "none" plus their mount configuration, and must populate/label their filesystem. The fstab-aware behavior proposed for PR2 is absent here.

The core marker design looks correct by inspection: it precedes the first deploy; libostree v2026.1 skips seeding any nonempty stateroot var, independently relabels /var when its own stamp is absent, and writes that stamp itself. Bootc's logical /var override labels the state root and marker correctly. Later ostree deploys remain nonempty; composefs upgrade/switch already pass initialize_var=false; rollback selects existing deployments. No reseeding path found. Keeping the marker is appropriate for ostree. The Features: implementation, option/man-page generation, and two-commit organization look reasonable.

Verification: read the full diff, commit messages, AGENTS.md, REVIEW.md, REVIEW_RUST.md and CONTRIBUTING.md; bot-git check passed for both commits. Independently reran cargo fmt --check, shell syntax checking and git diff --check against the reviewed head, all passed. Inspected the linked devspace logs recording 336 + 10 passing tests and both install-only tmt passes. I did not rerun builds or VMs; the known boot failure remains unresolved.

[jmarrero-bot]

Review guide for head a20a4617f1: where to look closely and what is safe to skim. It is advice from the bot's reviewer and doesn't replace reading the diff; the review app walks it.

Adds install to-filesystem --defer-var: before the first deployment bootc writes an empty .bootc-defer-var into the stateroot var (ostree/deploy/STATEROOT/var or state/os/default/var) and labels it as /var. libostree's prepare_deployment_var() seeds only an empty stateroot var, and bootc's composefs install now checks the marker, so neither backend seeds /var at install or later. Also adds a Features: list to bootc --version. The core change is small and correct on both backends. The risk is in what it promises: on composefs, a separate /var filesystem is still shadowed at boot by setup-root's bind mount of the (now nearly empty) state directory until PR 2's setup-root commit lands, and an unfilled deferred /var on composefs can boot with /var/lib unlabeled when a volume sits below a directory the empty /var lacks. The tmt test runs with --disable-selinux and never boots the result, so labels and boot behavior rest on the VM runs in the gist.

Hotspots

1. look-closely · logic — crates/lib/src/install.rs:1153-1156 (6e150f6): On ostree the marker works only because libostree's prepare_deployment_var() stops at any entry in the stateroot var (ostree-sysroot-deploy.c, e6be8dbe, ~3130). It must be written after init_osname and before the first deploy, which it is. libostree still relabels /var and writes .ostree-selabeled afterwards, so the bootc labeling here is redundant but harmless on ostree.
2. note · logic — crates/lib/src/install.rs:2347-2354 (6e150f6): composefs: the marker goes in before setup_composefs_boot, which is where write_composefs_state() checks it. Upgrades and switches pass initialize_var=false anyway, so on composefs the marker only matters at install.
3. note · logic — crates/lib/src/install/defer_var.rs:27-40 (6e150f6): On composefs this is the only thing that labels the state var root (var_t). The state var is bind-mounted in the initramfs, so directories created under the empty /var before the policy loads (here /var/lib and /var/lib/nfs, for the rpc_pipefs mount carried across switch-root) come out unlabeled until tmpfiles fixes them. That window is what breaks the gist's /var/lib/appdb case.
4. look-closely · api — docs/src/man/bootc-install-to-filesystem.8.md:19-22 (6e150f6): Says /var can be filled by the caller 'for example on a separate filesystem'. On composefs without PR 2's setup-root commit, a /var filesystem from fstab is shadowed at boot by the bind mount of the state directory, which with --defer-var is empty. Either say so (with the [var] mount = "none" workaround) or move that commit here.
5. note · api — crates/lib/src/cli.rs:1081-1095 (6e150f6): New external interface: osbuild and image-builder will parse the Features: list from bootc --version. The first line stays the bare version, and the format is pinned by test_long_version.
6. note · test-gap — tmt/tests/booted/test-install-to-filesystem-var-mount.sh:243-266 (a20a461): Checks the state directory's content only. It runs with --disable-selinux and doesn't boot, so labels, and the absence of reseeding on later deployments, are covered only by the manual VM runs.

Safe to skim

• crates/lib/src/install/baseline.rs: struct field default
• tmt/tests/tests.fmf: duration bump
• tmt/tests/booted/test-install-to-filesystem-var-mount.sh:145-210: mechanical: existing steps moved into mount_target/run_install functions

[jmarrero-bot]

Reviewer verdict at a20a461: fix first (docs only, small). The code is correct on both backends.

• ostree: the marker is written after init_osname and before the first deploy. libostree's prepare_deployment_var() stops at any entry (e6be8dbe, ~3130), so nothing is seeded on install, upgrade, switch or rollback. libostree still relabels /var and writes .ostree-selabeled on the first deploy (~822-866), so bootc's own labeling there is redundant but harmless.
• composefs: the marker is checked only at install. Updates already pass initialize_var=false.

Findings:

1. docs/src/man/bootc-install-to-filesystem.8.md:19-22 (and the body): the man page offers "for example on a separate filesystem", but on composefs a /var fstab filesystem is still shadowed by setup-root's bind mount of the state directory, which --defer-var leaves empty. bootc-dev#2599 put the setup-root change "in PR 2 unless PR 1 already needs it", and for composefs it does. Either move 6ab7f0a here, or document [var] mount = "none" for composefs until it lands.
2. The composefs emergency mode with --defer-var alone, layout D: the cause is right. I booted layout E on a build of the PR 2 head. /var/lib and /var/lib/nfs have birth times about 3.6s after boot, before the policy loaded at 4.2s, which is the rpc_pipefs mount carried across switch-root. composefs bind-mounts /var in the initramfs, while ostree mounts it after the policy loads. tmpfiles later relabels /var/lib to var_lib_t, so layout E boots. var-lib-appdb.mount runs before tmpfiles, so layout D fails. This shouldn't block this PR, because --defer-var alone means "the caller fills /var". Fix: document that the caller must fill /var before the first boot, and file an upstream issue about composefs writes to /var before the policy loads. Don't make mark() create mountpoint parents: that only hides the problem.
3. tmt/.../test-install-to-filesystem-var-mount.sh:243-266 runs with --disable-selinux and doesn't boot, so labels and later deployments rest on the manual VM runs. That's acceptable.

Re-run on a 16-core devspace: fmt, clippy (Makefile flags), doc -D warnings, --no-default-features, update-generated (no diff), and 336 + 10 unit tests all pass. The body's claims match r4-static.txt, the tmt outputs and the od/cd logs. bot-git check: clean.

The SystemNotUEFI failure is a real upstream regression, not test setup. 3b40e99 (bootc-dev#2554) made find_booted_from_esp() fall back only on MissingVar (crates/lib/src/store/mod.rs:514), but classify_bootloader already supports non-EFI composefs (bootc-dev#2375). Treat SystemNotUEFI the same way. composefs switch and rollback do run on UEFI: with OVMF, the PR 2 head passed boot, bootc status, switch to v2 and rollback, with all checks passing under enforcing. One catch: a non-EFI install VM leaves the ESP empty (bootupd installs only the BIOS component), so the harness has to fill the ESP.

Nothing here blocks the e2e run (tracker#55).

Generated-by: https://github.com/jmarrero-bot#llms

[jmarrero-bot]

Review fixes pushed: a20a461 is now 3452994.

• The composefs setup-root fstab commit moved here from install: Initialize separately mounted /var filesystems from the image #2 (it's now the third commit). Without it, a /var volume in fstab is hidden behind the empty state var. setup-root now reads the fstab as bytes, decodes it lossily, and on any read error warns and keeps the bind mount, so the fstab can't fail a boot. Unit tests cover 0xff and Latin-1 bytes and an unreadable fstab. A VM case checks that a Latin-1 byte in fstab doesn't fail the boot or change what it mounts.
• With --defer-var alone, the man page and the body now say the caller must fill /var before the first boot. The early-boot /var/lib labeling is a composefs issue (dracut 107's nfs cleanup hook; dracut-ng fixed it in 110 with bf29f4b79a). An upstream issue is drafted.
• The tmt plain-install check now uses a known payload from the test image.
• New VM cases, enforcing, composefs on UEFI: ostree and composefs (caller-filled /var/lib) check labels of the state root, the marker and the directories created at boot, and data and absent image content through switch and rollback. A composefs case with a /var volume confirms /var is the volume. The unfilled composefs case still goes to emergency mode, as documented.

Results are in the body, logs in the gist (round 5).

[jmarrero-bot]

Re-review fixes pushed: 3452994 is now 9506ed3, squashed into the commits they belong to.

• Gate: setup-root leaves /var to fstab only when state/os/default/var/.bootc-defer-var exists. It checks through the deployment state's relative var symlink, which a unit test covers. Explicit [var] and transient still win. Existing composefs systems, and installs without --defer-var, keep the bind mount. Commit 3 is reworded, and the body says how PR 2 could widen the gate.
• Mount records only: an entry needs source, mountpoint and type, split on spaces and tabs only, and swap doesn't count. noauto counts, as in ostree. The man page says so.
• Symlinks: the fstab is opened with openat2(RESOLVE_IN_ROOT) from visible_root, so a symlinked fstab or /etc resolves within the deployment, then must be a regular file.
• Logging: the decision is printed with eprintln!, and it shows in the boot journal.
• Tests: the decision is a pure function with a table test. The fstab, comment, symlink, root-proof read-error and marker-path tests are new or fixed. Reverting each fix makes its test fail (ds-mut.py in the gist).
• Docs: the man pages cover switching or rolling back to an older initramfs, that the marker matters on composefs only at install, and that mount = "bind" keeps the old behavior. The deferred var root is now set to 0755 explicitly.

Tests on a fresh devspace, with images checked to be this build: static checks, 336 + 15 unit tests, tmt plan-32 on ostree and composefs BLS, and the od, cv and cd VM cases. A new gate case installs composefs from main with a /var fstab entry, switches to this build, and keeps the bind mount. Results are in the body, logs in the gist (round 6).

[jmarrero-bot]

Second-review low items pushed: 9506ed3 is now ff41dcc. Only commit 3 changed (git range-diff shows commits 1 and 2 identical).

• read_in_root() retries openat2 on EAGAIN, up to 5 attempts, as openat2(2) suggests for RESOLVE_IN_ROOT.
• Credit wording, in the commit message and the body: the three-field and no-swap rules follow systemd/libmount; the whitespace and noauto rules match ostree's fstab_has_mount_entry() (glibc getmntent).
• test_read_in_root has a socket (ENXIO) row, and a new test_var_config covers the setup-root wiring with and without the marker.

Static checks and unit tests (336 + 16) pass, and all 17 mutations fail a test, including the new var_config and ENXIO ones. The retry itself can't be induced in a test. Logs: https://gist.github.com/jmarrero-bot/cb0ed128e38cab8cc220d7b21e63de0b (round 7). The PR 2 VM cases ran on this commit's setup-root too.