install: Defer the /var copy and initialize separately mounted /var filesystems from the image #2599

Problem

bootc install to-filesystem seeds the image's /var into a state directory. On ostree, libostree's prepare_deployment_var() fills an empty ostree/deploy/<stateroot>/var. On composefs, bootc copies into state/os/default/var. Since #1727, filesystems the caller mounts at the target's /var or below are accepted, but they stay empty, and at boot they hide the seeded content. On composefs a separate /var filesystem isn't even used: the initramfs bind-mounts the state directory's var over /var, so systemd treats the fstab /var entry as already mounted.

This mostly comes up with hardening profiles such as DISA STIG and CIS, which ask for separate filesystems for /var, /var/log, /var/log/audit, /var/tmp and /home (/var/home on bootc systems). These are usually LVM volumes, often nested, sometimes with no /var volume at all (#1728), on one or more VGs. A single --var mount spec can't describe this layout, and neither can a DPS Variable Data Partition: DPS needs GPT partitions and has no partition types for /var/log or /var/log/audit. Installers already create this layout (Anaconda, and image-builder through customizations.disk), so the caller mounts it and bootc fills it: bootc knows the image's /var content and how to label it.

#2521 adds an opt-in --initialize-var-mounts that copies the seeded /var into the mounted filesystems after install. It works, but every file under a mount is then stored twice: once on the volume, and once on the root filesystem in the state directory, hidden under the mount.

Proposal

Two PRs.

1. install to-filesystem --defer-var

Install without seeding /var. bootc writes a bootc-owned marker that records the deferred /var, the same on the ostree and composefs backends. On composefs, bootc skips its copy. On ostree, libostree seeds any empty stateroot var when it deploys, so bootc also has to keep it from doing that on this and later deploys. We wouldn't rely on libostree's .ostree-selabeled stamp for that, since libostree considers it slated for removal (ostreedev/ostree#872). The stateroot var root is still labeled as /var.

On its own, this is for callers that fill /var themselves, before the first boot. On composefs an empty /var is already visible in the initramfs, before the SELinux policy is loaded, so directories created there then come up unlabeled (on CentOS Stream 10, /var/lib for /var/lib/nfs/rpc_pipefs), and a volume mounted below one, such as /var/lib/appdb, can fail to mount. With PR 2, it gives a single-copy install.

2. --initialize-var-mounts, from the image, and a day-two command

This is the next revision of #2521, reworked, and would be force-pushed there. It copies from the image's /var (the deployment checkout on ostree, the mounted image on composefs) instead of from the seeded copy. It makes one copy per empty filesystem mounted at or below the target's /var, each in a private mount namespace where the mountpoints of the filesystems nested in it are hidden, so each file lands on the filesystem that holds it at boot. With --defer-var, the stateroot var gets only what isn't on a mounted filesystem. Without it, the stateroot var is still seeded in full, as with #2521 today.

From #2521: symlinked mountpoints are refused, and everything is SELinux-labeled by its logical path and synced. Two things change from #2521, both found while testing it:

• An empty parent of a non-empty volume is now filled. In install: Initialize separately mounted /var filesystems from the image #2521, data in one nested volume skipped the whole tree, including an empty /var volume above it.
• Hardlinks are kept within a filesystem on composefs. On ostree they become separate files, as with libostree's seeding, because a commit doesn't record them.

The two flags can be combined: bootc install to-filesystem --defer-var --initialize-var-mounts ....

bootc state initialize-var [PATH] runs the same code on a booted system, for anyone who needs it outside install, for example after adding a volume for /var/log/app. The source is the booted deployment's image. It only considers block-device filesystems mounted from their root, so tmpfs, NFS, bind mounts (such as kubelet's), btrfs subvolumes and ZFS datasets are left alone.

Each PR would list what it adds as a feature in bootc --version, as suggested in the #2521 review (main has no such list yet), so osbuild and image-builder can check for them (osbuild/image-builder#2733).

composefs: leave a provided /var to fstab

setup-root would skip its /var bind mount when the fstab that /etc will show has a /var entry. ostree's generator already does this for var.mount, and setup-root already skips the bind mount for systemd.volatile=state. This also matches bootc-filesystem(7): "If it is not a distinct partition, then it is automatically made a bind". The fstab has to be read through the effective /etc, including an overlay /etc. An entry counts when it has a source, a mountpoint and a type and isn't swap; noauto entries count.

PR 1 needs this, so it's in PR 1, as its own commit. It's gated: setup-root only skips the bind mount when the stateroot var has a bootc marker. --defer-var writes one, and PR 2's --initialize-var-mounts writes another when a filesystem is mounted on /var itself. Ungated, it would change boot for existing composefs systems with a /var fstab entry. They have been running on the state directory's var, and after an update they would boot on the fstab filesystem, where the /var data they wrote since install appears to vanish (it is still in state/os/default/var), and if that filesystem is still empty and unlabeled, the boot may fail. With the gate, they keep the bind mount. An initramfs that predates the change ignores the markers, so switching or rolling back to an older image bind-mounts the state directory's var over the volume again.

Behavior

• The installer runs the initialize step at install time, and there's no first-boot unit for now. By the time a deferred install has booted, systemd-tmpfiles has written to /var, so it isn't empty and bootc state initialize-var can't fill it. A first-boot unit could come later (see below).
• Only empty filesystems are filled. Nothing is merged. A re-run doesn't re-add files an admin deleted, and it doesn't apply /var content from newer images. That matches the VOLUME /var semantics in bootc-filesystem(7). A volume that is still empty, for example because the image has nothing for it, is initialized again on a re-run, which resets the owner, mode and SELinux label of its root.
• A failed or interrupted copy isn't repaired automatically, but it is reported. A failed copy fails the command with an error naming the filesystem, after the filesystems already filled are labeled and synced. After a failure or an interruption, later runs say they skip that filesystem because it isn't empty. Both are printed, and also logged to the journal when bootc runs on a system booted with systemd, as bootc state initialize-var does; an install run in a container only has its own output. To retry, empty the filesystem and run the command again.

Open questions

1. Should bootc state stay hidden? On main, the group is hidden and only has wipe-ostree. bootc state initialize-var is meant for admins, for example after adding a /var volume, so it would be easier to find if bootc state were visible.
2. Gating the composefs fstab change. The proposal gates it on bootc's markers, so existing systems are unchanged. Or should composefs follow fstab for every system, as ostree does, with a release note?

Future work

• --var and DPS. A --var mount option, and using a DPS Variable Data Partition by default (install to-filesystem: support separate /var for install #336, and the install: Initialize separately mounted /var filesystems from the image #2521 review), are wanted but don't block this. A DPS /var would be filled by the same routine.
• First boot. A unit that runs the routine before systemd-tmpfiles, when /var is empty (Seed /var behaviour and separate /var filesystems ostreedev/ostree#3222), for setups that create filesystems at first boot rather than at build time.
• Factory reset (install: Support reinstallation (factory reset) #404, the experimental bootc install reset, ostree only today). A reset creates a new stateroot, and libostree seeds its var on the root filesystem. The new deployment's fstab carries over only the /boot entry, so separately mounted /var filesystems aren't mounted after the reset, and they keep the old content. A reset that wants a fresh /var on them would carry their mounts over, wipe them and run the same routine, which the "only empty filesystems" rule already allows. That needs the routine to target a deployment other than the booted one, for example one mounted with bootc install mount (install: Add explicit offline deployment mount API #2483).

Prototype

A prototype of both parts and the composefs change was tested in VMs on ostree and composefs with synthetic LVM layouts, covering install, boot, the day-two command, switch and rollback, with the same flag and command names. It has since been reworked into the two PRs described above, which were also tested on composefs booted with UEFI, with SELinux enforcing, and with both flags on a /var volume with volumes nested in it. They'll be linked here once opened.

Related

• PR 2 would close install to-filesystem doesn't work when target /var is a separate mount #997 and to-filesystem: support pre-created /var and other mount points #1615. It is the next revision of install: Initialize separately mounted /var filesystems from the image #2521.
• install to-filesystem: support separate /var for install #336 stays open for --var.
• install: Handle mount points in rootfs operations #1727 (for Separate /var part required for ISO based installation but forbidden for 'bootc install to-filesystem' #1728) made mountpoints under /var accepted at install.
• bootc: populate separate /var filesystems in bootc disk images osbuild/image-builder#2733 is the image-builder side.
• sysroot: Rework /var handling to act like Docker VOLUME /var ostreedev/ostree#3166 is where the /var seeding was introduced. Its discussion covers separate /var mounts and splitting stateroot init from deploy.
• Seed /var behaviour and separate /var filesystems ostreedev/ostree#3222 covers seeding at first boot.

Generated-by: https://github.com/jmarrero-bot#llms