TODO — diskless fleet boot system

Persistent backlog, kept in sync with the working session task tracker. Status: [ ] pending · [~] in progress · [x] done.

[ ] 1. Port the dracut module off `network-legacy` → `network-manager`

Durable replacement for the current dhcp-client stopgap. dhclient is deprecated in RHEL 9 and will eventually leave the repos; the build only works today because the provisioner installs dhcp-client so network-legacy passes dracut's check().

Rewrite both network bring-up paths in ../files/dracut/95dot1x-macsec/dot1x-macsec-up.sh: the secure path's ifup macsec0 and the rd.dot1x=0 bypass path's ifup $NIC — use nmcli/NetworkManager-in-initramfs (or ip= cmdline + the nm module) instead.
Change depends() network-legacy → network-manager in ../files/dracut/95dot1x-macsec/module-setup.sh.
Update add_dracutmodules in ../files/etc/dracut.conf.d/90-diskless.conf.
Drop dhcp-client from the provisioner dnf install in ../diskless.pkr.hcl.
Re-verify the lsinitrd assertion.
Gate: full boot-chain smoke test (swtpm + hostapd + FreeRADIUS) before merge, per CLAUDE.md.

[ ] 2. Apply RHEL9 STIG hardening to the built image

OpenSCAP + SCAP Security Guide (xccdf_org.ssgproject.content_profile_stig), run as a Packer provisioner after dnf install and before the dracut/initramfs build and cleanup.

Tailoring (autotailor): unselect rules this architecture can't satisfy — separate-partition rules (single erofs root + tmpfs overlay; satisfy intent with nodev,nosuid,noexec tmpfs mounts), GRUB/kernel-cmdline rules (no GRUB), and AIDE/file-integrity rules (dm-verity is the documented compensating control).
Kernel params → UKI cmdline in ../scripts/extract-artifacts.sh before signing: audit=1 audit_backlog_limit=8192 slub_debug=P page_poison=1 vsyscall=none (+ fips=1 if FIPS).
Reproducibility guard: strip non-deterministic remediation output (SSH host keys → regen at boot into overlay, AIDE db, timestamps, fresh UUIDs) so the verity hash stays stable.
Audit logging: audisp-remote/rsyslog to a central collector addressed by DNS name (host-generic).
Optional FIPS: fips=1 cmdline + 01fips dracut module + FIPS crypto policy.
CI: run oscap eval (no --remediate) to publish an ARF+HTML STIG report as a build artifact; optionally hard-fail below a score threshold.
Decisions needed before implementing: FIPS on/off · remote-audit collector DNS name (or ephemeral/none for lab) · CI fail-on-score threshold vs report-only.
Note: orthogonal to the rd.dot1x=0 test toggle, but a full STIG production posture wants 802.1X re-enabled.
Credentials (partly done): root is now locked and access is via a baked sudo-enabled key-only admin user (var.ssh_authorized_key / var.admin_user, wheel + NOPASSWD sudo), with sshd drop-in PermitRootLogin no / PasswordAuthentication no. Two items for STIG to reconcile: NOPASSWD sudo is a finding (acceptable for a key-only account, but document/deviate), and the admin key/user are fleet-wide baked — for per-host/rotatable access without rebuilds, layer the signed HTTP/SMBIOS config-agent or an SSH CA (TrustedUserCAKeys) discussed in session.
SSH host keys (done): extract-artifacts.sh strips /etc/ssh/ssh_host_* so they're not baked (baked keys = shared fleet identity + the build's perms/group don't survive the tar→erofs round-trip, so sshd rejected them "Permissions 640 too open" and refused to start). RHEL's sshd-keygen@.service regenerates them into the overlay before sshd starts. ⚠ Caveat: the ephemeral overlay means they regenerate every boot → the host key changes per reboot (clients warn). Follow-up for a stable host identity: persist the host keys or derive them from the TPM (ties to the hostnqn/identity model).

[ ] 3. Roll the errata into design doc v3

Once the boot path is validated and item 1 lands, fold design-v2-errata.md into the source design document and re-export a v3 PDF (key change: rootovl → systemd.volatile=overlay in §3 step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source (not in this repo).

[~] 4. Kiosk mode — autologin + fullscreen Chromium loading a URL

WORKING / display-verified 2026-06-17 (kiosk renders Chromium fullscreen on the configured URL). Build-flagged OFF by default for now (enable_kiosk=false) while iterating on boot/network — set -var enable_kiosk=true to ship it.

Path to getting here (don't re-discover):

cage is NOT packaged for EL9 (dnf install cage → "Unable to find a match"). Tried weston instead.
EPEL 9 ships weston 8.0, which has NEITHER kiosk-shell.so NOR the [autolaunch] weston.ini section — both arrived in weston 9.0. The first cut wrongly assumed 8.0 had them; shell=kiosk-shell.so failed to load. rpm -ql weston shows only desktop-shell / fullscreen-shell / ivi-shell.
Resolution: desktop-shell.so (the only packaged shell Chromium's xdg-shell works with — fullscreen-shell needs a client protocol Chromium doesn't speak), stripped bare (panel-position=none, black bg, no animation/lock), and Chromium launched by a wrapper (kiosk-session) since there's no [autolaunch]. Chromium --kiosk fullscreens over the shell.
Seat fix (load-bearing): weston died with logind: failed to get session seat → failed to create compositor backend. logind's auto-derivation didn't attach a seat; Environment=XDG_SEAT=seat0 + XDG_VTNR=1 in the unit (put into the PAM env before pam_open_session) registers the session on seat0/vt1. /dev/dri/card0 was present all along (not the issue).
Tested live without a rebuild: the whole root is systemd.volatile=overlay, so /etc AND /usr are writable tmpfs at runtime — drop-in the files + restart.
enable_kiosk Packer var — currently false by DEFAULT (set 2026-06-17). The whole kiosk provisioner is feature-flagged off while iterating on the boot/network stack (DHCP-lease/keep-alive work), for faster builds and a smaller image. Re-enable with -var enable_kiosk=true when returning to kiosk work — CI does not pass the var, so CI currently builds lean (no kiosk) too; add -var enable_kiosk=true to the Gitea workflow to ship it.
kiosk_url Packer var (default https://www.reddit.com; empty disables the whole install). Wired in CI as KIOSK_URL env.
Provisioner enables EPEL, installs weston chromium mesa-dri-drivers mesa-libEGL mesa-libgbm dejavu-sans-fonts, creates a passwordless unprivileged kiosk user (video/input groups).
files/kiosk/kiosk.service — runs /usr/local/bin/kiosk-session as kiosk on tty1 via PAMName=login+TTYPath+Environment=XDG_SEAT=seat0/XDG_VTNR=1 (gets seat0 for DRM/input), Conflicts=getty@tty1, WantedBy=graphical.target; default target set to graphical.
files/kiosk/weston.ini — shell=desktop-shell.so, idle-time=0, require-input=false, xwayland=false; [shell] panel-position=none, black background-color, animation=none, locking=false.
files/kiosk/kiosk-session — execs weston --use-pixman --socket=wayland-kiosk in the foreground (owns the seat), and launches Chromium (--kiosk --ozone-platform=wayland --disable-gpu, tmpfs profile/cache, URL from /etc/kiosk/kiosk.env) in a background crash-restart loop.
Remaining polish (now that it renders): dropping --disable-gpu on real GPUs, /dev/dri present in the VM (weston needs a DRM device — set Display to VirtIO-GPU/std), and SELinux AVCs for the GUI stack (check ausearch -m avc; the kiosk user is unconfined so likely none — but do NOT set permissive if they appear; add a policy module). Admin SSH (the key user) still works for debug even if the kiosk fails (it lands at graphical.target either way).
Image size: Chromium+Mesa add hundreds of MB to the uncompressed erofs — feeds the "shrink the rootfs" concern (and #7); reconsider whether a lighter browser/render path is viable.

Original task

Make the diskless workstation boot straight into a locked-down kiosk: a fullscreen Chromium showing a configured site (initial target: Reddit). All baked at build time into the root tree (immutable erofs/verity root; runtime is the tmpfs overlay), fleet-generic (no per-host state).

Graphical stack (keep it minimal): add a Packer provisioner installing a lightweight kiosk compositor + browser. Preferred: cage (Wayland single-app kiosk compositor) + Mesa (software GL fallback for VMs/no-GPU); X11 + a minimal WM (matchbox/openbox) as the alternative. Chromium on Rocky 9 is in EPEL — enabling EPEL adds a repo/supply-chain surface (decision below).
Autologin + session: dedicated unprivileged kiosk user; launch the session via a kiosk.service running cage -s -- chromium --kiosk … (or greetd + cage). Stage units/configs under files/ like the other provisioned config.
Chromium flags: --kiosk --noerrdialogs --disable-infobars --incognito --no-first-run --disable-session-crashed-bubble <URL>. Writable profile/cache land on the systemd.volatile=overlay tmpfs (RAM-backed, ephemeral) — fine, but watch RAM sizing; pin --user-data-dir/--disk-cache-dir to a tmpfs path if needed.
Kiosk URL = Packer var (e.g. kiosk_url, default the target site), baked into the launch command; keep it host-generic (per-env via DNS if ever needed).
Verify in boot test: graphical session comes up on real/VM display, Chromium fullscreen loads the URL over the booted network (works with the current rd.dot1x=0 test path / open network).
Decisions: Wayland(cage) vs X11; accept EPEL for Chromium (vs another source); confirm the target URL.
⚠ Conflict with item 2 (STIG): passwordless autologin and a desktop browser are STIG findings and enlarge the attack surface/image. Reconcile — e.g. a tightly-constrained kiosk session + documented deviation — when hardening lands.

[x] 5. Deploy & validate the end-to-end boot path (open-network test)

✅ DONE (2026-06-15). Full chain validated on QEMU/Proxmox: UEFI HTTP boot → initramfs → DHCP → nvmf-connect.service auto-connects the NVMe/TCP subsystem → two namespaces (data + verity hash) appear by pinned UUID → systemd-veritysetup maps /dev/mapper/root → uncompressed erofs mounts → systemd.volatile=overlay → multi-user. Booted with rd.dot1x=0 (open network) and the IP discovery address. See bench findings below for the full list of fixes. Remaining cleanup to leave the lab-test posture (now unblocked): revert the hardcoded IP to the nvme.fleet.lab DNS name, and re-enable security (DISABLE_DOT1X=false) — the secure path needs the same connect trigger over macsec0 (see ⚠ note below).

Original task: stand up serving/ (dnsmasq + nginx + SPDK) per serving/README.md, rebuild the UKI with nvmet_traddr=nvme.fleet.lab (now a Packer var — older UKIs have the literal NVMET_TRADDR placeholder and won't connect), then boot a UEFI HTTP-boot client (Secure Boot OFF, unsigned UKI; no TPM needed in rd.dot1x=0 mode) on the boot LAN.

Pass criteria: UKI fetched over HTTP → initramfs → DHCP on NIC → NVMe/TCP namespace connected → dm-verity root mapped → systemd.volatile=overlay writable → multi-user.
This validation gates item #1 (network-manager port) and item #3 (errata → design v3).
Bench findings (boot debug, 2026-06-14):
- Initramfs has no resolver (dracut writes resolv.conf only via its own DHCP path, not a manual dhclient), so the NVMe discovery address must be an IP, not a DNS name — build with -var nvmet_traddr=192.168.50.10. Old UKIs carrying nvme.fleet.lab can't connect early in boot.
- NVMe controller indices are not stable: a transient discovery controller can take nvme0, pushing the I/O controller to nvme1. The hardcoded /dev/nvme0n1 in the cmdline was a latent brick. Fixed: namespace UUID pinned on the target (spdk-setup.sh add_ns -u f1ee7000-0000-4000-8000-000000000001) and the cmdline now references /dev/disk/by-id/nvme-uuid.<uuid> (extract-artifacts.sh). The two constants MUST stay in sync; make them version-specific for #6.
- nvme_tcp/nvme_fabrics must be loaded before /dev/nvme-fabrics exists (dracut's nvmf module does this in a normal boot; at rd.break modprobe manually).
- Don't double-configure the NIC in test mode. With ip=dhcp on the cmdline, dracut's network module brings the NIC up AND emits the "online" event that the nvmf module's initqueue/online hook waits on to run nvme connect-all. The rd.dot1x=0 bypass used to also ifup $NIC, which (a) raced dracut → Error: ipv4: Address already assigned, and (b) didn't fire the nvmf online hook, so the namespace never auto-connected and boot spun on /dev/mapper/root forever (manual connect-all worked, which is what masked it). Fixed: the bypass branch no longer touches the NIC; ip=dhcp owns networking. (The secure path still addresses macsec0 itself — dracut doesn't know about that interface.)
- dracut's nvmf module never auto-connects on this distro. Its cmdline hook (92-parse-nvmf-boot-connections.sh) parses rd.nvmf.discover into /etc/nvme/discovery.conf correctly, but initqueue/online/ is empty — no autoconnect hook is registered, so nvme connect-all is never run (no attempt in dmesg) and the namespace never appears. Drivers load and /dev/nvme-fabrics exists; only the connect trigger is missing. A job placed on the main initqueue from a settled hook gets registered but is never re-executed once the loop blocks on udev events (verified: job present, manual connect-all works, namespace stays absent). An initqueue/online hook also does NOT work: this initramfs is systemd- driven ("A start job is running for /dev/mapper/root") and never sources dracut online hooks on link-up (verified: hook installs as 90-nvmf-connect.sh but never fires — no log, no stamp). Fixed: the connect is now a systemd unit (nvmf-connect.service, ConditionKernelCommandLine=rd.dot1x=0, Before=initrd-root-device.target) whose ExecStart (/usr/sbin/nvmf-connect) polls for the network then runs nvme connect-all until the namespace appears. The dracut module installs the unit + script and enables it via an initrd-root-device.target.wants symlink; diskless.pkr.hcl hard-asserts the unit is in the initramfs. Gotcha: $systemdsystemunitdir is empty in this dracut's module-setup context — inst_simple then silently drops the unit (empty dest dir). Use a ${systemdsystemunitdir:-/usr/lib/systemd/system} fallback and return 1 if the unit didn't land, so the build fails loudly instead of shipping a UKI that can't connect its root. Second gotcha: the build assertion lsinitrd … | grep -q 'nvmf-connect.service' was a false negative — grep -q exits on first match, closes the pipe, lsinitrd dies of SIGPIPE, and under pipefail the pipeline reports failure even though the unit IS present. This masked the fact that the systemd unit install was working all along. Fix: lsinitrd > file; grep -q … file (no pipe). Never lsinitrd | grep -q under pipefail. Bonus: the survivor listing confirmed nvme-cli's native autoconnect bits ARE in the initramfs (/usr/sbin/nvmf-autoconnect.sh, /etc/udev/rules.d/95-nvmf-initqueue.rules) — so the fully-native rd.nvmf.discover autoconnect path is available as a fallback if the unit doesn't fire on boot.
- dm-verity: hash-offset is unsupported → two namespaces. With NVMe connecting and the right by-id device present, verity still failed: systemd-veritysetup logged unknown option 'hash-offset=…' then Failed to load verity superblock — exactly the CLAUDE.md bench-verify item. The in-file (data+hash in one namespace, split by offset) layout is not supported by Rocky 9.8 systemd. Fixed via the documented fallback: extract-artifacts.sh now emits rootfs.erofs (data) + rootfs.hash (hash tree) as separate files, served as two NVMe namespaces with distinct pinned UUIDs (f1ee7000-…-01 data, …-02 hash); the cmdline uses systemd.verity_root_data=/systemd.verity_root_hash= by-id devices with no offset. Touches extract-artifacts.sh, build-oci.sh (artifact list + SHA256SUMS), serving/spdk-setup.sh (2nd bdev/namespace), and invariant #9.
- SELinux labels must survive into the erofs. It booted to multi-user but root/packer-build-only login was accepted then the session was killed instantly. Cause: kickstart sets selinux --enforcing, but the build dropped security.selinux xattrs (virt-tar-out + plain tar -xpf don't carry them), so the enforcing target denied the login session on an unlabeled root — and a read-only erofs can't autorelabel. Confirmed by booting with enforcing=0 (SMBIOS-injected) → login worked. Fixed: export via guestfish tar-out … selinux:true xattrs:true acls:true, restore with tar --xattrs --xattrs-include='*' --acls --selinux, and a getfattr build-time assertion that /usr/bin/bash is labeled before mkfs.erofs (mkfs.erofs preserves xattrs). Needs attr on the build host. The image now boots enforcing.
- mkfs.erofs --all-root was flattening all ownership to root:root. Mode bits + (now) xattrs survive the export, but --all-root forced every inode to uid/gid 0 — discarding group ownership (ssh_keys on host keys → sshd "permissions 640 too open" → refused; tty/crontab setgid groups) and user ownership (home dirs). Fixed: dropped --all-root so real ownership is preserved from the tree. That now requires extract-artifacts.sh to run as root (added an id -u guard; tar -x needs CAP_CHOWN to restore ownership) — the post-processor runs it via sudo and chowns the output files back to the build user so build-oci.sh (rootless buildah + its own login) still works. Combined with the host-key strip above, sshd starts cleanly. ⚠ The secure path has the same dependency on the (absent) nvmf autoconnect — when 802.1X is re-enabled, the macsec0 path must likewise drive connect-all (or the network-manager port #1 must wire a working autoconnect). Currently only the test path is fixed.
- The target Rocky 9 kernel's erofs is built without CONFIG_EROFS_FS_ZIP — mounting a compressed image fails erofs: lz4 algorithm isn't enabled, which silently blocks the root mount. Fixed: extract-artifacts.sh now builds the erofs uncompressed (mkfs.erofs without -z). Tradeoff: larger rootfs namespace + more NVMe/TCP bandwidth per read. Re-enable compression only against a kernel verified to have EROFS_FS_ZIP (grep CONFIG_EROFS /boot/config-$(uname -r)). This is the CLAUDE.md bench-verify item ("erofs module availability on the target RHEL kernel"). Larger image size also feeds the "shrink the rootfs" angle of #7.
- SPDK: add the listener to the discovery subsystem explicitly (nqn.2014-08.org.nvmexpress.discovery) to silence the deprecation warning.

[ ] 6. Rolling fleet updates (multi-version serve → cohort reboots)

Image-based, reboot-to-update across many live workstations. Hosts are immutable (read-only erofs/verity root over NVMe/TCP, ephemeral overlay), so updates are atomic A/B reboots, never in-place. The released OCI artifact + digest-pinned manifest + roothash coupling already give the atomic unit and instant rollback; this item builds the rollout machinery around it.

Serve versions side-by-side (must not overwrite a live rootfs): extend serving/ so SPDK/nvmet exposes each release as its own subsystem/NQN (nqn…:rootfs-<ver>) and nginx serves multiple fleet-<ver>.efi (keep current + previous). Make the UKI's NQN version-specific so each signed UKI connects only its matching namespace (constant discovery DNS name).
Version selection at boot: dnsmasq cohort tagging (MAC / vendor class / assigned tag) hands each cohort a different option-67 UKI URL. Moving a cohort's URL = assigning the release.
Control plane + in-image reboot agent: a baked-in systemd timer polls a "desired version" endpoint and reboots when idle / in a maintenance window (or out-of-band via Redfish/BMC). Stagger + jitter reboots to avoid UKI-fetch / NVMe-connect storms.
Staged rollout: canary → waves (10/50/100%) with health gates (multi-user reached, NVMe connected, dm-verity mapped, kiosk URL rendered, prod: MACsec/802.1X up); auto-halt on threshold.
Rollback: repoint cohort URL to previous UKI + reboot (current+previous stay served); security rollback via dbx-revoke / TPM monotonic counter (design §6).
Drain & retire an old version only when SPDK/nvmet reports zero connected controllers on it.
Availability: run NVMe/TCP target replicas behind distinct discovery entries (design §7) — never serve the whole fleet from one unreplicated node, especially mid-rollout (N versions + reboot bursts).
Note: kiosk content changes are server-side (the website) and need no fleet update — only OS/UKI/rootfs changes require a reboot.

[ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience)

There is no production-grade memory-safe NVMe-oF target — the Rust NVMe work is all driver/initiator-side (vroom, RedLeaf, Rust-for-Linux); the only production targets are C (Linux kernel nvmet, SPDK). So this is serving-node hardening, not a language swap. dm-verity already removes the target from the clients' integrity TCB (verify-on-read rejects tampered blocks), so the risk is RCE/DoS on the node that holds the fleet's disk, not client integrity. The export is read-only + single-namespace = inherently small attack surface.

Evaluate kernel nvmet vs SPDK for the target: kernel nvmet keeps the data path in the hardened/fuzzed kernel with a tiny userspace footprint (arguably a better attack-surface story); SPDK is a userspace C daemon (current choice, needs hugepages + caps). Decide per environment.
Confine the target (whichever): unprivileged, drop caps, seccomp + SELinux/AppArmor profile, read-only container, minimal mounts; minimal immutable serving image, digest-pinned.
Slim the SPDK image (multi-stage build): the current single-stage from-source image is ~600 MB (it keeps the whole toolchain — gcc, meson/ninja, kernel headers, the python venv — plus the SPDK/DPDK source and build trees). Convert serving/spdk/Dockerfile to a builder stage + a minimal runtime stage that copies only nvmf_tgt + its ldd shared libs (libnuma, libssl/crypto, libuuid, libaio…) + rpc.py/python3. Expect ~150–250 MB (debian-slim base or a static SPDK link trims further). Smaller attack surface + faster pulls, and it pairs with the confinement above.
Network-isolate :4420 so only the boot VLAN reaches the target (compose with 802.1X/MACsec segmentation in prod).
Replicas behind distinct discovery entries (availability is the real risk — it's the fleet's disk). Ties into #6 (rolling updates) availability.
Revisit a memory-safe target only if/when an off-the-shelf one matures (e.g. a future Rust-for-Linux nvmet).

24 KiB Raw Blame History Unescape Escape