TODO — diskless fleet boot system

Persistent backlog, kept in sync with the working session task tracker. Status: [ ] pending · [~] in progress · [x] done.

[ ] 1. Port the dracut module off `network-legacy` → `network-manager`

Durable replacement for the current dhcp-client stopgap. dhclient is deprecated in RHEL 9 and will eventually leave the repos; the build only works today because the provisioner installs dhcp-client so network-legacy passes dracut's check().

Rewrite both network bring-up paths in ../files/dracut/95dot1x-macsec/dot1x-macsec-up.sh: the secure path's ifup macsec0 and the rd.dot1x=0 bypass path's ifup $NIC — use nmcli/NetworkManager-in-initramfs (or ip= cmdline + the nm module) instead.
Change depends() network-legacy → network-manager in ../files/dracut/95dot1x-macsec/module-setup.sh.
Update add_dracutmodules in ../files/etc/dracut.conf.d/90-diskless.conf.
Drop dhcp-client from the provisioner dnf install in ../diskless.pkr.hcl.
Re-verify the lsinitrd assertion.
Gate: full boot-chain smoke test (swtpm + hostapd + FreeRADIUS) before merge, per CLAUDE.md.

[ ] 2. Apply RHEL9 STIG hardening to the built image

OpenSCAP + SCAP Security Guide (xccdf_org.ssgproject.content_profile_stig), run as a Packer provisioner after dnf install and before the dracut/initramfs build and cleanup.

Tailoring (autotailor): unselect rules this architecture can't satisfy — separate-partition rules (single erofs root + tmpfs overlay; satisfy intent with nodev,nosuid,noexec tmpfs mounts), GRUB/kernel-cmdline rules (no GRUB), and AIDE/file-integrity rules (dm-verity is the documented compensating control).
Kernel params → UKI cmdline in ../scripts/extract-artifacts.sh before signing: audit=1 audit_backlog_limit=8192 slub_debug=P page_poison=1 vsyscall=none (+ fips=1 if FIPS).
Reproducibility guard: strip non-deterministic remediation output (SSH host keys → regen at boot into overlay, AIDE db, timestamps, fresh UUIDs) so the verity hash stays stable.
Audit logging: audisp-remote/rsyslog to a central collector addressed by DNS name (host-generic).
Optional FIPS: fips=1 cmdline + 01fips dracut module + FIPS crypto policy.
CI: run oscap eval (no --remediate) to publish an ARF+HTML STIG report as a build artifact; optionally hard-fail below a score threshold.
Decisions needed before implementing: FIPS on/off · remote-audit collector DNS name (or ephemeral/none for lab) · CI fail-on-score threshold vs report-only.
Note: orthogonal to the rd.dot1x=0 test toggle, but a full STIG production posture wants 802.1X re-enabled.

[ ] 3. Roll the errata into design doc v3

Once the boot path is validated and item 1 lands, fold design-v2-errata.md into the source design document and re-export a v3 PDF (key change: rootovl → systemd.volatile=overlay in §3 step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source (not in this repo).

[ ] 4. Kiosk mode — autologin + fullscreen Chromium loading a URL

Make the diskless workstation boot straight into a locked-down kiosk: a fullscreen Chromium showing a configured site (initial target: Reddit). All baked at build time into the root tree (immutable erofs/verity root; runtime is the tmpfs overlay), fleet-generic (no per-host state).

Graphical stack (keep it minimal): add a Packer provisioner installing a lightweight kiosk compositor + browser. Preferred: cage (Wayland single-app kiosk compositor) + Mesa (software GL fallback for VMs/no-GPU); X11 + a minimal WM (matchbox/openbox) as the alternative. Chromium on Rocky 9 is in EPEL — enabling EPEL adds a repo/supply-chain surface (decision below).
Autologin + session: dedicated unprivileged kiosk user; launch the session via a kiosk.service running cage -s -- chromium --kiosk … (or greetd + cage). Stage units/configs under files/ like the other provisioned config.
Chromium flags: --kiosk --noerrdialogs --disable-infobars --incognito --no-first-run --disable-session-crashed-bubble <URL>. Writable profile/cache land on the systemd.volatile=overlay tmpfs (RAM-backed, ephemeral) — fine, but watch RAM sizing; pin --user-data-dir/--disk-cache-dir to a tmpfs path if needed.
Kiosk URL = Packer var (e.g. kiosk_url, default the target site), baked into the launch command; keep it host-generic (per-env via DNS if ever needed).
Verify in boot test: graphical session comes up on real/VM display, Chromium fullscreen loads the URL over the booted network (works with the current rd.dot1x=0 test path / open network).
Decisions: Wayland(cage) vs X11; accept EPEL for Chromium (vs another source); confirm the target URL.
⚠ Conflict with item 2 (STIG): passwordless autologin and a desktop browser are STIG findings and enlarge the attack surface/image. Reconcile — e.g. a tightly-constrained kiosk session + documented deviation — when hardening lands.

[~] 5. Deploy & validate the end-to-end boot path (open-network test)

In progress. Stand up serving/ (dnsmasq + nginx + SPDK) per serving/README.md, rebuild the UKI with nvmet_traddr=nvme.fleet.lab (now a Packer var — older UKIs have the literal NVMET_TRADDR placeholder and won't connect), then boot a UEFI HTTP-boot client (Secure Boot OFF, unsigned UKI; no TPM needed in rd.dot1x=0 mode) on the boot LAN.

Pass criteria: UKI fetched over HTTP → initramfs → DHCP on NIC → NVMe/TCP namespace connected → dm-verity root mapped → systemd.volatile=overlay writable → multi-user.
This validation gates item #1 (network-manager port) and item #3 (errata → design v3).
Bench findings (boot debug, 2026-06-14):
- Initramfs has no resolver (dracut writes resolv.conf only via its own DHCP path, not a manual dhclient), so the NVMe discovery address must be an IP, not a DNS name — build with -var nvmet_traddr=192.168.50.10. Old UKIs carrying nvme.fleet.lab can't connect early in boot.
- NVMe controller indices are not stable: a transient discovery controller can take nvme0, pushing the I/O controller to nvme1. The hardcoded /dev/nvme0n1 in the cmdline was a latent brick. Fixed: namespace UUID pinned on the target (spdk-setup.sh add_ns -u f1ee7000-0000-4000-8000-000000000001) and the cmdline now references /dev/disk/by-id/nvme-uuid.<uuid> (extract-artifacts.sh). The two constants MUST stay in sync; make them version-specific for #6.
- nvme_tcp/nvme_fabrics must be loaded before /dev/nvme-fabrics exists (dracut's nvmf module does this in a normal boot; at rd.break modprobe manually).
- Don't double-configure the NIC in test mode. With ip=dhcp on the cmdline, dracut's network module brings the NIC up AND emits the "online" event that the nvmf module's initqueue/online hook waits on to run nvme connect-all. The rd.dot1x=0 bypass used to also ifup $NIC, which (a) raced dracut → Error: ipv4: Address already assigned, and (b) didn't fire the nvmf online hook, so the namespace never auto-connected and boot spun on /dev/mapper/root forever (manual connect-all worked, which is what masked it). Fixed: the bypass branch no longer touches the NIC; ip=dhcp owns networking. (The secure path still addresses macsec0 itself — dracut doesn't know about that interface.)
- dracut's nvmf module never auto-connects on this distro. Its cmdline hook (92-parse-nvmf-boot-connections.sh) parses rd.nvmf.discover into /etc/nvme/discovery.conf correctly, but initqueue/online/ is empty — no autoconnect hook is registered, so nvme connect-all is never run (no attempt in dmesg) and the namespace never appears. Drivers load and /dev/nvme-fabrics exists; only the connect trigger is missing. A job placed on the main initqueue from a settled hook gets registered but is never re-executed once the loop blocks on udev events (verified: job present, manual connect-all works, namespace stays absent). An initqueue/online hook also does NOT work: this initramfs is systemd- driven ("A start job is running for /dev/mapper/root") and never sources dracut online hooks on link-up (verified: hook installs as 90-nvmf-connect.sh but never fires — no log, no stamp). Fixed: the connect is now a systemd unit (nvmf-connect.service, ConditionKernelCommandLine=rd.dot1x=0, Before=initrd-root-device.target) whose ExecStart (/usr/sbin/nvmf-connect) polls for the network then runs nvme connect-all until the namespace appears. The dracut module installs the unit + script and enables it via an initrd-root-device.target.wants symlink; diskless.pkr.hcl hard-asserts the unit is in the initramfs. Gotcha: $systemdsystemunitdir is empty in this dracut's module-setup context — inst_simple then silently drops the unit (empty dest dir). Use a ${systemdsystemunitdir:-/usr/lib/systemd/system} fallback and return 1 if the unit didn't land, so the build fails loudly instead of shipping a UKI that can't connect its root. ⚠ The secure path has the same dependency on the (absent) nvmf autoconnect — when 802.1X is re-enabled, the macsec0 path must likewise drive connect-all (or the network-manager port #1 must wire a working autoconnect). Currently only the test path is fixed.
- The target Rocky 9 kernel's erofs is built without CONFIG_EROFS_FS_ZIP — mounting a compressed image fails erofs: lz4 algorithm isn't enabled, which silently blocks the root mount. Fixed: extract-artifacts.sh now builds the erofs uncompressed (mkfs.erofs without -z). Tradeoff: larger rootfs namespace + more NVMe/TCP bandwidth per read. Re-enable compression only against a kernel verified to have EROFS_FS_ZIP (grep CONFIG_EROFS /boot/config-$(uname -r)). This is the CLAUDE.md bench-verify item ("erofs module availability on the target RHEL kernel"). Larger image size also feeds the "shrink the rootfs" angle of #7.
- SPDK: add the listener to the discovery subsystem explicitly (nqn.2014-08.org.nvmexpress.discovery) to silence the deprecation warning.

[ ] 6. Rolling fleet updates (multi-version serve → cohort reboots)

Image-based, reboot-to-update across many live workstations. Hosts are immutable (read-only erofs/verity root over NVMe/TCP, ephemeral overlay), so updates are atomic A/B reboots, never in-place. The released OCI artifact + digest-pinned manifest + roothash coupling already give the atomic unit and instant rollback; this item builds the rollout machinery around it.

Serve versions side-by-side (must not overwrite a live rootfs): extend serving/ so SPDK/nvmet exposes each release as its own subsystem/NQN (nqn…:rootfs-<ver>) and nginx serves multiple fleet-<ver>.efi (keep current + previous). Make the UKI's NQN version-specific so each signed UKI connects only its matching namespace (constant discovery DNS name).
Version selection at boot: dnsmasq cohort tagging (MAC / vendor class / assigned tag) hands each cohort a different option-67 UKI URL. Moving a cohort's URL = assigning the release.
Control plane + in-image reboot agent: a baked-in systemd timer polls a "desired version" endpoint and reboots when idle / in a maintenance window (or out-of-band via Redfish/BMC). Stagger + jitter reboots to avoid UKI-fetch / NVMe-connect storms.
Staged rollout: canary → waves (10/50/100%) with health gates (multi-user reached, NVMe connected, dm-verity mapped, kiosk URL rendered, prod: MACsec/802.1X up); auto-halt on threshold.
Rollback: repoint cohort URL to previous UKI + reboot (current+previous stay served); security rollback via dbx-revoke / TPM monotonic counter (design §6).
Drain & retire an old version only when SPDK/nvmet reports zero connected controllers on it.
Availability: run NVMe/TCP target replicas behind distinct discovery entries (design §7) — never serve the whole fleet from one unreplicated node, especially mid-rollout (N versions + reboot bursts).
Note: kiosk content changes are server-side (the website) and need no fleet update — only OS/UKI/rootfs changes require a reboot.

[ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience)

There is no production-grade memory-safe NVMe-oF target — the Rust NVMe work is all driver/initiator-side (vroom, RedLeaf, Rust-for-Linux); the only production targets are C (Linux kernel nvmet, SPDK). So this is serving-node hardening, not a language swap. dm-verity already removes the target from the clients' integrity TCB (verify-on-read rejects tampered blocks), so the risk is RCE/DoS on the node that holds the fleet's disk, not client integrity. The export is read-only + single-namespace = inherently small attack surface.

Evaluate kernel nvmet vs SPDK for the target: kernel nvmet keeps the data path in the hardened/fuzzed kernel with a tiny userspace footprint (arguably a better attack-surface story); SPDK is a userspace C daemon (current choice, needs hugepages + caps). Decide per environment.
Confine the target (whichever): unprivileged, drop caps, seccomp + SELinux/AppArmor profile, read-only container, minimal mounts; minimal immutable serving image, digest-pinned.
Slim the SPDK image (multi-stage build): the current single-stage from-source image is ~600 MB (it keeps the whole toolchain — gcc, meson/ninja, kernel headers, the python venv — plus the SPDK/DPDK source and build trees). Convert serving/spdk/Dockerfile to a builder stage + a minimal runtime stage that copies only nvmf_tgt + its ldd shared libs (libnuma, libssl/crypto, libuuid, libaio…) + rpc.py/python3. Expect ~150–250 MB (debian-slim base or a static SPDK link trims further). Smaller attack surface + faster pulls, and it pairs with the confinement above.
Network-isolate :4420 so only the boot VLAN reaches the target (compose with 802.1X/MACsec segmentation in prod).
Replicas behind distinct discovery entries (availability is the real risk — it's the fleet's disk). Ties into #6 (rolling updates) availability.
Revisit a memory-safe target only if/when an off-the-shelf one matures (e.g. a future Rust-for-Linux nvmet).

15 KiB Raw Blame History Unescape Escape