diskless-workstation/docs/TODO.md

TODO — diskless fleet boot system

Persistent backlog, kept in sync with the working session task tracker. Status: [ ] pending · [~] in progress · [x] done.

---

Current build posture (LAB / test) — as of 2026-06-18

The image currently builds in open-network test mode. These toggles must be flipped back for a secure/production build (consolidated as item #8):

• disable_dot1x=true / CI DISABLE_DOT1X="true" → rd.dot1x=0 (NO 802.1X / MACsec; open NVMe-oF). TEMPORARY.
• nvmet_traddr / CI NVMET_TRADDR="192.168.50.10" → hardcoded IP (should be a per-env DNS name like nvme.fleet.lab once DNS-at-boot is trusted).
• Unsigned UKI (sb_key/sb_cert not passed) → Secure Boot must be OFF on the test target. Production needs CI/HSM signing.
• enable_kiosk=false by default (CI may set true); enable_stig=true; boot_splash=true; NTP via DHCP→serving-host chrony.

What's working/verified vs awaiting bench is per-item below. Cross-cutting bench-verify checklist lives in CLAUDE.md → "Known bench-verify items".

---

[x] 1. Port the dracut module off network-legacy → network-manager

DONE — test path VERIFIED ON THE BENCH 2026-06-17 (host stays up past the DHCP lease time; no more lease-expiry outage). Fixed the confirmed outage: network-legacy's one-shot DHCP lease was never renewed post-switch-root, so the address dropped at lease expiry → NVMe/TCP root died ("keep alive timeout" / "no route to host"). NM hands its initramfs connections to the real-root NM (/run/NetworkManager state), which renews.

REMAINING: the secure path (macsec0 DHCP via nmcli, physical NIC unmanaged via nm-unmanaged.sh) is only exercised once disable_dot1x=false — verify on the swtpm + hostapd + FreeRADIUS bench when re-enabling security (see #2/cleanup).

Done:

• 90-diskless.conf add_dracutmodules: network-legacy → network-manager.
• module-setup.sh depends() → network-manager; added nmcli; installs the new cmdline hook.
• NEW nm-unmanaged.sh (cmdline hook): secure mode writes /run/NetworkManager/conf.d/10-dot1x-unmanaged.conf (unmanaged-devices=interface-name:en*) before NM starts; no-op in test mode. Replaces the deleted static 99-dot1x-unmanaged.conf (couldn't be mode-aware — it wrongly unmanaged en* in test mode, blocking renewal).
• dot1x-macsec-up.sh secure path: ifup macsec0 → nmcli (managed + DHCP, poll for IPv4). macsec0 MANAGED so NM renews; physical NIC UNMANAGED.
• diskless.pkr.hcl: dropped dhcp-client, added explicit NetworkManager; removed the static-config install; lsinitrd now asserts NetworkManager + nm-unmanaged.sh present.
• Static lint clean (bash -n, shellcheck, packer validate).

BENCH-VERIFY before trusting (the gate):

• Test path (disable_dot1x=true): boot, nmcli device shows boot NIC MANAGED; run past the DHCP lease time (drop the lab dnsmasq lease low) — address persists, lease renews (journalctl -u NetworkManager), NVMe + SSH stay up.
• Secure path (swtpm + hostapd + FreeRADIUS, disable_dot1x=false): en* UNMANAGED, macsec0 MANAGED + renewing; MKA survives switch-root; verity root mapped; NVMe stays connected past lease time; no "Address already assigned".
• Risk: ordering of the unmanaged drop-in vs NM start in secure mode. If NM transiently grabs en*, fall back to also baking a static en*-unmanaged file for secure-only builds.

[~] 2. Apply RHEL9 STIG hardening to the built image

IMPLEMENTED 2026-06-18 — awaiting build/bench verification. OpenSCAP + SSG remediation in a provisioner (enable_stig var, default on), tailored for the diskless/UKI/kiosk architecture via files/stig/diskless-tailoring.xml. STIG kernel params baked into the UKI cmdline (extract-artifacts.sh). Decisions: FIPS off, audit local-only, CI report-only. Deviations are documented in stig-deviations.md (curated register) and reported per-build by ../scripts/stig-deviations.sh (non-pass rules tagged intentional vs REVIEW); CI publishes stig-report.html + stig-arf.xml + stig-deviations.md. Scanner tooling is removed from the shipped image. BENCH-VERIFY: boot/login still work; reproducible verity hash across two builds; triage REVIEW rows.

FINDING 2026-06-18 (FIXED) — false-clean scan. The first hardened build (branch hardening, ada4aa4) produced a report reading "0 REVIEW / clean", but the ARF had 0 pass, 0 fail — only 462 notapplicable + 1068 notselected. Root cause: the provisioner selected the datastream with ls ssg-rhel9-ds.xml ssg-rl9-ds.xml | head -1, which picks rhel9 alphabetically; the RHEL9 datastream's CPE dictionary has no Rocky entry, so on Rocky 9.8 every selected rule went notapplicable and nothing was actually evaluated or remediated — the image was unhardened despite a clean-looking report. Fixes: (a) provisioner now maps /etc/os-release $ID → datastream (Rocky→ssg-rl9-ds.xml); (b) provisioner aborts the build if the scan evaluates 0 rules; (c) stig-deviations.sh flags an INVALID SCAN and exits non-zero on 0-evaluated. Re-run the hardened build and confirm pass/fail counts are non-zero before trusting any deviations report.

FINDING 2026-06-18 (FIXED) — hardening locked the fleet out. The first genuinely-evaluated hardened image (app 215b0eb) broke both login paths: the Chromium kiosk dropped to a getty login prompt, and admin SSH was rejected with Permission denied (publickey). Root cause: STIG enables pam_faillock (account phase) + password aging/inactivity, and the kiosk + fleetadmin accounts are passwordless / key-only, so the hardened PAM account phase treats them as expired/lockable (sshd key auth succeeds, then pam_acct_mgmt fails). Fix (provisioner 2c-STIG-postfix, runs after remediation): exempt the two appliance accounts — chage -M -1 -m 0 -I -1 -E -1 + faillock --reset; kiosk uses a dedicated /etc/pam.d/kiosk (PAMName=kiosk, permissive account / full session for seat); sshd gets an account sufficient pam_succeed_if.so user = fleetadmin first-rule bypass. Scoped to those two accounts; everyone else keeps full STIG. Justified as register deviation #10. Bench-verify on rebuild: kiosk renders + admin SSH works with hardening on.

Remaining (future hardening, currently waived — see the register): FIPS, firewall, per-host/rotatable admin access. Remote audit offload — receiver DONE (serving/audit-collector/, rsyslog over TCP/514, per-source-host files; DNS name ${AUDIT_HOST}); the image-side sender (audisp-syslog + rsyslog forward to audit.fleet.lab) is still to wire — see that dir's README.

Original spec (retained for reference)

OpenSCAP + SCAP Security Guide (xccdf_org.ssgproject.content_profile_stig), run as a Packer provisioner after dnf install and before the dracut/initramfs build and cleanup.

• Tailoring (autotailor): unselect rules this architecture can't satisfy — separate-partition rules (single erofs root + tmpfs overlay; satisfy intent with nodev,nosuid,noexec tmpfs mounts), GRUB/kernel-cmdline rules (no GRUB), and AIDE/file-integrity rules (dm-verity is the documented compensating control).
• Kernel params → UKI cmdline in ../scripts/extract-artifacts.sh before signing: audit=1 audit_backlog_limit=8192 slub_debug=P page_poison=1 vsyscall=none (+ fips=1 if FIPS).
• Reproducibility guard: strip non-deterministic remediation output (SSH host keys → regen at boot into overlay, AIDE db, timestamps, fresh UUIDs) so the verity hash stays stable.
• Audit logging: audisp-remote/rsyslog to a central collector addressed by DNS name (host-generic).
• Optional FIPS: fips=1 cmdline + 01fips dracut module + FIPS crypto policy.
• CI: run oscap eval (no --remediate) to publish an ARF+HTML STIG report as a build artifact; optionally hard-fail below a score threshold.
• Decisions needed before implementing: FIPS on/off · remote-audit collector DNS name (or ephemeral/none for lab) · CI fail-on-score threshold vs report-only.
• Note: orthogonal to the rd.dot1x=0 test toggle, but a full STIG production posture wants 802.1X re-enabled.
• Credentials (partly done): root is now locked and access is via a baked sudo-enabled key-only admin user (var.ssh_authorized_key / var.admin_user, wheel + NOPASSWD sudo), with sshd drop-in PermitRootLogin no / PasswordAuthentication no. Two items for STIG to reconcile: NOPASSWD sudo is a finding (acceptable for a key-only account, but document/deviate), and the admin key/user are fleet-wide baked — for per-host/rotatable access without rebuilds, layer the signed HTTP/SMBIOS config-agent or an SSH CA (TrustedUserCAKeys) discussed in session.
• SSH host keys (done): extract-artifacts.sh strips /etc/ssh/ssh_host_* so they're not baked (baked keys = shared fleet identity + the build's perms/group don't survive the tar→erofs round-trip, so sshd rejected them "Permissions 640 too open" and refused to start). RHEL's sshd-keygen@.service regenerates them into the overlay before sshd starts. ⚠ Caveat: the ephemeral overlay means they regenerate every boot → the host key changes per reboot (clients warn). Follow-up for a stable host identity: persist the host keys or derive them from the TPM (ties to the hostnqn/identity model).

[ ] 3. Roll the errata into design doc v3

Once the boot path is validated and item 1 lands, fold design-v2-errata.md into the source design document and re-export a v3 PDF (key change: rootovl → systemd.volatile=overlay in §3 step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source (not in this repo).

[x] 4. Kiosk mode — autologin + fullscreen Chromium loading a URL

WORKING (display-verified 2026-06-17). Remaining bits are polish (GPU/mesa trim, ship via CI enable_kiosk=true) and the phone-home/NTP lockdown (done) — all captured in the sub-notes below. Secure-path render check is folded into #8.

WORKING / display-verified 2026-06-17 (kiosk renders Chromium fullscreen on the configured URL). Build-flagged OFF by default for now (enable_kiosk=false) while iterating on boot/network — set -var enable_kiosk=true to ship it.

Path to getting here (don't re-discover):

• cage is NOT packaged for EL9 (dnf install cage → "Unable to find a match"). Tried weston instead.
• EPEL 9 ships weston 8.0, which has NEITHER kiosk-shell.so NOR the [autolaunch] weston.ini section — both arrived in weston 9.0. The first cut wrongly assumed 8.0 had them; shell=kiosk-shell.so failed to load. rpm -ql weston shows only desktop-shell / fullscreen-shell / ivi-shell.
• Resolution: desktop-shell.so (the only packaged shell Chromium's xdg-shell works with — fullscreen-shell needs a client protocol Chromium doesn't speak), stripped bare (panel-position=none, black bg, no animation/lock), and Chromium launched by a wrapper (kiosk-session) since there's no [autolaunch]. Chromium --kiosk fullscreens over the shell.
• Seat fix (load-bearing): weston died with logind: failed to get session seat → failed to create compositor backend. logind's auto-derivation didn't attach a seat; Environment=XDG_SEAT=seat0 + XDG_VTNR=1 in the unit (put into the PAM env before pam_open_session) registers the session on seat0/vt1. /dev/dri/card0 was present all along (not the issue).
• Tested live without a rebuild: the whole root is systemd.volatile=overlay, so /etc AND /usr are writable tmpfs at runtime — drop-in the files + restart.
• enable_kiosk Packer var — currently false by DEFAULT (set 2026-06-17). The whole kiosk provisioner is feature-flagged off while iterating on the boot/network stack (DHCP-lease/keep-alive work), for faster builds and a smaller image. Re-enable with -var enable_kiosk=true when returning to kiosk work — CI does not pass the var, so CI currently builds lean (no kiosk) too; add -var enable_kiosk=true to the Gitea workflow to ship it.
• kiosk_url Packer var (default http://boot.fleet.lab/kiosk/ — the kiosk SPA (serving/kiosk-app/, Svelte 5 + Tailwind v4, a cyber sci-fi spaceship monitoring/control terminal) served by the boot HTTP server, which reverse-proxies /kiosk/ to the kiosk container; self-contained so a boot validates without external connectivity. Empty disables the whole install). Wired in CI as KIOSK_URL env.
• Kiosk SPA (serving/kiosk-app/) — replaced the old single static serving/http/kiosk/index.html (2026-06-18). Top navbar + 5 pages (Bridge/Reactor/Nav/Life Support/Comms) with dummy telemetry; hash routing; builds in a 2-stage Docker image (Vite → nginx), reverse-proxied at /kiosk/ by the main serving nginx (loopback 127.0.0.1:${KIOSK_APP_PORT:-8088}). npm run build verified locally; container build untested (no docker here). Dummy data lives in src/lib/data.js (swap for a fleet API later).
• Provisioner enables EPEL, installs weston chromium mesa-dri-drivers mesa-libEGL mesa-libgbm dejavu-sans-fonts, creates a passwordless unprivileged kiosk user (video/input groups).
• files/kiosk/kiosk.service — runs /usr/local/bin/kiosk-session as kiosk on tty1 via PAMName=login+TTYPath+Environment=XDG_SEAT=seat0/XDG_VTNR=1 (gets seat0 for DRM/input), Conflicts=getty@tty1, WantedBy=graphical.target; default target set to graphical.
• files/kiosk/weston.ini — shell=desktop-shell.so, idle-time=0, require-input=false, xwayland=false; [shell] panel-position=none, black background-color, animation=none, locking=false.
• files/kiosk/kiosk-session — execs weston --use-pixman --socket=wayland-kiosk in the foreground (owns the seat), and launches Chromium (--kiosk --ozone-platform=wayland --disable-gpu, tmpfs profile/cache, URL from /etc/kiosk/kiosk.env) in a background crash-restart loop.
• Remaining polish (now that it renders): dropping --disable-gpu on real GPUs, /dev/dri present in the VM (weston needs a DRM device — set Display to VirtIO-GPU/std), and SELinux AVCs for the GUI stack (check ausearch -m avc; the kiosk user is unconfined so likely none — but do NOT set permissive if they appear; add a policy module). Admin SSH (the key user) still works for debug even if the kiosk fails (it lands at graphical.target either way).
• Phone-home lockdown (done 2026-06-17/18): Chromium was dialing Google (accounts/variations/optimization-guide/component+auto update/safe browsing — seen in dnsmasq logs). Two layers:
  • Flags in kiosk-session (--disable-background-networking, --disable-component-update, --disable-features=OptimizationHints,...) — confirmed killing variations/optimization-guide/update queries.
  • Managed policy files/kiosk/chromium-kiosk-policy.json for the policy-governed bits (sign-in accounts.google.com, sync). FIX 2026-06-18: the managed-policy dir differs by build, so it's installed to BOTH /etc/chromium/policies/managed/ AND /etc/chromium-browser/policies/managed/ (a policy in the wrong dir is silently ignored — the reason accounts.google.com/android.clients.google.com persisted). Verify via chrome://policy (Status: OK per entry).
  • Serving-side sinkhole (defense-in-depth, serving/docker-compose.yml dnsmasq --address=/google.com|googleapis.com|gstatic.com/0.0.0.0): the kiosk is local-only, so block egress at the resolver. Stops the leak; the query still logs (answered 0.0.0.0, not forwarded) — eliminating the query is the browser policy's job. Production: do this at the boot-LAN firewall.
  • NTP egress closed (done 2026-06-18): the *.pool.ntp.org lookups were the OS (chrony), not Chromium. Now: a chrony NTP server in the serving stack (serving/chrony/), dnsmasq hands it via DHCP option 42, and the image's NM dispatcher (files/etc/NetworkManager/dispatcher.d/10-chrony-dhcp-ntp) feeds it into chrony's sourcedir /run/chrony-dhcp; the baked public pool is stripped from /etc/chrony.conf. Result: the workstation syncs time only from the lab server — no external NTP egress. BENCH-VERIFY: on a booted host chronyc sources shows the lab server (the DHCP one), not a public pool.
• Image size: Chromium+Mesa add hundreds of MB to the uncompressed erofs — feeds the "shrink the rootfs" concern (and #7); reconsider whether a lighter browser/render path is viable.

Original task

Make the diskless workstation boot straight into a locked-down kiosk: a fullscreen Chromium showing a configured site (default: the boot HTTP server's /kiosk/ test page). All baked at build time into the root tree (immutable erofs/verity root; runtime is the tmpfs overlay), fleet-generic (no per-host state).

• Graphical stack (keep it minimal): add a Packer provisioner installing a lightweight kiosk compositor + browser. Preferred: cage (Wayland single-app kiosk compositor) + Mesa (software GL fallback for VMs/no-GPU); X11 + a minimal WM (matchbox/openbox) as the alternative. Chromium on Rocky 9 is in EPEL — enabling EPEL adds a repo/supply-chain surface (decision below).
• Autologin + session: dedicated unprivileged kiosk user; launch the session via a kiosk.service running cage -s -- chromium --kiosk … (or greetd + cage). Stage units/configs under files/ like the other provisioned config.
• Chromium flags: --kiosk --noerrdialogs --disable-infobars --incognito --no-first-run --disable-session-crashed-bubble <URL>. Writable profile/cache land on the systemd.volatile=overlay tmpfs (RAM-backed, ephemeral) — fine, but watch RAM sizing; pin --user-data-dir/--disk-cache-dir to a tmpfs path if needed.
• Kiosk URL = Packer var (e.g. kiosk_url, default the target site), baked into the launch command; keep it host-generic (per-env via DNS if ever needed).
• Verify in boot test: graphical session comes up on real/VM display, Chromium fullscreen loads the URL over the booted network (works with the current rd.dot1x=0 test path / open network).
• Decisions: Wayland(cage) vs X11; accept EPEL for Chromium (vs another source); confirm the target URL.
• ⚠ Conflict with item 2 (STIG): passwordless autologin and a desktop browser are STIG findings and enlarge the attack surface/image. Reconcile — e.g. a tightly-constrained kiosk session + documented deviation — when hardening lands.

[x] 5. Deploy & validate the end-to-end boot path (open-network test)

✅ DONE (2026-06-15). Full chain validated on QEMU/Proxmox: UEFI HTTP boot → initramfs → DHCP → nvmf-connect.service auto-connects the NVMe/TCP subsystem → two namespaces (data + verity hash) appear by pinned UUID → systemd-veritysetup maps /dev/mapper/root → uncompressed erofs mounts → systemd.volatile=overlay → multi-user. Booted with rd.dot1x=0 (open network) and the IP discovery address. See bench findings below for the full list of fixes. Remaining cleanup to leave the lab-test posture (now unblocked): revert the hardcoded IP to the nvme.fleet.lab DNS name, and re-enable security (DISABLE_DOT1X=false) — the secure path needs the same connect trigger over macsec0 (see ⚠ note below).

Original task: stand up serving/ (dnsmasq + nginx + SPDK) per serving/README.md, rebuild the UKI with nvmet_traddr=nvme.fleet.lab (now a Packer var — older UKIs have the literal NVMET_TRADDR placeholder and won't connect), then boot a UEFI HTTP-boot client (Secure Boot OFF, unsigned UKI; no TPM needed in rd.dot1x=0 mode) on the boot LAN.

• Pass criteria: UKI fetched over HTTP → initramfs → DHCP on NIC → NVMe/TCP namespace connected → dm-verity root mapped → systemd.volatile=overlay writable → multi-user.
• This validation gates item #1 (network-manager port) and item #3 (errata → design v3).
• Bench findings (boot debug, 2026-06-14):
  • Initramfs has no resolver (dracut writes resolv.conf only via its own DHCP path, not a manual dhclient), so the NVMe discovery address must be an IP, not a DNS name — build with -var nvmet_traddr=192.168.50.10. Old UKIs carrying nvme.fleet.lab can't connect early in boot.
  • NVMe controller indices are not stable: a transient discovery controller can take nvme0, pushing the I/O controller to nvme1. The hardcoded /dev/nvme0n1 in the cmdline was a latent brick. Fixed: namespace UUID pinned on the target (spdk-setup.sh add_ns -u f1ee7000-0000-4000-8000-000000000001) and the cmdline now references /dev/disk/by-id/nvme-uuid.<uuid> (extract-artifacts.sh). The two constants MUST stay in sync; make them version-specific for #6.
  • nvme_tcp/nvme_fabrics must be loaded before /dev/nvme-fabrics exists (dracut's nvmf module does this in a normal boot; at rd.break modprobe manually).
  • Don't double-configure the NIC in test mode. With ip=dhcp on the cmdline, dracut's network module brings the NIC up AND emits the "online" event that the nvmf module's initqueue/online hook waits on to run nvme connect-all. The rd.dot1x=0 bypass used to also ifup $NIC, which (a) raced dracut → Error: ipv4: Address already assigned, and (b) didn't fire the nvmf online hook, so the namespace never auto-connected and boot spun on /dev/mapper/root forever (manual connect-all worked, which is what masked it). Fixed: the bypass branch no longer touches the NIC; ip=dhcp owns networking. (The secure path still addresses macsec0 itself — dracut doesn't know about that interface.)
  • dracut's nvmf module never auto-connects on this distro. Its cmdline hook (92-parse-nvmf-boot-connections.sh) parses rd.nvmf.discover into /etc/nvme/discovery.conf correctly, but initqueue/online/ is empty — no autoconnect hook is registered, so nvme connect-all is never run (no attempt in dmesg) and the namespace never appears. Drivers load and /dev/nvme-fabrics exists; only the connect trigger is missing. A job placed on the main initqueue from a settled hook gets registered but is never re-executed once the loop blocks on udev events (verified: job present, manual connect-all works, namespace stays absent). An initqueue/online hook also does NOT work: this initramfs is systemd- driven ("A start job is running for /dev/mapper/root") and never sources dracut online hooks on link-up (verified: hook installs as 90-nvmf-connect.sh but never fires — no log, no stamp). Fixed: the connect is now a systemd unit (nvmf-connect.service, ConditionKernelCommandLine=rd.dot1x=0, Before=initrd-root-device.target) whose ExecStart (/usr/sbin/nvmf-connect) polls for the network then runs nvme connect-all until the namespace appears. The dracut module installs the unit + script and enables it via an initrd-root-device.target.wants symlink; diskless.pkr.hcl hard-asserts the unit is in the initramfs. Gotcha: $systemdsystemunitdir is empty in this dracut's module-setup context — inst_simple then silently drops the unit (empty dest dir). Use a ${systemdsystemunitdir:-/usr/lib/systemd/system} fallback and return 1 if the unit didn't land, so the build fails loudly instead of shipping a UKI that can't connect its root. Second gotcha: the build assertion lsinitrd … | grep -q 'nvmf-connect.service' was a false negative — grep -q exits on first match, closes the pipe, lsinitrd dies of SIGPIPE, and under pipefail the pipeline reports failure even though the unit IS present. This masked the fact that the systemd unit install was working all along. Fix: lsinitrd > file; grep -q … file (no pipe). Never lsinitrd | grep -q under pipefail. Bonus: the survivor listing confirmed nvme-cli's native autoconnect bits ARE in the initramfs (/usr/sbin/nvmf-autoconnect.sh, /etc/udev/rules.d/95-nvmf-initqueue.rules) — so the fully-native rd.nvmf.discover autoconnect path is available as a fallback if the unit doesn't fire on boot.
  • dm-verity: hash-offset is unsupported → two namespaces. With NVMe connecting and the right by-id device present, verity still failed: systemd-veritysetup logged unknown option 'hash-offset=…' then Failed to load verity superblock — exactly the CLAUDE.md bench-verify item. The in-file (data+hash in one namespace, split by offset) layout is not supported by Rocky 9.8 systemd. Fixed via the documented fallback: extract-artifacts.sh now emits rootfs.erofs (data) + rootfs.hash (hash tree) as separate files, served as two NVMe namespaces with distinct pinned UUIDs (f1ee7000-…-01 data, …-02 hash); the cmdline uses systemd.verity_root_data=/systemd.verity_root_hash= by-id devices with no offset. Touches extract-artifacts.sh, build-oci.sh (artifact list + SHA256SUMS), serving/spdk-setup.sh (2nd bdev/namespace), and invariant #9.
  • SELinux labels must survive into the erofs. It booted to multi-user but root/packer-build-only login was accepted then the session was killed instantly. Cause: kickstart sets selinux --enforcing, but the build dropped security.selinux xattrs (virt-tar-out + plain tar -xpf don't carry them), so the enforcing target denied the login session on an unlabeled root — and a read-only erofs can't autorelabel. Confirmed by booting with enforcing=0 (SMBIOS-injected) → login worked. Fixed: export via guestfish tar-out … selinux:true xattrs:true acls:true, restore with tar --xattrs --xattrs-include='*' --acls --selinux, and a getfattr build-time assertion that /usr/bin/bash is labeled before mkfs.erofs (mkfs.erofs preserves xattrs). Needs attr on the build host. The image now boots enforcing.
  • mkfs.erofs --all-root was flattening all ownership to root:root. Mode bits + (now) xattrs survive the export, but --all-root forced every inode to uid/gid 0 — discarding group ownership (ssh_keys on host keys → sshd "permissions 640 too open" → refused; tty/crontab setgid groups) and user ownership (home dirs). Fixed: dropped --all-root so real ownership is preserved from the tree. That now requires extract-artifacts.sh to run as root (added an id -u guard; tar -x needs CAP_CHOWN to restore ownership) — the post-processor runs it via sudo and chowns the output files back to the build user so build-oci.sh (rootless buildah + its own login) still works. Combined with the host-key strip above, sshd starts cleanly. ⚠ The secure path has the same dependency on the (absent) nvmf autoconnect — when 802.1X is re-enabled, the macsec0 path must likewise drive connect-all (or the network-manager port #1 must wire a working autoconnect). Currently only the test path is fixed.
  • The target Rocky 9 kernel's erofs is built without CONFIG_EROFS_FS_ZIP — mounting a compressed image fails erofs: lz4 algorithm isn't enabled, which silently blocks the root mount. Fixed: extract-artifacts.sh now builds the erofs uncompressed (mkfs.erofs without -z). Tradeoff: larger rootfs namespace + more NVMe/TCP bandwidth per read. Re-enable compression only against a kernel verified to have EROFS_FS_ZIP (grep CONFIG_EROFS /boot/config-$(uname -r)). This is the CLAUDE.md bench-verify item ("erofs module availability on the target RHEL kernel"). Larger image size also feeds the "shrink the rootfs" angle of #7.
  • SPDK: add the listener to the discovery subsystem explicitly (nqn.2014-08.org.nvmexpress.discovery) to silence the deprecation warning.

[ ] 6. Rolling fleet updates (multi-version serve → cohort reboots)

Image-based, reboot-to-update across many live workstations. Hosts are immutable (read-only erofs/verity root over NVMe/TCP, ephemeral overlay), so updates are atomic A/B reboots, never in-place. The released OCI artifact + digest-pinned manifest + roothash coupling already give the atomic unit and instant rollback; this item builds the rollout machinery around it.

• Serve versions side-by-side (must not overwrite a live rootfs): extend serving/ so SPDK/nvmet exposes each release as its own subsystem/NQN (nqn…:rootfs-<ver>) and nginx serves multiple fleet-<ver>.efi (keep current + previous). Make the UKI's NQN version-specific so each signed UKI connects only its matching namespace (constant discovery DNS name).
• Version selection at boot: dnsmasq cohort tagging (MAC / vendor class / assigned tag) hands each cohort a different option-67 UKI URL. Moving a cohort's URL = assigning the release.
• Control plane + in-image reboot agent: a baked-in systemd timer polls a "desired version" endpoint and reboots when idle / in a maintenance window (or out-of-band via Redfish/BMC). Stagger + jitter reboots to avoid UKI-fetch / NVMe-connect storms.
• Staged rollout: canary → waves (10/50/100%) with health gates (multi-user reached, NVMe connected, dm-verity mapped, kiosk URL rendered, prod: MACsec/802.1X up); auto-halt on threshold.
• Rollback: repoint cohort URL to previous UKI + reboot (current+previous stay served); security rollback via dbx-revoke / TPM monotonic counter (design §6).
• Drain & retire an old version only when SPDK/nvmet reports zero connected controllers on it.
• Availability: run NVMe/TCP target replicas behind distinct discovery entries (design §7) — never serve the whole fleet from one unreplicated node, especially mid-rollout (N versions + reboot bursts).
• Note: kiosk content changes are server-side (the website) and need no fleet update — only OS/UKI/rootfs changes require a reboot.

[ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience)

There is no production-grade memory-safe NVMe-oF target — the Rust NVMe work is all driver/initiator-side (vroom, RedLeaf, Rust-for-Linux); the only production targets are C (Linux kernel nvmet, SPDK). So this is serving-node hardening, not a language swap. dm-verity already removes the target from the clients' integrity TCB (verify-on-read rejects tampered blocks), so the risk is RCE/DoS on the node that holds the fleet's disk, not client integrity. The export is read-only + single-namespace = inherently small attack surface.

• Evaluate kernel nvmet vs SPDK for the target: kernel nvmet keeps the data path in the hardened/fuzzed kernel with a tiny userspace footprint (arguably a better attack-surface story); SPDK is a userspace C daemon (current choice, needs hugepages + caps). Decide per environment.
• Confine the target (whichever): unprivileged, drop caps, seccomp + SELinux/AppArmor profile, read-only container, minimal mounts; minimal immutable serving image, digest-pinned.
• Slim the SPDK image (multi-stage build): the current single-stage from-source image is ~600 MB (it keeps the whole toolchain — gcc, meson/ninja, kernel headers, the python venv — plus the SPDK/DPDK source and build trees). Convert serving/spdk/Dockerfile to a builder stage + a minimal runtime stage that copies only nvmf_tgt + its ldd shared libs (libnuma, libssl/crypto, libuuid, libaio…) + rpc.py/python3. Expect ~150–250 MB (debian-slim base or a static SPDK link trims further). Smaller attack surface + faster pulls, and it pairs with the confinement above.
• Network-isolate :4420 so only the boot VLAN reaches the target (compose with 802.1X/MACsec segmentation in prod).
• Replicas behind distinct discovery entries (availability is the real risk — it's the fleet's disk). Ties into #6 (rolling updates) availability.
• Revisit a memory-safe target only if/when an off-the-shelf one matures (e.g. a future Rust-for-Linux nvmet).

[ ] 8. Return to secure / production posture (un-toggle the lab flags)

Consolidates the cleanup of the temporary lab toggles (see "Current build posture" at the top) and the secure-path bench-verifies deferred while testing the open-network path. Do these together, then run the full secure boot-chain smoke test.

• Re-enable 802.1X/MACsec: disable_dot1x=false (CI DISABLE_DOT1X="false"). Removes rd.dot1x=0/ip=dhcp; the hook owns the link (macsec0).
• NVMe discovery by DNS: nvmet_traddr=nvme.fleet.lab (CI), resolved at boot by the serving dnsmasq — drop the hardcoded 192.168.50.10.
• Signing: wire sb_key/sb_cert (CI/HSM secrets) for a signed UKI; enable Secure Boot on targets. Until then the loud unsigned warning must stay.
• Bench-verify on the secure path (swtpm + hostapd + FreeRADIUS, disable_dot1x=false) — the items deferred this session:
  • NM port secure path: en* UNMANAGED + macsec0 MANAGED/renewing, MKA survives switch-root, NVMe stays up past lease (item #1 risk).
  • NTP via DHCP option 42 reaches chrony over macsec0 (chronyc sources).
  • Plymouth fleet theme: test-path render VERIFIED 2026-06-18; still confirm the auth→macsec rows reach done live on the secure path.
  • STIG remediation didn't break the secure path; reproducible verity hash; triage the generated stig-deviations.md REVIEW rows (item #2).
• Secure-path NVMe connect trigger: the test path uses nvmf-connect.service (gated rd.dot1x=0); confirm the secure path's connect (dracut nvmf over macsec0) actually fires, or give it an equivalent trigger. Apply the same -k 30 -l -1 -c 10 keep-alive/reconnect flags to the secure nvme connect as the test path has.
• Future hardening waived for now (from the STIG register): FIPS, firewall, remote audit offload, per-host/rotatable admin (SSH CA / config-agent).

[ ] 9. (OPTIONAL / nice-to-have) Convert provisioning to Ansible where it pays off

Low priority — only the config-management half of the build is a candidate, and the ROI is modest because the build VM is single-shot (idempotency buys little) and reproducibility/auditability favour minimal shell. Not a wholesale rewrite: the load-bearing logic (dracut module hook ordering, extract-artifacts.sh, build-oci.sh) is initramfs-runtime / host-side build tooling, not provisioning, and stays shell.

The high-ROI, low-risk slice if/when we pick this up:

• STIG via the SSG Ansible playbook (ssg-rl9-playbook-stig.yml, shipped with scap-security-guide) instead of oscap --remediate — the upstream path and far more reviewable (you can diff exactly what each rule changes). Keep oscap eval (no remediate) for the report + the 0-eval guard (item #2).
• Restyle the brittle imperative bits as declarative tasks: chrony.conf edit (lineinfile/blockinfile vs sed), sshd drop-in / NM dispatcher / kiosk file installs (template/copy), datastream-by-os-release as assert. Bonus: sidesteps the %{...}→%%{ HCL-escaping footgun.

Tips the balance toward "do it" only if: we start managing the serving hosts (dnsmasq/nginx/SPDK/chrony/audit-collector) with config-management too (shared roles to amortise — currently they're docker-compose), provisioning grows much more branchy, or we want molecule-tested hardening roles validated without a full ~17-min Packer build.