diskless-workstation/docs/TODO.md

TODO — diskless fleet boot system

Persistent backlog, kept in sync with the working session task tracker. Status: [ ] pending · [~] in progress · [x] done.

---

Current build posture (LAB / test) — as of 2026-06-19

The image currently builds in open-network test mode. These toggles must be flipped back for a secure/production build (consolidated as item #8):

• disable_dot1x=true / CI DISABLE_DOT1X="true" → rd.dot1x=0 (NO 802.1X / MACsec; open NVMe-oF). TEMPORARY.
• nvmet_traddr / CI NVMET_TRADDR="192.168.50.10" → hardcoded IP (should be a per-env DNS name like nvme.fleet.lab once DNS-at-boot is trusted).
• Unsigned UKI (sb_key/sb_cert not passed) → Secure Boot must be OFF on the test target. Production needs CI/HSM signing.
• enable_kiosk=true; boot_splash=true; NTP via DHCP→serving-host chrony.
• enable_stig / CI ENABLE_STIG="false" RIGHT NOW — temporarily disabled to get a working box while the STIG regressions were diagnosed (now fixed, below). Flip back to true and bench-verify the hardened build.
• debug_shell=true (DEBUG HOLE — passwordless root on tty2). REMOVE before prod (-var debug_shell=false). Note: it didn't actually come up on tty2 in testing; diagnosis ended up done via offline rootfs.erofs mount + SMBIOS enforcing=0. SMBIOS cmdline-extra works on Proxmox only after a full VM stop/start (not reset).
• Admin SSH key MUST be ECDSA or RSA, NOT ed25519, when STIG is on — the STIG crypto policy drops ed25519 from sshd's accepted pubkey algorithms. Set the SSH_AUTHORIZED_KEY CI secret to an ECDSA/RSA key (see #2).
• Hostname: image is hostname-neutral (empty /etc/hostname, hostname-mode=default); the serving dnsmasq's fleet-namer (--dhcp-script) assigns a stable creative starship name per MAC (e.g. nebula-vanguard-07) via DHCP option 12 + DNS A/PTR. Host picks it up on next DHCP renewal/reboot.
• Serving .env: must set AUDIT_HOST=audit.fleet.lab (+ docker compose up -d dnsmasq to reload) for remote log offload to resolve/flow.

What's working/verified vs awaiting bench is per-item below. Cross-cutting bench-verify checklist lives in CLAUDE.md → "Known bench-verify items".

---

[x] 1. Port the dracut module off network-legacy → network-manager

DONE — test path VERIFIED ON THE BENCH 2026-06-17 (host stays up past the DHCP lease time; no more lease-expiry outage). Fixed the confirmed outage: network-legacy's one-shot DHCP lease was never renewed post-switch-root, so the address dropped at lease expiry → NVMe/TCP root died ("keep alive timeout" / "no route to host"). NM hands its initramfs connections to the real-root NM (/run/NetworkManager state), which renews.

REMAINING: the secure path (macsec0 DHCP via nmcli, physical NIC unmanaged via nm-unmanaged.sh) is only exercised once disable_dot1x=false — verify on the swtpm + hostapd + FreeRADIUS bench when re-enabling security (see #2/cleanup).

Done:

• 90-diskless.conf add_dracutmodules: network-legacy → network-manager.
• module-setup.sh depends() → network-manager; added nmcli; installs the new cmdline hook.
• NEW nm-unmanaged.sh (cmdline hook): secure mode writes /run/NetworkManager/conf.d/10-dot1x-unmanaged.conf (unmanaged-devices=interface-name:en*) before NM starts; no-op in test mode. Replaces the deleted static 99-dot1x-unmanaged.conf (couldn't be mode-aware — it wrongly unmanaged en* in test mode, blocking renewal).
• dot1x-macsec-up.sh secure path: ifup macsec0 → nmcli (managed + DHCP, poll for IPv4). macsec0 MANAGED so NM renews; physical NIC UNMANAGED.
• diskless.pkr.hcl: dropped dhcp-client, added explicit NetworkManager; removed the static-config install; lsinitrd now asserts NetworkManager + nm-unmanaged.sh present.
• Static lint clean (bash -n, shellcheck, packer validate).

BENCH-VERIFY before trusting (the gate):

• Test path (disable_dot1x=true): boot, nmcli device shows boot NIC MANAGED; run past the DHCP lease time (drop the lab dnsmasq lease low) — address persists, lease renews (journalctl -u NetworkManager), NVMe + SSH stay up.
• Secure path (swtpm + hostapd + FreeRADIUS, disable_dot1x=false): en* UNMANAGED, macsec0 MANAGED + renewing; MKA survives switch-root; verity root mapped; NVMe stays connected past lease time; no "Address already assigned".
• Risk: ordering of the unmanaged drop-in vs NM start in secure mode. If NM transiently grabs en*, fall back to also baking a static en*-unmanaged file for secure-only builds.

[~] 2. Apply RHEL9 STIG hardening to the built image

IMPLEMENTED 2026-06-18 — awaiting build/bench verification. OpenSCAP + SSG remediation in a provisioner (enable_stig var, default on), tailored for the diskless/UKI/kiosk architecture via files/stig/diskless-tailoring.xml. STIG kernel params baked into the UKI cmdline (extract-artifacts.sh). Decisions: FIPS off, audit local-only, CI report-only. Deviations are documented in stig-deviations.md (curated register) and reported per-build by ../scripts/stig-deviations.sh (non-pass rules tagged intentional vs REVIEW); CI publishes stig-report.html + stig-arf.xml + stig-deviations.md. Scanner tooling is removed from the shipped image. BENCH-VERIFY: boot/login VERIFIED ON THE BENCH 2026-06-19 — hardened build with enable_stig=true, host cygnus-cruiser-99: kiosk auto-starts + renders, admin SSH (ECDSA) logs in, audit offload reaches the collector, DHCP creative hostname applied, NVMe/TCP verity root mounts and the host stays up. STILL OPEN: reproducible verity hash across two builds; triage the REVIEW rows (below); the secure path (#8, 802.1X/MACsec — this was the disable_dot1x=true test path).

FINDING 2026-06-18 (FIXED) — false-clean scan. The first hardened build (branch hardening, ada4aa4) produced a report reading "0 REVIEW / clean", but the ARF had 0 pass, 0 fail — only 462 notapplicable + 1068 notselected. Root cause: the provisioner selected the datastream with ls ssg-rhel9-ds.xml ssg-rl9-ds.xml | head -1, which picks rhel9 alphabetically; the RHEL9 datastream's CPE dictionary has no Rocky entry, so on Rocky 9.8 every selected rule went notapplicable and nothing was actually evaluated or remediated — the image was unhardened despite a clean-looking report. Fixes: (a) provisioner now maps /etc/os-release $ID → datastream (Rocky→ssg-rl9-ds.xml); (b) provisioner aborts the build if the scan evaluates 0 rules; (c) stig-deviations.sh flags an INVALID SCAN and exits non-zero on 0-evaluated. Re-run the hardened build and confirm pass/fail counts are non-zero before trusting any deviations report.

FINDING 2026-06-18 (FIXED) — hardening locked the fleet out. The first genuinely-evaluated hardened image (app 215b0eb) broke both login paths: the Chromium kiosk dropped to a getty login prompt, and admin SSH was rejected with Permission denied (publickey). Root cause: STIG enables pam_faillock (account phase) + password aging/inactivity, and the kiosk + fleetadmin accounts are passwordless / key-only, so the hardened PAM account phase treats them as expired/lockable (sshd key auth succeeds, then pam_acct_mgmt fails). Fix (provisioner 2c-STIG-postfix, runs after remediation): exempt the two appliance accounts — chage -M -1 -m 0 -I -1 -E -1 + faillock --reset; kiosk uses a dedicated /etc/pam.d/kiosk (PAMName=kiosk, permissive account / full session for seat); sshd gets an account sufficient pam_succeed_if.so user = fleetadmin first-rule bypass. Scoped to those two accounts; everyone else keeps full STIG. Justified as register deviation #10. VERIFIED ON THE BENCH 2026-06-19 (enable_stig=true, host cygnus-cruiser-99): kiosk auto-starts + renders and admin SSH (ECDSA key) logs in with hardening on.**

UPDATE 2026-06-19 — offline erofs inspection of the hardened image (mounted rootfs.erofs read-only) pinned the real causes; the PAM-account theory above was incomplete:

• Kiosk had TWO causes, both FIXED:
  1. fapolicyd is enabled and kiosk-session wasn't in its trust → it blocked /usr/local/bin/kiosk-session. Fix: KEEP fapolicyd (it's the exec-guard for the writable overlay dm-verity can't cover) and add the kiosk binary to its trust DB at build (2c-STIG-postfix). Not waived.
  2. Auto-start: kiosk was WantedBy=graphical.target but the host boots to multi-user.target (graphical never reached) → unit inactive (dead) at boot, fine on manual restart. Fix: WantedBy=multi-user.target + dropped set-default graphical.target (kiosk launches weston directly, needs no DM target).
• SSH = STIG crypto policy excludes ed25519 (CONFIRMED via ssh -vvv). The key fingerprint MATCHED (not a mismatch); the server's server-sig-algs advertised only ecdsa + rsa-sha2 — no ssh-ed25519 — so the ed25519 admin key is rejected at the signature stage (before PAM/SELinux, which is why enforcing=0 and the account-bypass didn't help). Fix (your action): set the SSH_AUTHORIZED_KEY CI secret to an ECDSA (ssh-keygen -t ecdsa -b 384) or RSA key, then rebuild. No code change. (Alt: re-allow ed25519 via PubkeyAcceptedAlgorithms +ssh-ed25519 in the sshd drop-in — but that's a non-approved algo and would re-fail a STIG rule; prefer the ECDSA key.)
• rsyslog TLS-offload rules waived (lab): rsyslog_encrypt_offload_* force gtls+x509+CA and broke the plaintext 2f forward → unselected in the tailoring; production TLS offload tracked under #8.
• Diagnosis tooling notes: SMBIOS cmdline-extra works only after a Proxmox VM stop/start (not reset); the debug-shell-on-tty2 never came up — the decisive diagnosis was offline rootfs.erofs mount + ssh -vvv + enforcing=0.

REVIEW-ROW TRIAGE — DONE 2026-06-19 (scan app-9f7d8d1, 21 REVIEW → 1 open). The post-fix hardened scan was valid (398 pass / 12 fail / 9 notchecked). All 21 REVIEW rows dispositioned (full detail: stig-deviations.md #12). Mechanics: a rule only stops being flagged REVIEW if it's selected="false" in diskless-tailoring.xml (the register prose alone doesn't clear it) or made to pass / N/A.

• Bucket A (waived, register-justified): encrypt_partitions, auditd_audispd_configure_sufficiently_large_partition (#1); grub2_password (#2); sysctl_crypto_fips_enabled, harden_sshd_macs_openssh_conf_crypto_policy, harden_sshd_macs_opensshserver_conf_crypto_policy (#5); configured_firewalld_default_deny (#4); sssd_enable_certmap, sssd_has_trust_anchor (#7).
• Bucket B (waived N/A): account_temp_expire_date, accounts_user_home_paths_only, libreswan_approved_tunnels, security_patches_up_to_date, ssh_keys_passphrase_protected.
• Bucket C (decisions applied): installed_OS_is_vendor_supported (Rocky IS supported; RHEL-datastream CPE artifact — waived), network_configure_name_resolution (single DHCP DNS by design — waived), sysctl_user_max_user_namespaces_no_remediation (KEEP userns for Chromium sandbox — waived), accounts_authorized_local_users (appliance accounts authorized, #10 — waived; prod: set the authorized-users regex var instead), package_postfix_installed + postfix_prevent_unrestricted_relay (postfix removed in 2c-STIG — no MTA needed; audit goes off-box via 2f), USBGuard configure_usbguard_auditbackend + usbguard_allow_hid/_hub/ _hid_and_hub (ENABLED + enforcing: LinuxAudit backend + hardware-independent HID/hub allowlist written in 2c-STIG; ⚠ BENCH-VERIFY physical console keyboard/ mouse on real hardware — the build-VM-generated policy + default block target would otherwise lock out console input).
• selinux_context_elevation_for_sudo — REMEDIATED WITH CONFINEMENT (decision 2026-06-20). 2c-STIG now rewrites the admin sudoers to TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL and maps the admin to staff_u (semanage login) so the sudo role transition is authorized. ⚠ TWO BENCH GATES: (a) admin SSH login + sudo + privileged ops must work under staff_u confinement (escape hatch: enable_stig=false → reverts to a justified deviation); (b) reproducibility — semanage rebuilds the policy store; confirm the verity hash is stable across two builds (normalize in 2d if it drifts). Needs policycoreutils-python-utils (installed in 2c-STIG if absent).

Expected next-build report: 0–1 REVIEW (0 if both postfix→N/A and usbguard→pass land and sudo confinement holds; the triage was validated by re-running scripts/stig-deviations.sh against the app-9f7d8d1 ARF with the new tailoring — 21 REVIEW → 3, where the 3 are exactly postfix/usbguard/sudo, all addressed for the next build).

Remaining (future hardening, currently waived — see the register): FIPS, firewall, per-host/rotatable admin access. Remote audit offload — WIRED both ends (2026-06-19): receiver serving/audit-collector/ (rsyslog TCP/514, per-source-host files) + sender in the image (provisioner 2f, audit_host var: audisp-syslog→LOCAL6 + rsyslog omfwd to audit.fleet.lab). Lab is plaintext; TLS/RELP is the remaining hardening. VERIFIED END-TO-END ON THE BENCH 2026-06-19 — booted host cygnus-cruiser-99 (192.168.50.148) forwards to the collector, which writes audit-collector/logs/192.168.50.148/{audit,messages}.log. Serving-host firewall gotcha hit + fixed: the collector is host-networked so the serving host's ufw governs :514 (Docker doesn't manage iptables for host-net containers); 514/tcp was bound to the wrong NIC (ens18 instead of the boot-LAN ens19), so SYNs were silently dropped (SYN-SENT / "connection timed out", not refused). Fix: ufw allow in on ens19 to any port 514 proto tcp. The sender's action.resumeRetryCount="-1" buffered the backlog and drained on reconnect — no records lost. Documented in ../serving/audit-collector/README.md. When the workstation firewall is re-enabled (#4 / register #4) it needs the reciprocal egress allow for 514/tcp to the collector.

Original spec (retained for reference)

OpenSCAP + SCAP Security Guide (xccdf_org.ssgproject.content_profile_stig), run as a Packer provisioner after dnf install and before the dracut/initramfs build and cleanup.

• Tailoring (autotailor): unselect rules this architecture can't satisfy — separate-partition rules (single erofs root + tmpfs overlay; satisfy intent with nodev,nosuid,noexec tmpfs mounts), GRUB/kernel-cmdline rules (no GRUB), and AIDE/file-integrity rules (dm-verity is the documented compensating control).
• Kernel params → UKI cmdline in ../scripts/extract-artifacts.sh before signing: audit=1 audit_backlog_limit=8192 slub_debug=P page_poison=1 vsyscall=none (+ fips=1 if FIPS).
• Reproducibility guard: strip non-deterministic remediation output (SSH host keys → regen at boot into overlay, AIDE db, timestamps, fresh UUIDs) so the verity hash stays stable.
• Audit logging: audisp-remote/rsyslog to a central collector addressed by DNS name (host-generic).
• Optional FIPS: fips=1 cmdline + 01fips dracut module + FIPS crypto policy.
• CI: run oscap eval (no --remediate) to publish an ARF+HTML STIG report as a build artifact; optionally hard-fail below a score threshold.
• Decisions needed before implementing: FIPS on/off · remote-audit collector DNS name (or ephemeral/none for lab) · CI fail-on-score threshold vs report-only.
• Note: orthogonal to the rd.dot1x=0 test toggle, but a full STIG production posture wants 802.1X re-enabled.
• Credentials (partly done): root is now locked and access is via a baked sudo-enabled key-only admin user (var.ssh_authorized_key / var.admin_user, wheel + NOPASSWD sudo), with sshd drop-in PermitRootLogin no / PasswordAuthentication no. Two items for STIG to reconcile: NOPASSWD sudo is a finding (acceptable for a key-only account, but document/deviate), and the admin key/user are fleet-wide baked — for per-host/rotatable access without rebuilds, layer the signed HTTP/SMBIOS config-agent or an SSH CA (TrustedUserCAKeys) discussed in session.
• SSH host keys (done): extract-artifacts.sh strips /etc/ssh/ssh_host_* so they're not baked (baked keys = shared fleet identity + the build's perms/group don't survive the tar→erofs round-trip, so sshd rejected them "Permissions 640 too open" and refused to start). RHEL's sshd-keygen@.service regenerates them into the overlay before sshd starts. ⚠ Caveat: the ephemeral overlay means they regenerate every boot → the host key changes per reboot (clients warn). Follow-up for a stable host identity: persist the host keys or derive them from the TPM (ties to the hostnqn/identity model).

[ ] 3. Roll the errata into design doc v3

Once the boot path is validated and item 1 lands, fold design-v2-errata.md into the source design document and re-export a v3 PDF (key change: rootovl → systemd.volatile=overlay in §3 step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source (not in this repo).

[x] 4. Kiosk mode — autologin + fullscreen Chromium loading a URL

WORKING (display-verified 2026-06-17). Remaining bits are polish (GPU/mesa trim, ship via CI enable_kiosk=true) and the phone-home/NTP lockdown (done) — all captured in the sub-notes below. Secure-path render check is folded into #8.

WORKING / display-verified 2026-06-17 (kiosk renders Chromium fullscreen on the configured URL). Build-flagged OFF by default for now (enable_kiosk=false) while iterating on boot/network — set -var enable_kiosk=true to ship it.

Path to getting here (don't re-discover):

• cage is NOT packaged for EL9 (dnf install cage → "Unable to find a match"). Tried weston instead.
• EPEL 9 ships weston 8.0, which has NEITHER kiosk-shell.so NOR the [autolaunch] weston.ini section — both arrived in weston 9.0. The first cut wrongly assumed 8.0 had them; shell=kiosk-shell.so failed to load. rpm -ql weston shows only desktop-shell / fullscreen-shell / ivi-shell.
• Resolution: desktop-shell.so (the only packaged shell Chromium's xdg-shell works with — fullscreen-shell needs a client protocol Chromium doesn't speak), stripped bare (panel-position=none, black bg, no animation/lock), and Chromium launched by a wrapper (kiosk-session) since there's no [autolaunch]. Chromium --kiosk fullscreens over the shell.
• Seat fix (load-bearing): weston died with logind: failed to get session seat → failed to create compositor backend. logind's auto-derivation didn't attach a seat; Environment=XDG_SEAT=seat0 + XDG_VTNR=1 in the unit (put into the PAM env before pam_open_session) registers the session on seat0/vt1. /dev/dri/card0 was present all along (not the issue).
• Tested live without a rebuild: the whole root is systemd.volatile=overlay, so /etc AND /usr are writable tmpfs at runtime — drop-in the files + restart.
• enable_kiosk Packer var — currently false by DEFAULT (set 2026-06-17). The whole kiosk provisioner is feature-flagged off while iterating on the boot/network stack (DHCP-lease/keep-alive work), for faster builds and a smaller image. Re-enable with -var enable_kiosk=true when returning to kiosk work — CI does not pass the var, so CI currently builds lean (no kiosk) too; add -var enable_kiosk=true to the Gitea workflow to ship it.
• kiosk_url Packer var (default http://boot.fleet.lab/kiosk/ — the kiosk SPA (serving/kiosk-app/, Svelte 5 + Tailwind v4, a cyber sci-fi spaceship monitoring/control terminal) served by the boot HTTP server, which reverse-proxies /kiosk/ to the kiosk container; self-contained so a boot validates without external connectivity. Empty disables the whole install). Wired in CI as KIOSK_URL env.
• Kiosk SPA (serving/kiosk-app/) — replaced the old single static serving/http/kiosk/index.html (2026-06-18). Top navbar + 5 pages (Bridge/Reactor/Nav/Life Support/Comms) with dummy telemetry; hash routing; builds in a 2-stage Docker image (Vite → nginx), reverse-proxied at /kiosk/ by the main serving nginx (loopback 127.0.0.1:${KIOSK_APP_PORT:-8088}). npm run build verified locally; container build untested (no docker here). Dummy data lives in src/lib/data.js (swap for a fleet API later).
• Provisioner enables EPEL, installs weston chromium mesa-dri-drivers mesa-libEGL mesa-libgbm dejavu-sans-fonts, creates a passwordless unprivileged kiosk user (video/input groups).
• files/kiosk/kiosk.service — runs /usr/local/bin/kiosk-session as kiosk on tty1 via PAMName=login+TTYPath+Environment=XDG_SEAT=seat0/XDG_VTNR=1 (gets seat0 for DRM/input), Conflicts=getty@tty1, WantedBy=graphical.target; default target set to graphical.
• files/kiosk/weston.ini — shell=desktop-shell.so, idle-time=0, require-input=false, xwayland=false; [shell] panel-position=none, black background-color, animation=none, locking=false.
• files/kiosk/kiosk-session — execs weston --use-pixman --socket=wayland-kiosk in the foreground (owns the seat), and launches Chromium (--kiosk --ozone-platform=wayland --disable-gpu, tmpfs profile/cache, URL from /etc/kiosk/kiosk.env) in a background crash-restart loop.
• Remaining polish (now that it renders): dropping --disable-gpu on real GPUs, /dev/dri present in the VM (weston needs a DRM device — set Display to VirtIO-GPU/std), and SELinux AVCs for the GUI stack (check ausearch -m avc; the kiosk user is unconfined so likely none — but do NOT set permissive if they appear; add a policy module). Admin SSH (the key user) still works for debug even if the kiosk fails (it lands at graphical.target either way).
• Phone-home lockdown (done 2026-06-17/18): Chromium was dialing Google (accounts/variations/optimization-guide/component+auto update/safe browsing — seen in dnsmasq logs). Two layers:
  • Flags in kiosk-session (--disable-background-networking, --disable-component-update, --disable-features=OptimizationHints,...) — confirmed killing variations/optimization-guide/update queries.
  • Managed policy files/kiosk/chromium-kiosk-policy.json for the policy-governed bits (sign-in accounts.google.com, sync). FIX 2026-06-18: the managed-policy dir differs by build, so it's installed to BOTH /etc/chromium/policies/managed/ AND /etc/chromium-browser/policies/managed/ (a policy in the wrong dir is silently ignored — the reason accounts.google.com/android.clients.google.com persisted). Verify via chrome://policy (Status: OK per entry).
  • Serving-side sinkhole (defense-in-depth, serving/docker-compose.yml dnsmasq --address=/google.com|googleapis.com|gstatic.com/0.0.0.0): the kiosk is local-only, so block egress at the resolver. Stops the leak; the query still logs (answered 0.0.0.0, not forwarded) — eliminating the query is the browser policy's job. Production: do this at the boot-LAN firewall.
  • NTP egress closed (done 2026-06-18): the *.pool.ntp.org lookups were the OS (chrony), not Chromium. Now: a chrony NTP server in the serving stack (serving/chrony/), dnsmasq hands it via DHCP option 42, and the image's NM dispatcher (files/etc/NetworkManager/dispatcher.d/10-chrony-dhcp-ntp) feeds it into chrony's sourcedir /run/chrony-dhcp; the baked public pool is stripped from /etc/chrony.conf. Result: the workstation syncs time only from the lab server — no external NTP egress. BENCH-VERIFY: on a booted host chronyc sources shows the lab server (the DHCP one), not a public pool.
• Image size: Chromium+Mesa add hundreds of MB to the uncompressed erofs — feeds the "shrink the rootfs" concern (and #7); reconsider whether a lighter browser/render path is viable.

Original task

Make the diskless workstation boot straight into a locked-down kiosk: a fullscreen Chromium showing a configured site (default: the boot HTTP server's /kiosk/ test page). All baked at build time into the root tree (immutable erofs/verity root; runtime is the tmpfs overlay), fleet-generic (no per-host state).

• Graphical stack (keep it minimal): add a Packer provisioner installing a lightweight kiosk compositor + browser. Preferred: cage (Wayland single-app kiosk compositor) + Mesa (software GL fallback for VMs/no-GPU); X11 + a minimal WM (matchbox/openbox) as the alternative. Chromium on Rocky 9 is in EPEL — enabling EPEL adds a repo/supply-chain surface (decision below).
• Autologin + session: dedicated unprivileged kiosk user; launch the session via a kiosk.service running cage -s -- chromium --kiosk … (or greetd + cage). Stage units/configs under files/ like the other provisioned config.
• Chromium flags: --kiosk --noerrdialogs --disable-infobars --incognito --no-first-run --disable-session-crashed-bubble <URL>. Writable profile/cache land on the systemd.volatile=overlay tmpfs (RAM-backed, ephemeral) — fine, but watch RAM sizing; pin --user-data-dir/--disk-cache-dir to a tmpfs path if needed.
• Kiosk URL = Packer var (e.g. kiosk_url, default the target site), baked into the launch command; keep it host-generic (per-env via DNS if ever needed).
• Verify in boot test: graphical session comes up on real/VM display, Chromium fullscreen loads the URL over the booted network (works with the current rd.dot1x=0 test path / open network).
• Decisions: Wayland(cage) vs X11; accept EPEL for Chromium (vs another source); confirm the target URL.
• ⚠ Conflict with item 2 (STIG): passwordless autologin and a desktop browser are STIG findings and enlarge the attack surface/image. Reconcile — e.g. a tightly-constrained kiosk session + documented deviation — when hardening lands.

[x] 5. Deploy & validate the end-to-end boot path (open-network test)

✅ DONE (2026-06-15). Full chain validated on QEMU/Proxmox: UEFI HTTP boot → initramfs → DHCP → nvmf-connect.service auto-connects the NVMe/TCP subsystem → two namespaces (data + verity hash) appear by pinned UUID → systemd-veritysetup maps /dev/mapper/root → uncompressed erofs mounts → systemd.volatile=overlay → multi-user. Booted with rd.dot1x=0 (open network) and the IP discovery address. See bench findings below for the full list of fixes. Remaining cleanup to leave the lab-test posture (now unblocked): revert the hardcoded IP to the nvme.fleet.lab DNS name, and re-enable security (DISABLE_DOT1X=false) — the secure path needs the same connect trigger over macsec0 (see ⚠ note below).

Original task: stand up serving/ (dnsmasq + nginx + SPDK) per serving/README.md, rebuild the UKI with nvmet_traddr=nvme.fleet.lab (now a Packer var — older UKIs have the literal NVMET_TRADDR placeholder and won't connect), then boot a UEFI HTTP-boot client (Secure Boot OFF, unsigned UKI; no TPM needed in rd.dot1x=0 mode) on the boot LAN.

• Pass criteria: UKI fetched over HTTP → initramfs → DHCP on NIC → NVMe/TCP namespace connected → dm-verity root mapped → systemd.volatile=overlay writable → multi-user.
• This validation gates item #1 (network-manager port) and item #3 (errata → design v3).
• Bench findings (boot debug, 2026-06-14):
  • Initramfs has no resolver (dracut writes resolv.conf only via its own DHCP path, not a manual dhclient), so the NVMe discovery address must be an IP, not a DNS name — build with -var nvmet_traddr=192.168.50.10. Old UKIs carrying nvme.fleet.lab can't connect early in boot.
  • NVMe controller indices are not stable: a transient discovery controller can take nvme0, pushing the I/O controller to nvme1. The hardcoded /dev/nvme0n1 in the cmdline was a latent brick. Fixed: namespace UUID pinned on the target (spdk-setup.sh add_ns -u f1ee7000-0000-4000-8000-000000000001) and the cmdline now references /dev/disk/by-id/nvme-uuid.<uuid> (extract-artifacts.sh). The two constants MUST stay in sync; make them version-specific for #6.
  • nvme_tcp/nvme_fabrics must be loaded before /dev/nvme-fabrics exists (dracut's nvmf module does this in a normal boot; at rd.break modprobe manually).
  • Don't double-configure the NIC in test mode. With ip=dhcp on the cmdline, dracut's network module brings the NIC up AND emits the "online" event that the nvmf module's initqueue/online hook waits on to run nvme connect-all. The rd.dot1x=0 bypass used to also ifup $NIC, which (a) raced dracut → Error: ipv4: Address already assigned, and (b) didn't fire the nvmf online hook, so the namespace never auto-connected and boot spun on /dev/mapper/root forever (manual connect-all worked, which is what masked it). Fixed: the bypass branch no longer touches the NIC; ip=dhcp owns networking. (The secure path still addresses macsec0 itself — dracut doesn't know about that interface.)
  • dracut's nvmf module never auto-connects on this distro. Its cmdline hook (92-parse-nvmf-boot-connections.sh) parses rd.nvmf.discover into /etc/nvme/discovery.conf correctly, but initqueue/online/ is empty — no autoconnect hook is registered, so nvme connect-all is never run (no attempt in dmesg) and the namespace never appears. Drivers load and /dev/nvme-fabrics exists; only the connect trigger is missing. A job placed on the main initqueue from a settled hook gets registered but is never re-executed once the loop blocks on udev events (verified: job present, manual connect-all works, namespace stays absent). An initqueue/online hook also does NOT work: this initramfs is systemd- driven ("A start job is running for /dev/mapper/root") and never sources dracut online hooks on link-up (verified: hook installs as 90-nvmf-connect.sh but never fires — no log, no stamp). Fixed: the connect is now a systemd unit (nvmf-connect.service, ConditionKernelCommandLine=rd.dot1x=0, Before=initrd-root-device.target) whose ExecStart (/usr/sbin/nvmf-connect) polls for the network then runs nvme connect-all until the namespace appears. The dracut module installs the unit + script and enables it via an initrd-root-device.target.wants symlink; diskless.pkr.hcl hard-asserts the unit is in the initramfs. Gotcha: $systemdsystemunitdir is empty in this dracut's module-setup context — inst_simple then silently drops the unit (empty dest dir). Use a ${systemdsystemunitdir:-/usr/lib/systemd/system} fallback and return 1 if the unit didn't land, so the build fails loudly instead of shipping a UKI that can't connect its root. Second gotcha: the build assertion lsinitrd … | grep -q 'nvmf-connect.service' was a false negative — grep -q exits on first match, closes the pipe, lsinitrd dies of SIGPIPE, and under pipefail the pipeline reports failure even though the unit IS present. This masked the fact that the systemd unit install was working all along. Fix: lsinitrd > file; grep -q … file (no pipe). Never lsinitrd | grep -q under pipefail. Bonus: the survivor listing confirmed nvme-cli's native autoconnect bits ARE in the initramfs (/usr/sbin/nvmf-autoconnect.sh, /etc/udev/rules.d/95-nvmf-initqueue.rules) — so the fully-native rd.nvmf.discover autoconnect path is available as a fallback if the unit doesn't fire on boot.
  • dm-verity: hash-offset is unsupported → two namespaces. With NVMe connecting and the right by-id device present, verity still failed: systemd-veritysetup logged unknown option 'hash-offset=…' then Failed to load verity superblock — exactly the CLAUDE.md bench-verify item. The in-file (data+hash in one namespace, split by offset) layout is not supported by Rocky 9.8 systemd. Fixed via the documented fallback: extract-artifacts.sh now emits rootfs.erofs (data) + rootfs.hash (hash tree) as separate files, served as two NVMe namespaces with distinct pinned UUIDs (f1ee7000-…-01 data, …-02 hash); the cmdline uses systemd.verity_root_data=/systemd.verity_root_hash= by-id devices with no offset. Touches extract-artifacts.sh, build-oci.sh (artifact list + SHA256SUMS), serving/spdk-setup.sh (2nd bdev/namespace), and invariant #9.
  • SELinux labels must survive into the erofs. It booted to multi-user but root/packer-build-only login was accepted then the session was killed instantly. Cause: kickstart sets selinux --enforcing, but the build dropped security.selinux xattrs (virt-tar-out + plain tar -xpf don't carry them), so the enforcing target denied the login session on an unlabeled root — and a read-only erofs can't autorelabel. Confirmed by booting with enforcing=0 (SMBIOS-injected) → login worked. Fixed: export via guestfish tar-out … selinux:true xattrs:true acls:true, restore with tar --xattrs --xattrs-include='*' --acls --selinux, and a getfattr build-time assertion that /usr/bin/bash is labeled before mkfs.erofs (mkfs.erofs preserves xattrs). Needs attr on the build host. The image now boots enforcing.
  • mkfs.erofs --all-root was flattening all ownership to root:root. Mode bits + (now) xattrs survive the export, but --all-root forced every inode to uid/gid 0 — discarding group ownership (ssh_keys on host keys → sshd "permissions 640 too open" → refused; tty/crontab setgid groups) and user ownership (home dirs). Fixed: dropped --all-root so real ownership is preserved from the tree. That now requires extract-artifacts.sh to run as root (added an id -u guard; tar -x needs CAP_CHOWN to restore ownership) — the post-processor runs it via sudo and chowns the output files back to the build user so build-oci.sh (rootless buildah + its own login) still works. Combined with the host-key strip above, sshd starts cleanly. ⚠ The secure path has the same dependency on the (absent) nvmf autoconnect — when 802.1X is re-enabled, the macsec0 path must likewise drive connect-all (or the network-manager port #1 must wire a working autoconnect). Currently only the test path is fixed.
  • The target Rocky 9 kernel's erofs is built without CONFIG_EROFS_FS_ZIP — mounting a compressed image fails erofs: lz4 algorithm isn't enabled, which silently blocks the root mount. Fixed: extract-artifacts.sh now builds the erofs uncompressed (mkfs.erofs without -z). Tradeoff: larger rootfs namespace + more NVMe/TCP bandwidth per read. Re-enable compression only against a kernel verified to have EROFS_FS_ZIP (grep CONFIG_EROFS /boot/config-$(uname -r)). This is the CLAUDE.md bench-verify item ("erofs module availability on the target RHEL kernel"). Larger image size also feeds the "shrink the rootfs" angle of #7.
  • SPDK: add the listener to the discovery subsystem explicitly (nqn.2014-08.org.nvmexpress.discovery) to silence the deprecation warning.

[ ] 6. Rolling fleet updates (multi-version serve → cohort reboots)

Image-based, reboot-to-update across many live workstations. Hosts are immutable (read-only erofs/verity root over NVMe/TCP, ephemeral overlay), so updates are atomic A/B reboots, never in-place. The released OCI artifact + digest-pinned manifest + roothash coupling already give the atomic unit and instant rollback; this item builds the rollout machinery around it.

• Serve versions side-by-side (must not overwrite a live rootfs): extend serving/ so SPDK/nvmet exposes each release as its own subsystem/NQN (nqn…:rootfs-<ver>) and nginx serves multiple fleet-<ver>.efi (keep current + previous). Make the UKI's NQN version-specific so each signed UKI connects only its matching namespace (constant discovery DNS name).
• Version selection at boot: dnsmasq cohort tagging (MAC / vendor class / assigned tag) hands each cohort a different option-67 UKI URL. Moving a cohort's URL = assigning the release.
• Control plane + in-image reboot agent: a baked-in systemd timer polls a "desired version" endpoint and reboots when idle / in a maintenance window (or out-of-band via Redfish/BMC). Stagger + jitter reboots to avoid UKI-fetch / NVMe-connect storms.
• Staged rollout: canary → waves (10/50/100%) with health gates (multi-user reached, NVMe connected, dm-verity mapped, kiosk URL rendered, prod: MACsec/802.1X up); auto-halt on threshold.
• Rollback: repoint cohort URL to previous UKI + reboot (current+previous stay served); security rollback via dbx-revoke / TPM monotonic counter (design §6).
• Drain & retire an old version only when SPDK/nvmet reports zero connected controllers on it.
• Availability: run NVMe/TCP target replicas behind distinct discovery entries (design §7) — never serve the whole fleet from one unreplicated node, especially mid-rollout (N versions + reboot bursts).
• Note: kiosk content changes are server-side (the website) and need no fleet update — only OS/UKI/rootfs changes require a reboot.

[ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience)

There is no production-grade memory-safe NVMe-oF target — the Rust NVMe work is all driver/initiator-side (vroom, RedLeaf, Rust-for-Linux); the only production targets are C (Linux kernel nvmet, SPDK). So this is serving-node hardening, not a language swap. dm-verity already removes the target from the clients' integrity TCB (verify-on-read rejects tampered blocks), so the risk is RCE/DoS on the node that holds the fleet's disk, not client integrity. The export is read-only + single-namespace = inherently small attack surface.

• Evaluate kernel nvmet vs SPDK for the target: kernel nvmet keeps the data path in the hardened/fuzzed kernel with a tiny userspace footprint (arguably a better attack-surface story); SPDK is a userspace C daemon (current choice, needs hugepages + caps). Decide per environment.
• Confine the target (whichever): unprivileged, drop caps, seccomp + SELinux/AppArmor profile, read-only container, minimal mounts; minimal immutable serving image, digest-pinned.
• Slim the SPDK image (multi-stage build): the current single-stage from-source image is ~600 MB (it keeps the whole toolchain — gcc, meson/ninja, kernel headers, the python venv — plus the SPDK/DPDK source and build trees). Convert serving/spdk/Dockerfile to a builder stage + a minimal runtime stage that copies only nvmf_tgt + its ldd shared libs (libnuma, libssl/crypto, libuuid, libaio…) + rpc.py/python3. Expect ~150–250 MB (debian-slim base or a static SPDK link trims further). Smaller attack surface + faster pulls, and it pairs with the confinement above.
• Network-isolate :4420 so only the boot VLAN reaches the target (compose with 802.1X/MACsec segmentation in prod).
• Replicas behind distinct discovery entries (availability is the real risk — it's the fleet's disk). Ties into #6 (rolling updates) availability.
• Revisit a memory-safe target only if/when an off-the-shelf one matures (e.g. a future Rust-for-Linux nvmet).

[ ] 8. Return to secure / production posture (un-toggle the lab flags)

Consolidates the cleanup of the temporary lab toggles (see "Current build posture" at the top) and the secure-path bench-verifies deferred while testing the open-network path. Do these together, then run the full secure boot-chain smoke test.

• Re-enable 802.1X/MACsec: disable_dot1x=false (CI DISABLE_DOT1X="false"). Removes rd.dot1x=0/ip=dhcp; the hook owns the link (macsec0).
• NVMe discovery by DNS: nvmet_traddr=nvme.fleet.lab (CI), resolved at boot by the serving dnsmasq — drop the hardcoded 192.168.50.10.
• Signing: wire sb_key/sb_cert (CI/HSM secrets) for a signed UKI; enable Secure Boot on targets. Until then the loud unsigned warning must stay. Key generation + enrolment procedure: secure-boot-keys.md (the PK/KEK/db trio; db = sb_key/sb_cert, PK/KEK stay offline).
• PXE-only iPXE fallback — BENCH-VERIFIED on the EliteBook 820 G4 (2026-06-20). Firmware without UEFI HTTP boot chainloads serving/tftp/snponly.efi → renders autoexec.ipxe (ifopen + DHCP-retry + imgargs <real cmdline>) → UKI over HTTP → NVMe/verity root. dnsmasq tags pxe/ipxe/httpboot give graceful per-client fallback; HTTP-boot hosts are unaffected (firmware → UKI direct). Gotchas that cost real time (all captured in ../serving/tftp/README.md): (i) the I219-LM needs snponly.efi (native iPXE driver won't bind); (ii) PortFast/edge on the switch port is REQUIRED — iPXE's PHY bounce triggers STP forwarding delay that drops DHCP (this was the real blocker); (iii) iPXE aborts the script on any unguarded command failure (use ||); (iv) Secure Boot OFF makes systemd-stub take iPXE's load options as the cmdline, so the entrypoint extracts the UKI's real .cmdline and feeds it via imgargs.
• Sign iPXE for the PXE-only fallback (Secure Boot, production). The lab snponly.efi is UNSIGNED → that path needs Secure Boot OFF. For production: sbsign the iPXE binary with the fleet db key (only db needed — no MS CA / shim), so SB stays enforced firmware → iPXE → UKI; then the cmdline contraption becomes a no-op (the stub ignores loader cmdline under SB). See secure-boot-keys.md "PXE-only hosts".
• Bench-verify on the secure path (swtpm + hostapd + FreeRADIUS, disable_dot1x=false) — the items deferred this session:
  • NM port secure path: en* UNMANAGED + macsec0 MANAGED/renewing, MKA survives switch-root, NVMe stays up past lease (item #1 risk).
  • NTP via DHCP option 42 reaches chrony over macsec0 (chronyc sources).
  • Plymouth fleet theme: test-path render VERIFIED 2026-06-18; still confirm the auth→macsec rows reach done live on the secure path.
  • STIG remediation didn't break the secure path; reproducible verity hash; triage the generated stig-deviations.md REVIEW rows (item #2).
• Secure-path NVMe connect trigger: the test path uses nvmf-connect.service (gated rd.dot1x=0); confirm the secure path's connect (dracut nvmf over macsec0) actually fires, or give it an equivalent trigger. Apply the same -k 30 -l -1 -c 10 keep-alive/reconnect flags to the secure nvme connect as the test path has.
• Future hardening waived for now (from the STIG register): FIPS, firewall, per-host/rotatable admin (SSH CA / config-agent).
• TLS-encrypted audit/log offload (production). The lab forwards logs to serving/audit-collector in plaintext, and the STIG TLS-offload rules (rsyslog_encrypt_offload_defaultnetstreamdriver, ..._actionsendstreamdrivermode, ..._actionsendstreamdriverauthmode) are waived in the tailoring because applying them forces gtls + x509 peer auth + a CA file and breaks the plaintext 2f forward. For production, do it properly and DROP the waivers:
  1. Stand up a CA (or reuse the boot-infra CA); issue the collector a server cert (CN/SAN = audit.fleet.lab).
  2. Configure the collector's rsyslog for TLS: imtcp with StreamDriver.Name="gtls", StreamDriver.Mode="1", server cert/key + CA; listen on the syslog-TLS port (6514) — add a TLS receiver to serving/audit-collector/rsyslog.conf + compose.
  3. Bundle the CA into the image (host-generic, no per-host state — same model as the RADIUS CA) and point $DefaultNetstreamDriverCAFile at it.
  4. Set the 2f omfwd action to StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" + the CA file, target port 6514.
  5. Re-select the three rsyslog_encrypt_offload_* rules (remove from files/stig/diskless-tailoring.xml) — they should then PASS, and update docs/stig-deviations.md #6 to reflect encrypted offload. (RELP+TLS via omrelp/imrelp is the alternative if guaranteed delivery is wanted alongside encryption.)

[ ] 9. (OPTIONAL / nice-to-have) Convert provisioning to Ansible where it pays off

Low priority — only the config-management half of the build is a candidate, and the ROI is modest because the build VM is single-shot (idempotency buys little) and reproducibility/auditability favour minimal shell. Not a wholesale rewrite: the load-bearing logic (dracut module hook ordering, extract-artifacts.sh, build-oci.sh) is initramfs-runtime / host-side build tooling, not provisioning, and stays shell.

The high-ROI, low-risk slice if/when we pick this up:

• STIG via the SSG Ansible playbook (ssg-rl9-playbook-stig.yml, shipped with scap-security-guide) instead of oscap --remediate — the upstream path and far more reviewable (you can diff exactly what each rule changes). Keep oscap eval (no remediate) for the report + the 0-eval guard (item #2).
• Restyle the brittle imperative bits as declarative tasks: chrony.conf edit (lineinfile/blockinfile vs sed), sshd drop-in / NM dispatcher / kiosk file installs (template/copy), datastream-by-os-release as assert. Bonus: sidesteps the %{...}→%%{ HCL-escaping footgun.

Tips the balance toward "do it" only if: we start managing the serving hosts (dnsmasq/nginx/SPDK/chrony/audit-collector) with config-management too (shared roles to amortise — currently they're docker-compose), provisioning grows much more branchy, or we want molecule-tested hardening roles validated without a full ~17-min Packer build.

[ ] 10. (OPTIONAL) Secure Boot cert auto-enrollment for provisioning

Speed up bringing up new workstations: bake the public PK/KEK/db certs into an image, detect Secure Boot Setup Mode at boot, and auto-enroll them so a fresh/reset box adopts the fleet trust root without a manual firmware-setup visit. Pairs with #8 (signing) + secure-boot-keys.md.

Tooling gap (found 2026-06-20): efitools is NOT packaged for EL9 — no efi-updatevar/efi-readvar/KeyTool. Resolved for the OS-side path by writing the signed .auth variables to efivarfs directly (de-immutable + single write), which needs no extra package. Done 2026-06-20: baked /usr/local/bin/fleet-sb-enroll (efivarfs method, Setup-Mode-gated, fapolicyd- trusted) + the public PK/KEK/db.auth at /usr/share/fleet-secureboot/ (supplied via files/secureboot/keys/ or SB_*_AUTH_B64 CI secrets); mokutil covers reading. So manual OS-side enrollment over SSH now works (sudo fleet-sb-enroll). What's left for auto-enroll: drive fleet-sb-enroll unattended on a Setup-Mode detection at boot, gated to a provisioning mode/UKI (below), with read-back verify + audit — NOT armed in the production image.

What's safe: the certs are public (no key material — only the db private key signs, and it stays in CI/HSM, invariant #7); Setup Mode accepts unauthenticated writes to PK/KEK/db, so public certs (ESL) are sufficient. It's fleet-generic (no per-host state, invariant #1) and deterministic (reproducible verity hash).

The real risk — provisioning-time TOFU (trust-on-first-use): in Setup Mode Secure Boot isn't enforcing, so the enroller runs as unverified code, and the fleet's UEFI HTTP boot is unauthenticated in that state. An on-path attacker during provisioning can serve a malicious image that enrolls their PK/KEK/db — the box then "has Secure Boot" but trusts the attacker. Auto-enroll is only as trustworthy as the provisioning channel, and it removes the physical-presence approval gate. So treat it as a controlled-environment provisioning step, not an always-on behaviour.

Optional separate provisioning UKI (preferred shape): build a dedicated provisioning UKI that carries the certs + the enroll logic, instead of arming the enroller in the production image. Keeps the standing footgun out of the normal image (a factory reset + tampered image can't silently re-root trust), and lets the enroller be a one-shot, attended/isolated step. The production UKI never enrolls.

• Build it as a separate Packer target / flag (e.g. provisioning=true) → its own small UKI; signed or unsigned (it runs in Setup Mode anyway, but sign it so it also boots once keys are enrolled / for re-runs).
• Or a tiny standalone EFI tool (KeyTool-style) served alongside, not a full OS.

Mitigations (do these regardless of UKI vs separate UKI):

• Run provisioning on a trusted, isolated VLAN (or a physically controlled bench) — the integrity Setup Mode can't give must come from the environment.
• Gate enrollment behind an explicit provisioning mode / one-shot token; never arm it on every boot of the production image.
• Read-back verify before declaring success: efi-readvar the enrolled PK/KEK/db and compare fingerprints to the values baked in; mismatch = fail loudly (turns silent TOFU into tamper-evidence).
• Audit the enrollment event + read-back fingerprints to the collector.
• Keep the Microsoft UEFI CA in db (or enroll your db alongside it) unless every NIC/GPU option ROM is covered by your keys — clearing MS db can break signed option ROMs (NIC PXE/HTTP-boot, GPU init), bricking the boot path.
• PCR 7 check: enrollment changes the Secure Boot measurement in PCR 7; confirm nothing the fleet seals to the TPM (the EAP-TLS key at 0x81010001, NV cert) is PCR-7-policy-bound before relying on this.

Decision before building: auto-enroll trades assurance for convenience and is reasonable for a controlled fleet; the highest-assurance path is attended/firmware or manufacturer pre-provisioning. Default recommendation: separate provisioning UKI + isolated VLAN + read-back verify.