diskless-workstation/docs/TODO.md

TODO — diskless fleet boot system

Persistent backlog, kept in sync with the working session task tracker. Status: [ ] pending · [~] in progress · [x] done.

---

Current build posture (LAB / test) — as of 2026-06-19

The image currently builds in open-network test mode. These toggles must be flipped back for a secure/production build (consolidated as item #8):

• disable_dot1x=true / CI DISABLE_DOT1X="true" → rd.dot1x=0 (NO 802.1X / MACsec; open NVMe-oF). TEMPORARY.
• nvmet_traddr / CI NVMET_TRADDR="192.168.50.10" → hardcoded IP (should be a per-env DNS name like nvme.fleet.lab once DNS-at-boot is trusted).
• Unsigned UKI (sb_key/sb_cert not passed) → Secure Boot must be OFF on the test target. Production needs CI/HSM signing. NOTE (2026-07-11, poam-hardening): the main CI now fails closed — a "Production build gate" aborts unless SB_KEY/SB_CERT/FLEET_CA are set, so an unsigned/placeholder-CA build is no longer producible via CI (only a local packer build without keys still yields the lab warning). The fleet-ipxe build is gated the same way.
• enable_kiosk=true; boot_splash=true.
• secure_ntp — DONE (2026-07-11, secure-ntp). NTP is now NTS (CA-authenticated) by default: the host verifies the serving NTP server's fleet-signed NTS-KE cert against radius-ca.pem (closes F-03/P-18; docs/secure-ntp-design.md). No baked FQDN — the NM dispatcher derives the server name from the DHCP domain at runtime (ntp.<domain>), so a site-domain change needs no rebuild. secure_ntp=false reverts to the legacy unauthenticated DHCP option-42 path. Serving needs serving/chrony/nts-cert.sh (DNS SAN ntp.<domain>), DHCP option-15 domain handout, + tcp/4460 open. BENCH-VERIFY chronyc -N authdata (NTS cookies) + $DHCP4_DOMAIN_NAME populated.
• enable_stig — DONE (2026-07-11, poam-hardening). CI ENABLE_STIG="true" and, more importantly, the Packer template now enforces enable_stig=true via variable validation — a build with it false fails packer validate/build. There is no unhardened build path anymore. (Was temporarily off during the STIG regression diagnosis; that escape hatch is gone.)
• debug_shell — REMOVED (2026-07-11, poam-hardening). The DEBUG HOLE (passwordless root shell) and its debug_shell variable are deleted from the template entirely; no build can bake it (closes POA&M P-02 / R-02). Offline diagnosis via rootfs.erofs mount + SMBIOS enforcing=0 remains the supported path (SMBIOS cmdline-extra works on Proxmox only after a full VM stop/start, not reset).
• Admin SSH key MUST be ECDSA or RSA, NOT ed25519, when STIG is on — the STIG crypto policy drops ed25519 from sshd's accepted pubkey algorithms. Set the SSH_AUTHORIZED_KEY CI secret to an ECDSA/RSA key (see #2).
• Hostname: image is hostname-neutral (empty /etc/hostname, hostname-mode=default); the serving dnsmasq's fleet-namer (--dhcp-script) assigns a stable creative starship name per MAC (e.g. nebula-vanguard-07) via DHCP option 12 + DNS A/PTR. Host picks it up on next DHCP renewal/reboot.
• Serving .env: must set AUDIT_HOST=audit.fleet.lab (+ docker compose up -d dnsmasq to reload) for remote log offload to resolve/flow.

What's working/verified vs awaiting bench is per-item below. Cross-cutting bench-verify checklist lives in CLAUDE.md → "Known bench-verify items".

---

[x] 1. Port the dracut module off network-legacy → network-manager

DONE — test path VERIFIED ON THE BENCH 2026-06-17 (host stays up past the DHCP lease time; no more lease-expiry outage). Fixed the confirmed outage: network-legacy's one-shot DHCP lease was never renewed post-switch-root, so the address dropped at lease expiry → NVMe/TCP root died ("keep alive timeout" / "no route to host"). NM hands its initramfs connections to the real-root NM (/run/NetworkManager state), which renews.

REMAINING: the secure path (macsec0 DHCP via nmcli, physical NIC unmanaged via nm-unmanaged.sh) is only exercised once disable_dot1x=false — verify on the swtpm + hostapd + FreeRADIUS bench when re-enabling security (see #2/cleanup).

Done:

• 90-diskless.conf add_dracutmodules: network-legacy → network-manager.
• module-setup.sh depends() → network-manager; added nmcli; installs the new cmdline hook.
• NEW nm-unmanaged.sh (cmdline hook): secure mode writes /run/NetworkManager/conf.d/10-dot1x-unmanaged.conf (unmanaged-devices=interface-name:en*) before NM starts; no-op in test mode. Replaces the deleted static 99-dot1x-unmanaged.conf (couldn't be mode-aware — it wrongly unmanaged en* in test mode, blocking renewal).
• dot1x-macsec-up.sh secure path: ifup macsec0 → nmcli (managed + DHCP, poll for IPv4). macsec0 MANAGED so NM renews; physical NIC UNMANAGED.
• diskless.pkr.hcl: dropped dhcp-client, added explicit NetworkManager; removed the static-config install; lsinitrd now asserts NetworkManager + nm-unmanaged.sh present.
• Static lint clean (bash -n, shellcheck, packer validate).

BENCH-VERIFY before trusting (the gate):

• Test path (disable_dot1x=true): boot, nmcli device shows boot NIC MANAGED; run past the DHCP lease time (drop the lab dnsmasq lease low) — address persists, lease renews (journalctl -u NetworkManager), NVMe + SSH stay up.
• Secure path (swtpm + hostapd + FreeRADIUS, disable_dot1x=false): en* UNMANAGED, macsec0 MANAGED + renewing; MKA survives switch-root; verity root mapped; NVMe stays connected past lease time; no "Address already assigned".
• Risk: ordering of the unmanaged drop-in vs NM start in secure mode. If NM transiently grabs en*, fall back to also baking a static en*-unmanaged file for secure-only builds.

[~] 2. Apply RHEL9 STIG hardening to the built image

IMPLEMENTED 2026-06-18 — awaiting build/bench verification. OpenSCAP + SSG remediation in a provisioner (enable_stig var, default on), tailored for the diskless/UKI/kiosk architecture via files/stig/diskless-tailoring.xml. STIG kernel params baked into the UKI cmdline (extract-artifacts.sh). Decisions: FIPS off, audit local-only, CI report-only. Deviations are documented in stig-deviations.md (curated register) and reported per-build by ../scripts/stig-deviations.sh (non-pass rules tagged intentional vs REVIEW); CI publishes stig-report.html + stig-arf.xml + stig-deviations.md. Scanner tooling is removed from the shipped image. BENCH-VERIFY: boot/login VERIFIED ON THE BENCH 2026-06-19 — hardened build with enable_stig=true, host cygnus-cruiser-99: kiosk auto-starts + renders, admin SSH (ECDSA) logs in, audit offload reaches the collector, DHCP creative hostname applied, NVMe/TCP verity root mounts and the host stays up. STILL OPEN: reproducible verity hash across two builds; triage the REVIEW rows (below); the secure path (#8, 802.1X/MACsec — this was the disable_dot1x=true test path).

FINDING 2026-06-18 (FIXED) — false-clean scan. The first hardened build (branch hardening, ada4aa4) produced a report reading "0 REVIEW / clean", but the ARF had 0 pass, 0 fail — only 462 notapplicable + 1068 notselected. Root cause: the provisioner selected the datastream with ls ssg-rhel9-ds.xml ssg-rl9-ds.xml | head -1, which picks rhel9 alphabetically; the RHEL9 datastream's CPE dictionary has no Rocky entry, so on Rocky 9.8 every selected rule went notapplicable and nothing was actually evaluated or remediated — the image was unhardened despite a clean-looking report. Fixes: (a) provisioner now maps /etc/os-release $ID → datastream (Rocky→ssg-rl9-ds.xml); (b) provisioner aborts the build if the scan evaluates 0 rules; (c) stig-deviations.sh flags an INVALID SCAN and exits non-zero on 0-evaluated. Re-run the hardened build and confirm pass/fail counts are non-zero before trusting any deviations report.

FINDING 2026-06-18 (FIXED) — hardening locked the fleet out. The first genuinely-evaluated hardened image (app 215b0eb) broke both login paths: the Chromium kiosk dropped to a getty login prompt, and admin SSH was rejected with Permission denied (publickey). Root cause: STIG enables pam_faillock (account phase) + password aging/inactivity, and the kiosk + fleetadmin accounts are passwordless / key-only, so the hardened PAM account phase treats them as expired/lockable (sshd key auth succeeds, then pam_acct_mgmt fails). Fix (provisioner 2c-STIG-postfix, runs after remediation): exempt the two appliance accounts — chage -M -1 -m 0 -I -1 -E -1 + faillock --reset; kiosk uses a dedicated /etc/pam.d/kiosk (PAMName=kiosk, permissive account / full session for seat); sshd gets an account sufficient pam_succeed_if.so user = fleetadmin first-rule bypass. Scoped to those two accounts; everyone else keeps full STIG. Justified as register deviation #10. VERIFIED ON THE BENCH 2026-06-19 (enable_stig=true, host cygnus-cruiser-99): kiosk auto-starts + renders and admin SSH (ECDSA key) logs in with hardening on.**

UPDATE 2026-06-19 — offline erofs inspection of the hardened image (mounted rootfs.erofs read-only) pinned the real causes; the PAM-account theory above was incomplete:

• Kiosk had TWO causes, both FIXED:
  1. fapolicyd is enabled and kiosk-session wasn't in its trust → it blocked /usr/local/bin/kiosk-session. Fix: KEEP fapolicyd (it's the exec-guard for the writable overlay dm-verity can't cover) and add the kiosk binary to its trust DB at build (2c-STIG-postfix). Not waived.
  2. Auto-start: kiosk was WantedBy=graphical.target but the host boots to multi-user.target (graphical never reached) → unit inactive (dead) at boot, fine on manual restart. Fix: WantedBy=multi-user.target + dropped set-default graphical.target (kiosk launches weston directly, needs no DM target).
• SSH = STIG crypto policy excludes ed25519 (CONFIRMED via ssh -vvv). The key fingerprint MATCHED (not a mismatch); the server's server-sig-algs advertised only ecdsa + rsa-sha2 — no ssh-ed25519 — so the ed25519 admin key is rejected at the signature stage (before PAM/SELinux, which is why enforcing=0 and the account-bypass didn't help). Fix (your action): set the SSH_AUTHORIZED_KEY CI secret to an ECDSA (ssh-keygen -t ecdsa -b 384) or RSA key, then rebuild. No code change. (Alt: re-allow ed25519 via PubkeyAcceptedAlgorithms +ssh-ed25519 in the sshd drop-in — but that's a non-approved algo and would re-fail a STIG rule; prefer the ECDSA key.)
• rsyslog TLS-offload rules waived (lab): rsyslog_encrypt_offload_* force gtls+x509+CA and broke the plaintext 2f forward → unselected in the tailoring; production TLS offload tracked under #8.
• Diagnosis tooling notes: SMBIOS cmdline-extra works only after a Proxmox VM stop/start (not reset); the debug-shell-on-tty2 never came up (and is now removed from the template — see the build-posture section) — the decisive diagnosis was offline rootfs.erofs mount + ssh -vvv + enforcing=0.

REVIEW-ROW TRIAGE — DONE 2026-06-19 (scan app-9f7d8d1, 21 REVIEW → 1 open). The post-fix hardened scan was valid (398 pass / 12 fail / 9 notchecked). All 21 REVIEW rows dispositioned (full detail: stig-deviations.md #12). Mechanics: a rule only stops being flagged REVIEW if it's selected="false" in diskless-tailoring.xml (the register prose alone doesn't clear it) or made to pass / N/A.

• Bucket A (waived, register-justified): encrypt_partitions, auditd_audispd_configure_sufficiently_large_partition (#1); grub2_password (#2); sysctl_crypto_fips_enabled, harden_sshd_macs_openssh_conf_crypto_policy, harden_sshd_macs_opensshserver_conf_crypto_policy (#5); configured_firewalld_default_deny (#4); sssd_enable_certmap, sssd_has_trust_anchor (#7).
• Bucket B (waived N/A): account_temp_expire_date, accounts_user_home_paths_only, libreswan_approved_tunnels, security_patches_up_to_date, ssh_keys_passphrase_protected.
• Bucket C (decisions applied): installed_OS_is_vendor_supported (Rocky IS supported; RHEL-datastream CPE artifact — waived), network_configure_name_resolution (single DHCP DNS by design — waived), sysctl_user_max_user_namespaces_no_remediation (KEEP userns for Chromium sandbox — waived), accounts_authorized_local_users (appliance accounts authorized, #10 — waived; prod: set the authorized-users regex var instead), package_postfix_installed + postfix_prevent_unrestricted_relay (postfix removed in 2c-STIG — no MTA needed; audit goes off-box via 2f), USBGuard configure_usbguard_auditbackend + usbguard_allow_hid/_hub/ _hid_and_hub (ENABLED + enforcing: LinuxAudit backend + hardware-independent HID/hub allowlist written in 2c-STIG; ⚠ BENCH-VERIFY physical console keyboard/ mouse on real hardware — the build-VM-generated policy + default block target would otherwise lock out console input).
• selinux_context_elevation_for_sudo — REMEDIATED WITH CONFINEMENT (decision 2026-06-20). 2c-STIG now rewrites the admin sudoers to TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL and maps the admin to staff_u (semanage login) so the sudo role transition is authorized. ⚠ TWO BENCH GATES: (a) admin SSH login + sudo + privileged ops must work under staff_u confinement (if it locks out, waive the offending rule in the tailoring as a justified deviation — enable_stig=false is no longer available, STIG is enforced-true); (b) reproducibility — semanage rebuilds the policy store; confirm the verity hash is stable across two builds (normalize in 2d if it drifts). Needs policycoreutils-python-utils (installed in 2c-STIG if absent).

Expected next-build report: 0–1 REVIEW (0 if both postfix→N/A and usbguard→pass land and sudo confinement holds; the triage was validated by re-running scripts/stig-deviations.sh against the app-9f7d8d1 ARF with the new tailoring — 21 REVIEW → 3, where the 3 are exactly postfix/usbguard/sudo, all addressed for the next build).

Remaining (future hardening, currently waived — see the register): FIPS, firewall, per-host/rotatable admin access. Remote audit offload — WIRED both ends (2026-06-19): receiver serving/audit-collector/ (rsyslog TCP/514, per-source-host files) + sender in the image (provisioner 2f, audit_host var: audisp-syslog→LOCAL6 + rsyslog omfwd to audit.fleet.lab). Lab is plaintext; TLS/RELP is the remaining hardening. VERIFIED END-TO-END ON THE BENCH 2026-06-19 — booted host cygnus-cruiser-99 (192.168.50.148) forwards to the collector, which writes audit-collector/logs/192.168.50.148/{audit,messages}.log. Serving-host firewall gotcha hit + fixed: the collector is host-networked so the serving host's ufw governs :514 (Docker doesn't manage iptables for host-net containers); 514/tcp was bound to the wrong NIC (ens18 instead of the boot-LAN ens19), so SYNs were silently dropped (SYN-SENT / "connection timed out", not refused). Fix: ufw allow in on ens19 to any port 514 proto tcp. The sender's action.resumeRetryCount="-1" buffered the backlog and drained on reconnect — no records lost. Documented in ../serving/audit-collector/README.md. When the workstation firewall is re-enabled (#4 / register #4) it needs the reciprocal egress allow for 514/tcp to the collector.

SIEM (Wazuh) added 2026-06-27 — serving/siem/. Upgrades the "Detect" pillar from file-collection-only to parse/search/correlate/alert. Runs on a DEDICATED node — wazuh1.bitasylum.org (managed independently), keeping the memory-hungry indexer JVM off the serving host so it can't OOM SPDK (the fleet's root disk / SPOF, #6/#7). Agentless (diskless hosts stay stateless, invariant #1): the serving-host audit-collector relays a copy to wazuh1's manager syslog listener (omfwd → wazuh1.bitasylum.org:514 in audit-collector/rsyslog.conf; rebuild the collector to apply) while still writing durable per-host files. serving/siem/ is the INTEGRATION (the <remote> syslog block to add on wazuh1 + reference deploy overlay), not a local deployment. DECODING BENCH-VERIFIED 2026-06-27 (wazuh-logtest): the audisp-syslog records carry a node=<host> prefix (STIG name_format=hostname, SSG auditd_name_format — kept, it's compliance) that breaks Wazuh's built-in auditd decoder, so a custom fleet-auditd decoder + rules (serving/siem/config/local_decoder.xml + local_rules.xml, mounted declaratively via /wazuh-config-mount) decode them into the fleet_audit group. Decode + alert confirmed: wazuh-logtest shows decoder.name=fleet-auditd + audit.type extracted (base rule 100100); and a live sudo on a booted host (cygnus-cruiser-99) surfaced USER_CMD → rule 100130 and CRED_*/USER_ACCT → rule 100110 (level 5) in the dashboard Threat Hunting view. Wazuh runs multi-node on wazuh1.bitasylum.org. REMAINING (prod): the relay→wazuh1 hop is plaintext — TLS it (gtls/6514, fleet CA; ISM protect-logs-in-transit) since it's now on the wire; rotate all Wazuh creds; restrict dashboard/API to admin hosts; set allowed-ips→/32 of the serving host; ISM retention/archival; then update the SSP "Detect" + ISM event-logging controls.

Original spec (retained for reference)

OpenSCAP + SCAP Security Guide (xccdf_org.ssgproject.content_profile_stig), run as a Packer provisioner after dnf install and before the dracut/initramfs build and cleanup.

• Tailoring (autotailor): unselect rules this architecture can't satisfy — separate-partition rules (single erofs root + tmpfs overlay; satisfy intent with nodev,nosuid,noexec tmpfs mounts), GRUB/kernel-cmdline rules (no GRUB), and AIDE/file-integrity rules (dm-verity is the documented compensating control).
• Kernel params → UKI cmdline in ../scripts/extract-artifacts.sh before signing: audit=1 audit_backlog_limit=8192 slub_debug=P page_poison=1 vsyscall=none (+ fips=1 if FIPS).
• Reproducibility guard: strip non-deterministic remediation output (SSH host keys → regen at boot into overlay, AIDE db, timestamps, fresh UUIDs) so the verity hash stays stable.
• Audit logging: audisp-remote/rsyslog to a central collector addressed by DNS name (host-generic).
• Optional FIPS: fips=1 cmdline + 01fips dracut module + FIPS crypto policy.
• CI: run oscap eval (no --remediate) to publish an ARF+HTML STIG report as a build artifact; optionally hard-fail below a score threshold.
• Decisions needed before implementing: FIPS on/off · remote-audit collector DNS name (or ephemeral/none for lab) · CI fail-on-score threshold vs report-only.
• Note: orthogonal to the rd.dot1x=0 test toggle, but a full STIG production posture wants 802.1X re-enabled.
• Credentials (partly done): root is now locked and access is via a baked sudo-enabled key-only admin user (var.ssh_authorized_key / var.admin_user, wheel + NOPASSWD sudo), with sshd drop-in PermitRootLogin no / PasswordAuthentication no. Two items for STIG to reconcile: NOPASSWD sudo is a finding (acceptable for a key-only account, but document/deviate), and the admin key/user are fleet-wide baked — for per-host/rotatable access without rebuilds, layer the signed HTTP/SMBIOS config-agent or an SSH CA (TrustedUserCAKeys) discussed in session.
• SSH host keys (done): extract-artifacts.sh strips /etc/ssh/ssh_host_* so they're not baked (baked keys = shared fleet identity + the build's perms/group don't survive the tar→erofs round-trip, so sshd rejected them "Permissions 640 too open" and refused to start). RHEL's sshd-keygen@.service regenerates them into the overlay before sshd starts. ⚠ Caveat: the ephemeral overlay means they regenerate every boot → the host key changes per reboot (clients warn). RESOLVED (F-02/P-17, bench-verified 2026-07-11 on comet-beacon-54): rather than persist a key, each per-boot key is signed into an OpenSSH host certificate by the fleet SSH CA via the enrol-broker host plane at boot (files/ssh-hostcert → fleet-ssh-hostcert); admins trust one @cert-authority line instead of TOFU. See ssh-host-certificate-design.md. Bench-surfaced fixes on branch poam-hardening (28d31e4).

[ ] 3. Roll the errata into design doc v3

Once the boot path is validated and item 1 lands, fold design-v2-errata.md into the source design document and re-export a v3 PDF (key change: rootovl → systemd.volatile=overlay in §3 step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source (not in this repo).

[x] 4. Kiosk mode — autologin + fullscreen Chromium loading a URL

WORKING (display-verified 2026-06-17). Remaining bits are polish (GPU/mesa trim, ship via CI enable_kiosk=true) and the phone-home/NTP lockdown (done) — all captured in the sub-notes below. Secure-path render check is folded into #8.

Package-footprint trim (2026-07-03): the kiosk dnf transaction was measured at 229 pkgs / 1.0 GB (weak deps ON). Added --setopt=install_weak_deps=False to the install in diskless.pkr.hcl — drops the flatpak/ostree sandbox runtime, tracker-miners (fs indexer = dead weight + threat surface), the pipewire/pulseaudio audio stack, and libproxy-webkitgtk4 → webkit2gtk3-jsc (an 8.9M second browser engine). Also added an rpm -qa --qf %%{SIZE} size-manifest diagnostic to the build log to quantify the result. The immovable floor is chromium + chromium-common + llvm-libs (mesa sw-GL) + gtk3/adwaita + libicu — all HARD deps. BENCH-VERIFY: with weak deps off, confirm the kiosk still renders (no genuine runtime dep was a weak dep — watch for a missing font or libva). NEXT trim candidates (measure via the new manifest): gstreamer/media codecs if the SPA plays no HTML5 media; mesa-dri-drivers + --use-pixman-only if the GPU path is abandoned.

WORKING / display-verified 2026-06-17 (kiosk renders Chromium fullscreen on the configured URL). Build-flagged OFF by default for now (enable_kiosk=false) while iterating on boot/network — set -var enable_kiosk=true to ship it.

Path to getting here (don't re-discover):

• cage is NOT packaged for EL9 (dnf install cage → "Unable to find a match"). Tried weston instead.
• EPEL 9 ships weston 8.0, which has NEITHER kiosk-shell.so NOR the [autolaunch] weston.ini section — both arrived in weston 9.0. The first cut wrongly assumed 8.0 had them; shell=kiosk-shell.so failed to load. rpm -ql weston shows only desktop-shell / fullscreen-shell / ivi-shell.
• Resolution: desktop-shell.so (the only packaged shell Chromium's xdg-shell works with — fullscreen-shell needs a client protocol Chromium doesn't speak), stripped bare (panel-position=none, black bg, no animation/lock), and Chromium launched by a wrapper (kiosk-session) since there's no [autolaunch]. Chromium --kiosk fullscreens over the shell.
• Seat fix (load-bearing): weston died with logind: failed to get session seat → failed to create compositor backend. logind's auto-derivation didn't attach a seat; Environment=XDG_SEAT=seat0 + XDG_VTNR=1 in the unit (put into the PAM env before pam_open_session) registers the session on seat0/vt1. /dev/dri/card0 was present all along (not the issue).
• Tested live without a rebuild: the whole root is systemd.volatile=overlay, so /etc AND /usr are writable tmpfs at runtime — drop-in the files + restart.
• enable_kiosk Packer var — currently false by DEFAULT (set 2026-06-17). The whole kiosk provisioner is feature-flagged off while iterating on the boot/network stack (DHCP-lease/keep-alive work), for faster builds and a smaller image. Re-enable with -var enable_kiosk=true when returning to kiosk work — CI does not pass the var, so CI currently builds lean (no kiosk) too; add -var enable_kiosk=true to the Gitea workflow to ship it.
• kiosk_url Packer var (default http://boot.fleet.lab/kiosk/ — the kiosk SPA (serving/kiosk-app/, Svelte 5 + Tailwind v4, a cyber sci-fi spaceship monitoring/control terminal) served by the boot HTTP server, which reverse-proxies /kiosk/ to the kiosk container; self-contained so a boot validates without external connectivity. Empty disables the whole install). Wired in CI as KIOSK_URL env.
• Kiosk SPA (serving/kiosk-app/) — replaced the old single static serving/http/kiosk/index.html (2026-06-18). Top navbar + 5 pages (Bridge/Reactor/Nav/Life Support/Comms) with dummy telemetry; hash routing; builds in a 2-stage Docker image (Vite → nginx), reverse-proxied at /kiosk/ by the main serving nginx (loopback 127.0.0.1:${KIOSK_APP_PORT:-8088}). npm run build verified locally; container build untested (no docker here). Dummy data lives in src/lib/data.js (swap for a fleet API later).
• Provisioner enables EPEL, installs weston chromium mesa-dri-drivers mesa-libEGL mesa-libgbm dejavu-sans-fonts, creates a passwordless unprivileged kiosk user (video/input groups).
• files/kiosk/kiosk.service — runs /usr/local/bin/kiosk-session as kiosk on tty1 via PAMName=login+TTYPath+Environment=XDG_SEAT=seat0/XDG_VTNR=1 (gets seat0 for DRM/input), Conflicts=getty@tty1, WantedBy=graphical.target; default target set to graphical.
• files/kiosk/weston.ini — shell=desktop-shell.so, idle-time=0, require-input=false, xwayland=false; [shell] panel-position=none, black background-color, animation=none, locking=false.
• files/kiosk/kiosk-session — execs weston --use-pixman --socket=wayland-kiosk in the foreground (owns the seat), and launches Chromium (--kiosk --ozone-platform=wayland --disable-gpu, tmpfs profile/cache, URL from /etc/kiosk/kiosk.env) in a background crash-restart loop.
• Remaining polish (now that it renders): dropping --disable-gpu on real GPUs, /dev/dri present in the VM (weston needs a DRM device — set Display to VirtIO-GPU/std), and SELinux AVCs for the GUI stack (check ausearch -m avc; the kiosk user is unconfined so likely none — but do NOT set permissive if they appear; add a policy module). Admin SSH (the key user) still works for debug even if the kiosk fails (it lands at graphical.target either way).
• Phone-home lockdown (done 2026-06-17/18): Chromium was dialing Google (accounts/variations/optimization-guide/component+auto update/safe browsing — seen in dnsmasq logs). Two layers:
  • Flags in kiosk-session (--disable-background-networking, --disable-component-update, --disable-features=OptimizationHints,...) — confirmed killing variations/optimization-guide/update queries.
  • Managed policy files/kiosk/chromium-kiosk-policy.json for the policy-governed bits (sign-in accounts.google.com, sync). FIX 2026-06-18: the managed-policy dir differs by build, so it's installed to BOTH /etc/chromium/policies/managed/ AND /etc/chromium-browser/policies/managed/ (a policy in the wrong dir is silently ignored — the reason accounts.google.com/android.clients.google.com persisted). Verify via chrome://policy (Status: OK per entry).
  • Serving-side sinkhole (defense-in-depth, serving/docker-compose.yml dnsmasq --address=/google.com|googleapis.com|gstatic.com/0.0.0.0): the kiosk is local-only, so block egress at the resolver. Stops the leak; the query still logs (answered 0.0.0.0, not forwarded) — eliminating the query is the browser policy's job. Production: do this at the boot-LAN firewall.
  • NTP egress closed (done 2026-06-18): the *.pool.ntp.org lookups were the OS (chrony), not Chromium. Now: a chrony NTP server in the serving stack (serving/chrony/), dnsmasq hands it via DHCP option 42, and the image's NM dispatcher (files/etc/NetworkManager/dispatcher.d/10-chrony-dhcp-ntp) feeds it into chrony's sourcedir /run/chrony-dhcp; the baked public pool is stripped from /etc/chrony.conf. Result: the workstation syncs time only from the lab server — no external NTP egress. BENCH-VERIFY: on a booted host chronyc sources shows the lab server (the DHCP one), not a public pool. SUPERSEDED (2026-07-11, secure-ntp) by secure_ntp=true: the DHCP option-42 path is now the unauthenticated fallback; the default is NTS/CA-authenticated time (the DHCP source is no longer mixed in). See the build-posture note above + docs/secure-ntp-design.md.
• Image size: Chromium+Mesa add hundreds of MB to the uncompressed erofs — feeds the "shrink the rootfs" concern (and #7); reconsider whether a lighter browser/render path is viable.

Original task

Make the diskless workstation boot straight into a locked-down kiosk: a fullscreen Chromium showing a configured site (default: the boot HTTP server's /kiosk/ test page). All baked at build time into the root tree (immutable erofs/verity root; runtime is the tmpfs overlay), fleet-generic (no per-host state).

• Graphical stack (keep it minimal): add a Packer provisioner installing a lightweight kiosk compositor + browser. Preferred: cage (Wayland single-app kiosk compositor) + Mesa (software GL fallback for VMs/no-GPU); X11 + a minimal WM (matchbox/openbox) as the alternative. Chromium on Rocky 9 is in EPEL — enabling EPEL adds a repo/supply-chain surface (decision below).
• Autologin + session: dedicated unprivileged kiosk user; launch the session via a kiosk.service running cage -s -- chromium --kiosk … (or greetd + cage). Stage units/configs under files/ like the other provisioned config.
• Chromium flags: --kiosk --noerrdialogs --disable-infobars --incognito --no-first-run --disable-session-crashed-bubble <URL>. Writable profile/cache land on the systemd.volatile=overlay tmpfs (RAM-backed, ephemeral) — fine, but watch RAM sizing; pin --user-data-dir/--disk-cache-dir to a tmpfs path if needed.
• Kiosk URL = Packer var (e.g. kiosk_url, default the target site), baked into the launch command; keep it host-generic (per-env via DNS if ever needed).
• Verify in boot test: graphical session comes up on real/VM display, Chromium fullscreen loads the URL over the booted network (works with the current rd.dot1x=0 test path / open network).
• Decisions: Wayland(cage) vs X11; accept EPEL for Chromium (vs another source); confirm the target URL.
• ⚠ Conflict with item 2 (STIG): passwordless autologin and a desktop browser are STIG findings and enlarge the attack surface/image. Reconcile — e.g. a tightly-constrained kiosk session + documented deviation — when hardening lands.

[x] 5. Deploy & validate the end-to-end boot path (open-network test)

✅ DONE (2026-06-15). Full chain validated on QEMU/Proxmox: UEFI HTTP boot → initramfs → DHCP → nvmf-connect.service auto-connects the NVMe/TCP subsystem → two namespaces (data + verity hash) appear by pinned UUID → systemd-veritysetup maps /dev/mapper/root → uncompressed erofs mounts → systemd.volatile=overlay → multi-user. Booted with rd.dot1x=0 (open network) and the IP discovery address. See bench findings below for the full list of fixes. Remaining cleanup to leave the lab-test posture (now unblocked): revert the hardcoded IP to the nvme.fleet.lab DNS name, and re-enable security (DISABLE_DOT1X=false) — the secure path needs the same connect trigger over macsec0 (see ⚠ note below).

Original task: stand up serving/ (dnsmasq + nginx + SPDK) per serving/README.md, rebuild the UKI with nvmet_traddr=nvme.fleet.lab (now a Packer var — older UKIs have the literal NVMET_TRADDR placeholder and won't connect), then boot a UEFI HTTP-boot client (Secure Boot OFF, unsigned UKI; no TPM needed in rd.dot1x=0 mode) on the boot LAN.

• Pass criteria: UKI fetched over HTTP → initramfs → DHCP on NIC → NVMe/TCP namespace connected → dm-verity root mapped → systemd.volatile=overlay writable → multi-user.
• This validation gates item #1 (network-manager port) and item #3 (errata → design v3).
• Bench findings (boot debug, 2026-06-14):
  • Initramfs has no resolver (dracut writes resolv.conf only via its own DHCP path, not a manual dhclient), so the NVMe discovery address must be an IP, not a DNS name — build with -var nvmet_traddr=192.168.50.10. Old UKIs carrying nvme.fleet.lab can't connect early in boot.
  • NVMe controller indices are not stable: a transient discovery controller can take nvme0, pushing the I/O controller to nvme1. The hardcoded /dev/nvme0n1 in the cmdline was a latent brick. Fixed: namespace UUID pinned on the target (spdk-setup.sh add_ns -u f1ee7000-0000-4000-8000-000000000001) and the cmdline now references /dev/disk/by-id/nvme-uuid.<uuid> (extract-artifacts.sh). The two constants MUST stay in sync; make them version-specific for #6.
  • nvme_tcp/nvme_fabrics must be loaded before /dev/nvme-fabrics exists (dracut's nvmf module does this in a normal boot; at rd.break modprobe manually).
  • Don't double-configure the NIC in test mode. With ip=dhcp on the cmdline, dracut's network module brings the NIC up AND emits the "online" event that the nvmf module's initqueue/online hook waits on to run nvme connect-all. The rd.dot1x=0 bypass used to also ifup $NIC, which (a) raced dracut → Error: ipv4: Address already assigned, and (b) didn't fire the nvmf online hook, so the namespace never auto-connected and boot spun on /dev/mapper/root forever (manual connect-all worked, which is what masked it). Fixed: the bypass branch no longer touches the NIC; ip=dhcp owns networking. (The secure path still addresses macsec0 itself — dracut doesn't know about that interface.)
  • dracut's nvmf module never auto-connects on this distro. Its cmdline hook (92-parse-nvmf-boot-connections.sh) parses rd.nvmf.discover into /etc/nvme/discovery.conf correctly, but initqueue/online/ is empty — no autoconnect hook is registered, so nvme connect-all is never run (no attempt in dmesg) and the namespace never appears. Drivers load and /dev/nvme-fabrics exists; only the connect trigger is missing. A job placed on the main initqueue from a settled hook gets registered but is never re-executed once the loop blocks on udev events (verified: job present, manual connect-all works, namespace stays absent). An initqueue/online hook also does NOT work: this initramfs is systemd- driven ("A start job is running for /dev/mapper/root") and never sources dracut online hooks on link-up (verified: hook installs as 90-nvmf-connect.sh but never fires — no log, no stamp). Fixed: the connect is now a systemd unit (nvmf-connect.service, ConditionKernelCommandLine=rd.dot1x=0, Before=initrd-root-device.target) whose ExecStart (/usr/sbin/nvmf-connect) polls for the network then runs nvme connect-all until the namespace appears. The dracut module installs the unit + script and enables it via an initrd-root-device.target.wants symlink; diskless.pkr.hcl hard-asserts the unit is in the initramfs. Gotcha: $systemdsystemunitdir is empty in this dracut's module-setup context — inst_simple then silently drops the unit (empty dest dir). Use a ${systemdsystemunitdir:-/usr/lib/systemd/system} fallback and return 1 if the unit didn't land, so the build fails loudly instead of shipping a UKI that can't connect its root. Second gotcha: the build assertion lsinitrd … | grep -q 'nvmf-connect.service' was a false negative — grep -q exits on first match, closes the pipe, lsinitrd dies of SIGPIPE, and under pipefail the pipeline reports failure even though the unit IS present. This masked the fact that the systemd unit install was working all along. Fix: lsinitrd > file; grep -q … file (no pipe). Never lsinitrd | grep -q under pipefail. Bonus: the survivor listing confirmed nvme-cli's native autoconnect bits ARE in the initramfs (/usr/sbin/nvmf-autoconnect.sh, /etc/udev/rules.d/95-nvmf-initqueue.rules) — so the fully-native rd.nvmf.discover autoconnect path is available as a fallback if the unit doesn't fire on boot.
  • dm-verity: hash-offset is unsupported → two namespaces. With NVMe connecting and the right by-id device present, verity still failed: systemd-veritysetup logged unknown option 'hash-offset=…' then Failed to load verity superblock — exactly the CLAUDE.md bench-verify item. The in-file (data+hash in one namespace, split by offset) layout is not supported by Rocky 9.8 systemd. Fixed via the documented fallback: extract-artifacts.sh now emits rootfs.erofs (data) + rootfs.hash (hash tree) as separate files, served as two NVMe namespaces with distinct pinned UUIDs (f1ee7000-…-01 data, …-02 hash); the cmdline uses systemd.verity_root_data=/systemd.verity_root_hash= by-id devices with no offset. Touches extract-artifacts.sh, build-oci.sh (artifact list + SHA256SUMS), serving/spdk-setup.sh (2nd bdev/namespace), and invariant #9.
  • SELinux labels must survive into the erofs. It booted to multi-user but root/packer-build-only login was accepted then the session was killed instantly. Cause: kickstart sets selinux --enforcing, but the build dropped security.selinux xattrs (virt-tar-out + plain tar -xpf don't carry them), so the enforcing target denied the login session on an unlabeled root — and a read-only erofs can't autorelabel. Confirmed by booting with enforcing=0 (SMBIOS-injected) → login worked. Fixed: export via guestfish tar-out … selinux:true xattrs:true acls:true, restore with tar --xattrs --xattrs-include='*' --acls --selinux, and a getfattr build-time assertion that /usr/bin/bash is labeled before mkfs.erofs (mkfs.erofs preserves xattrs). Needs attr on the build host. The image now boots enforcing.
  • mkfs.erofs --all-root was flattening all ownership to root:root. Mode bits + (now) xattrs survive the export, but --all-root forced every inode to uid/gid 0 — discarding group ownership (ssh_keys on host keys → sshd "permissions 640 too open" → refused; tty/crontab setgid groups) and user ownership (home dirs). Fixed: dropped --all-root so real ownership is preserved from the tree. That now requires extract-artifacts.sh to run as root (added an id -u guard; tar -x needs CAP_CHOWN to restore ownership) — the post-processor runs it via sudo and chowns the output files back to the build user so build-oci.sh (rootless buildah + its own login) still works. Combined with the host-key strip above, sshd starts cleanly. ⚠ The secure path has the same dependency on the (absent) nvmf autoconnect — when 802.1X is re-enabled, the macsec0 path must likewise drive connect-all (or the network-manager port #1 must wire a working autoconnect). Currently only the test path is fixed.
  • The target Rocky 9 kernel's erofs is built without CONFIG_EROFS_FS_ZIP — mounting a compressed image fails erofs: lz4 algorithm isn't enabled, which silently blocks the root mount. Fixed: extract-artifacts.sh now builds the erofs uncompressed (mkfs.erofs without -z). Tradeoff: larger rootfs namespace + more NVMe/TCP bandwidth per read. Re-enable compression only against a kernel verified to have EROFS_FS_ZIP (grep CONFIG_EROFS /boot/config-$(uname -r)). This is the CLAUDE.md bench-verify item ("erofs module availability on the target RHEL kernel"). Larger image size also feeds the "shrink the rootfs" angle of #7.
  • SPDK: add the listener to the discovery subsystem explicitly (nqn.2014-08.org.nvmexpress.discovery) to silence the deprecation warning.

[ ] 6. Rolling fleet updates (multi-version serve → cohort reboots)

Image-based, reboot-to-update across many live workstations. Hosts are immutable (read-only erofs/verity root over NVMe/TCP, ephemeral overlay), so updates are atomic A/B reboots, never in-place. The released OCI artifact + digest-pinned manifest + roothash coupling already give the atomic unit and instant rollback; this item builds the rollout machinery around it.

• Serve versions side-by-side (must not overwrite a live rootfs): extend serving/ so SPDK/nvmet exposes each release as its own subsystem/NQN (nqn…:rootfs-<ver>) and nginx serves multiple fleet-<ver>.efi (keep current + previous). Make the UKI's NQN version-specific so each signed UKI connects only its matching namespace (constant discovery DNS name).
• Version selection at boot: dnsmasq cohort tagging (MAC / vendor class / assigned tag) hands each cohort a different option-67 UKI URL. Moving a cohort's URL = assigning the release.
• Control plane + in-image reboot agent: a baked-in systemd timer polls a "desired version" endpoint and reboots when idle / in a maintenance window (or out-of-band via Redfish/BMC). Stagger + jitter reboots to avoid UKI-fetch / NVMe-connect storms.
• Staged rollout: canary → waves (10/50/100%) with health gates (multi-user reached, NVMe connected, dm-verity mapped, kiosk URL rendered, prod: MACsec/802.1X up); auto-halt on threshold.
• Rollback: repoint cohort URL to previous UKI + reboot (current+previous stay served); security rollback via dbx-revoke / TPM monotonic counter (design §6).
• Drain & retire an old version only when SPDK/nvmet reports zero connected controllers on it. Drain accuracy: account for hosts powered off mid-rollout — an assigned-but-offline host will reconnect to a version that may be gone, so don't remove a version while any host that could return is still assigned to it.
• Availability: run NVMe/TCP target replicas behind distinct discovery entries (design §7) — never serve the whole fleet from one unreplicated node, especially mid-rollout (N versions + reboot bursts).
• Note: kiosk content changes are server-side (the website) and need no fleet update — only OS/UKI/rootfs changes require a reboot.

SPDK restart / availability is the load-bearing constraint (analysis 2026-06-25)

SPDK sits in the live data path of every running host's root fs, so a process restart isn't like restarting a stateless service — it tears down every NVMe/TCP association at once (fleet-wide, NOT version-scoped). The rollout design must be built around never restarting the process that holds live connections.

• Core principle: a host changes root version only by rebooting (no live swap). So: version transitions = reboots; SPDK changes = additive/online; the process stays up. Routine rollouts never restart SPDK.
• Online add/remove via JSON-RPC, no restart. nvmf_tgt is fully runtime-configurable (bdev_aio_create, nvmf_create_subsystem, nvmf_subsystem_add_ns, ..._add_listener). Add vY's namespaces live (existing vX connections untouched); remove vX live once drained. Per-version subsystem (distinct NQN) matters here: adding vY as a new subsystem never disturbs the subsystem vX hosts are attached to — adding a namespace to a shared subsystem fires a namespace-change AER + rescan at running hosts. Per-version subsystem also gives clean per-release drain + per-release hostNQN allow-listing.
• What a connection loss does to a running host: root is read-only erofs + tmpfs overlay. tmpfs writes (RAM) + page-cached reads keep working; cold reads block. The nvme-tcp host driver reconnects (reconnect_delay) until ctrl_loss_tmo. Fast SPDK restart (same NQN/UUID/addr within the window) → host survives with a stall; long/changed/expired → root I/O errors, host needs a reboot ("keep alive timeout / no route to host"). Read-only + content-addressed makes reconnect unusually clean (no write/reservation state).
• Cheap host-side mitigation: tune ctrl_loss_tmo (even -1 = retry forever)
  • reconnect_delay at nvme connect time (dracut nvmf step). For a diskless box "hang and recover" beats "die" — there's nothing better to fail to.
• Structural answer — HA + NVMe-oF multipath (ANA): run ≥2 targets serving byte-identical, same-UUID namespaces; hosts multipath across them, so one target can be restarted/upgraded while the other serves (no root loss). The read-only, content-addressed root makes replication trivially coherent (no write-coherence problem) — this architecture is ideally suited to it. Wrinkle: the immutable UKI cmdline points at one discovery address, so multipath needs discovery to return multiple paths (or a DNS name resolving to several targets).
• Planned SPDK restarts only via drain/failover (multipath fail-over, or a maintenance window with generous ctrl_loss_tmo so hosts recover not die). There is no zero-impact single-node SPDK restart.
• [x] Long-lived target + online release control — IMPLEMENTED 2026-06-27. The old gap (SPDK ran spdk-setup.sh as entrypoint, single-version, configured-at-start → changing versions = container restart = drop ALL connections) is closed. Now in serving/spdk/:
  • spdk-target.sh — persistent entrypoint: starts nvmf_tgt ONCE (release-agnostic), creates the TCP transport + discovery listener, then reconciles every staged release. Never restarted to change versions; on (re)start re-adds everything under ./releases/ (a serving-host reboot restores all served versions).
  • spdk-release.sh — JSON-RPC control: add/remove/list/ drain-status/reconcile, one subsystem per version (distinct NQN, so adding/removing one never disturbs others' connections), drain-gated removal (refuses while a version has connected controllers unless --force).
  • spdk-ctl.sh — host ops: load/unload/list; stages a release's artifacts from its OCI image into ./releases/<ver>/ and drives the live target. Rolling update = load new → reboot hosts → unload old once drained.
  • spdk-loader.sh — DECLARATIVE rolling updates (the "init container that stays up", compose service spdk-loader). On compose up after an IMAGE_VERSION bump it stages + adds the new rootfs to the running target ONLINE, then polls and auto-reaps any non-current version once it has drained (0 conns) for a sustained grace window (REAP_GRACE_SECONDS, keep > host ctrl_loss_tmo). So migration = bump IMAGE_VERSION in .env + docker compose up -d (compose recreates the loader + the http/dnsmasq UKI servers, but NOT the nvmeof target → no dropped connections). Single served UKI ⇒ a 0-conn old version is genuinely orphaned (a returning host fetches the current UKI), so auto-reap is safe; removal is still drain-gated + grace-debounced. The loader + target share the RPC socket on a named volume (nvmf_tgt -r); the loader reuses the SPDK image (has python3 + rpc.py).
  • BENCH FINDING 2026-07-03 (FIXED) — hosts did connect-all, so NO version ever drained → auto-reap could never fire. Observed: spdk-release list showed more controllers than workstations, with every box appearing on every served version's subsystem (e.g. 1/3/3/1/3 across 5 versions for ~3 boxes). Root cause: the initramfs connected with nvme connect-all (test path nvmf-connect.sh; secure path delegated to the dracut nvmf module's discovery autoconnect), which attaches EVERY subsystem in the discovery log — the host then mounts only its own by-UUID verity namespaces but leaves a live controller on all the others. So each running host pinned every version above 0 conns, and spdk-loader's "auto-reap once drained (0 conns)" could never trigger while the fleet was up — the served-version list grew without bound. This defeats the whole drain/reap model, not just cosmetics. Fix: targeted connect. The per-version subsystem NQN (SUBNQN, already in verity.json .subnqn) is now also baked into the signed UKI cmdline as rd.nvmf.nqn= (extract-artifacts.sh), and both connect paths use nvme connect -n <nqn> — nvmf-connect.sh (test) and a new explicit step 6 in dot1x-macsec-up.sh (secure, replacing the module delegation; runs after macsec0 is addressed per invariant #2). Legacy UKIs without rd.nvmf.nqn fall back to connect-all (with a warning) so they still boot. Now each host = 1 controller on its own subsystem → versions drain to 0 when their hosts move off → auto-reap works; connections scale O(hosts) not O(hosts × versions). Requires a rebuild + re-sign (cmdline change, invariant #6). (a) TEST PATH BENCH-VERIFIED 2026-07-03 (commit 0bfe872): builds, deploys, and the host mounts ONLY ONE image (its own version's subsystem) instead of connect- all'ing every served version — the targeted nvme connect -n rd.nvmf.nqn works. So each host now = 1 controller on its own subsystem, which is what lets old versions drain to 0 and spdk-loader auto-reap. Still to confirm: the reap payoff itself (migrate a host, watch its old version drain to 0 and get reaped); (b) SECURE PATH — the new step-6 connect brings up the NVMe root over macsec0 (this path is not yet bench-confirmed end-to-end); (c) that the dracut nvmf module isn't ALSO connect-all-ing on the secure path (if it is, suppress it — the targeted connect is meant to be the sole connector).
  • BENCH FINDING 2026-07-03 (FIXED) — the target WAS still recreated on an IMAGE_VERSION bump. Despite the above, the nvmeof (spdk-target) service still carried a versioned type: image bootstrap mount (source: ${OCI_IMAGE}:${IMAGE_VERSION} → /artifacts) and an IMAGE_VERSION env, so bumping the version changed THIS service's config and compose up -d recreated it → nvmf_tgt restarted → every connected host's NVMe/TCP root dropped (the exact fleet-wide outage the long-lived-target design exists to prevent). Fix: made the target version-agnostic — removed the /artifacts image-volume + IMAGE_VERSION env from nvmeof in serving/docker-compose.yml. Bootstrap AND rolling updates are now solely the spdk-loader's job (stages the current release into ./spdk/releases, adds it online); spdk-target.sh's seed block is guarded ([ -s /artifacts/verity.json ] && [ -n IMAGE_VERSION ]) so it no-ops when unmounted. Verified nvmeof no longer references IMAGE_VERSION (only spdk-loader does). BENCH-VERIFY: bump IMAGE_VERSION + up -d with a host connected → nvmeof container NOT recreated (docker ps ID/uptime unchanged), host root survives, loader adds the new version online.
  • No jq dependency: spdk-release.sh parses RPC JSON with python3 (the SPDK image ships no jq); legacy artifacts (no per-version UUIDs in verity.json) fall back to the fixed UUID pair with a warning.
  • Per-version namespace UUIDs (scripts/extract-artifacts.sh): derived deterministically (UUIDv5 of image_version), baked into the UKI cmdline + verity.json (now also carries subnqn); the serving side reads them back from each release's verity.json. Required so two versions don't collide on the same /dev/disk/by-id/nvme-uuid.<uuid> host symlink.
  • Bootstrap unchanged: docker compose up with one version still works — the /artifacts image volume is seeded into ./releases/${IMAGE_VERSION}.
  • PARTIALLY BENCH-VERIFIED 2026-06-28: long-lived target boots a host end to end — UKI served over HTTP → host boots → NVMe/TCP root connects to the persistent nvmf_tgt (single version spdk-permanence-02ffd7d). Surfaced + fixed several real bugs along the way: bootstrap seed must COPY not symlink (a symlink into the version-pinned /artifacts re-pointed a stale release at new data); spdk-release parses RPC JSON via python3 (SPDK image has no jq); legacy verity.json (no per-version UUIDs) falls back to the fixed pair; mislabeled-dir guard (dir name vs verity.json image_version, case-insensitive); dropped the redundant discovery-listener re-add (daemon logged "Listener already exists"). CI lowercases image_version (gitea-main-ci vars step) — a mixed-case branch (SPDK-permanence) had baked a capital fleet-SPDK-…​.efi the lowercase boot URL couldn't fetch (iPXE "no such file"); booted via a local retag workaround pending a lowercase rebuild.
  • STILL BENCH-VERIFY: (a) load a 2nd version while a host is connected to the 1st → 1st's connection survives (no AER/reset on the running host); (b) unload/auto-reap drain-gate blocks while CONNS>0 and reaps after grace on a LIVE host; (c) serving-host reboot reconciles all staged versions; (d) per-version UUID build reproducible (two builds, same version → identical data_ns_uuid/hash_ns_uuid + root_hash); (e) the full fix on a clean LOWERCASE CI rebuild (no retag). See serving/spdk/README.md.
• Capacity: each concurrently-served version costs target memory/hugepages + file-backed bdev overhead — plan for N live versions (current + previous + canary) during long rollouts (raise HUGEPAGES).
• Attach gating + discovery: with several subsystems advertised, the dracut connect + per-subsystem hostNQN allow-list must ensure a vX host attaches only vX's namespace (the by-UUID verity ref protects what mounts, but gate the attach too); rd.nvmf.discover selection must be deterministic, not connect-all.

[ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience)

There is no production-grade memory-safe NVMe-oF target — the Rust NVMe work is all driver/initiator-side (vroom, RedLeaf, Rust-for-Linux); the only production targets are C (Linux kernel nvmet, SPDK). So this is serving-node hardening, not a language swap. dm-verity already removes the target from the clients' integrity TCB (verify-on-read rejects tampered blocks), so the risk is RCE/DoS on the node that holds the fleet's disk, not client integrity. The export is read-only + single-namespace = inherently small attack surface.

• Evaluate kernel nvmet vs SPDK for the target: kernel nvmet keeps the data path in the hardened/fuzzed kernel with a tiny userspace footprint (arguably a better attack-surface story); SPDK is a userspace C daemon (current choice, needs hugepages + caps). Decide per environment.
• Confine the target (whichever): unprivileged, drop caps, seccomp + SELinux/AppArmor profile, read-only container, minimal mounts; minimal immutable serving image, digest-pinned.
• Slim the SPDK image (multi-stage build): the current single-stage from-source image is ~600 MB (it keeps the whole toolchain — gcc, meson/ninja, kernel headers, the python venv — plus the SPDK/DPDK source and build trees). Convert serving/spdk/Dockerfile to a builder stage + a minimal runtime stage that copies only nvmf_tgt + its ldd shared libs (libnuma, libssl/crypto, libuuid, libaio…) + rpc.py/python3. Expect ~150–250 MB (debian-slim base or a static SPDK link trims further). Smaller attack surface + faster pulls, and it pairs with the confinement above.
• Network-isolate :4420 so only the boot VLAN reaches the target (compose with 802.1X/MACsec segmentation in prod).
• Replicas behind distinct discovery entries (availability is the real risk — it's the fleet's disk). The target is a fleet-wide SPOF in every running host's root data path; a process restart drops all NVMe/TCP associations at once. Structural fix = HA + NVMe-oF multipath (ANA) over byte-identical, same-UUID, read-only namespaces (replication is coherent precisely because the root is read-only + content-addressed); cheap mitigation = generous host ctrl_loss_tmo/reconnect_delay. See the "SPDK restart / availability" analysis under #6 (rolling updates) — same constraint, fuller treatment.
• Revisit a memory-safe target only if/when an off-the-shelf one matures (e.g. a future Rust-for-Linux nvmet).

[ ] 8. Return to secure / production posture (un-toggle the lab flags)

Consolidates the cleanup of the temporary lab toggles (see "Current build posture" at the top) and the secure-path bench-verifies deferred while testing the open-network path. Do these together, then run the full secure boot-chain smoke test.

• Re-enable 802.1X/MACsec: disable_dot1x=false (CI DISABLE_DOT1X="false"). Removes rd.dot1x=0/ip=dhcp; the hook owns the link (macsec0).
  • Bench harness scaffolded 2026-06-22: serving/dot1x-bench/. step-ca (fleet CA, on the serving host) signs each host's TPM-generated CSR
    • issues the RADIUS server cert; FreeRADIUS does EAP-TLS (trusts step-ca root, returns MSK); a hostapd wired authenticator does 802.1X + MACsec on an isolated bridge per workstation (flat VLAN can't auth — EAPOL dropped + MACsec is P2P). enrol/tpm-provision.sh creates the key in the TPM at 0x81000100 + writes the cert to NV 0x01800100. Starting solution signs a CSR (key stays in TPM); TPM attestation (ACME device-attest-01) is the documented upgrade on the same CA. BENCH-VERIFY items flagged in the files: hostapd MACsec build, openssl tpm2 provider vs pkcs11 key ref, EAP-TLS version. PKI product decision (greenfield): start step-ca; OpenBao/EJBCA/Dogtag later.
  • First step (DONE 2026-06-22): enrol/tpm-selftest.sh — non-destructive TPM capability check (run on the booted open-port image, no disable_dot1x needed) to de-risk the TPM half before the full path. Likely first finding: no openssl TPM2 provider in the image (tpm2-tss ≠ the provider) — add tpm2-openssl.
  • Enrolment of the per-host identity (key@0x81000100 + cert@NV) is its own design — see #11 (TOFU + OOB fingerprint broker, pre-auth permit-set, provisioning UKI). tpm-provision.sh is the manual bench stand-in for it.
• NVMe discovery by DNS: nvmet_traddr=nvme.fleet.lab (CI), resolved at boot by the serving dnsmasq — drop the hardcoded 192.168.50.10.
• Signing — SIGNED-UKI + SECURE-BOOT-ON PATH BENCH-VERIFIED on the Proxmox VM (HTTP boot, 2026-06-20). CI signs the UKI when SB_KEY/SB_CERT secrets are set (db key); fleet-sb-enroll enrols db→KEK→PK from the baked public .auth files; OVMF then boots the signed UKI under enforcing Secure Boot. Confirmed end-to-end: CI-signed UKI verifies (sbverify), enrolment succeeds, VM boots with mokutil --sb-state = enabled. Key generation + enrolment procedure: secure-boot-keys.md (the PK/KEK/db trio; db = sb_key/sb_cert, PK/KEK stay offline).
  • Bug fixed during bring-up (2026-06-20): fleet-sb-enroll built its 4-byte EFI attribute prefix in a bash variable (ATTR=$'\x27\x00\x00\x00'). Bash strings can't hold NULs → only 0x27 survived → efivarfs read a corrupt attribute u32 → every write failed with EINVAL ("dd: Invalid argument"). Now emitted directly with printf at write time. Baked fix VERIFIED on the 820 (2026-06-22): sudo fleet-sb-enroll enrolled db→KEK→PK cleanly, no inline workaround. (Red herring on the way: Proxmox EFI disks default to pre-enrolled-keys=1, so mokutil --db shows the MS cert even in Setup Mode — looked like a non-empty-db rejection but was the NUL bug all along; it failed identically on a clean pre-enrolled-keys=0 disk.)
  • iPXE signing WIRED (2026-06-21): snponly.efi is now built + signed (db) + pushed by CI as a separate FROM-scratch fleet-ipxe image (scripts/build-ipxe-oci.sh, .gitea/workflows/ipxe-image.yaml), mounted into dnsmasq at /ipxe. dnsmasq itself moved to a generic CI-built/pulled image (.gitea/workflows/dnsmasq-image.yaml) — no longer built at deploy time, and carries no keys/binary. (Decision: separate fleet-ipxe image, not baked into dnsmasq, so the daemon image stays key-agnostic across SB environments.)
  • 820 / PXE signed chain — BENCH-VERIFIED 2026-06-22. Real hardware (HP EliteBook 820 G4), Secure Boot enforcing (mokutil --sb-state = enabled, Setup Mode no, db/KEK/PK enrolled): firmware verifies the signed snponly.efi (fleet-ipxe image) → signed UKI and boots. Confirms the whole fleet-ipxe signing pipeline + the PXE→iPXE→UKI chain under SB on real silicon (the VM only covered the HTTP path, which has no iPXE stage).
  • REMAINING for production: (a) secure path — re-verify signed boot with disable_dot1x=false (802.1X/MACsec), not just the test path; (b) confirm signing is reproducible (verity hash stable across two signed builds).
• PXE-only iPXE fallback — BENCH-VERIFIED on the EliteBook 820 G4 (2026-06-20). Firmware without UEFI HTTP boot chainloads serving/tftp/snponly.efi → renders autoexec.ipxe (ifopen + DHCP-retry + imgargs <real cmdline>) → UKI over HTTP → NVMe/verity root. dnsmasq tags pxe/ipxe/httpboot give graceful per-client fallback; HTTP-boot hosts are unaffected (firmware → UKI direct). Gotchas that cost real time (all captured in ../serving/tftp/README.md): (i) the I219-LM needs snponly.efi (native iPXE driver won't bind); (ii) PortFast/edge on the switch port is REQUIRED — iPXE's PHY bounce triggers STP forwarding delay that drops DHCP (this was the real blocker); (iii) iPXE aborts the script on any unguarded command failure (use ||); (iv) Secure Boot OFF makes systemd-stub take iPXE's load options as the cmdline, so the entrypoint extracts the UKI's real .cmdline and feeds it via imgargs. Tidied 2026-06-20 once PortFast was identified: dropped the autoexec.ipxe + ifopen + DHCP-retry STP workaround; boot.ipxe is now just dhcp+imgfetch+imgargs+boot reached via normal iPXE autoboot. Re-verified on the 820 G4. SB-on simplification: when Secure Boot is enabled the cmdline injection is unnecessary (the stub ignores the loader cmdline) — collapse boot.ipxe to a plain chain of the UKI and drop the .cmdline extraction + binutils + the artifact mount from the dnsmasq service.
• Sign iPXE for the PXE-only fallback (Secure Boot, production). The lab snponly.efi is UNSIGNED → that path needs Secure Boot OFF. For production: sbsign the iPXE binary with the fleet db key (only db needed — no MS CA / shim), so SB stays enforced firmware → iPXE → UKI; then the cmdline contraption becomes a no-op (the stub ignores loader cmdline under SB). See secure-boot-keys.md "PXE-only hosts".
• Bench-verify on the secure path (swtpm + hostapd + FreeRADIUS, disable_dot1x=false) — the items deferred this session:
  • NM port secure path: en* UNMANAGED + macsec0 MANAGED/renewing, MKA survives switch-root, NVMe stays up past lease (item #1 risk).
  • NTP via DHCP option 42 reaches chrony over macsec0 (chronyc sources).
  • Plymouth fleet theme: test-path render VERIFIED 2026-06-18; still confirm the auth→macsec rows reach done live on the secure path.
  • STIG remediation didn't break the secure path; reproducible verity hash; triage the generated stig-deviations.md REVIEW rows (item #2).
• Secure-path NVMe connect trigger: the test path uses nvmf-connect.service (gated rd.dot1x=0); confirm the secure path's connect (dracut nvmf over macsec0) actually fires, or give it an equivalent trigger. Apply the same -k 30 -l -1 -c 10 keep-alive/reconnect flags to the secure nvme connect as the test path has.
• Future hardening waived for now (from the STIG register): FIPS, firewall, per-host/rotatable admin (SSH CA / config-agent).
• TLS-encrypted audit/log offload (production). The lab forwards logs to serving/audit-collector in plaintext, and the STIG TLS-offload rules (rsyslog_encrypt_offload_defaultnetstreamdriver, ..._actionsendstreamdrivermode, ..._actionsendstreamdriverauthmode) are waived in the tailoring because applying them forces gtls + x509 peer auth + a CA file and breaks the plaintext 2f forward. For production, do it properly and DROP the waivers:
  1. Stand up a CA (or reuse the boot-infra CA); issue the collector a server cert (CN/SAN = audit.fleet.lab).
  2. Configure the collector's rsyslog for TLS: imtcp with StreamDriver.Name="gtls", StreamDriver.Mode="1", server cert/key + CA; listen on the syslog-TLS port (6514) — add a TLS receiver to serving/audit-collector/rsyslog.conf + compose.
  3. Bundle the CA into the image (host-generic, no per-host state — same model as the RADIUS CA) and point $DefaultNetstreamDriverCAFile at it.
  4. Set the 2f omfwd action to StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" + the CA file, target port 6514.
  5. Re-select the three rsyslog_encrypt_offload_* rules (remove from files/stig/diskless-tailoring.xml) — they should then PASS, and update docs/stig-deviations.md #6 to reflect encrypted offload. (RELP+TLS via omrelp/imrelp is the alternative if guaranteed delivery is wanted alongside encryption.)
• fleet-hostapi: scope CORS to the kiosk origin. The host-info agent ships with FLEET_HOSTAPI_ALLOW_ORIGIN=* (lab). For prod set it to the exact kiosk origin (default http://boot.fleet.lab) via the unit's commented Environment= line (or a drop-in). See #12.

[ ] 9. (OPTIONAL / nice-to-have) Convert provisioning to Ansible where it pays off

Low priority — only the config-management half of the build is a candidate, and the ROI is modest because the build VM is single-shot (idempotency buys little) and reproducibility/auditability favour minimal shell. Not a wholesale rewrite: the load-bearing logic (dracut module hook ordering, extract-artifacts.sh, build-oci.sh) is initramfs-runtime / host-side build tooling, not provisioning, and stays shell.

The high-ROI, low-risk slice if/when we pick this up:

• STIG via the SSG Ansible playbook (ssg-rl9-playbook-stig.yml, shipped with scap-security-guide) instead of oscap --remediate — the upstream path and far more reviewable (you can diff exactly what each rule changes). Keep oscap eval (no remediate) for the report + the 0-eval guard (item #2).
• Restyle the brittle imperative bits as declarative tasks: chrony.conf edit (lineinfile/blockinfile vs sed), sshd drop-in / NM dispatcher / kiosk file installs (template/copy), datastream-by-os-release as assert. Bonus: sidesteps the %{...}→%%{ HCL-escaping footgun.

Tips the balance toward "do it" only if: we start managing the serving hosts (dnsmasq/nginx/SPDK/chrony/audit-collector) with config-management too (shared roles to amortise — currently they're docker-compose), provisioning grows much more branchy, or we want molecule-tested hardening roles validated without a full ~17-min Packer build.

[ ] 10. (OPTIONAL) Secure Boot cert auto-enrollment for provisioning

Speed up bringing up new workstations: bake the public PK/KEK/db certs into an image, detect Secure Boot Setup Mode at boot, and auto-enroll them so a fresh/reset box adopts the fleet trust root without a manual firmware-setup visit. Pairs with #8 (signing) + secure-boot-keys.md. Shares the provisioning-UKI + isolated-environment vehicle with #11 (which enrols the network identity); build them together.

Tooling gap (found 2026-06-20): efitools is NOT packaged for EL9 — no efi-updatevar/efi-readvar/KeyTool. Resolved for the OS-side path by writing the signed .auth variables to efivarfs directly (de-immutable + single write), which needs no extra package. Done 2026-06-20: baked /usr/local/bin/fleet-sb-enroll (efivarfs method, Setup-Mode-gated, fapolicyd- trusted) + the public PK/KEK/db.auth at /usr/share/fleet-secureboot/ (supplied via files/secureboot/keys/ or SB_*_AUTH_B64 CI secrets); mokutil covers reading. So manual OS-side enrollment over SSH now works (sudo fleet-sb-enroll). What's left for auto-enroll: drive fleet-sb-enroll unattended on a Setup-Mode detection at boot, gated to a provisioning mode/UKI (below), with read-back verify + audit — NOT armed in the production image.

What's safe: the certs are public (no key material — only the db private key signs, and it stays in CI/HSM, invariant #7); Setup Mode accepts unauthenticated writes to PK/KEK/db, so public certs (ESL) are sufficient. It's fleet-generic (no per-host state, invariant #1) and deterministic (reproducible verity hash).

The real risk — provisioning-time TOFU (trust-on-first-use): in Setup Mode Secure Boot isn't enforcing, so the enroller runs as unverified code, and the fleet's UEFI HTTP boot is unauthenticated in that state. An on-path attacker during provisioning can serve a malicious image that enrolls their PK/KEK/db — the box then "has Secure Boot" but trusts the attacker. Auto-enroll is only as trustworthy as the provisioning channel, and it removes the physical-presence approval gate. So treat it as a controlled-environment provisioning step, not an always-on behaviour.

Optional separate provisioning UKI (preferred shape): build a dedicated provisioning UKI that carries the certs + the enroll logic, instead of arming the enroller in the production image. Keeps the standing footgun out of the normal image (a factory reset + tampered image can't silently re-root trust), and lets the enroller be a one-shot, attended/isolated step. The production UKI never enrolls.

• Build it as a separate Packer target / flag (e.g. provisioning=true) → its own small UKI; signed or unsigned (it runs in Setup Mode anyway, but sign it so it also boots once keys are enrolled / for re-runs).
• Or a tiny standalone EFI tool (KeyTool-style) served alongside, not a full OS.

Mitigations (do these regardless of UKI vs separate UKI):

• Run provisioning on a trusted, isolated VLAN (or a physically controlled bench) — the integrity Setup Mode can't give must come from the environment.
• Gate enrollment behind an explicit provisioning mode / one-shot token; never arm it on every boot of the production image.
• Read-back verify before declaring success: efi-readvar the enrolled PK/KEK/db and compare fingerprints to the values baked in; mismatch = fail loudly (turns silent TOFU into tamper-evidence).
• Audit the enrollment event + read-back fingerprints to the collector.
• Keep the Microsoft UEFI CA in db (or enroll your db alongside it) unless every NIC/GPU option ROM is covered by your keys — clearing MS db can break signed option ROMs (NIC PXE/HTTP-boot, GPU init), bricking the boot path.
• PCR 7 check: enrollment changes the Secure Boot measurement in PCR 7; confirm nothing the fleet seals to the TPM (the EAP-TLS key at 0x81000100, NV cert) is PCR-7-policy-bound before relying on this.

Decision before building: auto-enroll trades assurance for convenience and is reasonable for a controlled fleet; the highest-assurance path is attended/firmware or manufacturer pre-provisioning. Default recommendation: separate provisioning UKI + isolated VLAN + read-back verify.

[~] 11. (DESIGN) TPM 802.1X identity enrolment (TOFU + OOB fingerprint)

Authoritative design consolidated 2026-06-28: tpm-enrolment-design.md — unifies this item with the fleet-manager (the broker = the fleet-manager's one unauthenticated bootstrap endpoint; approval births the inventory record) and the full identity lifecycle (enrol / renew / alert). v1 decisions (2026-06-28): fingerprint-only OOB assurance (TPM attestation via ACME device-attest-01 is the documented later upgrade); inline single-UKI vehicle for now (empty-NV early-boot branch — keeps provisioning network-boot-only, no USB; a separate provisioning UKI behind an enrol_mode build flag is the planned later upgrade); enrolment transport is plain stdlib HTTP/JSON (Go, the one unauthenticated edge — gRPC reserved for the steady-state agent path). The notes below are retained as the originating decisions; the design note is canonical.

Broker BUILT (server side, 2026-06-28): serving/enrol-broker/ — Go/stdlib; public unauth POST /request+GET /status/{id} (behind the serving nginx /enrol/) + admin mTLS web UI (/, /pending, /approve|deny/{id}) with a dedicated admin CA (admin-pki.sh); wordlist fingerprint, time-boxed store, per-IP rate limit; signs via step (dev mock signer). Ships in serving/dot1x-bench/docker-compose.yml; end-to-end smoke-tested (CSR → mTLS approve → cert). BENCH-VERIFY: run against real step-ca (bootstrap.sh root + enrol provisioner password readable by the broker) and confirm an issued cert authenticates EAP-TLS.

HOST SIDE BUILT 2026-07-02 — Go binary (fleet-enrol); TEST-PATH ENROLMENT BENCH-VERIFIED ON REAL HARDWARE (Dell 820, rd.dot1x=0, 2026-07-03). Secure-path (802.1X/MACsec) enrolment still to verify — see the NV-index bench finding below. Pivoted from a shell hook to a static Go + go-tpm binary (files/enrol/): it talks to /dev/tpmrm0 DIRECTLY (via transport.FromReadWriter on an os.File — no linuxtpm, so go-tpm is the only vendored dep), so the enrolment path needs no tpm2-tools key-gen / no openssl tpm2 provider / no curl in the image (removes the biggest bench risk — the provider handle: key-ref). It ensures the ECDSA key at 0x81000100, builds the CSR, POSTs CSR+context to rd.fleet.enrol_url, shows the broker fingerprint on Plymouth, polls /status, and writes the leaf cert to NV 0x01800100. The dracut hook tpm-enrol.sh (initqueue/settled prio 25) is now a thin GATE: empty-NV check (tpm2_nvread, still needed for the secure-path read) → pre-auth DHCP on en* (nmcli override of the secure-mode unmanaged drop-in) → exec /usr/sbin/fleet-enrol → reboot to converge on exit 0. No-op when a cert is present or under rd.dot1x=0.

Built OUT OF THE VM (2026-07-02, per decision): fleet-enrol AND fleet-hostapi now compile on the CI runner (.gitea/workflows/gitea-main-ci.yaml "Build host Go binaries" step, Go 1.26.4, CGO_ENABLED=0, vendored/stdlib, GOPROXY=off), and Packer file-provisions the static binaries in — the Rocky build image no longer carries the Go toolchain (dropped the in-VM dnf install golang + the 2d removal). fleet-enrol lands at /usr/sbin before the initramfs build (dracut inst_simples it; asserted in lsinitrd), fapolicyd-trusted. New enrol_url Packer var → rd.fleet.enrol_url cmdline.

TEST-MODE ENROLMENT ENABLED (2026-07-02, temporary): the hook no longer no-ops under rd.dot1x=0 — enrolment runs on the open/test path too, so it can be exercised on a host with a TPM + the broker WITHOUT standing up 802.1X/MACsec/RADIUS. In test mode ip=dhcp owns the NIC (the hook skips its own DHCP to avoid the dracut "address already assigned" race) and DNS resolves the broker via the lease. rd.fleet.enrol=0 disables it. Production intent is still secure-path-only; revisit when the secure path is bench-verified.

TESTED (2026-07-02): transaction logic (CSR → mock broker → approve → cert extract) as stdlib unit tests, AND the full TPM ceremony (key-gen → persist → ECDSA sign → CSR verify → chunked NV write → re-enrol) against the go-tpm software simulator (separate files/enrol/simtest/ module so the CGO simulator never enters the shipped build). Both run in CI.

BENCH FINDING 2026-07-02 (FIXED) — the signing-key handle 0x81000100 was the EK slot; moved to 0x81000100. First real-TPM boot (host cygnus-cruiser-99, Proxmox vTPM, rd.dot1x=0) failed: fleet-enrol: ensure key at 0x81010001: key at handle is not ECC: did not contain eccDetail (selector value was 1). EnsureKey died reading an existing RSA object as ECC → enrolment died before the CSR/POST, so the splash flashed "FIRST-BOOT TPM ENROLMENT" then the test system booted with no fingerprint and nothing at the broker. Root cause (confirmed via tpm2_readpublic under enforcing=0): 0x81000100 is the TCG-standard RSA Endorsement Key handle — the object there was the swtpm-provisioned EK (restricted|decrypt|adminwithpolicy, RSA-2048/AES128CFB, canonical EK authPolicy 837197674484b3f8…), with its EK cert at NV 0x1c00002. Our design squatted the EK slot fleet-wide (most firmware/swtpm TPMs have their EK there), so enrolment could never create our key.

• Fix (commit c574fa4 + follow-up): the fleet signing key moved to 0x81000100 (non-reserved owner handle; threaded through fleet-enrol -handle default, rd.dot1x.handle= cmdline knob baked by extract-artifacts.sh, wpa_supplicant-dot1x.conf, the bench scripts, invariant #1, and the design docs). NV index was 0x01800001 here (owner range); it MOVED to 0x01800100 in a later finding (OEM platform-NV collision on the 820 — see below).
• EnsureKey hardening: (i) evicts a genuine stale signing key at the handle and recreates (self-heal for old provisioning leftovers); (ii) refuses to evict any restricted-decrypt object (EK/SRK) as a safety net — never clobber the TPM's endorsement/storage identity (also protects the future EK/AK attestation). Regressions: simtest/TestEnsureKeyEvictsStaleNonECC + TestEnsureKeyRefusesToEvictProtectedKey.
• IDevID note (future): if EK/AK attestation (device-attest-01, §"Stronger assurance") lands, the natural home for a TPM device-identity key is the TCG IDevID handle 0x81020001 (LDevID 0x81030001) — see tpm-enrolment-design.md. Not adopted for v1 (implies the IDevID template/cert conventions).
• Also seen — leftover NV suppresses enrolment: a different host (the 820) booted tpm-enrol: client cert present at NV 0x01800001 — normal boot with nothing enrolled this session — leftover NV state (from an earlier tpm-provision.sh / tpm-selftest.sh --use-real-handles) persists in the hardware TPM across reinstalls. Our presence check is content-blind (index defined ⇒ "enrolled"), so foreign/stale NV silently skips enrolment (→ secure-path auth would then fail). See the content-aware NV check item below.
• Needs a rebuild + re-boot to confirm (the sim proves the ceremony, not the device). ⚠ Don't hand-run sudo fleet-enrol on the booted STIG host — the confined admin domain (sysadm_t) is SELinux-denied write to /dev/tpmrm0 (avc: denied … tpm_device_t); the faithful path is the initramfs run (reboot, unconfined), or enforcing=0 for hand-run TPM inspection.

FOLLOW-UP HARDENING:

• Content-aware NV presence check — BUILT 2026-07-03. fleet-enrol -check-only (no network / no key-gen) reads NV 0x01800100 via tpmkey.ReadNVCert, parses it as X.509, checks the validity window, and — when a real fleet CA is bundled (-cacert, default radius-ca.pem) — verifies it chains to it. Only a valid fleet cert counts as enrolled (rc 0); empty/garbage/foreign/expired → rc 20 (enrol + overwrite; WriteNVCert undefine+redefines). The dracut hook now calls it as the single gate, replacing the blind tpm2_nvread (removed from tpm-enrol.sh; the binary's NVPresent stays for cheap existence checks). Lab fallback: the shipped radius-ca.pem is a placeholder (comments, not a cert) → loadCAPool returns nil → chain-less "parseable
  • unexpired" acceptance, so the reboot-converge doesn't loop on a bench-CA cert; production must replace radius-ca.pem with the real fleet CA for strict chaining. Tests: classifyNVCert unit table (valid/expired/garbage/foreign/no-CA) + simtest ReadNVCert roundtrip across the 512B chunk. BENCH-VERIFY on a real boot: a garbage/ foreign NV now enrols instead of "normal boot".
• De-enrol / TPM-reset procedure for re-testing: tpm2_nvundefine 0x01800100 -C o (+ tpm2_evictcontrol -C o -c 0x81000100 for a stale app key — never the EK), or BIOS "Clear TPM". Document in the bench README.

BENCH FINDING 2026-07-03 (FIXED + BENCH-VERIFIED) — NV index 0x01800001 collided with OEM platform NV; moved to 0x01800100. The Dell 820 kept "finding a cert" and bypassing enrolment even after repeated BIOS TPM clears. tpm2_nvreadpublic showed the truth: 0x01800001 on that hardware is a firmware/platform index (platformcreate|policydelete|writelocked, 70 bytes — NOT our cert; siblings at 0x01800003/4 + 0x01810000 share it), which no owner/BIOS clear can remove (TPM2_Clear spares platformcreate; policydelete needs the platform's secret policy). So our chosen index was permanently occupied by Dell — the blind check false-positived on it, and even the content-aware check would have failed at the NV write (can't undefine a platform index). Same class as the EK-handle collision. Fix: moved the NV cert index default 0x01800001 → 0x01800100 (free on the 820; owner range; overridable via rd.dot1x.nvindex=), threaded through the cmdline (extract-artifacts.sh), fleet-enrol -nvindex, the dracut hooks, bench scripts, invariant #1, and docs. Also hardened WriteNVCert to refuse a platformcreate index with a clear "pick a free owner index" error instead of a confusing undefine failure (regression simtest/ TestWriteNVCertRefusesPlatformIndex). BENCH-VERIFIED 2026-07-03 — the Dell 820 enrols end-to-end at 0x01800100. First success on REAL TPM hardware (past the un-clearable OEM platform-NV at 0x01800001): validates BOTH the NV-index move AND the 0x81000100 key-handle move on a physical TPM, not just the Proxmox swtpm — the full host ceremony (ensure ECDSA key → CSR → broker fingerprint → approve → leaf cert → NV write) runs on the metal. This was the hard case (the 820 previously "found a cert" and silently bypassed enrolment every boot). STILL BENCH-VERIFY: (a) tpm2_nvreadpublic on any OTHER hardware model in the fleet — if a vendor also squats 0x018001xx, pick a value free on all of them; (b) the enrolled cert actually authenticates EAP-TLS on the secure path (this run was test-mode rd.dot1x=0; 802.1X/MACsec enrolment still unverified — see the test-mode note above).

BENCH-VERIFY (host side, before trusting): (a) go-tpm against a REAL TPM + swtpm on the target (the simulator proves the ceremony, not the device) — the stale-key fix (above) is the first real-TPM result; re-boot to confirm it now reaches the broker; (b) empty-NV create/persist on a fresh TPM with empty owner auth (the code assumes it — a set owner auth needs handling); (c) pre-auth DHCP on en* in secure mode (the nmcli set managed yes override of nm-unmanaged.sh actually leases); (d) fingerprint renders on the fleet Plymouth theme + admin can match it; (e) NV write + reboot converges (2nd boot runs the secure path); (f) prod: broker over HTTPS (fleet-enrol -cacert radius-ca.pem), currently lab-HTTP.

TEMPORARY (2026-06-28): broker also added to the MAIN serving stack (serving/docker-compose.yml, enrol-broker service + ENROL_* in serving/defaults.env) so a single compose up in serving/ brings the /enrol/ endpoint up while the host side is built. Defaults to the zero-dependency lab path — mock signer (ephemeral CA; certs NOT trusted by RADIUS/NVMe) + plaintext loopback admin (ENROL_ADMIN_INSECURE=true). The serving nginx already proxies /enrol/→127.0.0.1:9090. SPLIT BACK OUT: the broker's proper home is a CA-co-located stack (it's still defined in dot1x-bench/ next to step-ca); real signing needs the fleet CA. Don't run both copies on one host (they collide on 9090/9443). To use real signing from the serving stack meanwhile: set ENROL_SIGNER=step + ENROL_STEP_CA_URL at a reachable step-ca, populate serving/enrol-pki/, and ENROL_ADMIN_INSECURE=false.

STEP SIGNER VERIFIED end-to-end 2026-06-28 (broker → real step-ca → signed cert that chains to the fleet root; hostnqn derived from serial; EKU clientAuth). Re-VERIFIED ON THE SERVING HOST 2026-07-02 against step-ca 0.30.2 via verify-serving.sh through the nginx /enrol/ proxy: issuer = Fleet Bench CA Intermediate, cert validity exactly 720h (confirms the maxTLSCertDuration claim raise took effect), EKU clientAuth, enrol provisioner extension present, hostnqn = cert serial. Bring-up gotchas hit + fixed (all in the commits): the CA wouldn't init (compose used _PASSWORD_FILE for a nonexistent file → use _PASSWORD); Docker auto-created ./pki root-owned (bootstrap self-heals perms + pre-create the bind-mount dirs); the CLI couldn't resolve ca.fleet.lab on the serving host (connect via localhost, which is in the CA SAN). Verification found + fixed two things the mock path had hidden:

• BUG FIXED (signer.go): --not-after was passed to step ca token (validity capped ~1h → "token validity too long") instead of step ca sign. Moved to the sign step. Without it the step signer never issues a cert.
• CA CLAIM REQUIRED: a stock step-ca caps certs at 24h, so 720h is rejected ("authorized maximum certificate duration of 24h"). dot1x-bench/step-ca/bootstrap.sh now raises authority.claims.maxTLSCertDuration (+restarts the CA) — this also unblocks bootstrap's own 720h RADIUS cert. Any other CA needs the same claim.
• BENCH-VERIFY still open: issued cert actually authenticates EAP-TLS through FreeRADIUS (next, before host enrolment).

How a host gets its 802.1X EAP-TLS client identity the first time — the key at TPM handle 0x81000100 + cert at NV 0x01800100 that invariant #1 says must come from enrolment, never the image. Distinct from #10 (that enrols Secure Boot PK/KEK/db); this enrols the network identity. They share the provisioning UKI + isolated-environment vehicle. Pairs with #8 + the CA decision (start step-ca, serving/dot1x-bench/; OpenBao/EJBCA/Dogtag later).

Chosen pattern — trust-on-first-use with out-of-band fingerprint approval: on an empty TPM the host generates a key in the TPM, builds a CSR, submits it to an enrolment broker, and shows a fingerprint on the Plymouth splash; an admin visually cross-checks that fingerprint against the broker's pending-list and approves; the broker has step-ca sign the CSR and returns the cert; the host writes it to NV 0x01800100 and proceeds to authenticate. Sound, recognised pattern (SSH/WireGuard/Signal-style OOB verification); good fit for invariant #1.

Key design decisions (settled 2026-06-22):

• The pre-auth permit-set already exists — enrolment rides in it. The firmware can't do 802.1X, so the UKI fetch (DHCP/DNS/TFTP/HTTP to the serving host) is always pre-auth; the secure model therefore requires a pre-auth permit-set (low-impact 802.1X / pre-auth dACL / boot→secure dynamic VLAN; RADIUS returns the secure VLAN on Access-Accept). The enrolment endpoint is just another entry in that set — expose it as an /enrol/ location on the existing serving nginx, reverse-proxied to the broker. No new network mechanism.
• The broker is the only new component. step-ca signs but has no human-approval queue: broker = POST CSR + context → PENDING + fingerprint → admin approves → step-ca signs → host polls → cert. Exposed to unauthenticated devices by design → harden: OOB-fingerprint gate (no auto-sign), rate-limit, time-boxed pending, HTTPS (image trusts the serving CA), pre-auth permit scoped to only these services + the serving host.
• Trigger on EMPTY-NV only — split the three events: empty NV → enrol (human); valid key + expired/near-expiry cert → renew automatically (no human, it already holds a trusted key); auth fails with a present cert → alert + retry, do NOT re-enrol (auto-reenrol on failure would mask revocation and churn keys on a transient RADIUS outage). Re-enrol after the first time = explicit admin action (clear/flag the host).
• Fingerprint: what it proves + entropy. It proves "this CSR came from this physical device" (approver reads the screen) — it does NOT prove TPM residency. Pair it with TPM attestation (step-ca ACME device-attest-01, EK/AK) for the residency proof: fingerprint = human authorises the device, attestation = the key is really in a TPM (together → strong SSP R-01 story). Size for entropy — a short hex fp is grindable; use a word-list/BIP39-style encoding (reads short, more bits) + rate-limiting + a server nonce. Show device context (asset/port/location) so the admin knows what they're approving.
• Pre-auth traffic has no MACsec (plaintext at L2). Fine for enrol (CSR/cert are public; OOB fingerprint gives authenticity) — and it's exactly why the UKI must be Secure-Boot-signed (#8): integrity of the boot comes from the signature, not the network. Enrolment design + Secure Boot are two halves of the same "pre-auth path is hostile" assumption.
• Run it in a provisioning UKI/mode, not the production initramfs — the boot hook ordering is load-bearing (invariant #2); don't park it on a multi-minute human-approval wait. Shares the #10 provisioning-UKI vehicle.

Bench caveat: hostapd wired is strictly port-based (no pre-auth per-flow ACL) — before EAP success it passes nothing, so the bench can't reproduce "boot + enrol pre-auth on the same port." Model the pre-auth state with a separate provisioning bridge/VLAN on the rig; the in-place low-impact pre-auth permit is a real-switch feature to validate on the 820's switch.

First step DONE (2026-06-22): serving/dot1x-bench/enrol/tpm-selftest.sh — a non-destructive TPM capability check (device, key persist, NV roundtrip, openssl TPM2 provider, CSR signed by the TPM key) to de-risk the TPM half before any of this. Expected gap: the image installs tpm2-tss but no openssl TPM2 provider (tpm2-openssl) — add it to the build once the self-test confirms.

[~] 12. Workstation host-info API (fleet-hostapi) for the kiosk

IMPLEMENTED 2026-06-28. A small Go, read-only, loopback-only HTTP agent on the workstation that reports host/OS/hardware facts to the kiosk SPA, per the HOST-API.md contract (GET http://localhost:9099/v1/host). Source: files/hostapi/ (stdlib-only, reproducible static build). Built + enabled by provisioner 2b-hostapi; toolchain removed in 2d; binary fapolicyd-trusted in 2c-STIG-postfix (register #10). Chromium policy grants Private Network Access scoped to the kiosk origin (files/kiosk/chromium-kiosk-policy.json).

• Per-host safety: host-specific fields (kernel, hostname, IP, MAC, board, uptime) are read at RUNTIME; only image-generic facts (image version + weston/chromium versions) are baked, to /etc/fleet-hostinfo.json (invariant #1 preserved). Deliberately does NOT expose DMI serial (root-only/per-host) or any TPM/cert/identity material.
• Hardening shipped: loopback bind enforced in code; DynamicUser, no caps, ProtectSystem=strict, MemoryDenyWriteExecute, SystemCallFilter=@system-service, IPAddressDeny=any (loopback only). Runs unprivileged, reads world-readable files only.

Lab posture LEFT AS-IS (decided 2026-06-28) — revisit for prod:

• CORS Allow-Origin=* → scope to the kiosk origin (http://boot.fleet.lab). Consolidated into #8.
• SELinux: runs in the default service domain (no custom policy module). BENCH-VERIFY no AVC denials for fleet-hostapi on an enforcing booted host (Wazuh/fleet_audit rule 100120 will surface any). If denials appear, decide: write a confined SELinux policy module vs adjust the sandbox.
• No auth (justified: loopback + read-only + single-user appliance). Option if the threat model tightens: shared token in kiosk.env + header check.
• Info sensitivity: exposes hostname/IP/MAC/board/versions to the kiosk. Drop MAC/IP if that's later deemed too much.
• Reproducibility: the golang toolchain version is a reproducibility-sensitive input (a package bump changes the binary → verity hash); covered by the build-twice gate. Pin the go version only if drift becomes a problem.

BENCH-VERIFY (functional): (a) service starts under enforcing SELinux + fapolicyd (not blocked); (b) the kiosk SPA's System page fetches /v1/host successfully through the PNA allowance (preflight + GET); (c) facts render correctly on real hardware (board/CPU/GPU from DMI/DRM, not the virtio build VM).

[~] 13. Kiosk GPU compositing + Chromium flag ownership

GPU FIX APPLIED 2026-06-28. The 820 (Intel HD620/i915) ran Chromium at ~150% CPU with periodic crashes: --disable-gpu in our kiosk-session forced software raster/compositing. Root cause confirmed from the live cmdline. Fixes in files/kiosk/kiosk-session + diskless.pkr.hcl:

• Runtime GPU probe (fleet-generic image boots real HW and virtio VMs): a real KMS driver (i915/amdgpu/…) + /dev/dri/renderD128 → GPU path (--ignore-gpu-blocklist --enable-gpu-rasterization --enable-zero-copy, weston GL renderer); else software (--disable-gpu + weston --use-pixman) so the VMs still render. No per-host state baked.
• kiosk user added to render (renderD128 isn't logind-seat-ACL'd on EL9; video only covers card0/KMS for weston).
• Merged the distro wrapper's --disable-features into ours — Chromium takes the LAST --disable-features (not the union), so ours was silently dropping the wrapper's Lens/Gemini/Compose disables.

BENCH-VERIFY (on the 820): GPU process alive (chrome://gpu), Compositor/ ThreadPool threads near-idle + CPU drop, gt_act_freq_mhz rises under load, no SELinux AVC for dri_device_t from the kiosk domain (Wazuh fleet_audit 100120 will surface any — if present, add a scoped SELinux allow or confirm the kiosk user's SELinux mapping is unconfined). Confirm VMs still render (software path).

DECIDED 2026-06-28 — Option 1 (policy lock), APPLIED. The distro chromium wrapper (/usr/bin/chromium-browser) layers flags as $CHROMIUM_FLAGS (/etc/chromium/chromium.conf → VA-API enables + AI disables; overridable via CHROMIUM_USER_FLAGS) + hardcoded $CHROMIUM_DISTRO_FLAGS (--enable-plugins --enable-extensions --enable-user-scripts --enable-printing --enable-sync --auto-ssl-client-auth) + our "$@" (last). The hardcoded layer isn't removable via env/conf — only by bypassing the wrapper. Rather than bypass (Option 2, which would force us to replicate the wrapper's stdio sanitisation + own the VA-API feature list across updates), we neutralise in managed policy (policy beats cmdline flags, survives updates): ExtensionInstallBlocklist=["*"] added to files/kiosk/chromium-kiosk-policy.json (blocks all extensions → also kills the user-scripts surface). --enable-sync already locked by SyncDisabled; --auto-ssl-client-auth is moot (no browser-store client cert; the TPM 802.1X cert isn't one) — AutoSelectCertificateForUrls left unset. VA-API enables inherited from chromium.conf unchanged. Option 2 (full flag ownership, bypass the wrapper, own the VA-API --enable-features: AllowQt,WaylandLinuxDrmSyncobj,WaylandPerSurfaceScale,WaylandUiScale,AcceleratedVideoDecodeLinuxGL,AcceleratedVideoDecodeLinuxZeroCopyGL,AcceleratedVideoEncoder) remains documented here if a minimal/auditable cmdline is ever wanted. BENCH-VERIFY: chrome://policy shows ExtensionInstallBlocklist=["*"] active; extensions can't install.

[~] 14. (PARTLY BUILT) Centralised fleet management plane (health/version/usage/maintenance)

SERVER-SIDE CONTROL PLANE BUILT 2026-07 — serving/fleet-manager/ (Go service

• embedded Svelte SPA). See the as-built note at the top of docs/fleet-management-design.md. What shipped and how the design forks resolved:

• Two planes: admin (HTTPS + WebAuthn/passkey, YubiKey; server-side session

  • CSRF; register-admin CLI bootstrap; UI cert via Let's Encrypt DNS-01, hot-reloaded) and agent (bespoke gRPC bidi/mTLS, fleet CA, reused TPM cert — health/metrics up, desired-state down; no inbound host port).

• Metrics → InfluxDB (manager is the only reader/writer; host holds no Influx credential; identity-tagged writes; Flux reads behind a field whitelist + hex- serial guard). Graphs render in the SPA (bespoke SVG), not Grafana. This replaces the earlier Wazuh-Phase-1 / OTLP-Phase-2 sketch for this plane; Wazuh stays the audit/SIEM path.

• Authorise pillar: the enrol-broker (#11) folded in "alongside" — manager drives its admin plane over mTLS and births the inventory record on approve.

• Deployment: Dockerfile + docker-compose.yml (private InfluxDB + host-networked manager) + server-pki.sh + admin-UI cert acme side-car (LE DNS-01 via Route 53, auto-renew) + CI image build (fleet-manager-image.yaml, pulled by tag) + backup.sh (durable volumes) + DEPLOY.md.

• Hardening still open (operator/per-env): network-segment the admin (:9500)

  • agent (:9600) listeners to the mgmt net; run server-pki.sh when the agent lands; decide :9500 vs a 443 proxy; digest-pin the images (invariant #12). See DEPLOY.md "Hardening status".

• [~] (a) the on-host agent — TELEMETRY HALF BUILT 2026-07-05 (files/agent/). The gRPC client that pairs with fleet-hostapi and feeds this plane. It reuses the host TPM identity for mTLS (signing key at 0x81000100 via tpmkey.LoadSigner — a read-only copy of the enrol tpmkey; cert from NV 0x01800100), opens the long-lived Session stream, sends Hello + periodic HealthSnapshot (load/cpu/temp/mem/tmpfs/uptime/nvme/auth/selinux/kiosk from /proc+/sys + hostapi facts, plus chrony clock health — state/source/ stratum/offset — read from /var/log/chrony/tracking.log), and — telemetry-only v1 — receives DesiredState/Command but only LOGS them (zero host mutation). Static CGO_ENABLED=0 binary built out-of-VM in CI (like enrol/hostapi), staged by the 2b-agent provisioner (enable_agent, default on), fapolicyd-trusted, fleet-generic config in files/agent/agent.env. mTLS handshake proven end-to-end against the go-tpm software simulator (files/agent/simtest). No dracut/initramfs changes (post-switch-root only). STILL TO BUILD (deferred): the reconcile actuators — apply kiosk_url, reboot_at/REBOOT, drain — wired to be received but not acted on; land after the telemetry half is bench-verified. BENCH-VERIFIED END-TO-END ON REAL HARDWARE 2026-07-10 (Dell 820 comet-beacon-54, real TPM, image fleet-manager-254ec79): the agent loads its reused TPM identity (serial derived on-device via go-tpm), completes mTLS to the manager, streams health, AND receives desired-state (logged, not applied). Manager logs agent connected: fed27077… (comet-beacon-54); the host stays connected (no reconnect loop). This closes the TPM-identity path (802.1X = NVMe hostnqn = agent mTLS, "one identity, three stages") on hardware.

  Deploying the agent plane — gotchas hit + FIXED in commit a89c3d2 (don't rediscover): the serving-side bring-up cost a long debug chain, each now automated:

  1. Host CA was the placeholder. /etc/dot1x/radius-ca.pem shipped a non-CA placeholder → client TLS: no certificates parsed from fleet CA. FIX: fleet_ca build var bakes the real step-ca root (CI secret → file → -var fleet_ca; default = placeholder, lab-only). The agent's FLEET_AGENT_CA points here; only the ROOT is needed host-side (the manager sends its chain).
  2. tls: certificate required. step-ca signs leaf certs with an INTERMEDIATE and enrolment stores the leaf-only cert in NV, so a root-only client CA makes the manager advertise only the root in its CertificateRequest → the Go client sends NO cert. FIX: server-pki.sh installs the agent CLIENT CA (FLEET_MGR_AGENT_CLIENT_CA) as root+intermediate; bootstrap.sh exports intermediate_ca.crt.
  3. Distroless perms. The nonroot manager (uid 65532) can't read pki/agent/* if created 0600 by the host user → open …/server.crt: permission denied. chown 65532 the agent PKI (as with the fleet-data volume).
  4. Manager wasn't a fleet name. The agent's baked default pointed at the PUBLIC admin-UI name (diskless-server.bitasylum.org), which resolves to the serving host's OUTBOUND IP → no route to host from the fleet LAN. FIX: the agent plane is now just another *.fleet.lab — manager.fleet.lab (MGMT_HOST in serving/.env), served by the fleet dnsmasq to SERVER_IP exactly like boot/nvme/audit (--address=/${MGMT_HOST}/${SERVER_IP}), baked as FLEET_AGENT_MANAGER and set as the agent server cert SAN (AGENT_SANS). No per-host override, no server-name hack. (The public diskless-server… name stays the ADMIN-UI name on Traefik/:443 — a separate plane.) The transient bench workaround was FLEET_AGENT_MANAGER=<ip> + FLEET_AGENT_SERVER_NAME=<name>; no longer needed.
  5. Host not in the manager's inventory → PermissionDenied (surfaced as a bare EOF on the agent before the fix). A host enrolled OUT OF BAND (bench broker / migrated fleet) has no inventory row. FIX: fleet-manager register-host <serial> -name <starship> (writes through the real store to FLEET_MGR_INVENTORY). The manager caches inventory in memory — any edit needs a --force-recreate. Also: the agent now logs the real gRPC status, not EOF.

  Metrics + cert auto-renewal added 2026-07-10 (fleet-manager branch):

  • uptime + reboot-pending: the agent sends uptime_seconds; the manager/UI derive "reboot pending in X / rebooted" from boot_time (= last_seen − uptime) vs reboot_at, all in the manager clock frame (no host↔manager clock sync needed). Monitor metric graphs auto-refresh (10 s poll).
  • chrony clock health (sync state + NTP source/stratum/offset). The agent reads /var/log/chrony/tracking.log (image bakes log tracking), NOT chronyd's command socket (owner-write-only unix socket; STIG cmdport 0 closes the UDP port). Reaching that log from the hardened DynamicUser agent took a 4-layer chase, every layer silent-failing (memory fleet-agent-hardened-probe-access): (i) lab NTP reachable — a 123/udp rule on the serving-host firewall (host-networked ntp container; same trap as audit 514); (ii) DAC — chrony supplementary group on the unit to traverse /var/log/chrony (0750); (iii) SELinux TE — files/stig/fleet-agent-chronylog.te allow for chronyd_var_log_t (permissive masks this); (iv) the MANAGER image must be ≥ the chrony commit — one with uptime-but-not-chrony silently drops the fields (manager & workstation are SEPARATE images/deploys). Isolator: read the manager inventory.json off the fleet-data volume via a throwaway busybox (distroless manager, no shell). BENCH-VERIFIED on comet-beacon-54 2026-07-10.
  • agent-plane server-cert auto-renewal: the agent-cert-renewer compose sidecar (profile cert-auto) issues + step ca renew --daemon renews the cert from a co-located step-ca; the manager hot-reloads via GetCertificate (certreload.go), no restart. Replaces the manual server-pki.sh + docker exec … step dance and the 90-day re-mint. The server cert SANs MUST include manager.fleet.lab (the fleet-DNS name the agent dials) — a missing SAN fails the handshake.

• STILL TO BUILD: (b) per-host boot-version assignment — target_version is stored + pushed but does not change what a host boots; that needs dnsmasq cohort steering + multi-version UKI serving (#6). Kiosk URL / reboot / drain reconcile via the agent's deferred actuator half (above). Version pin is labelled "recorded intent" in the Lifecycle UI.

Depends on the step-ca fleet CA (#8) for the mTLS identity.

Original design capture (2026-06-28) — retained for reference

DESIGN CAPTURED 2026-06-28 — docs/fleet-management-design.md. Server-authoritative, pull-based, TPM-identified management plane reusing existing pieces: fleet-hostapi as the on-host data source, the audit-collector→Wazuh pipeline as telemetry transport, the digest-pinned deployment manifest as version truth, and the SPDK spdk-loader/spdk-release rolling-update flow (#6) as the redeploy/drain mechanics. Hosts stay stateless (identity from the TPM cert, no baked enrollment); control is pull/reconcile (no inbound surface); convergence to a new version = reboot. Phased build: (1) push hostapi snapshots as structured events into Wazuh + fleet dashboard (health/version-drift/offline/usage/alerting); (2) metrics push (OTLP/remote-write) → VictoriaMetrics + Grafana; (3) mTLS desired-state pull (version pin, reboot/drain, kiosk URL) wired to the deployment manifest + SPDK flow. Build-time decisions (telemetry encoding, metrics/desired- state store, per-host vs cohort, osquery/FleetDM eval) listed in the design note. Depends on the step-ca fleet CA (#8) for the mTLS identity.

[ ] 15. Scope the root overlay → per-subtree tmpfs/overlay (un-shadowable /usr + STIG mount options)

DESIGN CAPTURED 2026-07-02. Today the whole root is one systemd.volatile=overlay tmpfs over / (see #5, invariant on the volatile overlay). Replace it with a read-only verity root + a handful of scoped writable mounts, for two payoffs that land from the same change:

1. Security — un-shadowable /usr. dm-verity seals the backing store, but the whole-root overlay lets any root process write a tmpfs copy over a system binary (e.g. /usr/sbin/fleet-enrol) that then executes in preference to the sealed one until reboot. fapolicyd is the current compensating control (CLAUDE.md: "the writable-overlay exec guard dm-verity can't provide"). Keeping /usr (and /, /opt, /boot) strictly read-only makes the executable set both verity-sealed and un-shadowable — defence in depth, shrinks fapolicyd reliance. Also: writes outside the sanctioned set fail loudly (EROFS) instead of silently landing in RAM and vanishing at reboot — enforces invariant #1 (no per-host state) architecturally.

2. STIG — converts the partition/mount family from notapplicable/waived to real pass. Quantified against the scan in reports/stig-arf.xml (NOT estimates): the mount_option_* rules currently score notapplicable (no separate mount to check — the "false-clean via notapplicable" pattern #2 warns about), and partition_for_* are notselected (waived in diskless-tailoring.xml lines 61-66). Scoped tmpfs mounts with noexec,nosuid,nodev + re-selecting the partition rules convert ~21 rules to genuine pass:

   • 15 mount_option_* N/A→pass: /tmp (nodev,nosuid[,noexec*]), /var/tmp (all 3), /var/log (all 3), /var/log/audit (all 3), /var (nodev), /home (nodev,nosuid).
   • 6 partition_for_* re-selects→pass: tmp, var, var_log, var_log_audit, var_tmp, home (+1 easy: partition_for_dev_shm — already a separate tmpfs).
   • Retires deviation-register entry #1 (the narrative hand-wave over the whole partition/mount family) — one fewer waiver to defend at ATO.
   • /dev/shm's three option rules already pass — no work there.

Writable-set + flag matrix (the design):

• / /usr /opt /boot → read-only (verity as-is, no overlay). /usr keeps suid for legit setuid bins (sudo/su/passwd).
• /etc → overlay (lower=verity /etc, upper=tmpfs) — needs defaults + writability (machine-id, regenerated ssh host keys, NM connections, resolv.conf, chrony).
• /var → overlay (NOT bare tmpfs — bare tmpfs hides /var/lib/rpm, selinux store); nodev. Nest the noexec leaf mounts on top (order: /var first, then children).
• /var/tmp /var/log /var/log/audit → tmpfs noexec,nosuid,nodev.
• /tmp → tmpfs nosuid,nodev (+noexec only after the kiosk repoint, below).
• /home /root /srv → tmpfs nodev,nosuid.
• Each tmpfs needs the correct SELinux context (tmp_t, var_log_t, auditd_log_t, …) via context= or restorecon — this is the login-deny hazard class (#5); a mislabeled tree denies logins / stops auditd.

Prerequisites / caveats that gate the count:

• mount_option_tmp_noexec (1 of the 21) is conditional: register entry #1 notes the Chromium kiosk uses /tmp for profile/cache and breaks under noexec. Must repoint the kiosk scratch (Chromium --user-data-dir/--disk-cache-dir) to a dedicated exec-capable tmpfs (e.g. /run/kiosk) first → ties to #13/#4. So ~20 land immediately, +1 after the repoint.
• fstab-vs-runtime OVAL — mostly de-risked: the rules return notapplicable (evaluating the live mountpoint) not fail (missing fstab line), evidence they read the runtime mount table — so systemd mount units / initramfs mounts should score without fstab entries. Confirm per-rule on the bench; add fstab lines if any rule is fstab-based (compatible with the empty-fstab invariant — root still comes from netroot+verity).
• Does NOT fix red findings (this family is ~0 fail today) — the value is evidenced compliance + one fewer deviation + kernel-enforced noexec, not clearing fails. Framed honestly so it isn't oversold.
• partition_for_usr stays waived — /usr should be verity-sealed, not a separate writable partition (a stronger control than the rule asks). Same for encrypt_partitions, auditd_..._large_partition, boot/boot_efi (genuinely N/A).

Mechanism: systemd.volatile= has no per-dir knob (it's all-or-var-only), so this means dropping the global volatile overlay for explicit early mount units (ordered before local-fs.target) or initramfs mounts. Reproducibility unaffected (runtime mounts, not image content → no verity-hash drift).

Load-bearing / bench-gated: touches the boot path (#5 territory) — full boot-chain bench (serving/dot1x-bench/) + careful SELinux-label verification + a reconciliation note in design-v2-errata.md. Do AFTER TPM enrolment (#11) is bench-verified. Tailoring edits: re-select the 6 partition_for_* (+dev_shm), delete/gut register entry #1, re-run scripts/stig-deviations.sh to confirm the new passes.

[ ] 16. Enable a locked-down host firewall (firewalld) — defense-in-depth + STIG passes

DECISION 2026-07-03: enable a configured firewalld. Currently REMOVED + masked (diskless.pkr.hcl 2d, commit 4346ce7) because an UNCONFIGURED weak-dep firewalld was running and flooding the SIEM with lib_t .pyc AVCs. The design deferred the firewall as "future hardening, currently waived" (stig-deviations.md #4) — NOT incompatible, just unverified during boot-chain bring-up. A diskless kiosk is an ideal firewall candidate: near-zero legit inbound (admin SSH only; fleet-hostapi is loopback), and the load-bearing paths are outbound (NVMe/TCP, DHCP) or L2 (EAPOL/MACsec).

Payoff: defense-in-depth (limits a compromised same-segment peer) + converts 4 STIG waivers → passes (service_firewalld_enabled, package_firewalld_installed, set_firewalld_default_zone, configured_firewalld_default_deny — re-select in the tailoring; retires deviation-register #4).

Must do together:

1. Config: default-DENY inbound zone; allow SSH (key-only admin) + established/related; assign macsec0 (secure) / en* (test) to it. hostapi (:9099 loopback) + kiosk→hostapi PNA need no rule.
2. Fix the .pyc AVC root cause (REQUIRED — an enabled firewalld re-hits it): the reproducible erofs -T 0 zeroes mtimes, so Python thinks every site-package .pyc is stale and rewrites it → firewalld_t (and ANY Python daemon, e.g. tuned) write/unlink AVCs on lib_t. Fix once with python3 -m compileall --invalidation-mode unchecked-hash over the Python trees BEFORE the erofs build → mtime-independent .pyc, no rewrites, and MORE reproducible. (Worth doing regardless of firewalld — it's the true root cause of the firewalld AVCs seen 2026-07-03.) [x] DONE (2026-07-03): implemented in diskless.pkr.hcl — (a) the compileall --invalidation-mode unchecked-hash pass over /usr/lib*/python3.* at the end of the 2d hygiene block (last step before erofs extraction), and (b) masked dnf-makecache.timer next to the existing rhsmcertd mask (it was the observed ~10-min post-boot trigger that re-imported the dnf Python stack). Confirmed live on a booted host via the overlay audit: dnf/libdnf/librepo/rhsm __pycache__ rewritten at boot+10min. BENCH-VERIFY: after a rebuild, no more .pyc files appear in the overlay (find / -xdev -newermt '1980-01-01' -path '*__pycache__*' empty) and the lib_t .pyc AVCs stop in the SIEM. Items 1 & 3 (firewalld enablement + tailoring) still OPEN.
3. Tailoring: re-select the 4 firewalld rules.

THE load-bearing bench gate (why #4 deferred it): the NVMe/TCP root is an OUTBOUND connection to :4420 established in the initramfs BEFORE firewalld starts. Verify it SURVIVES firewalld start + firewall-cmd --reload (established/related allowed by default → should, but UNVERIFIED); a reconnect within ctrl_loss_tmo re-establishes through the firewall; DHCP renew + MKA keepalives unaffected; SSH in; no AVCs; verity hash reproducible. If it drops the root, that is the fleet-wide outage risk — gate hard.

[~] 17. Cap fapolicyd trust-DB RAM (db_max_size) — biggest overlay consumer

Finding (overlay RAM audit, 2026-07-03, host cygnus-cruiser-99): the writable root is systemd.volatile=overlay (a tmpfs). Enumerating runtime-written files via the -T 0 trick — every baked file has mtime = epoch, so anything newer is an overlay write:

sudo find / -xdev -type f -newermt '1980-01-01' -printf '%b\n' | awk '{s+=$1*512} END{print s/1048576" MiB"}'

showed 124.5 MiB total, of which /var/lib/fapolicyd/data.mdb = 100 MiB, FULLY allocated (du == du --apparent-size, not sparse) — ~80% of the entire overlay. fapolicyd's LMDB trust DB is rebuilt at boot onto the tmpfs overlay, so its db_max_size (MB, in /etc/fapolicyd/fapolicyd.conf) is a fixed RAM reservation, not a ceiling it grows into. The EL9 default is ~100; the minimal+kiosk trust set is a fraction of that.

[x] DONE (2026-07-03): diskless.pkr.hcl 2c-STIG-postfix now seds db_max_size = 50 into fapolicyd.conf BEFORE the fapolicyd-cli --update rebuild (so the rebuilt db uses the smaller map). Value 50 chosen conservatively — it was fapolicyd's upstream default for years and covered a FULL workstation rpmdb (larger than this image), so it has real headroom. Reclaims ~50 MiB of overlay RAM.

[ ] BENCH-VERIFY (STIG-critical — fapolicyd is the writable-overlay exec guard):

1. Boot the rebuilt image; confirm kiosk renders, admin SSH (ECDSA) logs in, sudo works, and custom binaries (kiosk-session, fleet-hostapi, fleet-enrol) still execute — i.e. the trust DB rebuilt cleanly at 50 MiB with no MDB_MAP_FULL. Undersizing bricks exec (login/kiosk) — that's the hard gate.
2. Re-run the overlay audit: data.mdb should now be ~50 MiB and total overlay ~75 MiB.
3. Can go lower: measure the real trust set with fapolicyd-cli --dump | wc -l on the booted host; if the content is only ~15–25 MiB, 40 (or less) is safe and reclaims more.

Secondary lever (free, already in flight): the trust DB scales with trusted files, so the kiosk weak-deps trim (commit a193dc3, ~100 fewer packages) shrinks the actual data.mdb content — widening the headroom under db_max_size.

Other overlay consumers noted (not fapolicyd, for context): Chromium /dev/shm (~38 MiB, its own tmpfs) + kiosk /tmp/kiosk-profile (on the root overlay here — /tmp is NOT a separate tmpfs, ties to #15's mount separation); /var/log/{messages,audit} grow unbounded in RAM (audit is local-only) — logrotate caps or SIEM offload on long-lived hosts.

[~] 18. Slim the image — locale + firmware (biggest erofs-size wins)

Finding (image-size audit via the 2b rpm -qa --qf '%{SIZE}' manifest, 2026-07-03): the largest packages in the 534-package image are NOT the kiosk — they're base bloat: linux-firmware 678 MB, glibc-all-langpacks 228 MB, then chromium (370) + llvm-libs (129). The uncompressed erofs is read over NVMe/TCP on every boot of every host, so every dead byte is fleet-wide waste + attack surface.

[x] DONE — glibc langpacks → en-only (diskless.pkr.hcl step 2). @^minimal-environment pulls glibc-all-langpacks (all languages); swapped for glibc-minimal-langpack + glibc-langpack-en. ~220 MB reclaimed; image is English-only (we also strip /usr/share/locale in 2d). BENCH-VERIFY: locale still en_US.UTF-8; kiosk + login OK.

[x] DONE — linux-firmware strip to a hardware-audited keep-list (diskless.pkr.hcl step 2, run BEFORE the dracut build so the UKI initramfs shrinks too — not just the rootfs. --no-hostonly + plymouth's drm dep instmods a broad DRM set and dracut bundles each module's firmware into the initramfs, amdgpu being the classic UKI bloater; the strip runs first so dracut never sees it. A post-build diagnostic reports the .img size + any firmware still bundled — expected i915 only.) Live fleet audit (modinfo -F firmware over all loaded modules + sysfs/lspci driver bindings on the real 820-class hosts AND the VMs) proved the only loadable firmware any host requests is Intel i915 (GPU). Every other driver is firmware-free — crucially e1000e (on-chip NVM, not a fw file) and virtio_net, so NO NIC needs firmware and the fleet-brick case is off the table. Keep-list: i915 + intel/ (ice DDP insurance for the ice/mlx5 server-NIC build variant + small Intel blobs) + mellanox + intel-ucode; strip the rest (amdgpu/nvidia/radeon/ath/qcom/rtl/iwlwifi… ~600 MB). Wi-Fi (iwl*-firmware, ~83 MB) removed as packages (wired-only fleet). Strip-in-place (find … -exec rm -rf, not rpm -e — can't drop the package without losing i915); a du -sh /usr/lib/firmware line reports the result in the build log. BENCH-VERIFIED 2026-07-03 (commit f59eb79): built, booted, AND the kiosk SPA RENDERS on both the VM (virtio-gpu) and the 820 (i915). This clears the firmware strip's one real risk — the i915 DMC/GuC blobs load on real hardware, so the keep-list (i915/intel/mellanox/intel-ucode, pre-dracut) is validated end-to-end (firmware 678→147 MB, initramfs bundles only the keep-list). Remaining caveat: if a future host has a non-Intel GPU or an ice/mlx5 NIC, widen the keep-list before trusting fleet-wide. NOT yet in f59eb79 (bench-verify separately — none touch i915, so render should be safe): the nss-mdns/avahi drop (5cc72cc) and the omit_drivers GPU-kmod trim + keep-list tightening — dropped mellanox/, narrowed intel/ to ice only (853ae20). Audited models: HP-820-class (i915 + e1000e + iwlwifi-dropped) and virtio VMs.

Build-verified 2026-07-03: firmware 678 MB → 147 MB; the initramfs diagnostic confirms only the keep-list is bundled (i915 all-gens + intel/ice + intel/ish + mellanox) — proving the pre-dracut ordering works. But the initramfs came out 97 MB, and it's now kernel-MODULE-bound, not firmware (~10 MB fw): the --no-hostonly + plymouth drm set pulls every GPU driver (amdgpu.ko etc.) even though the fleet is i915-only.

[x] DONE — follow-on trims (2026-07-03):

• omit_drivers+=" amdgpu radeon nouveau " in 90-diskless.conf — drops the discrete-GPU kmods plymouth's drm dep would instmod (amdgpu.ko is the biggest single module); keeps i915 + virtio-gpu/bochs for the splash. Targets the 97 MB initramfs/UKI.
• Firmware keep-list tightened: dropped mellanox/ (those are mlxsw_spectrum SWITCH-ASIC blobs, NOT NIC fw — mlx5 NIC fw is on-card) and narrowed intel/ to just ice (dropped ish/sof/ibt = sensor-hub/audio/bluetooth, none loaded at boot).
• Initramfs top-25-files diagnostic added after the dracut build (sorts lsinitrd.txt by size) so the next build shows what dominates the .img — measure before the next cut.
• BENCH-VERIFY: kiosk still renders on the 820 (i915 kept); check the new "initramfs largest files" log line for the post-omit size + next targets.

[x] FIXED — the nss-mdns drop bricked the kiosk (regression found+fixed 2026-07-04): the "render should be safe" call above (5cc72cc) was WRONG — it broke the kiosk, just not via i915. trim-rpm-48dc3e1 booted to a flashing cursor, no kiosk, and the image was a suspicious 500 MB. Root cause: the 2d dnf remove -y nss-mdns removed chromium as a dependent package — the EPEL chromium hard-Requires nss-mdns (Chromium uses .local mDNS hostnames for WebRTC ICE candidates) — then reaped chromium's orphaned GUI stack (avahi, at-spi2, atk, adwaita, chromium-common; ~49 pkgs, 523→472). With no browser, kiosk-session died at the ${BROWSER:?} guard in 4 ms, before weston. The original commit checked what nss-mdns NEEDS (avahi) but not what NEEDS it (chromium). GPU/i915/firmware/plymouth were all healthy — a red herring. Fix (2bc2daf):

• Keep nss-mdns + avahi (satisfy chromium's Requires); mask the avahi daemon + socket instead — nss-mdns is only an NSS module, needs no listener, and :5353 was the real attack-surface concern. ~1.5 MB kept.
• FINAL GUARD in 2d: rpm -q chromium chromium-common weston mesa-{dri-drivers,libEGL,libgbm} asserts the kiosk stack survived the whole pipeline, else fail the build. A missing browser must brick the BUILD, never the fleet — catches any future remove-cascade / oscap package_*_removed / autoremove. Look for ==> guard: kiosk stack intact in CI. RE-VERIFIED 2026-07-04 (trim-rpm-2bc2daf): boots into the kiosk on the 820; image 755 MiB.

Measured end-to-end (2026-07-04, trim-rpm-2bc2daf): container artifact image 1.8 GiB → 755 MiB (~58%, >1 GB off every host's per-boot NVMe/TCP read).

Combined: ~800 MB off the uncompressed erofs (langpacks + firmware), dwarfing the kiosk weak-deps trim (#4), plus a smaller UKI/initramfs from the DRM-driver omit. NEXT candidates from the manifest (measure, don't guess): iso-codes (19 MB), samba-client-libs/intel-mediasdk/gstreamer (chromium-common HARD deps — only removable by dropping HTML5 media, and samba/rabbitmq/rist/srt/zmq are libavformat DT_NEEDED so they can't go without breaking future media — confirmed via ldd 2026-07-03), binutils (25 MB, build-only?), cracklib-dicts (9.8 MB, needed for pam pwquality — keep).

Stack note: committed on trim-rpm. When optimise-ram rebases onto the new trim-rpm, expect a trivial docs/TODO.md tail conflict (both branches appended after #16 — optimise-ram added #17 fapolicyd, this added #18); resolve by keeping both.