# TODO — diskless fleet boot system Persistent backlog, kept in sync with the working session task tracker. Status: `[ ]` pending · `[~]` in progress · `[x]` done. --- ## Current build posture (LAB / test) — as of 2026-06-19 The image currently builds in **open-network test mode**. These toggles must be flipped back for a secure/production build (consolidated as item #8): - `disable_dot1x=true` / CI `DISABLE_DOT1X="true"` → `rd.dot1x=0` (NO 802.1X / MACsec; open NVMe-oF). **TEMPORARY.** - `nvmet_traddr` / CI `NVMET_TRADDR="192.168.50.10"` → hardcoded IP (should be a per-env DNS name like `nvme.fleet.lab` once DNS-at-boot is trusted). - Unsigned UKI (`sb_key`/`sb_cert` not passed) → Secure Boot must be OFF on the test target. Production needs CI/HSM signing. **NOTE (2026-07-11, `poam-hardening`): the main CI now fails closed** — a "Production build gate" aborts unless `SB_KEY`/`SB_CERT`/`FLEET_CA` are set, so an unsigned/placeholder-CA build is no longer producible via CI (only a local `packer build` without keys still yields the lab warning). The `fleet-ipxe` build is gated the same way. - `enable_kiosk=true`; `boot_splash=true`. - **`secure_ntp` — DONE (2026-07-11, `secure-ntp`).** NTP is now **NTS (CA-authenticated)** by default: the host verifies the serving NTP server's fleet-signed NTS-KE cert against `radius-ca.pem` (closes F-03/P-18; docs/secure-ntp-design.md). **No baked FQDN** — the NM dispatcher derives the server name from the DHCP domain at runtime (`ntp.`), so a site-domain change needs no rebuild. `secure_ntp=false` reverts to the legacy unauthenticated DHCP option-42 path. Serving needs `serving/chrony/nts-cert.sh` (DNS SAN `ntp.`), DHCP option-15 domain handout, + tcp/4460 open. **BENCH-VERIFY** `chronyc -N authdata` (NTS cookies) + `$DHCP4_DOMAIN_NAME` populated. - **`enable_stig` — DONE (2026-07-11, `poam-hardening`).** CI `ENABLE_STIG="true"` and, more importantly, the Packer template now **enforces** `enable_stig=true` via variable validation — a build with it false fails `packer validate`/`build`. There is no unhardened build path anymore. (Was temporarily off during the STIG regression diagnosis; that escape hatch is gone.) - **`debug_shell` — REMOVED (2026-07-11, `poam-hardening`).** The DEBUG HOLE (passwordless root shell) and its `debug_shell` variable are deleted from the template entirely; no build can bake it (closes POA&M P-02 / R-02). Offline diagnosis via `rootfs.erofs` mount + SMBIOS `enforcing=0` remains the supported path (SMBIOS cmdline-extra works on Proxmox only after a full VM **stop/start**, not `reset`). - **Admin SSH key MUST be ECDSA or RSA, NOT ed25519**, when STIG is on — the STIG crypto policy drops ed25519 from sshd's accepted pubkey algorithms. Set the `SSH_AUTHORIZED_KEY` CI secret to an ECDSA/RSA key (see #2). - **Hostname:** image is hostname-neutral (empty `/etc/hostname`, `hostname-mode=default`); the serving dnsmasq's `fleet-namer` (`--dhcp-script`) assigns a stable creative starship name per MAC (e.g. `nebula-vanguard-07`) via DHCP option 12 + DNS A/PTR. Host picks it up on next DHCP renewal/reboot. - **Serving `.env`:** must set `AUDIT_HOST=audit.fleet.lab` (+ `docker compose up -d dnsmasq` to reload) for remote log offload to resolve/flow. Since the TLS upgrade (#8, 2026-07-12) the collector also needs its server cert issued (`audit-collector/issue-cert.sh`) + 6514/tcp open on the boot NIC, and the image must be built with the real fleet CA (`fleet_ca`) or senders spool. What's working/verified vs awaiting bench is per-item below. Cross-cutting bench-verify checklist lives in `CLAUDE.md` → "Known bench-verify items". --- ## [x] 1. Port the dracut module off `network-legacy` → `network-manager` **DONE — test path VERIFIED ON THE BENCH 2026-06-17** (host stays up past the DHCP lease time; no more lease-expiry outage). Fixed the confirmed outage: network-legacy's one-shot DHCP lease was never renewed post-switch-root, so the address dropped at lease expiry → NVMe/TCP root died ("keep alive timeout" / "no route to host"). NM hands its initramfs connections to the real-root NM (`/run/NetworkManager` state), which renews. REMAINING: the **secure path** (macsec0 DHCP via `nmcli`, physical NIC unmanaged via `nm-unmanaged.sh`) is only exercised once `disable_dot1x=false` — verify on the swtpm + hostapd + FreeRADIUS bench when re-enabling security (see #2/cleanup). Done: - `90-diskless.conf` `add_dracutmodules`: `network-legacy` → `network-manager`. - `module-setup.sh` `depends()` → `network-manager`; added `nmcli`; installs the new cmdline hook. - NEW `nm-unmanaged.sh` (cmdline hook): secure mode writes `/run/NetworkManager/conf.d/10-dot1x-unmanaged.conf` (`unmanaged-devices=interface-name:en*`) **before NM starts**; no-op in test mode. Replaces the deleted static `99-dot1x-unmanaged.conf` (couldn't be mode-aware — it wrongly unmanaged `en*` in test mode, blocking renewal). - `dot1x-macsec-up.sh` secure path: `ifup macsec0` → `nmcli` (managed + DHCP, poll for IPv4). macsec0 MANAGED so NM renews; physical NIC UNMANAGED. - `diskless.pkr.hcl`: dropped `dhcp-client`, added explicit `NetworkManager`; removed the static-config install; lsinitrd now asserts `NetworkManager` + `nm-unmanaged.sh` present. - Static lint clean (`bash -n`, `shellcheck`, `packer validate`). **BENCH-VERIFY before trusting (the gate):** - Test path (`disable_dot1x=true`): boot, `nmcli device` shows boot NIC MANAGED; run past the DHCP lease time (drop the lab dnsmasq lease low) — address persists, lease renews (`journalctl -u NetworkManager`), NVMe + SSH stay up. - Secure path (swtpm + hostapd + FreeRADIUS, `disable_dot1x=false`): `en*` UNMANAGED, `macsec0` MANAGED + renewing; MKA survives switch-root; verity root mapped; NVMe stays connected past lease time; no "Address already assigned". - **Risk:** ordering of the unmanaged drop-in vs NM start in secure mode. If NM transiently grabs `en*`, fall back to also baking a static `en*`-unmanaged file for secure-only builds. ## [~] 2. Apply RHEL9 STIG hardening to the built image **IMPLEMENTED 2026-06-18 — awaiting build/bench verification.** OpenSCAP + SSG remediation in a provisioner (`enable_stig` var, default on), tailored for the diskless/UKI/kiosk architecture via `files/stig/diskless-tailoring.xml`. STIG kernel params baked into the UKI cmdline (`extract-artifacts.sh`). Decisions: **FIPS off**, **audit local-only**, **CI report-only**. Deviations are documented in **[stig-deviations.md](stig-deviations.md)** (curated register) and reported per-build by **[../scripts/stig-deviations.sh](../scripts/stig-deviations.sh)** (non-pass rules tagged intentional vs REVIEW); CI publishes `stig-report.html` + `stig-arf.xml` + `stig-deviations.md`. Scanner tooling is removed from the shipped image. **BENCH-VERIFY:** boot/login **VERIFIED ON THE BENCH 2026-06-19** — hardened build with `enable_stig=true`, host `cygnus-cruiser-99`: kiosk auto-starts + renders, admin SSH (ECDSA) logs in, audit offload reaches the collector, DHCP creative hostname applied, NVMe/TCP verity root mounts and the host stays up. **STILL OPEN:** reproducible verity hash across two builds; triage the REVIEW rows (below); the secure path (#8, 802.1X/MACsec — this was the `disable_dot1x=true` test path). **FINDING 2026-06-18 (FIXED) — false-clean scan.** The first hardened build (branch `hardening`, `ada4aa4`) produced a report reading "0 REVIEW / clean", but the ARF had **0 pass, 0 fail** — only 462 `notapplicable` + 1068 `notselected`. Root cause: the provisioner selected the datastream with `ls ssg-rhel9-ds.xml ssg-rl9-ds.xml | head -1`, which picks **rhel9** alphabetically; the RHEL9 datastream's CPE dictionary has **no Rocky entry**, so on Rocky 9.8 every selected rule went `notapplicable` and **nothing was actually evaluated or remediated** — the image was unhardened despite a clean-looking report. Fixes: (a) provisioner now maps `/etc/os-release $ID` → datastream (Rocky→`ssg-rl9-ds.xml`); (b) provisioner **aborts the build** if the scan evaluates 0 rules; (c) `stig-deviations.sh` flags an **INVALID SCAN** and exits non-zero on 0-evaluated. **Re-run the hardened build** and confirm pass/fail counts are non-zero before trusting any deviations report. **FINDING 2026-06-18 (FIXED) — hardening locked the fleet out.** The first genuinely-evaluated hardened image (app `215b0eb`) broke **both** login paths: the Chromium kiosk dropped to a getty login prompt, and admin SSH was rejected with `Permission denied (publickey)`. Root cause: STIG enables `pam_faillock` (account phase) + password aging/inactivity, and the **kiosk + fleetadmin accounts are passwordless / key-only**, so the hardened PAM *account* phase treats them as expired/lockable (sshd key auth succeeds, then `pam_acct_mgmt` fails). Fix (provisioner `2c-STIG-postfix`, runs after remediation): exempt the two appliance accounts — `chage -M -1 -m 0 -I -1 -E -1` + `faillock --reset`; kiosk uses a dedicated `/etc/pam.d/kiosk` (`PAMName=kiosk`, permissive account / full session for seat); sshd gets an `account sufficient pam_succeed_if.so user = fleetadmin` first-rule bypass. Scoped to those two accounts; everyone else keeps full STIG. Justified as register deviation #10. **VERIFIED ON THE BENCH 2026-06-19** (`enable_stig=true`, host `cygnus-cruiser-99`): kiosk auto-starts + renders and admin SSH (ECDSA key) logs in with hardening on.** **UPDATE 2026-06-19 — offline erofs inspection of the hardened image (mounted `rootfs.erofs` read-only) pinned the real causes; the PAM-account theory above was incomplete:** - **Kiosk had TWO causes, both FIXED:** 1. **fapolicyd** is `enabled` and `kiosk-session` wasn't in its trust → it blocked `/usr/local/bin/kiosk-session`. Fix: KEEP fapolicyd (it's the exec-guard for the writable overlay dm-verity can't cover) and add the kiosk binary to its trust DB at build (2c-STIG-postfix). Not waived. 2. **Auto-start:** kiosk was `WantedBy=graphical.target` but the host boots to `multi-user.target` (graphical never reached) → unit `inactive (dead)` at boot, fine on manual `restart`. Fix: `WantedBy=multi-user.target` + dropped `set-default graphical.target` (kiosk launches weston directly, needs no DM target). - **SSH = STIG crypto policy excludes ed25519 (CONFIRMED via `ssh -vvv`).** The key fingerprint MATCHED (not a mismatch); the server's `server-sig-algs` advertised only ecdsa + rsa-sha2 — **no `ssh-ed25519`** — so the ed25519 admin key is rejected at the signature stage (before PAM/SELinux, which is why `enforcing=0` and the account-bypass didn't help). **Fix (your action): set the `SSH_AUTHORIZED_KEY` CI secret to an ECDSA (`ssh-keygen -t ecdsa -b 384`) or RSA key**, then rebuild. No code change. (Alt: re-allow ed25519 via `PubkeyAcceptedAlgorithms +ssh-ed25519` in the sshd drop-in — but that's a non-approved algo and would re-fail a STIG rule; prefer the ECDSA key.) - **rsyslog TLS-offload rules waived (lab):** `rsyslog_encrypt_offload_*` force gtls+x509+CA and broke the plaintext `2f` forward → unselected in the tailoring; production TLS offload tracked under #8. **(DONE 2026-07-12 — 2f is now gtls/6514 and the three waivers are dropped; see #8.)** - Diagnosis tooling notes: SMBIOS cmdline-extra works only after a Proxmox VM **stop/start** (not `reset`); the debug-shell-on-tty2 never came up (and is now removed from the template — see the build-posture section) — the decisive diagnosis was **offline `rootfs.erofs` mount** + `ssh -vvv` + `enforcing=0`. **REVIEW-ROW TRIAGE — DONE 2026-06-19 (scan `app-9f7d8d1`, 21 REVIEW → 1 open).** The post-fix hardened scan was valid (398 pass / 12 fail / 9 notchecked). All 21 REVIEW rows dispositioned (full detail: [stig-deviations.md](stig-deviations.md) #12). Mechanics: a rule only stops being flagged REVIEW if it's `selected="false"` in `diskless-tailoring.xml` (the register prose alone doesn't clear it) or made to pass / N/A. - **Bucket A (waived, register-justified):** `encrypt_partitions`, `auditd_audispd_configure_sufficiently_large_partition` (#1); `grub2_password` (#2); `sysctl_crypto_fips_enabled`, `harden_sshd_macs_openssh_conf_crypto_policy`, `harden_sshd_macs_opensshserver_conf_crypto_policy` (#5); `configured_firewalld_default_deny` (#4); `sssd_enable_certmap`, `sssd_has_trust_anchor` (#7). - **Bucket B (waived N/A):** `account_temp_expire_date`, `accounts_user_home_paths_only`, `libreswan_approved_tunnels`, `security_patches_up_to_date`, `ssh_keys_passphrase_protected`. - **Bucket C (decisions applied):** `installed_OS_is_vendor_supported` (Rocky IS supported; RHEL-datastream CPE artifact — waived), `network_configure_name_resolution` (single DHCP DNS by design — waived), `sysctl_user_max_user_namespaces_no_remediation` (KEEP userns for Chromium sandbox — waived), `accounts_authorized_local_users` (appliance accounts authorized, #10 — waived; prod: set the authorized-users regex var instead), `package_postfix_installed` + `postfix_prevent_unrestricted_relay` (**postfix removed** in 2c-STIG — no MTA needed; audit goes off-box via 2f), **USBGuard** `configure_usbguard_auditbackend` + `usbguard_allow_hid`/`_hub`/ `_hid_and_hub` (**ENABLED + enforcing**: LinuxAudit backend + hardware-independent HID/hub allowlist written in 2c-STIG; ⚠ BENCH-VERIFY physical console keyboard/ mouse on real hardware — the build-VM-generated policy + default block target would otherwise lock out console input). - **`selinux_context_elevation_for_sudo` — REMEDIATED WITH CONFINEMENT (decision 2026-06-20).** 2c-STIG now rewrites the admin sudoers to `TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL` and maps the admin to **`staff_u`** (`semanage login`) so the sudo role transition is authorized. ⚠ **TWO BENCH GATES:** (a) admin SSH login + sudo + privileged ops must work under `staff_u` confinement (if it locks out, waive the offending rule in the tailoring as a justified deviation — `enable_stig=false` is no longer available, STIG is enforced-true); (b) **reproducibility** — `semanage` rebuilds the policy store; confirm the verity hash is stable across two builds (normalize in 2d if it drifts). Needs `policycoreutils-python-utils` (installed in 2c-STIG if absent). **Expected next-build report: 0–1 REVIEW** (0 if both postfix→N/A and usbguard→pass land and sudo confinement holds; the triage was validated by re-running `scripts/stig-deviations.sh` against the app-9f7d8d1 ARF with the new tailoring — 21 REVIEW → 3, where the 3 are exactly postfix/usbguard/sudo, all addressed for the next build). Remaining (future hardening, currently waived — see the register): FIPS, firewall, per-host/rotatable admin access. **Remote audit offload — WIRED both ends** (2026-06-19): receiver `serving/audit-collector/` (rsyslog TCP/514, per-source-host files) + sender in the image (provisioner 2f, `audit_host` var: audisp-syslog→LOCAL6 + rsyslog `omfwd` to `audit.fleet.lab`). **UPGRADED TO TLS 2026-07-12** (gtls/6514, fleet-CA x509/name; see #8 — BENCH-VERIFY pending; the plaintext-era notes below record the original bring-up). **VERIFIED END-TO-END ON THE BENCH 2026-06-19** — booted host `cygnus-cruiser-99` (192.168.50.148) forwards to the collector, which writes `audit-collector/logs/192.168.50.148/{audit,messages}.log`. **Serving-host firewall gotcha hit + fixed:** the collector is host-networked so the serving host's **ufw** governs :514 (Docker doesn't manage iptables for host-net containers); 514/tcp was bound to the wrong NIC (`ens18` instead of the boot-LAN `ens19`), so SYNs were silently dropped (`SYN-SENT` / "connection timed out", not refused). Fix: `ufw allow in on ens19 to any port 514 proto tcp`. The sender's `action.resumeRetryCount="-1"` buffered the backlog and drained on reconnect — no records lost. Documented in [../serving/audit-collector/README.md](../serving/audit-collector/README.md). When the **workstation** firewall is re-enabled (#4 / register #4) it needs the reciprocal **egress** allow for 6514/tcp (TLS; was 514/tcp) to the collector. **SIEM (Wazuh) added 2026-06-27 — [serving/siem/](../serving/siem/).** Upgrades the "Detect" pillar from file-collection-only to parse/search/correlate/alert. **Runs on a DEDICATED node — `wazuh1.bitasylum.org`** (managed independently), keeping the memory-hungry indexer JVM **off** the serving host so it can't OOM SPDK (the fleet's root disk / SPOF, #6/#7). **Agentless** (diskless hosts stay stateless, invariant #1): the serving-host audit-collector **relays a copy** to wazuh1's manager syslog listener (`omfwd` → `wazuh1.bitasylum.org:514` in `audit-collector/rsyslog.conf`; rebuild the collector to apply) while still writing durable per-host files. `serving/siem/` is the INTEGRATION (the `` syslog block to add on wazuh1 + reference deploy overlay), not a local deployment. **DECODING BENCH-VERIFIED 2026-06-27** (`wazuh-logtest`): the audisp-syslog records carry a `node=` prefix (STIG `name_format=hostname`, SSG `auditd_name_format` — kept, it's compliance) that breaks Wazuh's built-in auditd decoder, so a custom **`fleet-auditd`** decoder + rules (`serving/siem/config/local_decoder.xml` + `local_rules.xml`, mounted declaratively via `/wazuh-config-mount`) decode them into the **`fleet_audit`** group. **Decode + alert confirmed:** `wazuh-logtest` shows `decoder.name=fleet-auditd` + `audit.type` extracted (base rule `100100`); and a live `sudo` on a booted host (`cygnus-cruiser-99`) surfaced `USER_CMD` → rule `100130` and `CRED_*`/`USER_ACCT` → rule `100110` (level 5) in the dashboard Threat Hunting view. Wazuh runs multi-node on `wazuh1.bitasylum.org`. **REMAINING (prod):** the relay→wazuh1 hop is **plaintext** — **TLS it** (gtls/6514, fleet CA; ISM protect-logs-in-transit) since it's now on the wire (NB: the host→collector hop is TLS since 2026-07-12, #8 — this is the separate collector→wazuh1 hop, and the collector-side `omfwd` must stay/`ptcp` until wazuh1 has a TLS listener); rotate all Wazuh creds; restrict dashboard/API to admin hosts; set `allowed-ips`→/32 of the serving host; ISM retention/archival; then update the SSP "Detect" + ISM event-logging controls. ### Original spec (retained for reference) OpenSCAP + SCAP Security Guide (`xccdf_org.ssgproject.content_profile_stig`), run as a Packer provisioner after `dnf install` and before the dracut/initramfs build and cleanup. - **Tailoring** (`autotailor`): unselect rules this architecture can't satisfy — separate-partition rules (single erofs root + tmpfs overlay; satisfy intent with `nodev,nosuid,noexec` tmpfs mounts), GRUB/kernel-cmdline rules (no GRUB), and AIDE/file-integrity rules (dm-verity is the documented compensating control). - **Kernel params → UKI cmdline** in [../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh) before signing: `audit=1 audit_backlog_limit=8192 slub_debug=P page_poison=1 vsyscall=none` (+ `fips=1` if FIPS). - **Reproducibility guard:** strip non-deterministic remediation output (SSH host keys → regen at boot into overlay, AIDE db, timestamps, fresh UUIDs) so the verity hash stays stable. - **Audit logging:** `audisp-remote`/rsyslog to a central collector addressed by DNS name (host-generic). - **Optional FIPS:** `fips=1` cmdline + `01fips` dracut module + FIPS crypto policy. - **CI:** run `oscap eval` (no `--remediate`) to publish an ARF+HTML STIG report as a build artifact; optionally hard-fail below a score threshold. - **Decisions needed before implementing:** FIPS on/off · remote-audit collector DNS name (or ephemeral/none for lab) · CI fail-on-score threshold vs report-only. - Note: orthogonal to the `rd.dot1x=0` test toggle, but a full STIG production posture wants 802.1X re-enabled. - **Credentials (partly done):** root is now **locked** and access is via a baked sudo-enabled key-only admin user (`var.ssh_authorized_key` / `var.admin_user`, `wheel` + NOPASSWD sudo), with sshd drop-in `PermitRootLogin no` / `PasswordAuthentication no`. Two items for STIG to reconcile: **NOPASSWD sudo** is a finding (acceptable for a key-only account, but document/deviate), and the admin key/user are **fleet-wide baked** — for per-host/rotatable access without rebuilds, layer the signed HTTP/SMBIOS config-agent or an SSH CA (`TrustedUserCAKeys`) discussed in session. - **SSH host keys (done):** `extract-artifacts.sh` strips `/etc/ssh/ssh_host_*` so they're not baked (baked keys = shared fleet identity + the build's perms/group don't survive the tar→erofs round-trip, so sshd rejected them "Permissions 640 too open" and refused to start). RHEL's `sshd-keygen@.service` regenerates them into the overlay before sshd starts. **⚠ Caveat:** the ephemeral overlay means they regenerate **every boot** → the host key changes per reboot (clients warn). **RESOLVED (F-02/P-17, bench-verified 2026-07-11 on comet-beacon-54):** rather than persist a key, each per-boot key is signed into an OpenSSH host certificate by the fleet SSH CA via the enrol-broker host plane at boot (`files/ssh-hostcert` → `fleet-ssh-hostcert`); admins trust one `@cert-authority` line instead of TOFU. See [ssh-host-certificate-design.md](ssh-host-certificate-design.md). Bench-surfaced fixes on branch `poam-hardening` (`28d31e4`). ## [ ] 3. Roll the errata into design doc v3 Once the boot path is validated and item 1 lands, fold [design-v2-errata.md](design-v2-errata.md) into the source design document and re-export a **v3** PDF (key change: `rootovl` → `systemd.volatile=overlay` in §3 step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source (not in this repo). ## [x] 4. Kiosk mode — autologin + fullscreen Chromium loading a URL **WORKING (display-verified 2026-06-17).** Remaining bits are polish (GPU/mesa trim, ship via CI `enable_kiosk=true`) and the phone-home/NTP lockdown (done) — all captured in the sub-notes below. Secure-path render check is folded into #8. **Package-footprint trim (2026-07-03):** the kiosk dnf transaction was measured at **229 pkgs / 1.0 GB** (weak deps ON). Added `--setopt=install_weak_deps=False` to the install in `diskless.pkr.hcl` — drops the flatpak/ostree sandbox runtime, tracker-miners (fs indexer = dead weight + threat surface), the pipewire/pulseaudio audio stack, and `libproxy-webkitgtk4 → webkit2gtk3-jsc` (an 8.9M *second* browser engine). Also added an `rpm -qa --qf %%{SIZE}` size-manifest diagnostic to the build log to quantify the result. The immovable floor is chromium + chromium-common + llvm-libs (mesa sw-GL) + gtk3/adwaita + libicu — all HARD deps. **BENCH-VERIFY:** with weak deps off, confirm the kiosk still renders (no genuine runtime dep was a weak dep — watch for a missing font or `libva`). NEXT trim candidates (measure via the new manifest): gstreamer/media codecs if the SPA plays no HTML5 media; mesa-dri-drivers + `--use-pixman`-only if the GPU path is abandoned. **WORKING / display-verified 2026-06-17** (kiosk renders Chromium fullscreen on the configured URL). Build-flagged OFF by default for now (`enable_kiosk=false`) while iterating on boot/network — set `-var enable_kiosk=true` to ship it. Path to getting here (don't re-discover): - **cage is NOT packaged for EL9** (`dnf install cage` → "Unable to find a match"). Tried weston instead. - **EPEL 9 ships weston 8.0**, which has NEITHER `kiosk-shell.so` NOR the `[autolaunch]` weston.ini section — both arrived in **weston 9.0**. The first cut wrongly assumed 8.0 had them; `shell=kiosk-shell.so` failed to load. `rpm -ql weston` shows only desktop-shell / fullscreen-shell / ivi-shell. - **Resolution: `desktop-shell.so`** (the only packaged shell Chromium's xdg-shell works with — fullscreen-shell needs a client protocol Chromium doesn't speak), stripped bare (`panel-position=none`, black bg, no animation/lock), and Chromium launched by a **wrapper** (kiosk-session) since there's no `[autolaunch]`. Chromium `--kiosk` fullscreens over the shell. - **Seat fix (load-bearing):** weston died with `logind: failed to get session seat` → `failed to create compositor backend`. logind's auto-derivation didn't attach a seat; `Environment=XDG_SEAT=seat0` + `XDG_VTNR=1` in the unit (put into the PAM env before pam_open_session) registers the session on seat0/vt1. `/dev/dri/card0` was present all along (not the issue). - Tested live without a rebuild: the whole root is `systemd.volatile=overlay`, so /etc AND /usr are writable tmpfs at runtime — drop-in the files + restart. - **`enable_kiosk` Packer var — currently `false` by DEFAULT (set 2026-06-17).** The whole kiosk provisioner is feature-flagged off while iterating on the boot/network stack (DHCP-lease/keep-alive work), for faster builds and a smaller image. **Re-enable with `-var enable_kiosk=true` when returning to kiosk work** — CI does not pass the var, so CI currently builds lean (no kiosk) too; add `-var enable_kiosk=true` to the Gitea workflow to ship it. - `kiosk_url` Packer var (default `http://boot.fleet.lab/kiosk/` — the kiosk **SPA** (`serving/kiosk-app/`, Svelte 5 + Tailwind v4, a cyber sci-fi spaceship monitoring/control terminal) served by the boot HTTP server, which **reverse-proxies** `/kiosk/` to the `kiosk` container; self-contained so a boot validates without external connectivity. Empty disables the whole install). Wired in CI as `KIOSK_URL` env. - **Kiosk SPA (`serving/kiosk-app/`)** — replaced the old single static `serving/http/kiosk/index.html` (2026-06-18). Top navbar + 5 pages (Bridge/Reactor/Nav/Life Support/Comms) with dummy telemetry; hash routing; builds in a 2-stage Docker image (Vite → nginx), reverse-proxied at `/kiosk/` by the main serving nginx (loopback `127.0.0.1:${KIOSK_APP_PORT:-8088}`). `npm run build` verified locally; container build untested (no docker here). Dummy data lives in `src/lib/data.js` (swap for a fleet API later). - Provisioner enables EPEL, installs `weston chromium mesa-dri-drivers mesa-libEGL mesa-libgbm dejavu-sans-fonts`, creates a passwordless unprivileged `kiosk` user (video/input groups). - `files/kiosk/kiosk.service` — runs `/usr/local/bin/kiosk-session` as `kiosk` on tty1 via `PAMName=login`+`TTYPath`+`Environment=XDG_SEAT=seat0`/`XDG_VTNR=1` (gets seat0 for DRM/input), `Conflicts=getty@tty1`, `WantedBy=graphical.target`; default target set to graphical. - `files/kiosk/weston.ini` — `shell=desktop-shell.so`, `idle-time=0`, `require-input=false`, `xwayland=false`; `[shell] panel-position=none`, black `background-color`, `animation=none`, `locking=false`. - `files/kiosk/kiosk-session` — `exec`s `weston --use-pixman --socket=wayland-kiosk` in the foreground (owns the seat), and launches Chromium (`--kiosk --ozone-platform=wayland --disable-gpu`, tmpfs profile/cache, URL from `/etc/kiosk/kiosk.env`) in a background crash-restart loop. - **Remaining polish (now that it renders):** dropping `--disable-gpu` on real GPUs, `/dev/dri` present in the VM (weston needs a DRM device — set Display to VirtIO-GPU/std), and **SELinux AVCs** for the GUI stack (check `ausearch -m avc`; the kiosk user is unconfined so likely none — but do NOT set permissive if they appear; add a policy module). Admin SSH (the key user) still works for debug even if the kiosk fails (it lands at graphical.target either way). - **Phone-home lockdown (done 2026-06-17/18):** Chromium was dialing Google (accounts/variations/optimization-guide/component+auto update/safe browsing — seen in dnsmasq logs). Two layers: - **Flags** in `kiosk-session` (`--disable-background-networking`, `--disable-component-update`, `--disable-features=OptimizationHints,...`) — confirmed killing variations/optimization-guide/update queries. - **Managed policy** `files/kiosk/chromium-kiosk-policy.json` for the policy-governed bits (sign-in `accounts.google.com`, sync). FIX 2026-06-18: the managed-policy dir differs by build, so it's installed to BOTH `/etc/chromium/policies/managed/` AND `/etc/chromium-browser/policies/managed/` (a policy in the wrong dir is silently ignored — the reason `accounts.google.com`/`android.clients.google.com` persisted). Verify via `chrome://policy` (Status: OK per entry). - **Serving-side sinkhole** (defense-in-depth, `serving/docker-compose.yml` dnsmasq `--address=/google.com|googleapis.com|gstatic.com/0.0.0.0`): the kiosk is local-only, so block egress at the resolver. Stops the *leak*; the query still logs (answered 0.0.0.0, not forwarded) — eliminating the query is the browser policy's job. Production: do this at the boot-LAN firewall. - **NTP egress closed (done 2026-06-18):** the `*.pool.ntp.org` lookups were the OS (chrony), not Chromium. Now: a `chrony` NTP server in the serving stack (`serving/chrony/`), dnsmasq hands it via **DHCP option 42**, and the image's NM dispatcher (`files/etc/NetworkManager/dispatcher.d/10-chrony-dhcp-ntp`) feeds it into chrony's `sourcedir /run/chrony-dhcp`; the baked public pool is stripped from `/etc/chrony.conf`. Result: the workstation syncs time only from the lab server — no external NTP egress. BENCH-VERIFY: on a booted host `chronyc sources` shows the lab server (the DHCP one), not a public pool. **SUPERSEDED (2026-07-11, `secure-ntp`) by `secure_ntp=true`:** the DHCP option-42 path is now the *unauthenticated fallback*; the default is NTS/CA-authenticated time (the DHCP source is no longer mixed in). See the build-posture note above + docs/secure-ntp-design.md. - **Image size:** Chromium+Mesa add hundreds of MB to the *uncompressed* erofs — feeds the "shrink the rootfs" concern (and #7); reconsider whether a lighter browser/render path is viable. ### Original task Make the diskless workstation boot straight into a locked-down kiosk: a fullscreen Chromium showing a configured site (default: the boot HTTP server's `/kiosk/` test page). All baked at **build time** into the root tree (immutable erofs/verity root; runtime is the tmpfs overlay), fleet-generic (no per-host state). - **Graphical stack (keep it minimal):** add a Packer provisioner installing a lightweight kiosk compositor + browser. Preferred: `cage` (Wayland single-app kiosk compositor) + Mesa (software GL fallback for VMs/no-GPU); X11 + a minimal WM (matchbox/openbox) as the alternative. Chromium on Rocky 9 is in **EPEL** — enabling EPEL adds a repo/supply-chain surface (decision below). - **Autologin + session:** dedicated unprivileged `kiosk` user; launch the session via a `kiosk.service` running `cage -s -- chromium --kiosk …` (or `greetd` + cage). Stage units/configs under `files/` like the other provisioned config. - **Chromium flags:** `--kiosk --noerrdialogs --disable-infobars --incognito --no-first-run --disable-session-crashed-bubble `. Writable profile/cache land on the `systemd.volatile=overlay` tmpfs (RAM-backed, ephemeral) — fine, but watch RAM sizing; pin `--user-data-dir`/`--disk-cache-dir` to a tmpfs path if needed. - **Kiosk URL = Packer var** (e.g. `kiosk_url`, default the target site), baked into the launch command; keep it host-generic (per-env via DNS if ever needed). - **Verify in boot test:** graphical session comes up on real/VM display, Chromium fullscreen loads the URL over the booted network (works with the current `rd.dot1x=0` test path / open network). - **Decisions:** Wayland(cage) vs X11; accept EPEL for Chromium (vs another source); confirm the target URL. - **⚠ Conflict with item 2 (STIG):** passwordless autologin and a desktop browser are STIG findings and enlarge the attack surface/image. Reconcile — e.g. a tightly-constrained kiosk session + documented deviation — when hardening lands. ## [x] 5. Deploy & validate the end-to-end boot path (open-network test) ✅ **DONE (2026-06-15).** Full chain validated on QEMU/Proxmox: UEFI HTTP boot → initramfs → DHCP → `nvmf-connect.service` auto-connects the NVMe/TCP subsystem → two namespaces (data + verity hash) appear by pinned UUID → `systemd-veritysetup` maps `/dev/mapper/root` → uncompressed erofs mounts → `systemd.volatile=overlay` → multi-user. Booted with `rd.dot1x=0` (open network) and the IP discovery address. See bench findings below for the full list of fixes. Remaining cleanup to leave the lab-test posture (now unblocked): revert the hardcoded IP to the `nvme.fleet.lab` DNS name, and re-enable security (`DISABLE_DOT1X=false`) — the secure path needs the same connect trigger over macsec0 (see ⚠ note below). Original task: stand up `serving/` (dnsmasq + nginx + SPDK) per [serving/README.md](../serving/README.md), rebuild the UKI with `nvmet_traddr=nvme.fleet.lab` (now a Packer var — older UKIs have the literal `NVMET_TRADDR` placeholder and won't connect), then boot a UEFI HTTP-boot client (**Secure Boot OFF**, unsigned UKI; no TPM needed in `rd.dot1x=0` mode) on the boot LAN. - **Pass criteria:** UKI fetched over HTTP → initramfs → DHCP on NIC → NVMe/TCP namespace connected → dm-verity root mapped → `systemd.volatile=overlay` writable → multi-user. - This validation **gates** item #1 (network-manager port) and item #3 (errata → design v3). - **Bench findings (boot debug, 2026-06-14):** - Initramfs has no resolver (dracut writes `resolv.conf` only via its own DHCP path, not a manual `dhclient`), so the NVMe discovery address must be an **IP**, not a DNS name — build with `-var nvmet_traddr=192.168.50.10`. Old UKIs carrying `nvme.fleet.lab` can't connect early in boot. - NVMe controller indices are **not stable**: a transient discovery controller can take `nvme0`, pushing the I/O controller to `nvme1`. The hardcoded `/dev/nvme0n1` in the cmdline was a latent brick. **Fixed:** namespace UUID pinned on the target (`spdk-setup.sh` `add_ns -u f1ee7000-0000-4000-8000-000000000001`) and the cmdline now references `/dev/disk/by-id/nvme-uuid.` (`extract-artifacts.sh`). The two constants MUST stay in sync; make them version-specific for #6. - `nvme_tcp`/`nvme_fabrics` must be loaded before `/dev/nvme-fabrics` exists (dracut's `nvmf` module does this in a normal boot; at `rd.break` modprobe manually). - **Don't double-configure the NIC in test mode.** With `ip=dhcp` on the cmdline, dracut's network module brings the NIC up AND emits the "online" event that the `nvmf` module's `initqueue/online` hook waits on to run `nvme connect-all`. The `rd.dot1x=0` bypass used to *also* `ifup $NIC`, which (a) raced dracut → `Error: ipv4: Address already assigned`, and (b) didn't fire the nvmf online hook, so the namespace never auto-connected and boot spun on `/dev/mapper/root` forever (manual `connect-all` worked, which is what masked it). **Fixed:** the bypass branch no longer touches the NIC; `ip=dhcp` owns networking. (The secure path still addresses macsec0 itself — dracut doesn't know about that interface.) - **dracut's `nvmf` module never auto-connects on this distro.** Its cmdline hook (`92-parse-nvmf-boot-connections.sh`) parses `rd.nvmf.discover` into `/etc/nvme/discovery.conf` correctly, but `initqueue/online/` is **empty** — no autoconnect hook is registered, so `nvme connect-all` is never run (no attempt in dmesg) and the namespace never appears. Drivers load and `/dev/nvme-fabrics` exists; only the connect trigger is missing. A job placed on the *main* initqueue from a settled hook gets registered but is **never re-executed** once the loop blocks on udev events (verified: job present, manual `connect-all` works, namespace stays absent). An `initqueue/online` hook also does NOT work: this initramfs is **systemd- driven** ("A start job is running for /dev/mapper/root") and never sources dracut online hooks on link-up (verified: hook installs as `90-nvmf-connect.sh` but never fires — no log, no stamp). **Fixed:** the connect is now a **systemd unit** (`nvmf-connect.service`, `ConditionKernelCommandLine=rd.dot1x=0`, `Before=initrd-root-device.target`) whose ExecStart (`/usr/sbin/nvmf-connect`) polls for the network then runs `nvme connect-all` until the namespace appears. The dracut module installs the unit + script and enables it via an `initrd-root-device.target.wants` symlink; `diskless.pkr.hcl` hard-asserts the unit is in the initramfs. Gotcha: `$systemdsystemunitdir` is **empty** in this dracut's module-setup context — `inst_simple` then silently drops the unit (empty dest dir). Use a `${systemdsystemunitdir:-/usr/lib/systemd/system}` fallback and `return 1` if the unit didn't land, so the build fails loudly instead of shipping a UKI that can't connect its root. Second gotcha: the build assertion `lsinitrd … | grep -q 'nvmf-connect.service'` was a **false negative** — `grep -q` exits on first match, closes the pipe, `lsinitrd` dies of SIGPIPE, and under `pipefail` the pipeline reports failure even though the unit IS present. This masked the fact that the systemd unit install was working all along. Fix: `lsinitrd > file; grep -q … file` (no pipe). Never `lsinitrd | grep -q` under pipefail. Bonus: the survivor listing confirmed nvme-cli's native autoconnect bits ARE in the initramfs (`/usr/sbin/nvmf-autoconnect.sh`, `/etc/udev/rules.d/95-nvmf-initqueue.rules`) — so the fully-native `rd.nvmf.discover` autoconnect path is available as a fallback if the unit doesn't fire on boot. - **dm-verity: `hash-offset` is unsupported → two namespaces.** With NVMe connecting and the right by-id device present, verity still failed: `systemd-veritysetup` logged `unknown option 'hash-offset=…'` then `Failed to load verity superblock` — exactly the CLAUDE.md bench-verify item. The in-file (data+hash in one namespace, split by offset) layout is not supported by Rocky 9.8 systemd. **Fixed** via the documented fallback: `extract-artifacts.sh` now emits `rootfs.erofs` (data) + `rootfs.hash` (hash tree) as separate files, served as **two NVMe namespaces** with distinct pinned UUIDs (`f1ee7000-…-01` data, `…-02` hash); the cmdline uses `systemd.verity_root_data=`/`systemd.verity_root_hash=` by-id devices with no offset. Touches `extract-artifacts.sh`, `build-oci.sh` (artifact list + SHA256SUMS), `serving/spdk-setup.sh` (2nd bdev/namespace), and invariant #9. - **SELinux labels must survive into the erofs.** It booted to multi-user but `root`/`packer-build-only` login was accepted then the session was killed instantly. Cause: kickstart sets `selinux --enforcing`, but the build dropped `security.selinux` xattrs (`virt-tar-out` + plain `tar -xpf` don't carry them), so the enforcing target denied the login session on an unlabeled root — and a read-only erofs can't autorelabel. Confirmed by booting with `enforcing=0` (SMBIOS-injected) → login worked. **Fixed:** export via `guestfish tar-out … selinux:true xattrs:true acls:true`, restore with `tar --xattrs --xattrs-include='*' --acls --selinux`, and a `getfattr` build-time assertion that `/usr/bin/bash` is labeled before `mkfs.erofs` (mkfs.erofs preserves xattrs). Needs `attr` on the build host. The image now boots **enforcing**. - **`mkfs.erofs --all-root` was flattening all ownership to root:root.** Mode bits + (now) xattrs survive the export, but `--all-root` forced every inode to uid/gid 0 — discarding group ownership (`ssh_keys` on host keys → sshd "permissions 640 too open" → refused; `tty`/`crontab` setgid groups) and user ownership (home dirs). **Fixed:** dropped `--all-root` so real ownership is preserved from the tree. That now requires `extract-artifacts.sh` to run as **root** (added an `id -u` guard; `tar -x` needs CAP_CHOWN to restore ownership) — the post-processor runs it via `sudo` and `chown`s the output files back to the build user so `build-oci.sh` (rootless buildah + its own login) still works. Combined with the host-key strip above, sshd starts cleanly. **⚠ The secure path has the same dependency on the (absent) nvmf autoconnect** — when 802.1X is re-enabled, the macsec0 path must likewise drive `connect-all` (or the network-manager port #1 must wire a working autoconnect). Currently only the test path is fixed. - The target Rocky 9 kernel's erofs is built **without `CONFIG_EROFS_FS_ZIP`** — mounting a compressed image fails `erofs: lz4 algorithm isn't enabled`, which silently blocks the root mount. **Fixed:** `extract-artifacts.sh` now builds the erofs **uncompressed** (`mkfs.erofs` without `-z`). Tradeoff: larger rootfs namespace + more NVMe/TCP bandwidth per read. Re-enable compression only against a kernel verified to have `EROFS_FS_ZIP` (`grep CONFIG_EROFS /boot/config-$(uname -r)`). This is the CLAUDE.md bench-verify item ("erofs module availability on the target RHEL kernel"). Larger image size also feeds the "shrink the rootfs" angle of #7. - SPDK: add the listener to the discovery subsystem explicitly (`nqn.2014-08.org.nvmexpress.discovery`) to silence the deprecation warning. ## [ ] 6. Rolling fleet updates (multi-version serve → cohort reboots) Image-based, reboot-to-update across many live workstations. Hosts are immutable (read-only erofs/verity root over NVMe/TCP, ephemeral overlay), so updates are **atomic A/B reboots**, never in-place. The released OCI artifact + digest-pinned manifest + roothash coupling already give the atomic unit and instant rollback; this item builds the rollout machinery around it. - **Serve versions side-by-side** (must not overwrite a live rootfs): extend `serving/` so SPDK/nvmet exposes each release as its **own subsystem/NQN** (`nqn…:rootfs-`) and nginx serves multiple `fleet-.efi` (keep current + previous). Make the UKI's NQN version-specific so each signed UKI connects only its matching namespace (constant discovery DNS name). - **Version selection at boot**: dnsmasq **cohort tagging** (MAC / vendor class / assigned tag) hands each cohort a different option-67 UKI URL. Moving a cohort's URL = assigning the release. - **Control plane + in-image reboot agent**: a baked-in systemd timer polls a "desired version" endpoint and reboots **when idle / in a maintenance window** (or out-of-band via Redfish/BMC). Stagger + jitter reboots to avoid UKI-fetch / NVMe-connect storms. - **Staged rollout**: canary → waves (10/50/100%) with health gates (multi-user reached, NVMe connected, dm-verity mapped, kiosk URL rendered, prod: MACsec/802.1X up); auto-halt on threshold. - **Rollback**: repoint cohort URL to previous UKI + reboot (current+previous stay served); security rollback via dbx-revoke / TPM monotonic counter (design §6). - **Drain & retire** an old version only when SPDK/nvmet reports zero connected controllers on it. **Drain accuracy:** account for hosts powered off mid-rollout — an assigned-but-offline host will reconnect to a version that may be gone, so don't remove a version while any host that could return is still assigned to it. - **Availability**: run NVMe/TCP target replicas behind distinct discovery entries (design §7) — never serve the whole fleet from one unreplicated node, especially mid-rollout (N versions + reboot bursts). - Note: kiosk *content* changes are server-side (the website) and need no fleet update — only OS/UKI/rootfs changes require a reboot. ### SPDK restart / availability is the load-bearing constraint (analysis 2026-06-25) SPDK sits in the **live data path of every running host's root fs**, so a process restart isn't like restarting a stateless service — it tears down **every** NVMe/TCP association at once (fleet-wide, NOT version-scoped). The rollout design must be built around **never restarting the process that holds live connections.** - **Core principle:** a host changes root version *only by rebooting* (no live swap). So: **version transitions = reboots; SPDK changes = additive/online; the process stays up.** Routine rollouts never restart SPDK. - **Online add/remove via JSON-RPC, no restart.** `nvmf_tgt` is fully runtime-configurable (`bdev_aio_create`, `nvmf_create_subsystem`, `nvmf_subsystem_add_ns`, `..._add_listener`). Add vY's namespaces live (existing vX connections untouched); remove vX live once drained. **Per-version subsystem (distinct NQN)** matters here: adding vY as a *new subsystem* never disturbs the subsystem vX hosts are attached to — adding a *namespace* to a shared subsystem fires a namespace-change AER + rescan at running hosts. Per-version subsystem also gives clean per-release drain + per-release hostNQN allow-listing. - **What a connection loss does to a running host:** root is **read-only erofs + tmpfs overlay**. tmpfs writes (RAM) + page-cached reads keep working; **cold reads block**. The nvme-tcp host driver reconnects (`reconnect_delay`) until `ctrl_loss_tmo`. **Fast SPDK restart** (same NQN/UUID/addr within the window) → host **survives with a stall**; **long/changed/expired** → root I/O errors, host needs a reboot ("keep alive timeout / no route to host"). Read-only + content-addressed makes reconnect unusually clean (no write/reservation state). - **Cheap host-side mitigation:** tune `ctrl_loss_tmo` (even `-1` = retry forever) + `reconnect_delay` at `nvme connect` time (dracut nvmf step). For a diskless box "hang and recover" beats "die" — there's nothing better to fail to. - **Structural answer — HA + NVMe-oF multipath (ANA):** run ≥2 targets serving **byte-identical, same-UUID** namespaces; hosts multipath across them, so one target can be restarted/upgraded while the other serves (no root loss). The **read-only, content-addressed root makes replication trivially coherent** (no write-coherence problem) — this architecture is ideally suited to it. Wrinkle: the **immutable UKI cmdline** points at one discovery address, so multipath needs discovery to return multiple paths (or a DNS name resolving to several targets). - **Planned SPDK restarts only via drain/failover** (multipath fail-over, or a maintenance window with generous `ctrl_loss_tmo` so hosts recover not die). There is **no zero-impact single-node SPDK restart.** - **[x] Long-lived target + online release control — IMPLEMENTED 2026-06-27.** The old gap (SPDK ran `spdk-setup.sh` as entrypoint, single-version, configured-at-start → changing versions = container restart = drop ALL connections) is closed. Now in `serving/spdk/`: - **`spdk-target.sh`** — persistent entrypoint: starts `nvmf_tgt` ONCE (release-agnostic), creates the TCP transport + discovery listener, then reconciles every staged release. Never restarted to change versions; on (re)start re-adds everything under `./releases/` (a serving-host reboot restores all served versions). - **`spdk-release.sh`** — JSON-RPC control: `add`/`remove`/`list`/ `drain-status`/`reconcile`, **one subsystem per version** (distinct NQN, so adding/removing one never disturbs others' connections), **drain-gated removal** (refuses while a version has connected controllers unless `--force`). - **`spdk-ctl.sh`** — host ops: `load`/`unload`/`list`; stages a release's artifacts from its OCI image into `./releases//` and drives the live target. Rolling update = `load` new → reboot hosts → `unload` old once drained. - **`spdk-loader.sh`** — DECLARATIVE rolling updates (the "init container that stays up", compose service `spdk-loader`). On `compose up` after an `IMAGE_VERSION` bump it stages + adds the new rootfs to the running target ONLINE, then polls and **auto-reaps** any non-current version once it has drained (0 conns) for a sustained grace window (`REAP_GRACE_SECONDS`, keep > host `ctrl_loss_tmo`). So migration = bump `IMAGE_VERSION` in `.env` + `docker compose up -d` (compose recreates the loader + the http/dnsmasq UKI servers, but NOT the nvmeof target → no dropped connections). Single served UKI ⇒ a 0-conn old version is genuinely orphaned (a returning host fetches the current UKI), so auto-reap is safe; removal is still drain-gated + grace-debounced. The loader + target share the RPC socket on a named volume (`nvmf_tgt -r`); the loader reuses the SPDK image (has python3 + rpc.py). - **BENCH FINDING 2026-07-03 (FIXED) — hosts did `connect-all`, so NO version ever drained → auto-reap could never fire.** Observed: `spdk-release list` showed more controllers than workstations, with every box appearing on *every* served version's subsystem (e.g. 1/3/3/1/3 across 5 versions for ~3 boxes). Root cause: the initramfs connected with **`nvme connect-all`** (test path `nvmf-connect.sh`; secure path delegated to the dracut nvmf module's discovery autoconnect), which attaches EVERY subsystem in the discovery log — the host then mounts only its own by-UUID verity namespaces but leaves a live controller on all the others. So each running host pinned every version above 0 conns, and `spdk-loader`'s "auto-reap once drained (0 conns)" **could never trigger** while the fleet was up — the served-version list grew without bound. This defeats the whole drain/reap model, not just cosmetics. **Fix:** targeted connect. The per-version subsystem NQN (`SUBNQN`, already in `verity.json .subnqn`) is now also baked into the signed UKI cmdline as **`rd.nvmf.nqn=`** (`extract-artifacts.sh`), and both connect paths use **`nvme connect -n `** — `nvmf-connect.sh` (test) and a new explicit step 6 in `dot1x-macsec-up.sh` (secure, replacing the module delegation; runs after macsec0 is addressed per invariant #2). Legacy UKIs without `rd.nvmf.nqn` fall back to `connect-all` (with a warning) so they still boot. Now each host = 1 controller on its own subsystem → versions drain to 0 when their hosts move off → auto-reap works; connections scale O(hosts) not O(hosts × versions). Requires a rebuild + re-sign (cmdline change, invariant #6). **(a) TEST PATH BENCH-VERIFIED 2026-07-03 (commit `0bfe872`): builds, deploys, and the host mounts ONLY ONE image** (its own version's subsystem) instead of connect- all'ing every served version — the targeted `nvme connect -n rd.nvmf.nqn` works. So each host now = 1 controller on its own subsystem, which is what lets old versions drain to 0 and `spdk-loader` auto-reap. **Still to confirm:** the reap payoff itself (migrate a host, watch its old version drain to 0 and get reaped); (b) SECURE PATH — the new step-6 connect brings up the NVMe root over macsec0 (this path is not yet bench-confirmed end-to-end); (c) that the dracut nvmf module isn't ALSO connect-all-ing on the secure path (if it is, suppress it — the targeted connect is meant to be the sole connector). - **BENCH FINDING 2026-07-03 (FIXED) — the target WAS still recreated on an `IMAGE_VERSION` bump.** Despite the above, the `nvmeof` (spdk-target) service still carried a versioned `type: image` bootstrap mount (`source: ${OCI_IMAGE}:${IMAGE_VERSION}` → `/artifacts`) **and** an `IMAGE_VERSION` env, so bumping the version changed THIS service's config and `compose up -d` recreated it → `nvmf_tgt` restarted → **every connected host's NVMe/TCP root dropped** (the exact fleet-wide outage the long-lived-target design exists to prevent). Fix: **made the target version-agnostic** — removed the `/artifacts` image-volume + `IMAGE_VERSION` env from `nvmeof` in `serving/docker-compose.yml`. Bootstrap AND rolling updates are now solely the `spdk-loader`'s job (stages the current release into `./spdk/releases`, adds it online); `spdk-target.sh`'s seed block is guarded (`[ -s /artifacts/verity.json ] && [ -n IMAGE_VERSION ]`) so it no-ops when unmounted. Verified `nvmeof` no longer references `IMAGE_VERSION` (only `spdk-loader` does). **BENCH-VERIFY:** bump `IMAGE_VERSION` + `up -d` with a host connected → `nvmeof` container NOT recreated (`docker ps` ID/uptime unchanged), host root survives, loader adds the new version online. - **No jq dependency:** `spdk-release.sh` parses RPC JSON with **python3** (the SPDK image ships no jq); legacy artifacts (no per-version UUIDs in verity.json) fall back to the fixed UUID pair with a warning. - **Per-version namespace UUIDs** (`scripts/extract-artifacts.sh`): derived deterministically (UUIDv5 of `image_version`), baked into the UKI cmdline + `verity.json` (now also carries `subnqn`); the serving side reads them back from each release's `verity.json`. Required so two versions don't collide on the same `/dev/disk/by-id/nvme-uuid.` host symlink. - **Bootstrap unchanged:** `docker compose up` with one version still works — the `/artifacts` image volume is seeded into `./releases/${IMAGE_VERSION}`. - **PARTIALLY BENCH-VERIFIED 2026-06-28:** long-lived target boots a host end to end — UKI served over HTTP → host boots → NVMe/TCP root connects to the persistent `nvmf_tgt` (single version `spdk-permanence-02ffd7d`). Surfaced + fixed several real bugs along the way: bootstrap seed must COPY not symlink (a symlink into the version-pinned `/artifacts` re-pointed a stale release at new data); `spdk-release` parses RPC JSON via **python3** (SPDK image has no jq); legacy `verity.json` (no per-version UUIDs) falls back to the fixed pair; mislabeled-dir guard (dir name vs verity.json `image_version`, case-insensitive); dropped the redundant discovery-listener re-add (daemon logged "Listener already exists"). **CI lowercases `image_version`** (gitea-main-ci vars step) — a mixed-case branch (`SPDK-permanence`) had baked a capital `fleet-SPDK-…​.efi` the lowercase boot URL couldn't fetch (iPXE "no such file"); booted via a local retag workaround pending a lowercase rebuild. - **STILL BENCH-VERIFY:** (a) `load` a 2nd version while a host is connected to the 1st → 1st's connection survives (no AER/reset on the running host); (b) `unload`/auto-reap drain-gate blocks while CONNS>0 and reaps after grace on a LIVE host; (c) serving-host reboot reconciles all staged versions; (d) per-version UUID build reproducible (two builds, same version → identical `data_ns_uuid`/`hash_ns_uuid` + `root_hash`); (e) the full fix on a clean LOWERCASE CI rebuild (no retag). See `serving/spdk/README.md`. - **Capacity:** each concurrently-served version costs target memory/hugepages + file-backed bdev overhead — plan for N live versions (current + previous + canary) during long rollouts (raise `HUGEPAGES`). - **Attach gating + discovery:** with several subsystems advertised, the dracut connect + per-subsystem hostNQN allow-list must ensure a vX host attaches **only** vX's namespace (the by-UUID verity ref protects *what mounts*, but gate the *attach* too); `rd.nvmf.discover` selection must be deterministic, not connect-all. ## [ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience) There is **no production-grade memory-safe NVMe-oF target** — the Rust NVMe work is all driver/initiator-side (vroom, RedLeaf, Rust-for-Linux); the only production targets are C (Linux kernel `nvmet`, SPDK). So this is **serving-node hardening, not a language swap**. dm-verity already removes the target from the clients' integrity TCB (verify-on-read rejects tampered blocks), so the risk is RCE/DoS on the node that holds the fleet's disk, not client integrity. The export is read-only + single-namespace = inherently small attack surface. - **Evaluate kernel `nvmet` vs SPDK** for the target: kernel `nvmet` keeps the data path in the hardened/fuzzed kernel with a tiny userspace footprint (arguably a better attack-surface story); SPDK is a userspace C daemon (current choice, needs hugepages + caps). Decide per environment. - **Confine the target** (whichever): unprivileged, drop caps, seccomp + SELinux/AppArmor profile, read-only container, minimal mounts; minimal immutable serving image, digest-pinned. - **Slim the SPDK image (multi-stage build)**: the current single-stage from-source image is ~600 MB (it keeps the whole toolchain — gcc, meson/ninja, kernel headers, the python venv — plus the SPDK/DPDK source and build trees). Convert `serving/spdk/Dockerfile` to a builder stage + a minimal runtime stage that copies only `nvmf_tgt` + its `ldd` shared libs (libnuma, libssl/crypto, libuuid, libaio…) + `rpc.py`/`python3`. Expect ~150–250 MB (debian-slim base or a static SPDK link trims further). Smaller attack surface + faster pulls, and it pairs with the confinement above. - **Network-isolate :4420** so only the boot VLAN reaches the target (compose with 802.1X/MACsec segmentation in prod). - **Replicas** behind distinct discovery entries (availability is the real risk — it's the fleet's disk). The target is a **fleet-wide SPOF in every running host's root data path**; a process restart drops *all* NVMe/TCP associations at once. Structural fix = **HA + NVMe-oF multipath (ANA)** over byte-identical, same-UUID, read-only namespaces (replication is coherent precisely because the root is read-only + content-addressed); cheap mitigation = generous host `ctrl_loss_tmo`/`reconnect_delay`. See the **"SPDK restart / availability"** analysis under #6 (rolling updates) — same constraint, fuller treatment. - Revisit a memory-safe target only if/when an off-the-shelf one matures (e.g. a future Rust-for-Linux `nvmet`). ## [ ] 8. Return to secure / production posture (un-toggle the lab flags) Consolidates the cleanup of the temporary lab toggles (see "Current build posture" at the top) and the secure-path bench-verifies deferred while testing the open-network path. Do these together, then run the full secure boot-chain smoke test. - **Re-enable 802.1X/MACsec:** `disable_dot1x=false` (CI `DISABLE_DOT1X="false"`). Removes `rd.dot1x=0`/`ip=dhcp`; the hook owns the link (macsec0). - **Bench harness scaffolded 2026-06-22: [serving/dot1x-bench/](../serving/dot1x-bench/).** step-ca (fleet CA, on the serving host) signs each host's **TPM-generated CSR** + issues the RADIUS server cert; FreeRADIUS does EAP-TLS (trusts step-ca root, returns MSK); a `hostapd` wired authenticator does 802.1X + MACsec on an **isolated bridge per workstation** (flat VLAN can't auth — EAPOL dropped + MACsec is P2P). `enrol/tpm-provision.sh` creates the key in the TPM at `0x81000100` + writes the cert to NV `0x01800100`. Starting solution **signs a CSR** (key stays in TPM); **TPM attestation** (ACME `device-attest-01`) is the documented upgrade on the same CA. **BENCH-VERIFY** items flagged in the files: hostapd MACsec build, openssl tpm2 provider vs pkcs11 key ref, EAP-TLS version. PKI product decision (greenfield): start step-ca; OpenBao/EJBCA/Dogtag later. - **First step (DONE 2026-06-22): `enrol/tpm-selftest.sh`** — non-destructive TPM capability check (run on the booted open-port image, no `disable_dot1x` needed) to de-risk the TPM half before the full path. Likely first finding: no openssl TPM2 **provider** in the image (`tpm2-tss` ≠ the provider) — add `tpm2-openssl`. - **Enrolment of the per-host identity** (key@`0x81000100` + cert@NV) is its own design — see **#11** (TOFU + OOB fingerprint broker, pre-auth permit-set, provisioning UKI). `tpm-provision.sh` is the manual bench stand-in for it. - **NVMe discovery by DNS:** `nvmet_traddr=nvme.fleet.lab` (CI), resolved at boot by the serving dnsmasq — drop the hardcoded `192.168.50.10`. - **Signing — SIGNED-UKI + SECURE-BOOT-ON PATH BENCH-VERIFIED on the Proxmox VM (HTTP boot, 2026-06-20).** CI signs the UKI when `SB_KEY`/`SB_CERT` secrets are set (`db` key); `fleet-sb-enroll` enrols db→KEK→PK from the baked public `.auth` files; OVMF then boots the signed UKI under enforcing Secure Boot. Confirmed end-to-end: CI-signed UKI verifies (`sbverify`), enrolment succeeds, VM boots with `mokutil --sb-state` = enabled. **Key generation + enrolment procedure: [secure-boot-keys.md](secure-boot-keys.md)** (the PK/KEK/db trio; `db` = `sb_key`/`sb_cert`, PK/KEK stay offline). - **Bug fixed during bring-up (2026-06-20):** `fleet-sb-enroll` built its 4-byte EFI attribute prefix in a bash variable (`ATTR=$'\x27\x00\x00\x00'`). Bash strings can't hold NULs → only `0x27` survived → efivarfs read a corrupt attribute u32 → every write failed with `EINVAL` ("dd: Invalid argument"). Now emitted directly with `printf` at write time. **Baked fix VERIFIED on the 820 (2026-06-22): `sudo fleet-sb-enroll` enrolled db→KEK→PK cleanly, no inline workaround.** (Red herring on the way: Proxmox EFI disks default to `pre-enrolled-keys=1`, so `mokutil --db` shows the MS cert even in Setup Mode — looked like a non-empty-db rejection but was the NUL bug all along; it failed identically on a clean `pre-enrolled-keys=0` disk.) - **iPXE signing WIRED (2026-06-21):** `snponly.efi` is now built + signed (`db`) + pushed by CI as a separate FROM-scratch **`fleet-ipxe`** image (`scripts/build-ipxe-oci.sh`, `.gitea/workflows/ipxe-image.yaml`), mounted into dnsmasq at `/ipxe`. dnsmasq itself moved to a generic CI-built/pulled image (`.gitea/workflows/dnsmasq-image.yaml`) — no longer built at deploy time, and carries no keys/binary. (Decision: separate `fleet-ipxe` image, not baked into dnsmasq, so the daemon image stays key-agnostic across SB environments.) - **820 / PXE signed chain — BENCH-VERIFIED 2026-06-22.** Real hardware (HP EliteBook 820 G4), Secure Boot **enforcing** (`mokutil --sb-state` = enabled, Setup Mode no, db/KEK/PK enrolled): firmware verifies the signed `snponly.efi` (fleet-ipxe image) → signed UKI and boots. Confirms the whole `fleet-ipxe` signing pipeline + the PXE→iPXE→UKI chain under SB on real silicon (the VM only covered the HTTP path, which has no iPXE stage). - **REMAINING for production:** (a) **secure path** — re-verify signed boot with `disable_dot1x=false` (802.1X/MACsec), not just the test path; (b) confirm signing is reproducible (verity hash stable across two signed builds). - **PXE-only iPXE fallback — BENCH-VERIFIED on the EliteBook 820 G4 (2026-06-20).** Firmware without UEFI HTTP boot chainloads `serving/tftp/snponly.efi` → renders `autoexec.ipxe` (ifopen + DHCP-retry + `imgargs `) → UKI over HTTP → NVMe/verity root. dnsmasq tags `pxe`/`ipxe`/`httpboot` give graceful per-client fallback; HTTP-boot hosts are unaffected (firmware → UKI direct). Gotchas that cost real time (all captured in [../serving/tftp/README.md](../serving/tftp/README.md)): (i) the I219-LM needs **`snponly.efi`** (native iPXE driver won't bind); (ii) **PortFast/edge on the switch port is REQUIRED** — iPXE's PHY bounce triggers STP forwarding delay that drops DHCP (this was the real blocker); (iii) iPXE aborts the script on any unguarded command failure (use `||`); (iv) Secure Boot OFF makes systemd-stub take iPXE's load options as the cmdline, so the entrypoint extracts the UKI's real `.cmdline` and feeds it via `imgargs`. **Tidied 2026-06-20** once PortFast was identified: dropped the `autoexec.ipxe` + `ifopen` + DHCP-retry STP workaround; `boot.ipxe` is now just `dhcp`+`imgfetch`+`imgargs`+`boot` reached via normal iPXE autoboot. Re-verified on the 820 G4. **SB-on simplification:** when Secure Boot is enabled the cmdline injection is unnecessary (the stub ignores the loader cmdline) — collapse `boot.ipxe` to a plain `chain` of the UKI and drop the `.cmdline` extraction + `binutils` + the artifact mount from the dnsmasq service. - **Sign iPXE for the PXE-only fallback (Secure Boot, production).** The lab `snponly.efi` is **UNSIGNED** → that path needs **Secure Boot OFF**. For production: `sbsign` the iPXE binary with the fleet **db** key (only `db` needed — no MS CA / shim), so SB stays enforced firmware → iPXE → UKI; then the cmdline contraption becomes a no-op (the stub ignores loader cmdline under SB). See [secure-boot-keys.md](secure-boot-keys.md) "PXE-only hosts". - **Bench-verify on the secure path** (swtpm + hostapd + FreeRADIUS, `disable_dot1x=false`) — the items deferred this session: - NM port secure path: `en*` UNMANAGED + `macsec0` MANAGED/renewing, MKA survives switch-root, NVMe stays up past lease (item #1 risk). - NTP via DHCP option 42 reaches chrony over macsec0 (`chronyc sources`). - Plymouth `fleet` theme: test-path render VERIFIED 2026-06-18; still confirm the auth→macsec rows reach `done` live on the secure path. - STIG remediation didn't break the secure path; reproducible verity hash; triage the generated `stig-deviations.md` REVIEW rows (item #2). - **Secure-path NVMe connect trigger:** the test path uses `nvmf-connect.service` (gated `rd.dot1x=0`); confirm the secure path's connect (dracut nvmf over macsec0) actually fires, or give it an equivalent trigger. Apply the same `-k 30 -l -1 -c 10` keep-alive/reconnect flags to the secure `nvme connect` as the test path has. - **Future hardening waived for now** (from the STIG register): FIPS, firewall, per-host/rotatable admin (SSH CA / config-agent). - **[x] TLS-encrypted audit/log offload — IMPLEMENTED 2026-07-12 (awaiting bench).** The host→collector hop is now syslog-over-TLS; the plaintext `2f` forward and the three `rsyslog_encrypt_offload_*` waivers are gone. As built (followed the original 5-step plan): - **CA reused, not new:** the sender trusts the collector via the fleet CA already baked at `/etc/dot1x/radius-ca.pem` (`var.fleet_ca` — now the trust anchor for RADIUS EAP-TLS, fleet-agent AND log offload). Collector server cert (SAN = `${AUDIT_HOST}`) issued from the bench step-ca by `serving/audit-collector/issue-cert.sh` → `tls/` (gitignored), mounted at `/etc/rsyslog-tls`; renew by re-running it (720h default — expiry shows as handshake errors + senders spooling). - **Collector** (`serving/audit-collector/`): gtls listener on **6514** (per-listener `streamDriver.*` params; `authMode="anon"` — hosts carry NO client cert: fleet-generic, invariant #1; client mTLS with the TPM identity was CONSIDERED + DEFERRED 2026-07-12 — rationale + revisit trigger in audit-collector/README.md "Decision: no client mTLS"). Plaintext :514 kept as a LEGACY listener for pre-TLS images during migration — close it once the fleet is on TLS images. Deliberately no global `DefaultNetstreamDriver="gtls"` (would silently flip the plaintext wazuh1 relay `omfwd` to TLS and break it). Alpine image needs `rsyslog-tls` (ships `lmnsd_gtls.so`; verified v3.20). - **Sender (2f):** `omfwd` → `$AUDIT_HOST:6514`, `StreamDriver="gtls"` mode 1 `StreamDriverAuthMode="x509/name"` + `StreamDriverPermittedPeers=$AUDIT_HOST`, CA via `global(DefaultNetstreamDriverCAFile="/etc/dot1x/radius-ca.pem")`. With the placeholder CA the handshake can't succeed → action suspends and the disk-assisted queue spools (same lab posture as RADIUS). `rsyslog` + `rsyslog-gnutls` moved to the step-2 baseline install so they're present at STIG-scan time. - **STIG:** the three `rsyslog_encrypt_offload_*` selects removed from `files/stig/diskless-tailoring.xml` (rules now evaluated; SSG remediation writes the legacy gtls/x509 globals, eval of record should PASS); register #6 rewritten (encrypted offload; audisp-remote rules + `rsyslog_remote_loghost` stay waived — intent met via rsyslog). - **Firewall:** serving host now needs **6514/tcp** open on the boot NIC (same host-networking trap as 514 — see audit-collector/README.md); the future workstation egress rule is 6514 too. - **BENCH-VERIFY (the gate):** build with the real fleet CA + run `issue-cert.sh` + `docker compose up -d --build audit`; on a booted host confirm records land in `audit-collector/logs//audit.log`, `ss -tnp` on the serving host shows the sender ESTABLISHED to **:6514** (not :514), a tcpdump of :6514 shows TLS (no cleartext syslog), a wrong/placeholder CA is REJECTED (sender spools, `rsyslogd` logs a cert-validation error — proves x509/name is actually enforced), and the STIG report shows the three `rsyslog_encrypt_offload_*` rules PASS (0 new REVIEW rows). (RELP+TLS via `omrelp`/`imrelp` remains the alternative if guaranteed delivery is wanted alongside encryption.) - **fleet-hostapi: scope CORS to the kiosk origin.** The host-info agent ships with `FLEET_HOSTAPI_ALLOW_ORIGIN=*` (lab). For prod set it to the exact kiosk origin (default `http://boot.fleet.lab`) via the unit's commented `Environment=` line (or a drop-in). See #12. ## [ ] 9. (OPTIONAL / nice-to-have) Convert provisioning to Ansible where it pays off Low priority — only the *config-management* half of the build is a candidate, and the ROI is modest because the build VM is single-shot (idempotency buys little) and reproducibility/auditability favour minimal shell. **Not** a wholesale rewrite: the load-bearing logic (dracut module hook ordering, `extract-artifacts.sh`, `build-oci.sh`) is initramfs-runtime / host-side build tooling, not provisioning, and stays shell. The high-ROI, low-risk slice if/when we pick this up: - **STIG via the SSG Ansible playbook** (`ssg-rl9-playbook-stig.yml`, shipped with scap-security-guide) instead of `oscap --remediate` — the upstream path and far more *reviewable* (you can diff exactly what each rule changes). Keep `oscap eval` (no remediate) for the report + the 0-eval guard (item #2). - Restyle the brittle imperative bits as declarative tasks: chrony.conf edit (`lineinfile`/`blockinfile` vs `sed`), sshd drop-in / NM dispatcher / kiosk file installs (`template`/`copy`), datastream-by-`os-release` as `assert`. Bonus: sidesteps the `%{...}`→`%%{` HCL-escaping footgun. Tips the balance toward "do it" only if: we start managing the **serving hosts** (dnsmasq/nginx/SPDK/chrony/audit-collector) with config-management too (shared roles to amortise — currently they're docker-compose), provisioning grows much more branchy, or we want molecule-tested hardening roles validated without a full ~17-min Packer build. ## [ ] 10. (OPTIONAL) Secure Boot cert auto-enrollment for provisioning Speed up bringing up new workstations: bake the **public** PK/KEK/db certs into an image, detect Secure Boot **Setup Mode** at boot, and auto-enroll them so a fresh/reset box adopts the fleet trust root without a manual firmware-setup visit. Pairs with #8 (signing) + [secure-boot-keys.md](secure-boot-keys.md). Shares the provisioning-UKI + isolated-environment vehicle with **#11** (which enrols the *network* identity); build them together. **Tooling gap (found 2026-06-20): `efitools` is NOT packaged for EL9** — no `efi-updatevar`/`efi-readvar`/`KeyTool`. Resolved for the OS-side path by writing the signed `.auth` variables to **efivarfs directly** (de-immutable + single write), which needs no extra package. **Done 2026-06-20:** baked `/usr/local/bin/fleet-sb-enroll` (efivarfs method, Setup-Mode-gated, fapolicyd- trusted) + the public `PK/KEK/db.auth` at `/usr/share/fleet-secureboot/` (supplied via `files/secureboot/keys/` or `SB_*_AUTH_B64` CI secrets); `mokutil` covers reading. So **manual** OS-side enrollment over SSH now works (`sudo fleet-sb-enroll`). What's left for *auto*-enroll: drive `fleet-sb-enroll` unattended on a **Setup-Mode detection** at boot, gated to a provisioning mode/UKI (below), with read-back verify + audit — NOT armed in the production image. **What's safe:** the certs are public (no key material — only the **db** private key signs, and it stays in CI/HSM, invariant #7); Setup Mode accepts *unauthenticated* writes to PK/KEK/db, so public certs (ESL) are sufficient. It's fleet-generic (no per-host state, invariant #1) and deterministic (reproducible verity hash). **The real risk — provisioning-time TOFU (trust-on-first-use):** in Setup Mode Secure Boot isn't enforcing, so the enroller runs as *unverified* code, and the fleet's UEFI **HTTP boot is unauthenticated** in that state. An on-path attacker during provisioning can serve a malicious image that enrolls **their** PK/KEK/db — the box then "has Secure Boot" but trusts the attacker. Auto-enroll is only as trustworthy as the provisioning channel, and it removes the physical-presence approval gate. So treat it as a controlled-environment provisioning step, not an always-on behaviour. **Optional separate provisioning UKI (preferred shape):** build a dedicated **provisioning UKI** that carries the certs + the enroll logic, instead of arming the enroller in the production image. Keeps the standing footgun out of the normal image (a factory reset + tampered image can't silently re-root trust), and lets the enroller be a one-shot, attended/isolated step. The production UKI never enrolls. - Build it as a separate Packer target / flag (e.g. `provisioning=true`) → its own small UKI; signed or unsigned (it runs in Setup Mode anyway, but sign it so it *also* boots once keys are enrolled / for re-runs). - Or a tiny standalone EFI tool (KeyTool-style) served alongside, not a full OS. **Mitigations (do these regardless of UKI vs separate UKI):** - Run provisioning on a **trusted, isolated VLAN** (or a physically controlled bench) — the integrity Setup Mode can't give must come from the environment. - **Gate** enrollment behind an explicit provisioning mode / one-shot token; never arm it on every boot of the production image. - **Read-back verify before declaring success:** `efi-readvar` the enrolled PK/KEK/db and compare fingerprints to the values baked in; mismatch = fail loudly (turns silent TOFU into tamper-evidence). - **Audit** the enrollment event + read-back fingerprints to the collector. - **Keep the Microsoft UEFI CA in `db`** (or enroll your db alongside it) unless every NIC/GPU option ROM is covered by your keys — clearing MS db can break signed **option ROMs** (NIC PXE/HTTP-boot, GPU init), bricking the boot path. - **PCR 7 check:** enrollment changes the Secure Boot measurement in PCR 7; confirm nothing the fleet seals to the TPM (the EAP-TLS key at `0x81000100`, NV cert) is PCR-7-policy-bound before relying on this. **Decision before building:** auto-enroll trades assurance for convenience and is reasonable for a controlled fleet; the highest-assurance path is attended/firmware or manufacturer pre-provisioning. Default recommendation: **separate provisioning UKI + isolated VLAN + read-back verify.** ## [~] 11. (DESIGN) TPM 802.1X identity enrolment (TOFU + OOB fingerprint) **Authoritative design consolidated 2026-06-28: [tpm-enrolment-design.md](tpm-enrolment-design.md)** — unifies this item with the fleet-manager (the broker = the fleet-manager's one unauthenticated bootstrap endpoint; approval births the inventory record) and the full identity lifecycle (enrol / renew / alert). **v1 decisions (2026-06-28):** fingerprint-only OOB assurance (TPM attestation via ACME `device-attest-01` is the documented later upgrade); **inline single-UKI** vehicle for now (empty-NV early-boot branch — keeps provisioning network-boot-only, no USB; a separate provisioning UKI behind an `enrol_mode` build flag is the planned later upgrade); enrolment transport is plain **stdlib HTTP/JSON** (Go, the one unauthenticated edge — gRPC reserved for the steady-state agent path). The notes below are retained as the originating decisions; the design note is canonical. **Broker BUILT (server side, 2026-06-28): [serving/enrol-broker/](../serving/enrol-broker/)** — Go/stdlib; public unauth `POST /request`+`GET /status/{id}` (behind the serving nginx `/enrol/`) + admin **mTLS web UI** (`/`, `/pending`, `/approve|deny/{id}`) with a **dedicated admin CA** (`admin-pki.sh`); wordlist fingerprint, time-boxed store, per-IP rate limit; signs via `step` (dev `mock` signer). Ships in `serving/dot1x-bench/docker-compose.yml`; end-to-end smoke-tested (CSR → mTLS approve → cert). **BENCH-VERIFY:** run against real step-ca (`bootstrap.sh` root + `enrol` provisioner password readable by the broker) and confirm an issued cert authenticates EAP-TLS. **HOST SIDE BUILT 2026-07-02 — Go binary (`fleet-enrol`); TEST-PATH ENROLMENT BENCH-VERIFIED ON REAL HARDWARE (Dell 820, `rd.dot1x=0`, 2026-07-03).** Secure-path (802.1X/MACsec) enrolment still to verify — see the NV-index bench finding below. Pivoted from a shell hook to a static **Go + go-tpm** binary (`files/enrol/`): it talks to `/dev/tpmrm0` DIRECTLY (via `transport.FromReadWriter` on an `os.File` — no `linuxtpm`, so go-tpm is the only vendored dep), so the enrolment path needs **no tpm2-tools key-gen / no openssl tpm2 provider / no curl** in the image (removes the biggest bench risk — the provider `handle:` key-ref). It ensures the ECDSA key at `0x81000100`, builds the CSR, POSTs CSR+context to `rd.fleet.enrol_url`, shows the broker fingerprint on Plymouth, polls `/status`, and writes the leaf cert to NV `0x01800100`. The dracut hook `tpm-enrol.sh` (initqueue/settled prio 25) is now a thin GATE: empty-NV check (`tpm2_nvread`, still needed for the secure-path read) → pre-auth DHCP on `en*` (nmcli override of the secure-mode unmanaged drop-in) → `exec /usr/sbin/fleet-enrol` → reboot to converge on exit 0. No-op when a cert is present or under `rd.dot1x=0`. **Built OUT OF THE VM (2026-07-02, per decision):** `fleet-enrol` AND `fleet-hostapi` now compile on the CI runner (`.gitea/workflows/gitea-main-ci.yaml` "Build host Go binaries" step, Go 1.26.4, `CGO_ENABLED=0`, vendored/stdlib, `GOPROXY=off`), and Packer **file-provisions** the static binaries in — the Rocky build image no longer carries the Go toolchain (dropped the in-VM `dnf install golang` + the 2d removal). `fleet-enrol` lands at `/usr/sbin` before the initramfs build (dracut `inst_simple`s it; asserted in `lsinitrd`), fapolicyd-trusted. New `enrol_url` Packer var → `rd.fleet.enrol_url` cmdline. **TEST-MODE ENROLMENT ENABLED (2026-07-02, temporary):** the hook no longer no-ops under `rd.dot1x=0` — enrolment runs on the open/test path too, so it can be exercised on a host with a TPM + the broker WITHOUT standing up 802.1X/MACsec/RADIUS. In test mode `ip=dhcp` owns the NIC (the hook skips its own DHCP to avoid the dracut "address already assigned" race) and DNS resolves the broker via the lease. `rd.fleet.enrol=0` disables it. Production intent is still secure-path-only; revisit when the secure path is bench-verified. **TESTED (2026-07-02):** transaction logic (CSR → mock broker → approve → cert extract) as stdlib unit tests, AND the **full TPM ceremony (key-gen → persist → ECDSA sign → CSR verify → chunked NV write → re-enrol) against the go-tpm software simulator** (separate `files/enrol/simtest/` module so the CGO simulator never enters the shipped build). Both run in CI. **BENCH FINDING 2026-07-02 (FIXED) — the signing-key handle `0x81000100` was the EK slot; moved to `0x81000100`.** First real-TPM boot (host `cygnus-cruiser-99`, Proxmox vTPM, `rd.dot1x=0`) failed: `fleet-enrol: ensure key at 0x81010001: key at handle is not ECC: did not contain eccDetail (selector value was 1)`. `EnsureKey` died reading an existing RSA object as ECC → enrolment died before the CSR/POST, so the splash flashed "FIRST-BOOT TPM ENROLMENT" then the test system booted with **no fingerprint and nothing at the broker**. Root cause (confirmed via `tpm2_readpublic` under `enforcing=0`): **`0x81000100` is the TCG-standard RSA Endorsement Key handle** — the object there was the swtpm-provisioned **EK** (`restricted|decrypt|adminwithpolicy`, RSA-2048/AES128CFB, canonical EK authPolicy `837197674484b3f8…`), with its EK cert at NV `0x1c00002`. Our design squatted the EK slot fleet-wide (most firmware/swtpm TPMs have their EK there), so enrolment could never create our key. - **Fix (commit c574fa4 + follow-up):** the fleet signing key moved to **`0x81000100`** (non-reserved owner handle; threaded through `fleet-enrol -handle` default, `rd.dot1x.handle=` cmdline knob baked by `extract-artifacts.sh`, `wpa_supplicant-dot1x.conf`, the bench scripts, invariant #1, and the design docs). NV index was `0x01800001` here (owner range); it MOVED to `0x01800100` in a later finding (OEM platform-NV collision on the 820 — see below). - **EnsureKey hardening:** (i) evicts a genuine *stale signing key* at the handle and recreates (self-heal for old provisioning leftovers); (ii) **refuses to evict any restricted-decrypt object (EK/SRK)** as a safety net — never clobber the TPM's endorsement/storage identity (also protects the future EK/AK attestation). Regressions: `simtest/TestEnsureKeyEvictsStaleNonECC` + `TestEnsureKeyRefusesToEvictProtectedKey`. - **IDevID note (future):** if EK/AK attestation (`device-attest-01`, §"Stronger assurance") lands, the natural home for a TPM device-identity key is the TCG **IDevID** handle `0x81020001` (LDevID `0x81030001`) — see tpm-enrolment-design.md. Not adopted for v1 (implies the IDevID template/cert conventions). - **Also seen — leftover NV suppresses enrolment:** a *different* host (the 820) booted `tpm-enrol: client cert present at NV 0x01800001 — normal boot` with nothing enrolled this session — leftover NV state (from an earlier `tpm-provision.sh` / `tpm-selftest.sh --use-real-handles`) **persists in the hardware TPM** across reinstalls. Our presence check is content-blind (index defined ⇒ "enrolled"), so foreign/stale NV silently skips enrolment (→ secure-path auth would then fail). See the **content-aware NV check** item below. - **Needs a rebuild + re-boot to confirm** (the sim proves the ceremony, not the device). ⚠ **Don't hand-run `sudo fleet-enrol` on the booted STIG host** — the confined admin domain (`sysadm_t`) is SELinux-**denied** write to `/dev/tpmrm0` (`avc: denied … tpm_device_t`); the faithful path is the **initramfs** run (reboot, unconfined), or `enforcing=0` for hand-run TPM inspection. **FOLLOW-UP HARDENING:** - **Content-aware NV presence check — BUILT 2026-07-03.** `fleet-enrol -check-only` (no network / no key-gen) reads NV `0x01800100` via `tpmkey.ReadNVCert`, parses it as X.509, checks the validity window, and — when a real fleet CA is bundled (`-cacert`, default `radius-ca.pem`) — verifies it **chains** to it. Only a valid fleet cert counts as enrolled (rc 0); empty/garbage/foreign/expired → rc 20 (enrol + overwrite; `WriteNVCert` undefine+redefines). The dracut hook now calls it as the single gate, replacing the blind `tpm2_nvread` (removed from `tpm-enrol.sh`; the binary's `NVPresent` stays for cheap existence checks). **Lab fallback:** the shipped `radius-ca.pem` is a *placeholder* (comments, not a cert) → `loadCAPool` returns nil → chain-less "parseable + unexpired" acceptance, so the reboot-converge doesn't loop on a bench-CA cert; **production must replace `radius-ca.pem` with the real fleet CA** for strict chaining. Tests: `classifyNVCert` unit table (valid/expired/garbage/foreign/no-CA) + `simtest` ReadNVCert roundtrip across the 512B chunk. **BENCH-VERIFY** on a real boot: a garbage/ foreign NV now enrols instead of "normal boot". - **De-enrol / TPM-reset procedure** for re-testing: `tpm2_nvundefine 0x01800100 -C o` (+ `tpm2_evictcontrol -C o -c 0x81000100` for a stale *app* key — never the EK), or BIOS "Clear TPM". Document in the bench README. **BENCH FINDING 2026-07-03 (FIXED + BENCH-VERIFIED) — NV index `0x01800001` collided with OEM platform NV; moved to `0x01800100`.** The Dell 820 kept "finding a cert" and bypassing enrolment even after repeated BIOS TPM clears. `tpm2_nvreadpublic` showed the truth: `0x01800001` on that hardware is a **firmware/platform** index (`platformcreate|policydelete|writelocked`, 70 bytes — NOT our cert; siblings at `0x01800003/4` + `0x01810000` share it), which **no owner/BIOS clear can remove** (`TPM2_Clear` spares `platformcreate`; `policydelete` needs the platform's secret policy). So our chosen index was permanently occupied by Dell — the blind check false-positived on it, and even the content-aware check would have failed at the NV **write** (can't undefine a platform index). Same class as the EK-handle collision. Fix: **moved the NV cert index default `0x01800001` → `0x01800100`** (free on the 820; owner range; overridable via `rd.dot1x.nvindex=`), threaded through the cmdline (`extract-artifacts.sh`), `fleet-enrol -nvindex`, the dracut hooks, bench scripts, invariant #1, and docs. Also **hardened `WriteNVCert`** to refuse a `platformcreate` index with a clear "pick a free owner index" error instead of a confusing undefine failure (regression `simtest/ TestWriteNVCertRefusesPlatformIndex`). **BENCH-VERIFIED 2026-07-03 — the Dell 820 enrols end-to-end at `0x01800100`.** First success on REAL TPM hardware (past the un-clearable OEM platform-NV at `0x01800001`): validates BOTH the NV-index move AND the `0x81000100` key-handle move on a physical TPM, not just the Proxmox swtpm — the full host ceremony (ensure ECDSA key → CSR → broker fingerprint → approve → leaf cert → NV write) runs on the metal. This was the hard case (the 820 previously "found a cert" and silently bypassed enrolment every boot). **STILL BENCH-VERIFY:** (a) `tpm2_nvreadpublic` on any OTHER hardware model in the fleet — if a vendor also squats `0x018001xx`, pick a value free on all of them; (b) the enrolled cert actually **authenticates EAP-TLS on the secure path** (this run was test-mode `rd.dot1x=0`; 802.1X/MACsec enrolment still unverified — see the test-mode note above). **BENCH-VERIFY (host side, before trusting):** (a) go-tpm against a REAL TPM + swtpm on the target (the simulator proves the ceremony, not the device) — **the stale-key fix (above) is the first real-TPM result; re-boot to confirm it now reaches the broker**; (b) empty-NV create/persist on a fresh TPM with **empty owner auth** (the code assumes it — a set owner auth needs handling); (c) pre-auth DHCP on `en*` in secure mode (the `nmcli set managed yes` override of nm-unmanaged.sh actually leases); (d) fingerprint renders on the `fleet` Plymouth theme + admin can match it; (e) NV write + **reboot converges** (2nd boot runs the secure path); (f) prod: broker over **HTTPS** (`fleet-enrol -cacert radius-ca.pem`), currently lab-HTTP. **TEMPORARY (2026-06-28): broker also added to the MAIN serving stack** (`serving/docker-compose.yml`, `enrol-broker` service + `ENROL_*` in `serving/defaults.env`) so a single `compose up` in `serving/` brings the `/enrol/` endpoint up while the host side is built. Defaults to the zero-dependency lab path — **`mock` signer** (ephemeral CA; certs NOT trusted by RADIUS/NVMe) + **plaintext loopback admin** (`ENROL_ADMIN_INSECURE=true`). The serving nginx already proxies `/enrol/`→`127.0.0.1:9090`. **SPLIT BACK OUT:** the broker's proper home is a CA-co-located stack (it's still defined in `dot1x-bench/` next to step-ca); real signing needs the fleet CA. Don't run both copies on one host (they collide on 9090/9443). To use real signing from the serving stack meanwhile: set `ENROL_SIGNER=step` + `ENROL_STEP_CA_URL` at a reachable step-ca, populate `serving/enrol-pki/`, and `ENROL_ADMIN_INSECURE=false`. **STEP SIGNER VERIFIED end-to-end 2026-06-28** (broker → real step-ca → signed cert that chains to the fleet root; hostnqn derived from serial; EKU clientAuth). **Re-VERIFIED ON THE SERVING HOST 2026-07-02** against step-ca 0.30.2 via `verify-serving.sh` through the nginx `/enrol/` proxy: issuer = Fleet Bench CA Intermediate, cert validity exactly 720h (confirms the maxTLSCertDuration claim raise took effect), EKU clientAuth, `enrol` provisioner extension present, hostnqn = cert serial. Bring-up gotchas hit + fixed (all in the commits): the CA wouldn't init (compose used `_PASSWORD_FILE` for a nonexistent file → use `_PASSWORD`); Docker auto-created `./pki` root-owned (bootstrap self-heals perms + pre-create the bind-mount dirs); the CLI couldn't resolve `ca.fleet.lab` on the serving host (connect via `localhost`, which is in the CA SAN). Verification found + fixed two things the `mock` path had hidden: - **BUG FIXED (`signer.go`):** `--not-after` was passed to `step ca token` (validity capped ~1h → "token validity too long") instead of `step ca sign`. Moved to the sign step. Without it the step signer never issues a cert. - **CA CLAIM REQUIRED:** a stock step-ca caps certs at 24h, so 720h is rejected ("authorized maximum certificate duration of 24h"). `dot1x-bench/step-ca/bootstrap.sh` now raises `authority.claims.maxTLSCertDuration` (+restarts the CA) — this also unblocks bootstrap's own 720h RADIUS cert. Any other CA needs the same claim. - **BENCH-VERIFY still open:** issued cert actually authenticates EAP-TLS through FreeRADIUS (next, before host enrolment). How a host gets its **802.1X EAP-TLS client identity** the first time — the key at TPM handle `0x81000100` + cert at NV `0x01800100` that invariant #1 says must come from enrolment, never the image. Distinct from #10 (that enrols *Secure Boot* PK/KEK/db); this enrols the *network identity*. They share the provisioning UKI + isolated-environment vehicle. Pairs with #8 + the CA decision (start **step-ca**, [serving/dot1x-bench/](../serving/dot1x-bench/); OpenBao/EJBCA/Dogtag later). **Chosen pattern — trust-on-first-use with out-of-band fingerprint approval:** on an empty TPM the host generates a key *in* the TPM, builds a CSR, submits it to an enrolment **broker**, and shows a **fingerprint** on the Plymouth splash; an admin visually cross-checks that fingerprint against the broker's pending-list and approves; the broker has step-ca sign the CSR and returns the cert; the host writes it to NV `0x01800100` and proceeds to authenticate. Sound, recognised pattern (SSH/WireGuard/Signal-style OOB verification); good fit for invariant #1. **Key design decisions (settled 2026-06-22):** - **The pre-auth permit-set already exists — enrolment rides in it.** The firmware can't do 802.1X, so the UKI fetch (DHCP/DNS/TFTP/HTTP to the serving host) is **always pre-auth**; the secure model therefore *requires* a pre-auth permit-set (low-impact 802.1X / pre-auth dACL / boot→secure dynamic VLAN; RADIUS returns the secure VLAN on Access-Accept). The enrolment endpoint is just **another entry in that set** — expose it as an `/enrol/` location on the **existing serving nginx**, reverse-proxied to the broker. No new network mechanism. - **The broker is the only new component.** step-ca signs but has no human-approval queue: broker = `POST CSR + context → PENDING + fingerprint → admin approves → step-ca signs → host polls → cert`. Exposed to *unauthenticated* devices by design → harden: OOB-fingerprint gate (no auto-sign), rate-limit, time-boxed pending, HTTPS (image trusts the serving CA), pre-auth permit scoped to only these services + the serving host. - **Trigger on EMPTY-NV only — split the three events:** empty NV → enrol (human); valid key + expired/near-expiry cert → **renew** automatically (no human, it already holds a trusted key); auth fails with a present cert → **alert + retry, do NOT re-enrol** (auto-reenrol on failure would mask revocation and churn keys on a transient RADIUS outage). Re-enrol after the first time = explicit admin action (clear/flag the host). - **Fingerprint: what it proves + entropy.** It proves "this CSR came from *this physical device*" (approver reads the screen) — it does NOT prove TPM residency. Pair it with **TPM attestation** (step-ca ACME `device-attest-01`, EK/AK) for the residency proof: fingerprint = human authorises the device, attestation = the key is really in a TPM (together → strong SSP R-01 story). Size for entropy — a short hex fp is grindable; use a word-list/BIP39-style encoding (reads short, more bits) + rate-limiting + a server nonce. Show device context (asset/port/location) so the admin knows *what* they're approving. - **Pre-auth traffic has no MACsec (plaintext at L2).** Fine for enrol (CSR/cert are public; OOB fingerprint gives authenticity) — and it's exactly **why the UKI must be Secure-Boot-signed** (#8): integrity of the boot comes from the signature, not the network. Enrolment design + Secure Boot are two halves of the same "pre-auth path is hostile" assumption. - **Run it in a provisioning UKI/mode, not the production initramfs** — the boot hook ordering is load-bearing (invariant #2); don't park it on a multi-minute human-approval wait. Shares the #10 provisioning-UKI vehicle. **Bench caveat:** `hostapd` wired is strictly port-based (no pre-auth per-flow ACL) — before EAP success it passes nothing, so the bench can't reproduce "boot + enrol pre-auth on the same port." Model the pre-auth state with a separate provisioning bridge/VLAN on the rig; the in-place low-impact pre-auth permit is a real-switch feature to validate on the 820's switch. **First step DONE (2026-06-22):** `serving/dot1x-bench/enrol/tpm-selftest.sh` — a non-destructive TPM capability check (device, key persist, NV roundtrip, openssl TPM2 provider, CSR signed by the TPM key) to de-risk the TPM half before any of this. Expected gap: the image installs `tpm2-tss` but no openssl TPM2 **provider** (`tpm2-openssl`) — add it to the build once the self-test confirms. ## [~] 12. Workstation host-info API (`fleet-hostapi`) for the kiosk **IMPLEMENTED 2026-06-28.** A small **Go, read-only, loopback-only** HTTP agent on the workstation that reports host/OS/hardware facts to the kiosk SPA, per the [HOST-API.md](../HOST-API.md) contract (`GET http://localhost:9099/v1/host`). Source: `files/hostapi/` (stdlib-only, reproducible static build). Built + enabled by provisioner **2b-hostapi**; toolchain removed in 2d; binary fapolicyd-trusted in 2c-STIG-postfix (register #10). Chromium policy grants Private Network Access scoped to the kiosk origin (`files/kiosk/chromium-kiosk-policy.json`). - **Per-host safety:** host-specific fields (kernel, hostname, IP, MAC, board, uptime) are read at RUNTIME; only image-generic facts (image version + weston/chromium versions) are baked, to `/etc/fleet-hostinfo.json` (invariant #1 preserved). Deliberately does NOT expose DMI serial (root-only/per-host) or any TPM/cert/identity material. - **Hardening shipped:** loopback bind enforced in code; `DynamicUser`, no caps, `ProtectSystem=strict`, `MemoryDenyWriteExecute`, `SystemCallFilter=@system-service`, `IPAddressDeny=any` (loopback only). Runs unprivileged, reads world-readable files only. **Lab posture LEFT AS-IS (decided 2026-06-28) — revisit for prod:** - **CORS `Allow-Origin=*`** → scope to the kiosk origin (`http://boot.fleet.lab`). Consolidated into #8. - **SELinux:** runs in the default service domain (no custom policy module). **BENCH-VERIFY** no AVC denials for `fleet-hostapi` on an enforcing booted host (Wazuh/`fleet_audit` rule 100120 will surface any). If denials appear, decide: write a confined SELinux policy module vs adjust the sandbox. - **No auth** (justified: loopback + read-only + single-user appliance). Option if the threat model tightens: shared token in `kiosk.env` + header check. - **Info sensitivity:** exposes hostname/IP/MAC/board/versions to the kiosk. Drop MAC/IP if that's later deemed too much. - **Reproducibility:** the `golang` toolchain version is a reproducibility-sensitive input (a package bump changes the binary → verity hash); covered by the build-twice gate. Pin the go version only if drift becomes a problem. **BENCH-VERIFY (functional):** (a) service starts under enforcing SELinux + fapolicyd (not blocked); (b) the kiosk SPA's System page fetches `/v1/host` successfully through the PNA allowance (preflight + GET); (c) facts render correctly on real hardware (board/CPU/GPU from DMI/DRM, not the virtio build VM). ## [~] 13. Kiosk GPU compositing + Chromium flag ownership **GPU FIX APPLIED 2026-06-28.** The 820 (Intel HD620/i915) ran Chromium at ~150% CPU with periodic crashes: `--disable-gpu` in our `kiosk-session` forced software raster/compositing. Root cause confirmed from the live cmdline. Fixes in `files/kiosk/kiosk-session` + `diskless.pkr.hcl`: - **Runtime GPU probe** (fleet-generic image boots real HW *and* virtio VMs): a real KMS driver (`i915`/`amdgpu`/…) + `/dev/dri/renderD128` → GPU path (`--ignore-gpu-blocklist --enable-gpu-rasterization --enable-zero-copy`, weston GL renderer); else software (`--disable-gpu` + `weston --use-pixman`) so the VMs still render. No per-host state baked. - **kiosk user added to `render`** (renderD128 isn't logind-seat-ACL'd on EL9; `video` only covers card0/KMS for weston). - **Merged the distro wrapper's `--disable-features` into ours** — Chromium takes the LAST `--disable-features` (not the union), so ours was silently dropping the wrapper's Lens/Gemini/Compose disables. **BENCH-VERIFY (on the 820):** GPU process alive (`chrome://gpu`), Compositor/ ThreadPool threads near-idle + CPU drop, `gt_act_freq_mhz` rises under load, no SELinux AVC for `dri_device_t` from the kiosk domain (Wazuh `fleet_audit` 100120 will surface any — if present, add a scoped SELinux allow or confirm the kiosk user's SELinux mapping is unconfined). Confirm VMs still render (software path). **DECIDED 2026-06-28 — Option 1 (policy lock), APPLIED.** The distro chromium wrapper (`/usr/bin/chromium-browser`) layers flags as `$CHROMIUM_FLAGS` (`/etc/chromium/chromium.conf` → VA-API enables + AI disables; overridable via `CHROMIUM_USER_FLAGS`) + **hardcoded** `$CHROMIUM_DISTRO_FLAGS` (`--enable-plugins --enable-extensions --enable-user-scripts --enable-printing --enable-sync --auto-ssl-client-auth`) + our `"$@"` (last). The hardcoded layer isn't removable via env/conf — only by bypassing the wrapper. Rather than bypass (Option 2, which would force us to replicate the wrapper's stdio sanitisation + own the VA-API feature list across updates), we **neutralise in managed policy** (policy beats cmdline flags, survives updates): `ExtensionInstallBlocklist=["*"]` added to `files/kiosk/chromium-kiosk-policy.json` (blocks all extensions → also kills the user-scripts surface). `--enable-sync` already locked by `SyncDisabled`; `--auto-ssl-client-auth` is moot (no browser-store client cert; the TPM 802.1X cert isn't one) — `AutoSelectCertificateForUrls` left unset. VA-API enables inherited from `chromium.conf` unchanged. Option 2 (full flag ownership, bypass the wrapper, own the VA-API `--enable-features`: `AllowQt,WaylandLinuxDrmSyncobj,WaylandPerSurfaceScale,WaylandUiScale,AcceleratedVideoDecodeLinuxGL,AcceleratedVideoDecodeLinuxZeroCopyGL,AcceleratedVideoEncoder`) remains documented here if a minimal/auditable cmdline is ever wanted. **BENCH-VERIFY:** `chrome://policy` shows `ExtensionInstallBlocklist=["*"]` active; extensions can't install. ## [~] 14. (PARTLY BUILT) Centralised fleet management plane (health/version/usage/maintenance) **SERVER-SIDE CONTROL PLANE BUILT 2026-07 — `serving/fleet-manager/`** (Go service + embedded Svelte SPA). See the **as-built note** at the top of [docs/fleet-management-design.md](fleet-management-design.md). What shipped and how the design forks resolved: - **Two planes:** admin (HTTPS + **WebAuthn/passkey**, YubiKey; server-side session + CSRF; `register-admin` CLI bootstrap; UI cert via **Let's Encrypt DNS-01**, hot-reloaded) and agent (**bespoke gRPC bidi/mTLS**, fleet CA, reused TPM cert — health/metrics up, desired-state down; no inbound host port). - **Metrics → InfluxDB** (manager is the only reader/writer; host holds no Influx credential; identity-tagged writes; Flux reads behind a field whitelist + hex- serial guard). Graphs render **in the SPA** (bespoke SVG), not Grafana. This replaces the earlier Wazuh-Phase-1 / OTLP-Phase-2 sketch for *this* plane; Wazuh stays the audit/SIEM path. - **Authorise pillar:** the enrol-broker (#11) folded in "alongside" — manager drives its admin plane over mTLS and **births the inventory record** on approve. - **Deployment:** `Dockerfile` + `docker-compose.yml` (private InfluxDB + host-networked manager) + `server-pki.sh` + admin-UI cert `acme` side-car (LE DNS-01 via Route 53, auto-renew) + CI image build (`fleet-manager-image.yaml`, pulled by tag) + `backup.sh` (durable volumes) + `DEPLOY.md`. - **Hardening still open (operator/per-env):** network-segment the admin (`:9500`) + agent (`:9600`) listeners to the mgmt net; run `server-pki.sh` when the agent lands; decide `:9500` vs a 443 proxy; digest-pin the images (invariant #12). See DEPLOY.md "Hardening status". - **[~] (a) the on-host agent — TELEMETRY HALF BUILT 2026-07-05 (`files/agent/`).** The gRPC client that pairs with `fleet-hostapi` and feeds this plane. It reuses the host **TPM identity** for mTLS (signing key at `0x81000100` via `tpmkey.LoadSigner` — a read-only copy of the enrol `tpmkey`; cert from NV `0x01800100`), opens the long-lived `Session` stream, sends `Hello` + periodic `HealthSnapshot` (load/cpu/temp/mem/tmpfs/**uptime**/nvme/auth/selinux/kiosk from `/proc`+`/sys` + hostapi facts, plus **chrony** clock health — state/source/ stratum/offset — read from `/var/log/chrony/tracking.log`), and — **telemetry-only v1** — **receives `DesiredState`/`Command` but only LOGS them** (zero host mutation). Static `CGO_ENABLED=0` binary built out-of-VM in CI (like enrol/hostapi), staged by the `2b-agent` provisioner (`enable_agent`, default on), fapolicyd-trusted, fleet-generic config in `files/agent/agent.env`. mTLS handshake proven end-to-end against the go-tpm software simulator (`files/agent/simtest`). No dracut/initramfs changes (post-switch-root only). **STILL TO BUILD (deferred):** the **reconcile actuators** — apply kiosk_url, reboot_at/REBOOT, drain — wired to be received but not acted on; land after the telemetry half is bench-verified. **BENCH-VERIFIED END-TO-END ON REAL HARDWARE 2026-07-10** (Dell 820 `comet-beacon-54`, real TPM, image `fleet-manager-254ec79`): the agent loads its reused TPM identity (serial derived on-device via go-tpm), completes mTLS to the manager, streams health, AND receives desired-state (logged, not applied). Manager logs `agent connected: fed27077… (comet-beacon-54)`; the host stays connected (no reconnect loop). This closes the TPM-identity path (802.1X = NVMe hostnqn = agent mTLS, "one identity, three stages") on hardware. **Deploying the agent plane — gotchas hit + FIXED in commit `a89c3d2` (don't rediscover):** the serving-side bring-up cost a long debug chain, each now automated: 1. **Host CA was the placeholder.** `/etc/dot1x/radius-ca.pem` shipped a non-CA placeholder → `client TLS: no certificates parsed from fleet CA`. FIX: `fleet_ca` build var bakes the real step-ca root (CI secret → file → `-var fleet_ca`; default = placeholder, lab-only). The agent's `FLEET_AGENT_CA` points here; only the ROOT is needed host-side (the manager sends its chain). 2. **`tls: certificate required`.** step-ca signs leaf certs with an INTERMEDIATE and enrolment stores the leaf-only cert in NV, so a root-only client CA makes the manager advertise only the root in its CertificateRequest → the Go client sends NO cert. FIX: `server-pki.sh` installs the agent CLIENT CA (`FLEET_MGR_AGENT_CLIENT_CA`) as **root+intermediate**; `bootstrap.sh` exports `intermediate_ca.crt`. 3. **Distroless perms.** The nonroot manager (uid 65532) can't read `pki/agent/*` if created `0600` by the host user → `open …/server.crt: permission denied`. `chown 65532` the agent PKI (as with the fleet-data volume). 4. **Manager wasn't a fleet name.** The agent's baked default pointed at the PUBLIC admin-UI name (`diskless-server.bitasylum.org`), which resolves to the serving host's OUTBOUND IP → `no route to host` from the fleet LAN. FIX: the agent plane is now just another `*.fleet.lab` — `manager.fleet.lab` (`MGMT_HOST` in serving/.env), served by the fleet dnsmasq to `SERVER_IP` exactly like boot/nvme/audit (`--address=/${MGMT_HOST}/${SERVER_IP}`), baked as `FLEET_AGENT_MANAGER` and set as the agent server cert SAN (`AGENT_SANS`). No per-host override, no server-name hack. (The public `diskless-server…` name stays the ADMIN-UI name on Traefik/:443 — a separate plane.) The transient bench workaround was `FLEET_AGENT_MANAGER= + FLEET_AGENT_SERVER_NAME=`; no longer needed. 5. **Host not in the manager's inventory** → `PermissionDenied` (surfaced as a bare `EOF` on the agent before the fix). A host enrolled OUT OF BAND (bench broker / migrated fleet) has no inventory row. FIX: `fleet-manager register-host -name ` (writes through the real store to `FLEET_MGR_INVENTORY`). The manager caches inventory in memory — any edit needs a `--force-recreate`. Also: the agent now logs the real gRPC status, not `EOF`. **Metrics + cert auto-renewal added 2026-07-10 (`fleet-manager` branch):** - **uptime + reboot-pending**: the agent sends `uptime_seconds`; the manager/UI derive "reboot pending in X / rebooted" from boot_time (= last_seen − uptime) vs `reboot_at`, all in the manager clock frame (no host↔manager clock sync needed). Monitor metric graphs auto-refresh (10 s poll). - **chrony clock health** (sync state + NTP source/stratum/offset). The agent reads `/var/log/chrony/tracking.log` (image bakes `log tracking`), NOT chronyd's command socket (owner-write-only unix socket; STIG `cmdport 0` closes the UDP port). **Reaching that log from the hardened DynamicUser agent took a 4-layer chase, every layer silent-failing** (memory `fleet-agent-hardened-probe-access`): (i) lab NTP reachable — a `123/udp` rule on the serving-host firewall (host-networked ntp container; same trap as audit `514`); (ii) DAC — `chrony` supplementary group on the unit to traverse `/var/log/chrony` (0750); (iii) SELinux TE — `files/stig/fleet-agent-chronylog.te` allow for `chronyd_var_log_t` (permissive masks this); (iv) the MANAGER image must be ≥ the chrony commit — one with uptime-but-not-chrony silently drops the fields (manager & workstation are SEPARATE images/deploys). Isolator: read the manager `inventory.json` off the `fleet-data` volume via a throwaway busybox (distroless manager, no shell). BENCH-VERIFIED on `comet-beacon-54` 2026-07-10. - **agent-plane server-cert auto-renewal**: the `agent-cert-renewer` compose sidecar (profile `cert-auto`) issues + `step ca renew --daemon` renews the cert from a co-located step-ca; the manager hot-reloads via `GetCertificate` (`certreload.go`), no restart. Replaces the manual `server-pki.sh` + `docker exec … step` dance and the 90-day re-mint. The server cert SANs MUST include `manager.fleet.lab` (the fleet-DNS name the agent dials) — a missing SAN fails the handshake. - **STILL TO BUILD:** (b) **per-host boot-version assignment** — `target_version` is stored + pushed but does not change what a host *boots*; that needs dnsmasq cohort steering + multi-version UKI serving (**#6**). Kiosk URL / reboot / drain reconcile via the agent's deferred actuator half (above). Version pin is labelled "recorded intent" in the Lifecycle UI. Depends on the step-ca fleet CA (#8) for the mTLS identity.
Original design capture (2026-06-28) — retained for reference **DESIGN CAPTURED 2026-06-28 — [docs/fleet-management-design.md](fleet-management-design.md).** Server-authoritative, pull-based, TPM-identified management plane reusing existing pieces: `fleet-hostapi` as the on-host data source, the audit-collector→Wazuh pipeline as telemetry transport, the digest-pinned deployment manifest as version truth, and the SPDK `spdk-loader`/`spdk-release` rolling-update flow (#6) as the redeploy/drain mechanics. Hosts stay stateless (identity from the TPM cert, no baked enrollment); control is pull/reconcile (no inbound surface); convergence to a new version = reboot. **Phased build:** (1) push hostapi snapshots as structured events into Wazuh + fleet dashboard (health/version-drift/offline/usage/alerting); (2) metrics push (OTLP/remote-write) → VictoriaMetrics + Grafana; (3) mTLS desired-state pull (version pin, reboot/drain, kiosk URL) wired to the deployment manifest + SPDK flow. Build-time decisions (telemetry encoding, metrics/desired- state store, per-host vs cohort, osquery/FleetDM eval) listed in the design note. Depends on the step-ca fleet CA (#8) for the mTLS identity.
## [ ] 15. Scope the root overlay → per-subtree tmpfs/overlay (un-shadowable `/usr` + STIG mount options) **DESIGN CAPTURED 2026-07-02.** Today the whole root is one `systemd.volatile=overlay` tmpfs over `/` (see #5, invariant on the volatile overlay). Replace it with a **read-only verity root + a handful of scoped writable mounts**, for two payoffs that land from the same change: 1. **Security — un-shadowable `/usr`.** dm-verity seals the backing store, but the whole-root overlay lets any root process write a tmpfs *copy* over a system binary (e.g. `/usr/sbin/fleet-enrol`) that then executes in preference to the sealed one until reboot. fapolicyd is the current compensating control (CLAUDE.md: "the writable-overlay exec guard dm-verity can't provide"). Keeping `/usr` (and `/`, `/opt`, `/boot`) strictly **read-only** makes the executable set both verity-sealed *and* un-shadowable — defence in depth, shrinks fapolicyd reliance. Also: writes outside the sanctioned set fail loudly (`EROFS`) instead of silently landing in RAM and vanishing at reboot — enforces invariant #1 (no per-host state) architecturally. 2. **STIG — converts the partition/mount family from `notapplicable`/waived to real `pass`.** Quantified against the scan in `reports/stig-arf.xml` (NOT estimates): the `mount_option_*` rules currently score **`notapplicable`** (no separate mount to check — the "false-clean via notapplicable" pattern #2 warns about), and `partition_for_*` are **`notselected`** (waived in `diskless-tailoring.xml` lines 61-66). Scoped tmpfs mounts with `noexec,nosuid,nodev` + re-selecting the partition rules convert **~21 rules to genuine `pass`**: - 15 `mount_option_*` N/A→pass: `/tmp` (nodev,nosuid[,noexec*]), `/var/tmp` (all 3), `/var/log` (all 3), `/var/log/audit` (all 3), `/var` (nodev), `/home` (nodev,nosuid). - 6 `partition_for_*` re-selects→pass: tmp, var, var_log, var_log_audit, var_tmp, home (+1 easy: `partition_for_dev_shm` — already a separate tmpfs). - Retires **deviation-register entry #1** (the narrative hand-wave over the whole partition/mount family) — one fewer waiver to defend at ATO. - `/dev/shm`'s three option rules **already pass** — no work there. **Writable-set + flag matrix (the design):** - `/` `/usr` `/opt` `/boot` → **read-only** (verity as-is, no overlay). `/usr` keeps suid for legit setuid bins (sudo/su/passwd). - `/etc` → overlay (lower=verity `/etc`, upper=tmpfs) — needs defaults + writability (machine-id, regenerated ssh host keys, NM connections, resolv.conf, chrony). - `/var` → overlay (NOT bare tmpfs — bare tmpfs hides `/var/lib/rpm`, selinux store); `nodev`. Nest the noexec leaf mounts on top (order: `/var` first, then children). - `/var/tmp` `/var/log` `/var/log/audit` → tmpfs `noexec,nosuid,nodev`. - `/tmp` → tmpfs `nosuid,nodev` (+`noexec` only after the kiosk repoint, below). - `/home` `/root` `/srv` → tmpfs `nodev,nosuid`. - Each tmpfs needs the **correct SELinux context** (`tmp_t`, `var_log_t`, `auditd_log_t`, …) via `context=` or restorecon — this is the login-deny hazard class (#5); a mislabeled tree denies logins / stops auditd. **Prerequisites / caveats that gate the count:** - `mount_option_tmp_noexec` (1 of the 21) is **conditional**: register entry #1 notes the Chromium kiosk uses `/tmp` for profile/cache and breaks under `noexec`. Must **repoint the kiosk scratch** (Chromium `--user-data-dir`/`--disk-cache-dir`) to a dedicated exec-capable tmpfs (e.g. `/run/kiosk`) first → ties to #13/#4. So ~20 land immediately, +1 after the repoint. - **fstab-vs-runtime OVAL — mostly de-risked:** the rules return `notapplicable` (evaluating the live mountpoint) not `fail` (missing fstab line), evidence they read the runtime mount table — so systemd mount units / initramfs mounts should score without fstab entries. Confirm per-rule on the bench; add fstab lines if any rule is fstab-based (compatible with the empty-fstab invariant — root still comes from netroot+verity). - **Does NOT fix red findings** (this family is ~0 fail today) — the value is evidenced compliance + one fewer deviation + kernel-enforced noexec, not clearing fails. Framed honestly so it isn't oversold. - **`partition_for_usr` stays waived** — `/usr` should be verity-sealed, not a separate writable partition (a *stronger* control than the rule asks). Same for `encrypt_partitions`, `auditd_..._large_partition`, boot/boot_efi (genuinely N/A). **Mechanism:** `systemd.volatile=` has no per-dir knob (it's all-or-var-only), so this means dropping the global volatile overlay for explicit early mount units (ordered before `local-fs.target`) or initramfs mounts. Reproducibility unaffected (runtime mounts, not image content → no verity-hash drift). **Load-bearing / bench-gated:** touches the boot path (#5 territory) — full boot-chain bench (`serving/dot1x-bench/`) + careful SELinux-label verification + a reconciliation note in `design-v2-errata.md`. **Do AFTER TPM enrolment (#11) is bench-verified.** Tailoring edits: re-select the 6 `partition_for_*` (+dev_shm), delete/gut register entry #1, re-run `scripts/stig-deviations.sh` to confirm the new passes. ## [ ] 16. Enable a locked-down host firewall (firewalld) — defense-in-depth + STIG passes **DECISION 2026-07-03: enable a *configured* firewalld.** Currently REMOVED + masked (`diskless.pkr.hcl` 2d, commit 4346ce7) because an UNCONFIGURED weak-dep firewalld was running and flooding the SIEM with `lib_t` `.pyc` AVCs. The design deferred the firewall as "future hardening, currently waived" ([stig-deviations.md](stig-deviations.md) #4) — NOT incompatible, just unverified during boot-chain bring-up. A diskless kiosk is an ideal firewall candidate: near-zero legit inbound (admin SSH only; `fleet-hostapi` is loopback), and the load-bearing paths are outbound (NVMe/TCP, DHCP) or L2 (EAPOL/MACsec). **Payoff:** defense-in-depth (limits a compromised same-segment peer) + converts **4 STIG waivers → passes** (`service_firewalld_enabled`, `package_firewalld_installed`, `set_firewalld_default_zone`, `configured_firewalld_default_deny` — re-select in the tailoring; retires deviation-register #4). **Must do together:** 1. **Config:** default-DENY inbound zone; allow **SSH** (key-only admin) + established/related; assign `macsec0` (secure) / `en*` (test) to it. hostapi (:9099 loopback) + kiosk→hostapi PNA need no rule. 2. **Fix the `.pyc` AVC root cause (REQUIRED — an enabled firewalld re-hits it):** the reproducible erofs `-T 0` zeroes mtimes, so Python thinks every site-package `.pyc` is stale and rewrites it → `firewalld_t` (and ANY Python daemon, e.g. `tuned`) write/unlink AVCs on `lib_t`. Fix once with `python3 -m compileall --invalidation-mode unchecked-hash` over the Python trees BEFORE the erofs build → mtime-independent `.pyc`, no rewrites, and MORE reproducible. (Worth doing regardless of firewalld — it's the true root cause of the firewalld AVCs seen 2026-07-03.) **[x] DONE (2026-07-03):** implemented in `diskless.pkr.hcl` — (a) the compileall `--invalidation-mode unchecked-hash` pass over `/usr/lib*/python3.*` at the end of the 2d hygiene block (last step before erofs extraction), and (b) masked `dnf-makecache.timer` next to the existing `rhsmcertd` mask (it was the observed ~10-min post-boot trigger that re-imported the dnf Python stack). Confirmed live on a booted host via the overlay audit: `dnf/libdnf/librepo/rhsm` `__pycache__` rewritten at boot+10min. **BENCH-VERIFY:** after a rebuild, no more `.pyc` files appear in the overlay (`find / -xdev -newermt '1980-01-01' -path '*__pycache__*'` empty) and the `lib_t` `.pyc` AVCs stop in the SIEM. Items 1 & 3 (firewalld enablement + tailoring) still OPEN. 3. **Tailoring:** re-select the 4 firewalld rules. **THE load-bearing bench gate (why #4 deferred it):** the NVMe/TCP root is an OUTBOUND connection to `:4420` established in the initramfs BEFORE firewalld starts. Verify it **SURVIVES** firewalld start + `firewall-cmd --reload` (established/related allowed by default → should, but UNVERIFIED); a reconnect within `ctrl_loss_tmo` re-establishes *through* the firewall; DHCP renew + MKA keepalives unaffected; SSH in; no AVCs; verity hash reproducible. If it drops the root, that is the fleet-wide outage risk — gate hard. ## [~] 17. Cap fapolicyd trust-DB RAM (`db_max_size`) — biggest overlay consumer **Finding (overlay RAM audit, 2026-07-03, host `cygnus-cruiser-99`):** the writable root is `systemd.volatile=overlay` (a tmpfs). Enumerating runtime-written files via the `-T 0` trick — every baked file has mtime = epoch, so anything newer is an overlay write: ```bash sudo find / -xdev -type f -newermt '1980-01-01' -printf '%b\n' | awk '{s+=$1*512} END{print s/1048576" MiB"}' ``` showed **124.5 MiB** total, of which **`/var/lib/fapolicyd/data.mdb` = 100 MiB, FULLY allocated** (`du` == `du --apparent-size`, not sparse) — **~80% of the entire overlay.** fapolicyd's LMDB trust DB is rebuilt at boot onto the tmpfs overlay, so its `db_max_size` (MB, in `/etc/fapolicyd/fapolicyd.conf`) is a **fixed RAM reservation**, not a ceiling it grows into. The EL9 default is ~100; the minimal+kiosk trust set is a fraction of that. **[x] DONE (2026-07-03):** `diskless.pkr.hcl` 2c-STIG-postfix now seds `db_max_size = 50` into `fapolicyd.conf` BEFORE the `fapolicyd-cli --update` rebuild (so the rebuilt db uses the smaller map). Value 50 chosen conservatively — it was fapolicyd's upstream default for years and covered a FULL workstation rpmdb (larger than this image), so it has real headroom. Reclaims ~50 MiB of overlay RAM. **[ ] BENCH-VERIFY (STIG-critical — fapolicyd is the writable-overlay exec guard):** 1. Boot the rebuilt image; confirm kiosk renders, admin SSH (ECDSA) logs in, `sudo` works, and custom binaries (`kiosk-session`, `fleet-hostapi`, `fleet-enrol`) still execute — i.e. the trust DB rebuilt cleanly at 50 MiB with no `MDB_MAP_FULL`. Undersizing bricks exec (login/kiosk) — that's the hard gate. 2. Re-run the overlay audit: `data.mdb` should now be ~50 MiB and total overlay ~75 MiB. 3. **Can go lower:** measure the real trust set with `fapolicyd-cli --dump | wc -l` on the booted host; if the content is only ~15–25 MiB, 40 (or less) is safe and reclaims more. **Secondary lever (free, already in flight):** the trust DB scales with trusted files, so the kiosk weak-deps trim (commit `a193dc3`, ~100 fewer packages) shrinks the *actual* `data.mdb` content — widening the headroom under `db_max_size`. **Other overlay consumers noted (not fapolicyd, for context):** Chromium `/dev/shm` (~38 MiB, its own tmpfs) + kiosk `/tmp/kiosk-profile` (on the root overlay here — `/tmp` is NOT a separate tmpfs, ties to #15's mount separation); `/var/log/{messages,audit}` grow unbounded in RAM (audit is local-only) — logrotate caps or SIEM offload on long-lived hosts. ## [~] 18. Slim the image — locale + firmware (biggest erofs-size wins) **Finding (image-size audit via the 2b `rpm -qa --qf '%{SIZE}'` manifest, 2026-07-03):** the largest packages in the 534-package image are NOT the kiosk — they're base bloat: `linux-firmware` **678 MB**, `glibc-all-langpacks` **228 MB**, then chromium (370) + llvm-libs (129). The uncompressed erofs is read over NVMe/TCP on every boot of every host, so every dead byte is fleet-wide waste + attack surface. **[x] DONE — glibc langpacks → en-only** (`diskless.pkr.hcl` step 2). `@^minimal-environment` pulls `glibc-all-langpacks` (all languages); swapped for `glibc-minimal-langpack` + `glibc-langpack-en`. ~220 MB reclaimed; image is English-only (we also strip `/usr/share/locale` in 2d). **BENCH-VERIFY:** locale still `en_US.UTF-8`; kiosk + login OK. **[x] DONE — linux-firmware strip to a hardware-audited keep-list** (`diskless.pkr.hcl` step 2, run **BEFORE the dracut build** so the UKI initramfs shrinks too — not just the rootfs. `--no-hostonly` + plymouth's `drm` dep instmods a broad DRM set and dracut bundles each module's firmware into the initramfs, amdgpu being the classic UKI bloater; the strip runs first so dracut never sees it. A post-build diagnostic reports the `.img` size + any firmware still bundled — expected i915 only.) Live fleet audit (`modinfo -F firmware` over all loaded modules + sysfs/`lspci` driver bindings on the real 820-class hosts AND the VMs) proved the **only** loadable firmware any host requests is Intel **i915** (GPU). Every other driver is firmware-free — crucially **e1000e** (on-chip NVM, not a fw file) and **virtio_net**, so NO NIC needs firmware and the fleet-brick case is off the table. Keep-list: `i915` + `intel/` (ice DDP insurance for the `ice`/`mlx5` server-NIC build variant + small Intel blobs) + `mellanox` + `intel-ucode`; strip the rest (amdgpu/nvidia/radeon/ath/qcom/rtl/iwlwifi… ~600 MB). Wi-Fi (`iwl*-firmware`, ~83 MB) removed as packages (wired-only fleet). Strip-in-place (`find … -exec rm -rf`, not `rpm -e` — can't drop the package without losing i915); a `du -sh /usr/lib/firmware` line reports the result in the build log. **BENCH-VERIFIED 2026-07-03 (commit `f59eb79`): built, booted, AND the kiosk SPA RENDERS on both the VM (virtio-gpu) and the 820 (i915).** This clears the firmware strip's one real risk — the i915 DMC/GuC blobs load on real hardware, so the keep-list (i915/intel/mellanox/intel-ucode, pre-dracut) is validated end-to-end (firmware 678→147 MB, initramfs bundles only the keep-list). Remaining caveat: if a future host has a non-Intel GPU or an `ice`/`mlx5` NIC, widen the keep-list before trusting fleet-wide. **NOT yet in `f59eb79`** (bench-verify separately — none touch i915, so render should be safe): the `nss-mdns`/avahi drop (`5cc72cc`) and the `omit_drivers` GPU-kmod trim + keep-list tightening — dropped `mellanox/`, narrowed `intel/` to `ice` only (`853ae20`). Audited models: HP-820-class (i915 + e1000e + iwlwifi-dropped) and virtio VMs. **Build-verified 2026-07-03:** firmware **678 MB → 147 MB**; the initramfs diagnostic confirms only the keep-list is bundled (i915 all-gens + intel/ice + intel/ish + mellanox) — proving the pre-dracut ordering works. But the initramfs came out **97 MB**, and it's now **kernel-MODULE-bound, not firmware** (~10 MB fw): the `--no-hostonly` + plymouth `drm` set pulls every GPU driver (`amdgpu.ko` etc.) even though the fleet is i915-only. **[x] DONE — follow-on trims (2026-07-03):** - **`omit_drivers+=" amdgpu radeon nouveau "`** in `90-diskless.conf` — drops the discrete-GPU kmods plymouth's `drm` dep would instmod (amdgpu.ko is the biggest single module); keeps i915 + virtio-gpu/bochs for the splash. Targets the 97 MB initramfs/UKI. - **Firmware keep-list tightened**: dropped `mellanox/` (those are `mlxsw_spectrum` SWITCH-ASIC blobs, NOT NIC fw — mlx5 NIC fw is on-card) and narrowed `intel/` to just `ice` (dropped ish/sof/ibt = sensor-hub/audio/bluetooth, none loaded at boot). - **Initramfs top-25-files diagnostic** added after the dracut build (sorts `lsinitrd.txt` by size) so the next build shows what dominates the `.img` — measure before the next cut. - **BENCH-VERIFY:** kiosk still renders on the 820 (i915 kept); check the new "initramfs largest files" log line for the post-omit size + next targets. **[x] FIXED — the `nss-mdns` drop bricked the kiosk (regression found+fixed 2026-07-04):** the "render should be safe" call above (5cc72cc) was WRONG — it broke the kiosk, just not via i915. `trim-rpm-48dc3e1` booted to a **flashing cursor, no kiosk**, and the image was a suspicious **500 MB**. Root cause: the 2d `dnf remove -y nss-mdns` removed **chromium** as a *dependent package* — the EPEL `chromium` **hard-Requires `nss-mdns`** (Chromium uses `.local` mDNS hostnames for WebRTC ICE candidates) — then reaped chromium's orphaned GUI stack (avahi, at-spi2, atk, adwaita, chromium-common; ~49 pkgs, 523→472). With no browser, `kiosk-session` died at the `${BROWSER:?}` guard in 4 ms, before weston. The original commit checked what `nss-mdns` NEEDS (avahi) but not what NEEDS it (chromium). GPU/i915/firmware/plymouth were all healthy — a red herring. Fix (`2bc2daf`): - **Keep `nss-mdns` + avahi** (satisfy chromium's Requires); **mask the avahi daemon + socket** instead — nss-mdns is only an NSS module, needs no listener, and :5353 was the real attack-surface concern. ~1.5 MB kept. - **FINAL GUARD in 2d**: `rpm -q chromium chromium-common weston mesa-{dri-drivers,libEGL,libgbm}` asserts the kiosk stack survived the whole pipeline, else **fail the build**. A missing browser must brick the BUILD, never the fleet — catches any future remove-cascade / oscap `package_*_removed` / autoremove. Look for `==> guard: kiosk stack intact` in CI. **RE-VERIFIED 2026-07-04 (`trim-rpm-2bc2daf`): boots into the kiosk on the 820; image 755 MiB.** **Measured end-to-end (2026-07-04, `trim-rpm-2bc2daf`): container artifact image 1.8 GiB → 755 MiB (~58%, >1 GB off every host's per-boot NVMe/TCP read).** **Combined:** ~**800 MB** off the uncompressed erofs (langpacks + firmware), dwarfing the kiosk weak-deps trim (#4), plus a smaller UKI/initramfs from the DRM-driver omit. NEXT candidates from the manifest (measure, don't guess): `iso-codes` (19 MB), `samba-client-libs`/`intel-mediasdk`/`gstreamer` (chromium-common HARD deps — only removable by dropping HTML5 media, and samba/rabbitmq/rist/srt/zmq are libavformat DT_NEEDED so they can't go without breaking future media — confirmed via ldd 2026-07-03), `binutils` (25 MB, build-only?), `cracklib-dicts` (9.8 MB, needed for pam pwquality — keep). **Stack note:** committed on `trim-rpm`. When `optimise-ram` rebases onto the new `trim-rpm`, expect a trivial `docs/TODO.md` tail conflict (both branches appended after #16 — `optimise-ram` added #17 fapolicyd, this added #18); resolve by keeping both.