# TODO — diskless fleet boot system Persistent backlog, kept in sync with the working session task tracker. Status: `[ ]` pending · `[~]` in progress · `[x]` done. --- ## Current build posture (LAB / test) — as of 2026-06-19 The image currently builds in **open-network test mode**. These toggles must be flipped back for a secure/production build (consolidated as item #8): - `disable_dot1x=true` / CI `DISABLE_DOT1X="true"` → `rd.dot1x=0` (NO 802.1X / MACsec; open NVMe-oF). **TEMPORARY.** - `nvmet_traddr` / CI `NVMET_TRADDR="192.168.50.10"` → hardcoded IP (should be a per-env DNS name like `nvme.fleet.lab` once DNS-at-boot is trusted). - Unsigned UKI (`sb_key`/`sb_cert` not passed) → Secure Boot must be OFF on the test target. Production needs CI/HSM signing. - `enable_kiosk=true`; `boot_splash=true`; NTP via DHCP→serving-host chrony. - **`enable_stig` / CI `ENABLE_STIG="false"` RIGHT NOW** — temporarily disabled to get a working box while the STIG regressions were diagnosed (now fixed, below). **Flip back to `true`** and bench-verify the hardened build. - **`debug_shell=true` (DEBUG HOLE — passwordless root on tty2). REMOVE before prod** (`-var debug_shell=false`). Note: it didn't actually come up on tty2 in testing; diagnosis ended up done via offline `rootfs.erofs` mount + SMBIOS `enforcing=0`. SMBIOS cmdline-extra works on Proxmox only after a full VM **stop/start** (not `reset`). - **Admin SSH key MUST be ECDSA or RSA, NOT ed25519**, when STIG is on — the STIG crypto policy drops ed25519 from sshd's accepted pubkey algorithms. Set the `SSH_AUTHORIZED_KEY` CI secret to an ECDSA/RSA key (see #2). - **Hostname:** image is hostname-neutral (empty `/etc/hostname`, `hostname-mode=default`); the serving dnsmasq's `fleet-namer` (`--dhcp-script`) assigns a stable creative starship name per MAC (e.g. `nebula-vanguard-07`) via DHCP option 12 + DNS A/PTR. Host picks it up on next DHCP renewal/reboot. - **Serving `.env`:** must set `AUDIT_HOST=audit.fleet.lab` (+ `docker compose up -d dnsmasq` to reload) for remote log offload to resolve/flow. What's working/verified vs awaiting bench is per-item below. Cross-cutting bench-verify checklist lives in `CLAUDE.md` → "Known bench-verify items". --- ## [x] 1. Port the dracut module off `network-legacy` → `network-manager` **DONE — test path VERIFIED ON THE BENCH 2026-06-17** (host stays up past the DHCP lease time; no more lease-expiry outage). Fixed the confirmed outage: network-legacy's one-shot DHCP lease was never renewed post-switch-root, so the address dropped at lease expiry → NVMe/TCP root died ("keep alive timeout" / "no route to host"). NM hands its initramfs connections to the real-root NM (`/run/NetworkManager` state), which renews. REMAINING: the **secure path** (macsec0 DHCP via `nmcli`, physical NIC unmanaged via `nm-unmanaged.sh`) is only exercised once `disable_dot1x=false` — verify on the swtpm + hostapd + FreeRADIUS bench when re-enabling security (see #2/cleanup). Done: - `90-diskless.conf` `add_dracutmodules`: `network-legacy` → `network-manager`. - `module-setup.sh` `depends()` → `network-manager`; added `nmcli`; installs the new cmdline hook. - NEW `nm-unmanaged.sh` (cmdline hook): secure mode writes `/run/NetworkManager/conf.d/10-dot1x-unmanaged.conf` (`unmanaged-devices=interface-name:en*`) **before NM starts**; no-op in test mode. Replaces the deleted static `99-dot1x-unmanaged.conf` (couldn't be mode-aware — it wrongly unmanaged `en*` in test mode, blocking renewal). - `dot1x-macsec-up.sh` secure path: `ifup macsec0` → `nmcli` (managed + DHCP, poll for IPv4). macsec0 MANAGED so NM renews; physical NIC UNMANAGED. - `diskless.pkr.hcl`: dropped `dhcp-client`, added explicit `NetworkManager`; removed the static-config install; lsinitrd now asserts `NetworkManager` + `nm-unmanaged.sh` present. - Static lint clean (`bash -n`, `shellcheck`, `packer validate`). **BENCH-VERIFY before trusting (the gate):** - Test path (`disable_dot1x=true`): boot, `nmcli device` shows boot NIC MANAGED; run past the DHCP lease time (drop the lab dnsmasq lease low) — address persists, lease renews (`journalctl -u NetworkManager`), NVMe + SSH stay up. - Secure path (swtpm + hostapd + FreeRADIUS, `disable_dot1x=false`): `en*` UNMANAGED, `macsec0` MANAGED + renewing; MKA survives switch-root; verity root mapped; NVMe stays connected past lease time; no "Address already assigned". - **Risk:** ordering of the unmanaged drop-in vs NM start in secure mode. If NM transiently grabs `en*`, fall back to also baking a static `en*`-unmanaged file for secure-only builds. ## [~] 2. Apply RHEL9 STIG hardening to the built image **IMPLEMENTED 2026-06-18 — awaiting build/bench verification.** OpenSCAP + SSG remediation in a provisioner (`enable_stig` var, default on), tailored for the diskless/UKI/kiosk architecture via `files/stig/diskless-tailoring.xml`. STIG kernel params baked into the UKI cmdline (`extract-artifacts.sh`). Decisions: **FIPS off**, **audit local-only**, **CI report-only**. Deviations are documented in **[stig-deviations.md](stig-deviations.md)** (curated register) and reported per-build by **[../scripts/stig-deviations.sh](../scripts/stig-deviations.sh)** (non-pass rules tagged intentional vs REVIEW); CI publishes `stig-report.html` + `stig-arf.xml` + `stig-deviations.md`. Scanner tooling is removed from the shipped image. **BENCH-VERIFY:** boot/login **VERIFIED ON THE BENCH 2026-06-19** — hardened build with `enable_stig=true`, host `cygnus-cruiser-99`: kiosk auto-starts + renders, admin SSH (ECDSA) logs in, audit offload reaches the collector, DHCP creative hostname applied, NVMe/TCP verity root mounts and the host stays up. **STILL OPEN:** reproducible verity hash across two builds; triage the REVIEW rows (below); the secure path (#8, 802.1X/MACsec — this was the `disable_dot1x=true` test path). **FINDING 2026-06-18 (FIXED) — false-clean scan.** The first hardened build (branch `hardening`, `ada4aa4`) produced a report reading "0 REVIEW / clean", but the ARF had **0 pass, 0 fail** — only 462 `notapplicable` + 1068 `notselected`. Root cause: the provisioner selected the datastream with `ls ssg-rhel9-ds.xml ssg-rl9-ds.xml | head -1`, which picks **rhel9** alphabetically; the RHEL9 datastream's CPE dictionary has **no Rocky entry**, so on Rocky 9.8 every selected rule went `notapplicable` and **nothing was actually evaluated or remediated** — the image was unhardened despite a clean-looking report. Fixes: (a) provisioner now maps `/etc/os-release $ID` → datastream (Rocky→`ssg-rl9-ds.xml`); (b) provisioner **aborts the build** if the scan evaluates 0 rules; (c) `stig-deviations.sh` flags an **INVALID SCAN** and exits non-zero on 0-evaluated. **Re-run the hardened build** and confirm pass/fail counts are non-zero before trusting any deviations report. **FINDING 2026-06-18 (FIXED) — hardening locked the fleet out.** The first genuinely-evaluated hardened image (app `215b0eb`) broke **both** login paths: the Chromium kiosk dropped to a getty login prompt, and admin SSH was rejected with `Permission denied (publickey)`. Root cause: STIG enables `pam_faillock` (account phase) + password aging/inactivity, and the **kiosk + fleetadmin accounts are passwordless / key-only**, so the hardened PAM *account* phase treats them as expired/lockable (sshd key auth succeeds, then `pam_acct_mgmt` fails). Fix (provisioner `2c-STIG-postfix`, runs after remediation): exempt the two appliance accounts — `chage -M -1 -m 0 -I -1 -E -1` + `faillock --reset`; kiosk uses a dedicated `/etc/pam.d/kiosk` (`PAMName=kiosk`, permissive account / full session for seat); sshd gets an `account sufficient pam_succeed_if.so user = fleetadmin` first-rule bypass. Scoped to those two accounts; everyone else keeps full STIG. Justified as register deviation #10. **VERIFIED ON THE BENCH 2026-06-19** (`enable_stig=true`, host `cygnus-cruiser-99`): kiosk auto-starts + renders and admin SSH (ECDSA key) logs in with hardening on.** **UPDATE 2026-06-19 — offline erofs inspection of the hardened image (mounted `rootfs.erofs` read-only) pinned the real causes; the PAM-account theory above was incomplete:** - **Kiosk had TWO causes, both FIXED:** 1. **fapolicyd** is `enabled` and `kiosk-session` wasn't in its trust → it blocked `/usr/local/bin/kiosk-session`. Fix: KEEP fapolicyd (it's the exec-guard for the writable overlay dm-verity can't cover) and add the kiosk binary to its trust DB at build (2c-STIG-postfix). Not waived. 2. **Auto-start:** kiosk was `WantedBy=graphical.target` but the host boots to `multi-user.target` (graphical never reached) → unit `inactive (dead)` at boot, fine on manual `restart`. Fix: `WantedBy=multi-user.target` + dropped `set-default graphical.target` (kiosk launches weston directly, needs no DM target). - **SSH = STIG crypto policy excludes ed25519 (CONFIRMED via `ssh -vvv`).** The key fingerprint MATCHED (not a mismatch); the server's `server-sig-algs` advertised only ecdsa + rsa-sha2 — **no `ssh-ed25519`** — so the ed25519 admin key is rejected at the signature stage (before PAM/SELinux, which is why `enforcing=0` and the account-bypass didn't help). **Fix (your action): set the `SSH_AUTHORIZED_KEY` CI secret to an ECDSA (`ssh-keygen -t ecdsa -b 384`) or RSA key**, then rebuild. No code change. (Alt: re-allow ed25519 via `PubkeyAcceptedAlgorithms +ssh-ed25519` in the sshd drop-in — but that's a non-approved algo and would re-fail a STIG rule; prefer the ECDSA key.) - **rsyslog TLS-offload rules waived (lab):** `rsyslog_encrypt_offload_*` force gtls+x509+CA and broke the plaintext `2f` forward → unselected in the tailoring; production TLS offload tracked under #8. - Diagnosis tooling notes: SMBIOS cmdline-extra works only after a Proxmox VM **stop/start** (not `reset`); the debug-shell-on-tty2 never came up — the decisive diagnosis was **offline `rootfs.erofs` mount** + `ssh -vvv` + `enforcing=0`. **REVIEW-ROW TRIAGE — DONE 2026-06-19 (scan `app-9f7d8d1`, 21 REVIEW → 1 open).** The post-fix hardened scan was valid (398 pass / 12 fail / 9 notchecked). All 21 REVIEW rows dispositioned (full detail: [stig-deviations.md](stig-deviations.md) #12). Mechanics: a rule only stops being flagged REVIEW if it's `selected="false"` in `diskless-tailoring.xml` (the register prose alone doesn't clear it) or made to pass / N/A. - **Bucket A (waived, register-justified):** `encrypt_partitions`, `auditd_audispd_configure_sufficiently_large_partition` (#1); `grub2_password` (#2); `sysctl_crypto_fips_enabled`, `harden_sshd_macs_openssh_conf_crypto_policy`, `harden_sshd_macs_opensshserver_conf_crypto_policy` (#5); `configured_firewalld_default_deny` (#4); `sssd_enable_certmap`, `sssd_has_trust_anchor` (#7). - **Bucket B (waived N/A):** `account_temp_expire_date`, `accounts_user_home_paths_only`, `libreswan_approved_tunnels`, `security_patches_up_to_date`, `ssh_keys_passphrase_protected`. - **Bucket C (decisions applied):** `installed_OS_is_vendor_supported` (Rocky IS supported; RHEL-datastream CPE artifact — waived), `network_configure_name_resolution` (single DHCP DNS by design — waived), `sysctl_user_max_user_namespaces_no_remediation` (KEEP userns for Chromium sandbox — waived), `accounts_authorized_local_users` (appliance accounts authorized, #10 — waived; prod: set the authorized-users regex var instead), `package_postfix_installed` + `postfix_prevent_unrestricted_relay` (**postfix removed** in 2c-STIG — no MTA needed; audit goes off-box via 2f), **USBGuard** `configure_usbguard_auditbackend` + `usbguard_allow_hid`/`_hub`/ `_hid_and_hub` (**ENABLED + enforcing**: LinuxAudit backend + hardware-independent HID/hub allowlist written in 2c-STIG; ⚠ BENCH-VERIFY physical console keyboard/ mouse on real hardware — the build-VM-generated policy + default block target would otherwise lock out console input). - **`selinux_context_elevation_for_sudo` — REMEDIATED WITH CONFINEMENT (decision 2026-06-20).** 2c-STIG now rewrites the admin sudoers to `TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL` and maps the admin to **`staff_u`** (`semanage login`) so the sudo role transition is authorized. ⚠ **TWO BENCH GATES:** (a) admin SSH login + sudo + privileged ops must work under `staff_u` confinement (escape hatch: `enable_stig=false` → reverts to a justified deviation); (b) **reproducibility** — `semanage` rebuilds the policy store; confirm the verity hash is stable across two builds (normalize in 2d if it drifts). Needs `policycoreutils-python-utils` (installed in 2c-STIG if absent). **Expected next-build report: 0–1 REVIEW** (0 if both postfix→N/A and usbguard→pass land and sudo confinement holds; the triage was validated by re-running `scripts/stig-deviations.sh` against the app-9f7d8d1 ARF with the new tailoring — 21 REVIEW → 3, where the 3 are exactly postfix/usbguard/sudo, all addressed for the next build). Remaining (future hardening, currently waived — see the register): FIPS, firewall, per-host/rotatable admin access. **Remote audit offload — WIRED both ends** (2026-06-19): receiver `serving/audit-collector/` (rsyslog TCP/514, per-source-host files) + sender in the image (provisioner 2f, `audit_host` var: audisp-syslog→LOCAL6 + rsyslog `omfwd` to `audit.fleet.lab`). Lab is plaintext; TLS/RELP is the remaining hardening. **VERIFIED END-TO-END ON THE BENCH 2026-06-19** — booted host `cygnus-cruiser-99` (192.168.50.148) forwards to the collector, which writes `audit-collector/logs/192.168.50.148/{audit,messages}.log`. **Serving-host firewall gotcha hit + fixed:** the collector is host-networked so the serving host's **ufw** governs :514 (Docker doesn't manage iptables for host-net containers); 514/tcp was bound to the wrong NIC (`ens18` instead of the boot-LAN `ens19`), so SYNs were silently dropped (`SYN-SENT` / "connection timed out", not refused). Fix: `ufw allow in on ens19 to any port 514 proto tcp`. The sender's `action.resumeRetryCount="-1"` buffered the backlog and drained on reconnect — no records lost. Documented in [../serving/audit-collector/README.md](../serving/audit-collector/README.md). When the **workstation** firewall is re-enabled (#4 / register #4) it needs the reciprocal **egress** allow for 514/tcp to the collector. **SIEM (Wazuh) added 2026-06-27 — [serving/siem/](../serving/siem/).** Upgrades the "Detect" pillar from file-collection-only to parse/search/correlate/alert. **Runs on a DEDICATED node — `wazuh1.bitasylum.org`** (managed independently), keeping the memory-hungry indexer JVM **off** the serving host so it can't OOM SPDK (the fleet's root disk / SPOF, #6/#7). **Agentless** (diskless hosts stay stateless, invariant #1): the serving-host audit-collector **relays a copy** to wazuh1's manager syslog listener (`omfwd` → `wazuh1.bitasylum.org:514` in `audit-collector/rsyslog.conf`; rebuild the collector to apply) while still writing durable per-host files. `serving/siem/` is the INTEGRATION (the `` syslog block to add on wazuh1 + reference deploy overlay), not a local deployment. **DECODING BENCH-VERIFIED 2026-06-27** (`wazuh-logtest`): the audisp-syslog records carry a `node=` prefix (STIG `name_format=hostname`, SSG `auditd_name_format` — kept, it's compliance) that breaks Wazuh's built-in auditd decoder, so a custom **`fleet-auditd`** decoder + rules (`serving/siem/config/local_decoder.xml` + `local_rules.xml`, mounted declaratively via `/wazuh-config-mount`) decode them into the **`fleet_audit`** group. **Decode + alert confirmed:** `wazuh-logtest` shows `decoder.name=fleet-auditd` + `audit.type` extracted (base rule `100100`); and a live `sudo` on a booted host (`cygnus-cruiser-99`) surfaced `USER_CMD` → rule `100130` and `CRED_*`/`USER_ACCT` → rule `100110` (level 5) in the dashboard Threat Hunting view. Wazuh runs multi-node on `wazuh1.bitasylum.org`. **REMAINING (prod):** the relay→wazuh1 hop is **plaintext** — **TLS it** (gtls/6514, fleet CA; ISM protect-logs-in-transit) since it's now on the wire; rotate all Wazuh creds; restrict dashboard/API to admin hosts; set `allowed-ips`→/32 of the serving host; ISM retention/archival; then update the SSP "Detect" + ISM event-logging controls. ### Original spec (retained for reference) OpenSCAP + SCAP Security Guide (`xccdf_org.ssgproject.content_profile_stig`), run as a Packer provisioner after `dnf install` and before the dracut/initramfs build and cleanup. - **Tailoring** (`autotailor`): unselect rules this architecture can't satisfy — separate-partition rules (single erofs root + tmpfs overlay; satisfy intent with `nodev,nosuid,noexec` tmpfs mounts), GRUB/kernel-cmdline rules (no GRUB), and AIDE/file-integrity rules (dm-verity is the documented compensating control). - **Kernel params → UKI cmdline** in [../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh) before signing: `audit=1 audit_backlog_limit=8192 slub_debug=P page_poison=1 vsyscall=none` (+ `fips=1` if FIPS). - **Reproducibility guard:** strip non-deterministic remediation output (SSH host keys → regen at boot into overlay, AIDE db, timestamps, fresh UUIDs) so the verity hash stays stable. - **Audit logging:** `audisp-remote`/rsyslog to a central collector addressed by DNS name (host-generic). - **Optional FIPS:** `fips=1` cmdline + `01fips` dracut module + FIPS crypto policy. - **CI:** run `oscap eval` (no `--remediate`) to publish an ARF+HTML STIG report as a build artifact; optionally hard-fail below a score threshold. - **Decisions needed before implementing:** FIPS on/off · remote-audit collector DNS name (or ephemeral/none for lab) · CI fail-on-score threshold vs report-only. - Note: orthogonal to the `rd.dot1x=0` test toggle, but a full STIG production posture wants 802.1X re-enabled. - **Credentials (partly done):** root is now **locked** and access is via a baked sudo-enabled key-only admin user (`var.ssh_authorized_key` / `var.admin_user`, `wheel` + NOPASSWD sudo), with sshd drop-in `PermitRootLogin no` / `PasswordAuthentication no`. Two items for STIG to reconcile: **NOPASSWD sudo** is a finding (acceptable for a key-only account, but document/deviate), and the admin key/user are **fleet-wide baked** — for per-host/rotatable access without rebuilds, layer the signed HTTP/SMBIOS config-agent or an SSH CA (`TrustedUserCAKeys`) discussed in session. - **SSH host keys (done):** `extract-artifacts.sh` strips `/etc/ssh/ssh_host_*` so they're not baked (baked keys = shared fleet identity + the build's perms/group don't survive the tar→erofs round-trip, so sshd rejected them "Permissions 640 too open" and refused to start). RHEL's `sshd-keygen@.service` regenerates them into the overlay before sshd starts. **⚠ Caveat:** the ephemeral overlay means they regenerate **every boot** → the host key changes per reboot (clients warn). Follow-up for a stable host identity: persist the host keys or derive them from the TPM (ties to the hostnqn/identity model). ## [ ] 3. Roll the errata into design doc v3 Once the boot path is validated and item 1 lands, fold [design-v2-errata.md](design-v2-errata.md) into the source design document and re-export a **v3** PDF (key change: `rootovl` → `systemd.volatile=overlay` in §3 step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source (not in this repo). ## [x] 4. Kiosk mode — autologin + fullscreen Chromium loading a URL **WORKING (display-verified 2026-06-17).** Remaining bits are polish (GPU/mesa trim, ship via CI `enable_kiosk=true`) and the phone-home/NTP lockdown (done) — all captured in the sub-notes below. Secure-path render check is folded into #8. **WORKING / display-verified 2026-06-17** (kiosk renders Chromium fullscreen on the configured URL). Build-flagged OFF by default for now (`enable_kiosk=false`) while iterating on boot/network — set `-var enable_kiosk=true` to ship it. Path to getting here (don't re-discover): - **cage is NOT packaged for EL9** (`dnf install cage` → "Unable to find a match"). Tried weston instead. - **EPEL 9 ships weston 8.0**, which has NEITHER `kiosk-shell.so` NOR the `[autolaunch]` weston.ini section — both arrived in **weston 9.0**. The first cut wrongly assumed 8.0 had them; `shell=kiosk-shell.so` failed to load. `rpm -ql weston` shows only desktop-shell / fullscreen-shell / ivi-shell. - **Resolution: `desktop-shell.so`** (the only packaged shell Chromium's xdg-shell works with — fullscreen-shell needs a client protocol Chromium doesn't speak), stripped bare (`panel-position=none`, black bg, no animation/lock), and Chromium launched by a **wrapper** (kiosk-session) since there's no `[autolaunch]`. Chromium `--kiosk` fullscreens over the shell. - **Seat fix (load-bearing):** weston died with `logind: failed to get session seat` → `failed to create compositor backend`. logind's auto-derivation didn't attach a seat; `Environment=XDG_SEAT=seat0` + `XDG_VTNR=1` in the unit (put into the PAM env before pam_open_session) registers the session on seat0/vt1. `/dev/dri/card0` was present all along (not the issue). - Tested live without a rebuild: the whole root is `systemd.volatile=overlay`, so /etc AND /usr are writable tmpfs at runtime — drop-in the files + restart. - **`enable_kiosk` Packer var — currently `false` by DEFAULT (set 2026-06-17).** The whole kiosk provisioner is feature-flagged off while iterating on the boot/network stack (DHCP-lease/keep-alive work), for faster builds and a smaller image. **Re-enable with `-var enable_kiosk=true` when returning to kiosk work** — CI does not pass the var, so CI currently builds lean (no kiosk) too; add `-var enable_kiosk=true` to the Gitea workflow to ship it. - `kiosk_url` Packer var (default `http://boot.fleet.lab/kiosk/` — the kiosk **SPA** (`serving/kiosk-app/`, Svelte 5 + Tailwind v4, a cyber sci-fi spaceship monitoring/control terminal) served by the boot HTTP server, which **reverse-proxies** `/kiosk/` to the `kiosk` container; self-contained so a boot validates without external connectivity. Empty disables the whole install). Wired in CI as `KIOSK_URL` env. - **Kiosk SPA (`serving/kiosk-app/`)** — replaced the old single static `serving/http/kiosk/index.html` (2026-06-18). Top navbar + 5 pages (Bridge/Reactor/Nav/Life Support/Comms) with dummy telemetry; hash routing; builds in a 2-stage Docker image (Vite → nginx), reverse-proxied at `/kiosk/` by the main serving nginx (loopback `127.0.0.1:${KIOSK_APP_PORT:-8088}`). `npm run build` verified locally; container build untested (no docker here). Dummy data lives in `src/lib/data.js` (swap for a fleet API later). - Provisioner enables EPEL, installs `weston chromium mesa-dri-drivers mesa-libEGL mesa-libgbm dejavu-sans-fonts`, creates a passwordless unprivileged `kiosk` user (video/input groups). - `files/kiosk/kiosk.service` — runs `/usr/local/bin/kiosk-session` as `kiosk` on tty1 via `PAMName=login`+`TTYPath`+`Environment=XDG_SEAT=seat0`/`XDG_VTNR=1` (gets seat0 for DRM/input), `Conflicts=getty@tty1`, `WantedBy=graphical.target`; default target set to graphical. - `files/kiosk/weston.ini` — `shell=desktop-shell.so`, `idle-time=0`, `require-input=false`, `xwayland=false`; `[shell] panel-position=none`, black `background-color`, `animation=none`, `locking=false`. - `files/kiosk/kiosk-session` — `exec`s `weston --use-pixman --socket=wayland-kiosk` in the foreground (owns the seat), and launches Chromium (`--kiosk --ozone-platform=wayland --disable-gpu`, tmpfs profile/cache, URL from `/etc/kiosk/kiosk.env`) in a background crash-restart loop. - **Remaining polish (now that it renders):** dropping `--disable-gpu` on real GPUs, `/dev/dri` present in the VM (weston needs a DRM device — set Display to VirtIO-GPU/std), and **SELinux AVCs** for the GUI stack (check `ausearch -m avc`; the kiosk user is unconfined so likely none — but do NOT set permissive if they appear; add a policy module). Admin SSH (the key user) still works for debug even if the kiosk fails (it lands at graphical.target either way). - **Phone-home lockdown (done 2026-06-17/18):** Chromium was dialing Google (accounts/variations/optimization-guide/component+auto update/safe browsing — seen in dnsmasq logs). Two layers: - **Flags** in `kiosk-session` (`--disable-background-networking`, `--disable-component-update`, `--disable-features=OptimizationHints,...`) — confirmed killing variations/optimization-guide/update queries. - **Managed policy** `files/kiosk/chromium-kiosk-policy.json` for the policy-governed bits (sign-in `accounts.google.com`, sync). FIX 2026-06-18: the managed-policy dir differs by build, so it's installed to BOTH `/etc/chromium/policies/managed/` AND `/etc/chromium-browser/policies/managed/` (a policy in the wrong dir is silently ignored — the reason `accounts.google.com`/`android.clients.google.com` persisted). Verify via `chrome://policy` (Status: OK per entry). - **Serving-side sinkhole** (defense-in-depth, `serving/docker-compose.yml` dnsmasq `--address=/google.com|googleapis.com|gstatic.com/0.0.0.0`): the kiosk is local-only, so block egress at the resolver. Stops the *leak*; the query still logs (answered 0.0.0.0, not forwarded) — eliminating the query is the browser policy's job. Production: do this at the boot-LAN firewall. - **NTP egress closed (done 2026-06-18):** the `*.pool.ntp.org` lookups were the OS (chrony), not Chromium. Now: a `chrony` NTP server in the serving stack (`serving/chrony/`), dnsmasq hands it via **DHCP option 42**, and the image's NM dispatcher (`files/etc/NetworkManager/dispatcher.d/10-chrony-dhcp-ntp`) feeds it into chrony's `sourcedir /run/chrony-dhcp`; the baked public pool is stripped from `/etc/chrony.conf`. Result: the workstation syncs time only from the lab server — no external NTP egress. BENCH-VERIFY: on a booted host `chronyc sources` shows the lab server (the DHCP one), not a public pool. - **Image size:** Chromium+Mesa add hundreds of MB to the *uncompressed* erofs — feeds the "shrink the rootfs" concern (and #7); reconsider whether a lighter browser/render path is viable. ### Original task Make the diskless workstation boot straight into a locked-down kiosk: a fullscreen Chromium showing a configured site (default: the boot HTTP server's `/kiosk/` test page). All baked at **build time** into the root tree (immutable erofs/verity root; runtime is the tmpfs overlay), fleet-generic (no per-host state). - **Graphical stack (keep it minimal):** add a Packer provisioner installing a lightweight kiosk compositor + browser. Preferred: `cage` (Wayland single-app kiosk compositor) + Mesa (software GL fallback for VMs/no-GPU); X11 + a minimal WM (matchbox/openbox) as the alternative. Chromium on Rocky 9 is in **EPEL** — enabling EPEL adds a repo/supply-chain surface (decision below). - **Autologin + session:** dedicated unprivileged `kiosk` user; launch the session via a `kiosk.service` running `cage -s -- chromium --kiosk …` (or `greetd` + cage). Stage units/configs under `files/` like the other provisioned config. - **Chromium flags:** `--kiosk --noerrdialogs --disable-infobars --incognito --no-first-run --disable-session-crashed-bubble `. Writable profile/cache land on the `systemd.volatile=overlay` tmpfs (RAM-backed, ephemeral) — fine, but watch RAM sizing; pin `--user-data-dir`/`--disk-cache-dir` to a tmpfs path if needed. - **Kiosk URL = Packer var** (e.g. `kiosk_url`, default the target site), baked into the launch command; keep it host-generic (per-env via DNS if ever needed). - **Verify in boot test:** graphical session comes up on real/VM display, Chromium fullscreen loads the URL over the booted network (works with the current `rd.dot1x=0` test path / open network). - **Decisions:** Wayland(cage) vs X11; accept EPEL for Chromium (vs another source); confirm the target URL. - **⚠ Conflict with item 2 (STIG):** passwordless autologin and a desktop browser are STIG findings and enlarge the attack surface/image. Reconcile — e.g. a tightly-constrained kiosk session + documented deviation — when hardening lands. ## [x] 5. Deploy & validate the end-to-end boot path (open-network test) ✅ **DONE (2026-06-15).** Full chain validated on QEMU/Proxmox: UEFI HTTP boot → initramfs → DHCP → `nvmf-connect.service` auto-connects the NVMe/TCP subsystem → two namespaces (data + verity hash) appear by pinned UUID → `systemd-veritysetup` maps `/dev/mapper/root` → uncompressed erofs mounts → `systemd.volatile=overlay` → multi-user. Booted with `rd.dot1x=0` (open network) and the IP discovery address. See bench findings below for the full list of fixes. Remaining cleanup to leave the lab-test posture (now unblocked): revert the hardcoded IP to the `nvme.fleet.lab` DNS name, and re-enable security (`DISABLE_DOT1X=false`) — the secure path needs the same connect trigger over macsec0 (see ⚠ note below). Original task: stand up `serving/` (dnsmasq + nginx + SPDK) per [serving/README.md](../serving/README.md), rebuild the UKI with `nvmet_traddr=nvme.fleet.lab` (now a Packer var — older UKIs have the literal `NVMET_TRADDR` placeholder and won't connect), then boot a UEFI HTTP-boot client (**Secure Boot OFF**, unsigned UKI; no TPM needed in `rd.dot1x=0` mode) on the boot LAN. - **Pass criteria:** UKI fetched over HTTP → initramfs → DHCP on NIC → NVMe/TCP namespace connected → dm-verity root mapped → `systemd.volatile=overlay` writable → multi-user. - This validation **gates** item #1 (network-manager port) and item #3 (errata → design v3). - **Bench findings (boot debug, 2026-06-14):** - Initramfs has no resolver (dracut writes `resolv.conf` only via its own DHCP path, not a manual `dhclient`), so the NVMe discovery address must be an **IP**, not a DNS name — build with `-var nvmet_traddr=192.168.50.10`. Old UKIs carrying `nvme.fleet.lab` can't connect early in boot. - NVMe controller indices are **not stable**: a transient discovery controller can take `nvme0`, pushing the I/O controller to `nvme1`. The hardcoded `/dev/nvme0n1` in the cmdline was a latent brick. **Fixed:** namespace UUID pinned on the target (`spdk-setup.sh` `add_ns -u f1ee7000-0000-4000-8000-000000000001`) and the cmdline now references `/dev/disk/by-id/nvme-uuid.` (`extract-artifacts.sh`). The two constants MUST stay in sync; make them version-specific for #6. - `nvme_tcp`/`nvme_fabrics` must be loaded before `/dev/nvme-fabrics` exists (dracut's `nvmf` module does this in a normal boot; at `rd.break` modprobe manually). - **Don't double-configure the NIC in test mode.** With `ip=dhcp` on the cmdline, dracut's network module brings the NIC up AND emits the "online" event that the `nvmf` module's `initqueue/online` hook waits on to run `nvme connect-all`. The `rd.dot1x=0` bypass used to *also* `ifup $NIC`, which (a) raced dracut → `Error: ipv4: Address already assigned`, and (b) didn't fire the nvmf online hook, so the namespace never auto-connected and boot spun on `/dev/mapper/root` forever (manual `connect-all` worked, which is what masked it). **Fixed:** the bypass branch no longer touches the NIC; `ip=dhcp` owns networking. (The secure path still addresses macsec0 itself — dracut doesn't know about that interface.) - **dracut's `nvmf` module never auto-connects on this distro.** Its cmdline hook (`92-parse-nvmf-boot-connections.sh`) parses `rd.nvmf.discover` into `/etc/nvme/discovery.conf` correctly, but `initqueue/online/` is **empty** — no autoconnect hook is registered, so `nvme connect-all` is never run (no attempt in dmesg) and the namespace never appears. Drivers load and `/dev/nvme-fabrics` exists; only the connect trigger is missing. A job placed on the *main* initqueue from a settled hook gets registered but is **never re-executed** once the loop blocks on udev events (verified: job present, manual `connect-all` works, namespace stays absent). An `initqueue/online` hook also does NOT work: this initramfs is **systemd- driven** ("A start job is running for /dev/mapper/root") and never sources dracut online hooks on link-up (verified: hook installs as `90-nvmf-connect.sh` but never fires — no log, no stamp). **Fixed:** the connect is now a **systemd unit** (`nvmf-connect.service`, `ConditionKernelCommandLine=rd.dot1x=0`, `Before=initrd-root-device.target`) whose ExecStart (`/usr/sbin/nvmf-connect`) polls for the network then runs `nvme connect-all` until the namespace appears. The dracut module installs the unit + script and enables it via an `initrd-root-device.target.wants` symlink; `diskless.pkr.hcl` hard-asserts the unit is in the initramfs. Gotcha: `$systemdsystemunitdir` is **empty** in this dracut's module-setup context — `inst_simple` then silently drops the unit (empty dest dir). Use a `${systemdsystemunitdir:-/usr/lib/systemd/system}` fallback and `return 1` if the unit didn't land, so the build fails loudly instead of shipping a UKI that can't connect its root. Second gotcha: the build assertion `lsinitrd … | grep -q 'nvmf-connect.service'` was a **false negative** — `grep -q` exits on first match, closes the pipe, `lsinitrd` dies of SIGPIPE, and under `pipefail` the pipeline reports failure even though the unit IS present. This masked the fact that the systemd unit install was working all along. Fix: `lsinitrd > file; grep -q … file` (no pipe). Never `lsinitrd | grep -q` under pipefail. Bonus: the survivor listing confirmed nvme-cli's native autoconnect bits ARE in the initramfs (`/usr/sbin/nvmf-autoconnect.sh`, `/etc/udev/rules.d/95-nvmf-initqueue.rules`) — so the fully-native `rd.nvmf.discover` autoconnect path is available as a fallback if the unit doesn't fire on boot. - **dm-verity: `hash-offset` is unsupported → two namespaces.** With NVMe connecting and the right by-id device present, verity still failed: `systemd-veritysetup` logged `unknown option 'hash-offset=…'` then `Failed to load verity superblock` — exactly the CLAUDE.md bench-verify item. The in-file (data+hash in one namespace, split by offset) layout is not supported by Rocky 9.8 systemd. **Fixed** via the documented fallback: `extract-artifacts.sh` now emits `rootfs.erofs` (data) + `rootfs.hash` (hash tree) as separate files, served as **two NVMe namespaces** with distinct pinned UUIDs (`f1ee7000-…-01` data, `…-02` hash); the cmdline uses `systemd.verity_root_data=`/`systemd.verity_root_hash=` by-id devices with no offset. Touches `extract-artifacts.sh`, `build-oci.sh` (artifact list + SHA256SUMS), `serving/spdk-setup.sh` (2nd bdev/namespace), and invariant #9. - **SELinux labels must survive into the erofs.** It booted to multi-user but `root`/`packer-build-only` login was accepted then the session was killed instantly. Cause: kickstart sets `selinux --enforcing`, but the build dropped `security.selinux` xattrs (`virt-tar-out` + plain `tar -xpf` don't carry them), so the enforcing target denied the login session on an unlabeled root — and a read-only erofs can't autorelabel. Confirmed by booting with `enforcing=0` (SMBIOS-injected) → login worked. **Fixed:** export via `guestfish tar-out … selinux:true xattrs:true acls:true`, restore with `tar --xattrs --xattrs-include='*' --acls --selinux`, and a `getfattr` build-time assertion that `/usr/bin/bash` is labeled before `mkfs.erofs` (mkfs.erofs preserves xattrs). Needs `attr` on the build host. The image now boots **enforcing**. - **`mkfs.erofs --all-root` was flattening all ownership to root:root.** Mode bits + (now) xattrs survive the export, but `--all-root` forced every inode to uid/gid 0 — discarding group ownership (`ssh_keys` on host keys → sshd "permissions 640 too open" → refused; `tty`/`crontab` setgid groups) and user ownership (home dirs). **Fixed:** dropped `--all-root` so real ownership is preserved from the tree. That now requires `extract-artifacts.sh` to run as **root** (added an `id -u` guard; `tar -x` needs CAP_CHOWN to restore ownership) — the post-processor runs it via `sudo` and `chown`s the output files back to the build user so `build-oci.sh` (rootless buildah + its own login) still works. Combined with the host-key strip above, sshd starts cleanly. **⚠ The secure path has the same dependency on the (absent) nvmf autoconnect** — when 802.1X is re-enabled, the macsec0 path must likewise drive `connect-all` (or the network-manager port #1 must wire a working autoconnect). Currently only the test path is fixed. - The target Rocky 9 kernel's erofs is built **without `CONFIG_EROFS_FS_ZIP`** — mounting a compressed image fails `erofs: lz4 algorithm isn't enabled`, which silently blocks the root mount. **Fixed:** `extract-artifacts.sh` now builds the erofs **uncompressed** (`mkfs.erofs` without `-z`). Tradeoff: larger rootfs namespace + more NVMe/TCP bandwidth per read. Re-enable compression only against a kernel verified to have `EROFS_FS_ZIP` (`grep CONFIG_EROFS /boot/config-$(uname -r)`). This is the CLAUDE.md bench-verify item ("erofs module availability on the target RHEL kernel"). Larger image size also feeds the "shrink the rootfs" angle of #7. - SPDK: add the listener to the discovery subsystem explicitly (`nqn.2014-08.org.nvmexpress.discovery`) to silence the deprecation warning. ## [ ] 6. Rolling fleet updates (multi-version serve → cohort reboots) Image-based, reboot-to-update across many live workstations. Hosts are immutable (read-only erofs/verity root over NVMe/TCP, ephemeral overlay), so updates are **atomic A/B reboots**, never in-place. The released OCI artifact + digest-pinned manifest + roothash coupling already give the atomic unit and instant rollback; this item builds the rollout machinery around it. - **Serve versions side-by-side** (must not overwrite a live rootfs): extend `serving/` so SPDK/nvmet exposes each release as its **own subsystem/NQN** (`nqn…:rootfs-`) and nginx serves multiple `fleet-.efi` (keep current + previous). Make the UKI's NQN version-specific so each signed UKI connects only its matching namespace (constant discovery DNS name). - **Version selection at boot**: dnsmasq **cohort tagging** (MAC / vendor class / assigned tag) hands each cohort a different option-67 UKI URL. Moving a cohort's URL = assigning the release. - **Control plane + in-image reboot agent**: a baked-in systemd timer polls a "desired version" endpoint and reboots **when idle / in a maintenance window** (or out-of-band via Redfish/BMC). Stagger + jitter reboots to avoid UKI-fetch / NVMe-connect storms. - **Staged rollout**: canary → waves (10/50/100%) with health gates (multi-user reached, NVMe connected, dm-verity mapped, kiosk URL rendered, prod: MACsec/802.1X up); auto-halt on threshold. - **Rollback**: repoint cohort URL to previous UKI + reboot (current+previous stay served); security rollback via dbx-revoke / TPM monotonic counter (design §6). - **Drain & retire** an old version only when SPDK/nvmet reports zero connected controllers on it. **Drain accuracy:** account for hosts powered off mid-rollout — an assigned-but-offline host will reconnect to a version that may be gone, so don't remove a version while any host that could return is still assigned to it. - **Availability**: run NVMe/TCP target replicas behind distinct discovery entries (design §7) — never serve the whole fleet from one unreplicated node, especially mid-rollout (N versions + reboot bursts). - Note: kiosk *content* changes are server-side (the website) and need no fleet update — only OS/UKI/rootfs changes require a reboot. ### SPDK restart / availability is the load-bearing constraint (analysis 2026-06-25) SPDK sits in the **live data path of every running host's root fs**, so a process restart isn't like restarting a stateless service — it tears down **every** NVMe/TCP association at once (fleet-wide, NOT version-scoped). The rollout design must be built around **never restarting the process that holds live connections.** - **Core principle:** a host changes root version *only by rebooting* (no live swap). So: **version transitions = reboots; SPDK changes = additive/online; the process stays up.** Routine rollouts never restart SPDK. - **Online add/remove via JSON-RPC, no restart.** `nvmf_tgt` is fully runtime-configurable (`bdev_aio_create`, `nvmf_create_subsystem`, `nvmf_subsystem_add_ns`, `..._add_listener`). Add vY's namespaces live (existing vX connections untouched); remove vX live once drained. **Per-version subsystem (distinct NQN)** matters here: adding vY as a *new subsystem* never disturbs the subsystem vX hosts are attached to — adding a *namespace* to a shared subsystem fires a namespace-change AER + rescan at running hosts. Per-version subsystem also gives clean per-release drain + per-release hostNQN allow-listing. - **What a connection loss does to a running host:** root is **read-only erofs + tmpfs overlay**. tmpfs writes (RAM) + page-cached reads keep working; **cold reads block**. The nvme-tcp host driver reconnects (`reconnect_delay`) until `ctrl_loss_tmo`. **Fast SPDK restart** (same NQN/UUID/addr within the window) → host **survives with a stall**; **long/changed/expired** → root I/O errors, host needs a reboot ("keep alive timeout / no route to host"). Read-only + content-addressed makes reconnect unusually clean (no write/reservation state). - **Cheap host-side mitigation:** tune `ctrl_loss_tmo` (even `-1` = retry forever) + `reconnect_delay` at `nvme connect` time (dracut nvmf step). For a diskless box "hang and recover" beats "die" — there's nothing better to fail to. - **Structural answer — HA + NVMe-oF multipath (ANA):** run ≥2 targets serving **byte-identical, same-UUID** namespaces; hosts multipath across them, so one target can be restarted/upgraded while the other serves (no root loss). The **read-only, content-addressed root makes replication trivially coherent** (no write-coherence problem) — this architecture is ideally suited to it. Wrinkle: the **immutable UKI cmdline** points at one discovery address, so multipath needs discovery to return multiple paths (or a DNS name resolving to several targets). - **Planned SPDK restarts only via drain/failover** (multipath fail-over, or a maintenance window with generous `ctrl_loss_tmo` so hosts recover not die). There is **no zero-impact single-node SPDK restart.** - **[x] Long-lived target + online release control — IMPLEMENTED 2026-06-27.** The old gap (SPDK ran `spdk-setup.sh` as entrypoint, single-version, configured-at-start → changing versions = container restart = drop ALL connections) is closed. Now in `serving/spdk/`: - **`spdk-target.sh`** — persistent entrypoint: starts `nvmf_tgt` ONCE (release-agnostic), creates the TCP transport + discovery listener, then reconciles every staged release. Never restarted to change versions; on (re)start re-adds everything under `./releases/` (a serving-host reboot restores all served versions). - **`spdk-release.sh`** — JSON-RPC control: `add`/`remove`/`list`/ `drain-status`/`reconcile`, **one subsystem per version** (distinct NQN, so adding/removing one never disturbs others' connections), **drain-gated removal** (refuses while a version has connected controllers unless `--force`). - **`spdk-ctl.sh`** — host ops: `load`/`unload`/`list`; stages a release's artifacts from its OCI image into `./releases//` and drives the live target. Rolling update = `load` new → reboot hosts → `unload` old once drained. - **`spdk-loader.sh`** — DECLARATIVE rolling updates (the "init container that stays up", compose service `spdk-loader`). On `compose up` after an `IMAGE_VERSION` bump it stages + adds the new rootfs to the running target ONLINE, then polls and **auto-reaps** any non-current version once it has drained (0 conns) for a sustained grace window (`REAP_GRACE_SECONDS`, keep > host `ctrl_loss_tmo`). So migration = bump `IMAGE_VERSION` in `.env` + `docker compose up -d` (compose recreates the loader + the http/dnsmasq UKI servers, but NOT the nvmeof target → no dropped connections). Single served UKI ⇒ a 0-conn old version is genuinely orphaned (a returning host fetches the current UKI), so auto-reap is safe; removal is still drain-gated + grace-debounced. The loader + target share the RPC socket on a named volume (`nvmf_tgt -r`); the loader reuses the SPDK image (has python3 + rpc.py). - **No jq dependency:** `spdk-release.sh` parses RPC JSON with **python3** (the SPDK image ships no jq); legacy artifacts (no per-version UUIDs in verity.json) fall back to the fixed UUID pair with a warning. - **Per-version namespace UUIDs** (`scripts/extract-artifacts.sh`): derived deterministically (UUIDv5 of `image_version`), baked into the UKI cmdline + `verity.json` (now also carries `subnqn`); the serving side reads them back from each release's `verity.json`. Required so two versions don't collide on the same `/dev/disk/by-id/nvme-uuid.` host symlink. - **Bootstrap unchanged:** `docker compose up` with one version still works — the `/artifacts` image volume is seeded into `./releases/${IMAGE_VERSION}`. - **PARTIALLY BENCH-VERIFIED 2026-06-28:** long-lived target boots a host end to end — UKI served over HTTP → host boots → NVMe/TCP root connects to the persistent `nvmf_tgt` (single version `spdk-permanence-02ffd7d`). Surfaced + fixed several real bugs along the way: bootstrap seed must COPY not symlink (a symlink into the version-pinned `/artifacts` re-pointed a stale release at new data); `spdk-release` parses RPC JSON via **python3** (SPDK image has no jq); legacy `verity.json` (no per-version UUIDs) falls back to the fixed pair; mislabeled-dir guard (dir name vs verity.json `image_version`, case-insensitive); dropped the redundant discovery-listener re-add (daemon logged "Listener already exists"). **CI lowercases `image_version`** (gitea-main-ci vars step) — a mixed-case branch (`SPDK-permanence`) had baked a capital `fleet-SPDK-…​.efi` the lowercase boot URL couldn't fetch (iPXE "no such file"); booted via a local retag workaround pending a lowercase rebuild. - **STILL BENCH-VERIFY:** (a) `load` a 2nd version while a host is connected to the 1st → 1st's connection survives (no AER/reset on the running host); (b) `unload`/auto-reap drain-gate blocks while CONNS>0 and reaps after grace on a LIVE host; (c) serving-host reboot reconciles all staged versions; (d) per-version UUID build reproducible (two builds, same version → identical `data_ns_uuid`/`hash_ns_uuid` + `root_hash`); (e) the full fix on a clean LOWERCASE CI rebuild (no retag). See `serving/spdk/README.md`. - **Capacity:** each concurrently-served version costs target memory/hugepages + file-backed bdev overhead — plan for N live versions (current + previous + canary) during long rollouts (raise `HUGEPAGES`). - **Attach gating + discovery:** with several subsystems advertised, the dracut connect + per-subsystem hostNQN allow-list must ensure a vX host attaches **only** vX's namespace (the by-UUID verity ref protects *what mounts*, but gate the *attach* too); `rd.nvmf.discover` selection must be deterministic, not connect-all. ## [ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience) There is **no production-grade memory-safe NVMe-oF target** — the Rust NVMe work is all driver/initiator-side (vroom, RedLeaf, Rust-for-Linux); the only production targets are C (Linux kernel `nvmet`, SPDK). So this is **serving-node hardening, not a language swap**. dm-verity already removes the target from the clients' integrity TCB (verify-on-read rejects tampered blocks), so the risk is RCE/DoS on the node that holds the fleet's disk, not client integrity. The export is read-only + single-namespace = inherently small attack surface. - **Evaluate kernel `nvmet` vs SPDK** for the target: kernel `nvmet` keeps the data path in the hardened/fuzzed kernel with a tiny userspace footprint (arguably a better attack-surface story); SPDK is a userspace C daemon (current choice, needs hugepages + caps). Decide per environment. - **Confine the target** (whichever): unprivileged, drop caps, seccomp + SELinux/AppArmor profile, read-only container, minimal mounts; minimal immutable serving image, digest-pinned. - **Slim the SPDK image (multi-stage build)**: the current single-stage from-source image is ~600 MB (it keeps the whole toolchain — gcc, meson/ninja, kernel headers, the python venv — plus the SPDK/DPDK source and build trees). Convert `serving/spdk/Dockerfile` to a builder stage + a minimal runtime stage that copies only `nvmf_tgt` + its `ldd` shared libs (libnuma, libssl/crypto, libuuid, libaio…) + `rpc.py`/`python3`. Expect ~150–250 MB (debian-slim base or a static SPDK link trims further). Smaller attack surface + faster pulls, and it pairs with the confinement above. - **Network-isolate :4420** so only the boot VLAN reaches the target (compose with 802.1X/MACsec segmentation in prod). - **Replicas** behind distinct discovery entries (availability is the real risk — it's the fleet's disk). The target is a **fleet-wide SPOF in every running host's root data path**; a process restart drops *all* NVMe/TCP associations at once. Structural fix = **HA + NVMe-oF multipath (ANA)** over byte-identical, same-UUID, read-only namespaces (replication is coherent precisely because the root is read-only + content-addressed); cheap mitigation = generous host `ctrl_loss_tmo`/`reconnect_delay`. See the **"SPDK restart / availability"** analysis under #6 (rolling updates) — same constraint, fuller treatment. - Revisit a memory-safe target only if/when an off-the-shelf one matures (e.g. a future Rust-for-Linux `nvmet`). ## [ ] 8. Return to secure / production posture (un-toggle the lab flags) Consolidates the cleanup of the temporary lab toggles (see "Current build posture" at the top) and the secure-path bench-verifies deferred while testing the open-network path. Do these together, then run the full secure boot-chain smoke test. - **Re-enable 802.1X/MACsec:** `disable_dot1x=false` (CI `DISABLE_DOT1X="false"`). Removes `rd.dot1x=0`/`ip=dhcp`; the hook owns the link (macsec0). - **Bench harness scaffolded 2026-06-22: [serving/dot1x-bench/](../serving/dot1x-bench/).** step-ca (fleet CA, on the serving host) signs each host's **TPM-generated CSR** + issues the RADIUS server cert; FreeRADIUS does EAP-TLS (trusts step-ca root, returns MSK); a `hostapd` wired authenticator does 802.1X + MACsec on an **isolated bridge per workstation** (flat VLAN can't auth — EAPOL dropped + MACsec is P2P). `enrol/tpm-provision.sh` creates the key in the TPM at `0x81010001` + writes the cert to NV `0x01800001`. Starting solution **signs a CSR** (key stays in TPM); **TPM attestation** (ACME `device-attest-01`) is the documented upgrade on the same CA. **BENCH-VERIFY** items flagged in the files: hostapd MACsec build, openssl tpm2 provider vs pkcs11 key ref, EAP-TLS version. PKI product decision (greenfield): start step-ca; OpenBao/EJBCA/Dogtag later. - **First step (DONE 2026-06-22): `enrol/tpm-selftest.sh`** — non-destructive TPM capability check (run on the booted open-port image, no `disable_dot1x` needed) to de-risk the TPM half before the full path. Likely first finding: no openssl TPM2 **provider** in the image (`tpm2-tss` ≠ the provider) — add `tpm2-openssl`. - **Enrolment of the per-host identity** (key@`0x81010001` + cert@NV) is its own design — see **#11** (TOFU + OOB fingerprint broker, pre-auth permit-set, provisioning UKI). `tpm-provision.sh` is the manual bench stand-in for it. - **NVMe discovery by DNS:** `nvmet_traddr=nvme.fleet.lab` (CI), resolved at boot by the serving dnsmasq — drop the hardcoded `192.168.50.10`. - **Signing — SIGNED-UKI + SECURE-BOOT-ON PATH BENCH-VERIFIED on the Proxmox VM (HTTP boot, 2026-06-20).** CI signs the UKI when `SB_KEY`/`SB_CERT` secrets are set (`db` key); `fleet-sb-enroll` enrols db→KEK→PK from the baked public `.auth` files; OVMF then boots the signed UKI under enforcing Secure Boot. Confirmed end-to-end: CI-signed UKI verifies (`sbverify`), enrolment succeeds, VM boots with `mokutil --sb-state` = enabled. **Key generation + enrolment procedure: [secure-boot-keys.md](secure-boot-keys.md)** (the PK/KEK/db trio; `db` = `sb_key`/`sb_cert`, PK/KEK stay offline). - **Bug fixed during bring-up (2026-06-20):** `fleet-sb-enroll` built its 4-byte EFI attribute prefix in a bash variable (`ATTR=$'\x27\x00\x00\x00'`). Bash strings can't hold NULs → only `0x27` survived → efivarfs read a corrupt attribute u32 → every write failed with `EINVAL` ("dd: Invalid argument"). Now emitted directly with `printf` at write time. **Baked fix VERIFIED on the 820 (2026-06-22): `sudo fleet-sb-enroll` enrolled db→KEK→PK cleanly, no inline workaround.** (Red herring on the way: Proxmox EFI disks default to `pre-enrolled-keys=1`, so `mokutil --db` shows the MS cert even in Setup Mode — looked like a non-empty-db rejection but was the NUL bug all along; it failed identically on a clean `pre-enrolled-keys=0` disk.) - **iPXE signing WIRED (2026-06-21):** `snponly.efi` is now built + signed (`db`) + pushed by CI as a separate FROM-scratch **`fleet-ipxe`** image (`scripts/build-ipxe-oci.sh`, `.gitea/workflows/ipxe-image.yaml`), mounted into dnsmasq at `/ipxe`. dnsmasq itself moved to a generic CI-built/pulled image (`.gitea/workflows/dnsmasq-image.yaml`) — no longer built at deploy time, and carries no keys/binary. (Decision: separate `fleet-ipxe` image, not baked into dnsmasq, so the daemon image stays key-agnostic across SB environments.) - **820 / PXE signed chain — BENCH-VERIFIED 2026-06-22.** Real hardware (HP EliteBook 820 G4), Secure Boot **enforcing** (`mokutil --sb-state` = enabled, Setup Mode no, db/KEK/PK enrolled): firmware verifies the signed `snponly.efi` (fleet-ipxe image) → signed UKI and boots. Confirms the whole `fleet-ipxe` signing pipeline + the PXE→iPXE→UKI chain under SB on real silicon (the VM only covered the HTTP path, which has no iPXE stage). - **REMAINING for production:** (a) **secure path** — re-verify signed boot with `disable_dot1x=false` (802.1X/MACsec), not just the test path; (b) confirm signing is reproducible (verity hash stable across two signed builds). - **PXE-only iPXE fallback — BENCH-VERIFIED on the EliteBook 820 G4 (2026-06-20).** Firmware without UEFI HTTP boot chainloads `serving/tftp/snponly.efi` → renders `autoexec.ipxe` (ifopen + DHCP-retry + `imgargs `) → UKI over HTTP → NVMe/verity root. dnsmasq tags `pxe`/`ipxe`/`httpboot` give graceful per-client fallback; HTTP-boot hosts are unaffected (firmware → UKI direct). Gotchas that cost real time (all captured in [../serving/tftp/README.md](../serving/tftp/README.md)): (i) the I219-LM needs **`snponly.efi`** (native iPXE driver won't bind); (ii) **PortFast/edge on the switch port is REQUIRED** — iPXE's PHY bounce triggers STP forwarding delay that drops DHCP (this was the real blocker); (iii) iPXE aborts the script on any unguarded command failure (use `||`); (iv) Secure Boot OFF makes systemd-stub take iPXE's load options as the cmdline, so the entrypoint extracts the UKI's real `.cmdline` and feeds it via `imgargs`. **Tidied 2026-06-20** once PortFast was identified: dropped the `autoexec.ipxe` + `ifopen` + DHCP-retry STP workaround; `boot.ipxe` is now just `dhcp`+`imgfetch`+`imgargs`+`boot` reached via normal iPXE autoboot. Re-verified on the 820 G4. **SB-on simplification:** when Secure Boot is enabled the cmdline injection is unnecessary (the stub ignores the loader cmdline) — collapse `boot.ipxe` to a plain `chain` of the UKI and drop the `.cmdline` extraction + `binutils` + the artifact mount from the dnsmasq service. - **Sign iPXE for the PXE-only fallback (Secure Boot, production).** The lab `snponly.efi` is **UNSIGNED** → that path needs **Secure Boot OFF**. For production: `sbsign` the iPXE binary with the fleet **db** key (only `db` needed — no MS CA / shim), so SB stays enforced firmware → iPXE → UKI; then the cmdline contraption becomes a no-op (the stub ignores loader cmdline under SB). See [secure-boot-keys.md](secure-boot-keys.md) "PXE-only hosts". - **Bench-verify on the secure path** (swtpm + hostapd + FreeRADIUS, `disable_dot1x=false`) — the items deferred this session: - NM port secure path: `en*` UNMANAGED + `macsec0` MANAGED/renewing, MKA survives switch-root, NVMe stays up past lease (item #1 risk). - NTP via DHCP option 42 reaches chrony over macsec0 (`chronyc sources`). - Plymouth `fleet` theme: test-path render VERIFIED 2026-06-18; still confirm the auth→macsec rows reach `done` live on the secure path. - STIG remediation didn't break the secure path; reproducible verity hash; triage the generated `stig-deviations.md` REVIEW rows (item #2). - **Secure-path NVMe connect trigger:** the test path uses `nvmf-connect.service` (gated `rd.dot1x=0`); confirm the secure path's connect (dracut nvmf over macsec0) actually fires, or give it an equivalent trigger. Apply the same `-k 30 -l -1 -c 10` keep-alive/reconnect flags to the secure `nvme connect` as the test path has. - **Future hardening waived for now** (from the STIG register): FIPS, firewall, per-host/rotatable admin (SSH CA / config-agent). - **TLS-encrypted audit/log offload (production).** The lab forwards logs to `serving/audit-collector` in **plaintext**, and the STIG TLS-offload rules (`rsyslog_encrypt_offload_defaultnetstreamdriver`, `..._actionsendstreamdrivermode`, `..._actionsendstreamdriverauthmode`) are **waived** in the tailoring because applying them forces `gtls` + x509 peer auth + a CA file and breaks the plaintext `2f` forward. For production, do it properly and DROP the waivers: 1. Stand up a CA (or reuse the boot-infra CA); issue the collector a server cert (CN/SAN = `audit.fleet.lab`). 2. Configure the collector's rsyslog for TLS: `imtcp` with `StreamDriver.Name="gtls"`, `StreamDriver.Mode="1"`, server cert/key + CA; listen on the syslog-TLS port (6514) — add a TLS receiver to `serving/audit-collector/rsyslog.conf` + compose. 3. Bundle the CA into the image (host-generic, no per-host state — same model as the RADIUS CA) and point `$DefaultNetstreamDriverCAFile` at it. 4. Set the `2f` `omfwd` action to `StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name"` + the CA file, target port 6514. 5. Re-select the three `rsyslog_encrypt_offload_*` rules (remove from `files/stig/diskless-tailoring.xml`) — they should then PASS, and update `docs/stig-deviations.md` #6 to reflect encrypted offload. (RELP+TLS via `omrelp`/`imrelp` is the alternative if guaranteed delivery is wanted alongside encryption.) - **fleet-hostapi: scope CORS to the kiosk origin.** The host-info agent ships with `FLEET_HOSTAPI_ALLOW_ORIGIN=*` (lab). For prod set it to the exact kiosk origin (default `http://boot.fleet.lab`) via the unit's commented `Environment=` line (or a drop-in). See #12. ## [ ] 9. (OPTIONAL / nice-to-have) Convert provisioning to Ansible where it pays off Low priority — only the *config-management* half of the build is a candidate, and the ROI is modest because the build VM is single-shot (idempotency buys little) and reproducibility/auditability favour minimal shell. **Not** a wholesale rewrite: the load-bearing logic (dracut module hook ordering, `extract-artifacts.sh`, `build-oci.sh`) is initramfs-runtime / host-side build tooling, not provisioning, and stays shell. The high-ROI, low-risk slice if/when we pick this up: - **STIG via the SSG Ansible playbook** (`ssg-rl9-playbook-stig.yml`, shipped with scap-security-guide) instead of `oscap --remediate` — the upstream path and far more *reviewable* (you can diff exactly what each rule changes). Keep `oscap eval` (no remediate) for the report + the 0-eval guard (item #2). - Restyle the brittle imperative bits as declarative tasks: chrony.conf edit (`lineinfile`/`blockinfile` vs `sed`), sshd drop-in / NM dispatcher / kiosk file installs (`template`/`copy`), datastream-by-`os-release` as `assert`. Bonus: sidesteps the `%{...}`→`%%{` HCL-escaping footgun. Tips the balance toward "do it" only if: we start managing the **serving hosts** (dnsmasq/nginx/SPDK/chrony/audit-collector) with config-management too (shared roles to amortise — currently they're docker-compose), provisioning grows much more branchy, or we want molecule-tested hardening roles validated without a full ~17-min Packer build. ## [ ] 10. (OPTIONAL) Secure Boot cert auto-enrollment for provisioning Speed up bringing up new workstations: bake the **public** PK/KEK/db certs into an image, detect Secure Boot **Setup Mode** at boot, and auto-enroll them so a fresh/reset box adopts the fleet trust root without a manual firmware-setup visit. Pairs with #8 (signing) + [secure-boot-keys.md](secure-boot-keys.md). Shares the provisioning-UKI + isolated-environment vehicle with **#11** (which enrols the *network* identity); build them together. **Tooling gap (found 2026-06-20): `efitools` is NOT packaged for EL9** — no `efi-updatevar`/`efi-readvar`/`KeyTool`. Resolved for the OS-side path by writing the signed `.auth` variables to **efivarfs directly** (de-immutable + single write), which needs no extra package. **Done 2026-06-20:** baked `/usr/local/bin/fleet-sb-enroll` (efivarfs method, Setup-Mode-gated, fapolicyd- trusted) + the public `PK/KEK/db.auth` at `/usr/share/fleet-secureboot/` (supplied via `files/secureboot/keys/` or `SB_*_AUTH_B64` CI secrets); `mokutil` covers reading. So **manual** OS-side enrollment over SSH now works (`sudo fleet-sb-enroll`). What's left for *auto*-enroll: drive `fleet-sb-enroll` unattended on a **Setup-Mode detection** at boot, gated to a provisioning mode/UKI (below), with read-back verify + audit — NOT armed in the production image. **What's safe:** the certs are public (no key material — only the **db** private key signs, and it stays in CI/HSM, invariant #7); Setup Mode accepts *unauthenticated* writes to PK/KEK/db, so public certs (ESL) are sufficient. It's fleet-generic (no per-host state, invariant #1) and deterministic (reproducible verity hash). **The real risk — provisioning-time TOFU (trust-on-first-use):** in Setup Mode Secure Boot isn't enforcing, so the enroller runs as *unverified* code, and the fleet's UEFI **HTTP boot is unauthenticated** in that state. An on-path attacker during provisioning can serve a malicious image that enrolls **their** PK/KEK/db — the box then "has Secure Boot" but trusts the attacker. Auto-enroll is only as trustworthy as the provisioning channel, and it removes the physical-presence approval gate. So treat it as a controlled-environment provisioning step, not an always-on behaviour. **Optional separate provisioning UKI (preferred shape):** build a dedicated **provisioning UKI** that carries the certs + the enroll logic, instead of arming the enroller in the production image. Keeps the standing footgun out of the normal image (a factory reset + tampered image can't silently re-root trust), and lets the enroller be a one-shot, attended/isolated step. The production UKI never enrolls. - Build it as a separate Packer target / flag (e.g. `provisioning=true`) → its own small UKI; signed or unsigned (it runs in Setup Mode anyway, but sign it so it *also* boots once keys are enrolled / for re-runs). - Or a tiny standalone EFI tool (KeyTool-style) served alongside, not a full OS. **Mitigations (do these regardless of UKI vs separate UKI):** - Run provisioning on a **trusted, isolated VLAN** (or a physically controlled bench) — the integrity Setup Mode can't give must come from the environment. - **Gate** enrollment behind an explicit provisioning mode / one-shot token; never arm it on every boot of the production image. - **Read-back verify before declaring success:** `efi-readvar` the enrolled PK/KEK/db and compare fingerprints to the values baked in; mismatch = fail loudly (turns silent TOFU into tamper-evidence). - **Audit** the enrollment event + read-back fingerprints to the collector. - **Keep the Microsoft UEFI CA in `db`** (or enroll your db alongside it) unless every NIC/GPU option ROM is covered by your keys — clearing MS db can break signed **option ROMs** (NIC PXE/HTTP-boot, GPU init), bricking the boot path. - **PCR 7 check:** enrollment changes the Secure Boot measurement in PCR 7; confirm nothing the fleet seals to the TPM (the EAP-TLS key at `0x81010001`, NV cert) is PCR-7-policy-bound before relying on this. **Decision before building:** auto-enroll trades assurance for convenience and is reasonable for a controlled fleet; the highest-assurance path is attended/firmware or manufacturer pre-provisioning. Default recommendation: **separate provisioning UKI + isolated VLAN + read-back verify.** ## [ ] 11. (DESIGN) TPM 802.1X identity enrolment (TOFU + OOB fingerprint) How a host gets its **802.1X EAP-TLS client identity** the first time — the key at TPM handle `0x81010001` + cert at NV `0x01800001` that invariant #1 says must come from enrolment, never the image. Distinct from #10 (that enrols *Secure Boot* PK/KEK/db); this enrols the *network identity*. They share the provisioning UKI + isolated-environment vehicle. Pairs with #8 + the CA decision (start **step-ca**, [serving/dot1x-bench/](../serving/dot1x-bench/); OpenBao/EJBCA/Dogtag later). **Chosen pattern — trust-on-first-use with out-of-band fingerprint approval:** on an empty TPM the host generates a key *in* the TPM, builds a CSR, submits it to an enrolment **broker**, and shows a **fingerprint** on the Plymouth splash; an admin visually cross-checks that fingerprint against the broker's pending-list and approves; the broker has step-ca sign the CSR and returns the cert; the host writes it to NV `0x01800001` and proceeds to authenticate. Sound, recognised pattern (SSH/WireGuard/Signal-style OOB verification); good fit for invariant #1. **Key design decisions (settled 2026-06-22):** - **The pre-auth permit-set already exists — enrolment rides in it.** The firmware can't do 802.1X, so the UKI fetch (DHCP/DNS/TFTP/HTTP to the serving host) is **always pre-auth**; the secure model therefore *requires* a pre-auth permit-set (low-impact 802.1X / pre-auth dACL / boot→secure dynamic VLAN; RADIUS returns the secure VLAN on Access-Accept). The enrolment endpoint is just **another entry in that set** — expose it as an `/enrol/` location on the **existing serving nginx**, reverse-proxied to the broker. No new network mechanism. - **The broker is the only new component.** step-ca signs but has no human-approval queue: broker = `POST CSR + context → PENDING + fingerprint → admin approves → step-ca signs → host polls → cert`. Exposed to *unauthenticated* devices by design → harden: OOB-fingerprint gate (no auto-sign), rate-limit, time-boxed pending, HTTPS (image trusts the serving CA), pre-auth permit scoped to only these services + the serving host. - **Trigger on EMPTY-NV only — split the three events:** empty NV → enrol (human); valid key + expired/near-expiry cert → **renew** automatically (no human, it already holds a trusted key); auth fails with a present cert → **alert + retry, do NOT re-enrol** (auto-reenrol on failure would mask revocation and churn keys on a transient RADIUS outage). Re-enrol after the first time = explicit admin action (clear/flag the host). - **Fingerprint: what it proves + entropy.** It proves "this CSR came from *this physical device*" (approver reads the screen) — it does NOT prove TPM residency. Pair it with **TPM attestation** (step-ca ACME `device-attest-01`, EK/AK) for the residency proof: fingerprint = human authorises the device, attestation = the key is really in a TPM (together → strong SSP R-01 story). Size for entropy — a short hex fp is grindable; use a word-list/BIP39-style encoding (reads short, more bits) + rate-limiting + a server nonce. Show device context (asset/port/location) so the admin knows *what* they're approving. - **Pre-auth traffic has no MACsec (plaintext at L2).** Fine for enrol (CSR/cert are public; OOB fingerprint gives authenticity) — and it's exactly **why the UKI must be Secure-Boot-signed** (#8): integrity of the boot comes from the signature, not the network. Enrolment design + Secure Boot are two halves of the same "pre-auth path is hostile" assumption. - **Run it in a provisioning UKI/mode, not the production initramfs** — the boot hook ordering is load-bearing (invariant #2); don't park it on a multi-minute human-approval wait. Shares the #10 provisioning-UKI vehicle. **Bench caveat:** `hostapd` wired is strictly port-based (no pre-auth per-flow ACL) — before EAP success it passes nothing, so the bench can't reproduce "boot + enrol pre-auth on the same port." Model the pre-auth state with a separate provisioning bridge/VLAN on the rig; the in-place low-impact pre-auth permit is a real-switch feature to validate on the 820's switch. **First step DONE (2026-06-22):** `serving/dot1x-bench/enrol/tpm-selftest.sh` — a non-destructive TPM capability check (device, key persist, NV roundtrip, openssl TPM2 provider, CSR signed by the TPM key) to de-risk the TPM half before any of this. Expected gap: the image installs `tpm2-tss` but no openssl TPM2 **provider** (`tpm2-openssl`) — add it to the build once the self-test confirms. ## [~] 12. Workstation host-info API (`fleet-hostapi`) for the kiosk **IMPLEMENTED 2026-06-28.** A small **Go, read-only, loopback-only** HTTP agent on the workstation that reports host/OS/hardware facts to the kiosk SPA, per the [HOST-API.md](../HOST-API.md) contract (`GET http://localhost:9099/v1/host`). Source: `files/hostapi/` (stdlib-only, reproducible static build). Built + enabled by provisioner **2b-hostapi**; toolchain removed in 2d; binary fapolicyd-trusted in 2c-STIG-postfix (register #10). Chromium policy grants Private Network Access scoped to the kiosk origin (`files/kiosk/chromium-kiosk-policy.json`). - **Per-host safety:** host-specific fields (kernel, hostname, IP, MAC, board, uptime) are read at RUNTIME; only image-generic facts (image version + weston/chromium versions) are baked, to `/etc/fleet-hostinfo.json` (invariant #1 preserved). Deliberately does NOT expose DMI serial (root-only/per-host) or any TPM/cert/identity material. - **Hardening shipped:** loopback bind enforced in code; `DynamicUser`, no caps, `ProtectSystem=strict`, `MemoryDenyWriteExecute`, `SystemCallFilter=@system-service`, `IPAddressDeny=any` (loopback only). Runs unprivileged, reads world-readable files only. **Lab posture LEFT AS-IS (decided 2026-06-28) — revisit for prod:** - **CORS `Allow-Origin=*`** → scope to the kiosk origin (`http://boot.fleet.lab`). Consolidated into #8. - **SELinux:** runs in the default service domain (no custom policy module). **BENCH-VERIFY** no AVC denials for `fleet-hostapi` on an enforcing booted host (Wazuh/`fleet_audit` rule 100120 will surface any). If denials appear, decide: write a confined SELinux policy module vs adjust the sandbox. - **No auth** (justified: loopback + read-only + single-user appliance). Option if the threat model tightens: shared token in `kiosk.env` + header check. - **Info sensitivity:** exposes hostname/IP/MAC/board/versions to the kiosk. Drop MAC/IP if that's later deemed too much. - **Reproducibility:** the `golang` toolchain version is a reproducibility-sensitive input (a package bump changes the binary → verity hash); covered by the build-twice gate. Pin the go version only if drift becomes a problem. **BENCH-VERIFY (functional):** (a) service starts under enforcing SELinux + fapolicyd (not blocked); (b) the kiosk SPA's System page fetches `/v1/host` successfully through the PNA allowance (preflight + GET); (c) facts render correctly on real hardware (board/CPU/GPU from DMI/DRM, not the virtio build VM). ## [~] 13. Kiosk GPU compositing + Chromium flag ownership **GPU FIX APPLIED 2026-06-28.** The 820 (Intel HD620/i915) ran Chromium at ~150% CPU with periodic crashes: `--disable-gpu` in our `kiosk-session` forced software raster/compositing. Root cause confirmed from the live cmdline. Fixes in `files/kiosk/kiosk-session` + `diskless.pkr.hcl`: - **Runtime GPU probe** (fleet-generic image boots real HW *and* virtio VMs): a real KMS driver (`i915`/`amdgpu`/…) + `/dev/dri/renderD128` → GPU path (`--ignore-gpu-blocklist --enable-gpu-rasterization --enable-zero-copy`, weston GL renderer); else software (`--disable-gpu` + `weston --use-pixman`) so the VMs still render. No per-host state baked. - **kiosk user added to `render`** (renderD128 isn't logind-seat-ACL'd on EL9; `video` only covers card0/KMS for weston). - **Merged the distro wrapper's `--disable-features` into ours** — Chromium takes the LAST `--disable-features` (not the union), so ours was silently dropping the wrapper's Lens/Gemini/Compose disables. **BENCH-VERIFY (on the 820):** GPU process alive (`chrome://gpu`), Compositor/ ThreadPool threads near-idle + CPU drop, `gt_act_freq_mhz` rises under load, no SELinux AVC for `dri_device_t` from the kiosk domain (Wazuh `fleet_audit` 100120 will surface any — if present, add a scoped SELinux allow or confirm the kiosk user's SELinux mapping is unconfined). Confirm VMs still render (software path). **DECISION OPEN — Chromium flag ownership (security).** The distro chromium wrapper PREPENDS vendor defaults we don't control, including capability-broadening ones a locked kiosk arguably shouldn't carry: `--enable-extensions`, `--enable-user-scripts`, `--enable-plugins`, `--enable-printing`, `--auto-ssl-client-auth`, `--enable-sync`. Today the managed policy neutralises the security-relevant ones (sync off via `SyncDisabled`; sign-in/metrics/etc.), but extensions/user-scripts/auto-ssl-client-auth are NOT policy-locked. Option to **own the full flag set**: bypass/neutralise the wrapper (invoke the real binary, or empty its default-flags source) and specify everything ourselves incl. the VA-API `--enable-features` (now known: `AllowQt,WaylandLinuxDrmSyncobj,WaylandPerSurfaceScale,WaylandUiScale,AcceleratedVideoDecodeLinuxGL,AcceleratedVideoDecodeLinuxZeroCopyGL,AcceleratedVideoEncoder`). Tradeoff: full determinism + drop unwanted capabilities, vs we then track Chromium feature renames ourselves instead of inheriting the distro's. Also fully dedupes `--disable-features`/`--enable-sync`. Needs locating the wrapper's flag source on the box first (so we don't lose VA-API or break its env setup). Pairs with policy adds: `ExtensionInstallBlocklist=["*"]`, disable user scripts, restrict AutoSelectCertificateForUrls.