# TODO — diskless fleet boot system Persistent backlog, kept in sync with the working session task tracker. Status: `[ ]` pending · `[~]` in progress · `[x]` done. --- ## [x] 1. Port the dracut module off `network-legacy` → `network-manager` **DONE — test path VERIFIED ON THE BENCH 2026-06-17** (host stays up past the DHCP lease time; no more lease-expiry outage). Fixed the confirmed outage: network-legacy's one-shot DHCP lease was never renewed post-switch-root, so the address dropped at lease expiry → NVMe/TCP root died ("keep alive timeout" / "no route to host"). NM hands its initramfs connections to the real-root NM (`/run/NetworkManager` state), which renews. REMAINING: the **secure path** (macsec0 DHCP via `nmcli`, physical NIC unmanaged via `nm-unmanaged.sh`) is only exercised once `disable_dot1x=false` — verify on the swtpm + hostapd + FreeRADIUS bench when re-enabling security (see #2/cleanup). Done: - `90-diskless.conf` `add_dracutmodules`: `network-legacy` → `network-manager`. - `module-setup.sh` `depends()` → `network-manager`; added `nmcli`; installs the new cmdline hook. - NEW `nm-unmanaged.sh` (cmdline hook): secure mode writes `/run/NetworkManager/conf.d/10-dot1x-unmanaged.conf` (`unmanaged-devices=interface-name:en*`) **before NM starts**; no-op in test mode. Replaces the deleted static `99-dot1x-unmanaged.conf` (couldn't be mode-aware — it wrongly unmanaged `en*` in test mode, blocking renewal). - `dot1x-macsec-up.sh` secure path: `ifup macsec0` → `nmcli` (managed + DHCP, poll for IPv4). macsec0 MANAGED so NM renews; physical NIC UNMANAGED. - `diskless.pkr.hcl`: dropped `dhcp-client`, added explicit `NetworkManager`; removed the static-config install; lsinitrd now asserts `NetworkManager` + `nm-unmanaged.sh` present. - Static lint clean (`bash -n`, `shellcheck`, `packer validate`). **BENCH-VERIFY before trusting (the gate):** - Test path (`disable_dot1x=true`): boot, `nmcli device` shows boot NIC MANAGED; run past the DHCP lease time (drop the lab dnsmasq lease low) — address persists, lease renews (`journalctl -u NetworkManager`), NVMe + SSH stay up. - Secure path (swtpm + hostapd + FreeRADIUS, `disable_dot1x=false`): `en*` UNMANAGED, `macsec0` MANAGED + renewing; MKA survives switch-root; verity root mapped; NVMe stays connected past lease time; no "Address already assigned". - **Risk:** ordering of the unmanaged drop-in vs NM start in secure mode. If NM transiently grabs `en*`, fall back to also baking a static `en*`-unmanaged file for secure-only builds. ## [~] 2. Apply RHEL9 STIG hardening to the built image **IMPLEMENTED 2026-06-18 — awaiting build/bench verification.** OpenSCAP + SSG remediation in a provisioner (`enable_stig` var, default on), tailored for the diskless/UKI/kiosk architecture via `files/stig/diskless-tailoring.xml`. STIG kernel params baked into the UKI cmdline (`extract-artifacts.sh`). Decisions: **FIPS off**, **audit local-only**, **CI report-only**. Deviations are documented in **[stig-deviations.md](stig-deviations.md)** (curated register) and reported per-build by **[../scripts/stig-deviations.sh](../scripts/stig-deviations.sh)** (non-pass rules tagged intentional vs REVIEW); CI publishes `stig-report.html` + `stig-arf.xml` + `stig-deviations.md`. Scanner tooling is removed from the shipped image. **BENCH-VERIFY:** SSG datastream name; boot/login still work; reproducible verity hash across two builds; triage REVIEW rows. Remaining (future hardening, currently waived — see the register): FIPS, firewall, remote audit offload, per-host/rotatable admin access. ### Original spec (retained for reference) OpenSCAP + SCAP Security Guide (`xccdf_org.ssgproject.content_profile_stig`), run as a Packer provisioner after `dnf install` and before the dracut/initramfs build and cleanup. - **Tailoring** (`autotailor`): unselect rules this architecture can't satisfy — separate-partition rules (single erofs root + tmpfs overlay; satisfy intent with `nodev,nosuid,noexec` tmpfs mounts), GRUB/kernel-cmdline rules (no GRUB), and AIDE/file-integrity rules (dm-verity is the documented compensating control). - **Kernel params → UKI cmdline** in [../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh) before signing: `audit=1 audit_backlog_limit=8192 slub_debug=P page_poison=1 vsyscall=none` (+ `fips=1` if FIPS). - **Reproducibility guard:** strip non-deterministic remediation output (SSH host keys → regen at boot into overlay, AIDE db, timestamps, fresh UUIDs) so the verity hash stays stable. - **Audit logging:** `audisp-remote`/rsyslog to a central collector addressed by DNS name (host-generic). - **Optional FIPS:** `fips=1` cmdline + `01fips` dracut module + FIPS crypto policy. - **CI:** run `oscap eval` (no `--remediate`) to publish an ARF+HTML STIG report as a build artifact; optionally hard-fail below a score threshold. - **Decisions needed before implementing:** FIPS on/off · remote-audit collector DNS name (or ephemeral/none for lab) · CI fail-on-score threshold vs report-only. - Note: orthogonal to the `rd.dot1x=0` test toggle, but a full STIG production posture wants 802.1X re-enabled. - **Credentials (partly done):** root is now **locked** and access is via a baked sudo-enabled key-only admin user (`var.ssh_authorized_key` / `var.admin_user`, `wheel` + NOPASSWD sudo), with sshd drop-in `PermitRootLogin no` / `PasswordAuthentication no`. Two items for STIG to reconcile: **NOPASSWD sudo** is a finding (acceptable for a key-only account, but document/deviate), and the admin key/user are **fleet-wide baked** — for per-host/rotatable access without rebuilds, layer the signed HTTP/SMBIOS config-agent or an SSH CA (`TrustedUserCAKeys`) discussed in session. - **SSH host keys (done):** `extract-artifacts.sh` strips `/etc/ssh/ssh_host_*` so they're not baked (baked keys = shared fleet identity + the build's perms/group don't survive the tar→erofs round-trip, so sshd rejected them "Permissions 640 too open" and refused to start). RHEL's `sshd-keygen@.service` regenerates them into the overlay before sshd starts. **⚠ Caveat:** the ephemeral overlay means they regenerate **every boot** → the host key changes per reboot (clients warn). Follow-up for a stable host identity: persist the host keys or derive them from the TPM (ties to the hostnqn/identity model). ## [ ] 3. Roll the errata into design doc v3 Once the boot path is validated and item 1 lands, fold [design-v2-errata.md](design-v2-errata.md) into the source design document and re-export a **v3** PDF (key change: `rootovl` → `systemd.volatile=overlay` in §3 step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source (not in this repo). ## [~] 4. Kiosk mode — autologin + fullscreen Chromium loading a URL **WORKING / display-verified 2026-06-17** (kiosk renders Chromium fullscreen on the configured URL). Build-flagged OFF by default for now (`enable_kiosk=false`) while iterating on boot/network — set `-var enable_kiosk=true` to ship it. Path to getting here (don't re-discover): - **cage is NOT packaged for EL9** (`dnf install cage` → "Unable to find a match"). Tried weston instead. - **EPEL 9 ships weston 8.0**, which has NEITHER `kiosk-shell.so` NOR the `[autolaunch]` weston.ini section — both arrived in **weston 9.0**. The first cut wrongly assumed 8.0 had them; `shell=kiosk-shell.so` failed to load. `rpm -ql weston` shows only desktop-shell / fullscreen-shell / ivi-shell. - **Resolution: `desktop-shell.so`** (the only packaged shell Chromium's xdg-shell works with — fullscreen-shell needs a client protocol Chromium doesn't speak), stripped bare (`panel-position=none`, black bg, no animation/lock), and Chromium launched by a **wrapper** (kiosk-session) since there's no `[autolaunch]`. Chromium `--kiosk` fullscreens over the shell. - **Seat fix (load-bearing):** weston died with `logind: failed to get session seat` → `failed to create compositor backend`. logind's auto-derivation didn't attach a seat; `Environment=XDG_SEAT=seat0` + `XDG_VTNR=1` in the unit (put into the PAM env before pam_open_session) registers the session on seat0/vt1. `/dev/dri/card0` was present all along (not the issue). - Tested live without a rebuild: the whole root is `systemd.volatile=overlay`, so /etc AND /usr are writable tmpfs at runtime — drop-in the files + restart. - **`enable_kiosk` Packer var — currently `false` by DEFAULT (set 2026-06-17).** The whole kiosk provisioner is feature-flagged off while iterating on the boot/network stack (DHCP-lease/keep-alive work), for faster builds and a smaller image. **Re-enable with `-var enable_kiosk=true` when returning to kiosk work** — CI does not pass the var, so CI currently builds lean (no kiosk) too; add `-var enable_kiosk=true` to the Gitea workflow to ship it. - `kiosk_url` Packer var (default `http://boot.fleet.lab/kiosk/` — the self-contained test page served by the boot HTTP server at `serving/http/kiosk/`, via the `/kiosk/` nginx location; validates a boot without external connectivity. Empty disables the whole install). Wired in CI as `KIOSK_URL` env. - Provisioner enables EPEL, installs `weston chromium mesa-dri-drivers mesa-libEGL mesa-libgbm dejavu-sans-fonts`, creates a passwordless unprivileged `kiosk` user (video/input groups). - `files/kiosk/kiosk.service` — runs `/usr/local/bin/kiosk-session` as `kiosk` on tty1 via `PAMName=login`+`TTYPath`+`Environment=XDG_SEAT=seat0`/`XDG_VTNR=1` (gets seat0 for DRM/input), `Conflicts=getty@tty1`, `WantedBy=graphical.target`; default target set to graphical. - `files/kiosk/weston.ini` — `shell=desktop-shell.so`, `idle-time=0`, `require-input=false`, `xwayland=false`; `[shell] panel-position=none`, black `background-color`, `animation=none`, `locking=false`. - `files/kiosk/kiosk-session` — `exec`s `weston --use-pixman --socket=wayland-kiosk` in the foreground (owns the seat), and launches Chromium (`--kiosk --ozone-platform=wayland --disable-gpu`, tmpfs profile/cache, URL from `/etc/kiosk/kiosk.env`) in a background crash-restart loop. - **Remaining polish (now that it renders):** dropping `--disable-gpu` on real GPUs, `/dev/dri` present in the VM (weston needs a DRM device — set Display to VirtIO-GPU/std), and **SELinux AVCs** for the GUI stack (check `ausearch -m avc`; the kiosk user is unconfined so likely none — but do NOT set permissive if they appear; add a policy module). Admin SSH (the key user) still works for debug even if the kiosk fails (it lands at graphical.target either way). - **Phone-home lockdown (done 2026-06-17/18):** Chromium was dialing Google (accounts/variations/optimization-guide/component+auto update/safe browsing — seen in dnsmasq logs). Two layers: - **Flags** in `kiosk-session` (`--disable-background-networking`, `--disable-component-update`, `--disable-features=OptimizationHints,...`) — confirmed killing variations/optimization-guide/update queries. - **Managed policy** `files/kiosk/chromium-kiosk-policy.json` for the policy-governed bits (sign-in `accounts.google.com`, sync). FIX 2026-06-18: the managed-policy dir differs by build, so it's installed to BOTH `/etc/chromium/policies/managed/` AND `/etc/chromium-browser/policies/managed/` (a policy in the wrong dir is silently ignored — the reason `accounts.google.com`/`android.clients.google.com` persisted). Verify via `chrome://policy` (Status: OK per entry). - **Serving-side sinkhole** (defense-in-depth, `serving/docker-compose.yml` dnsmasq `--address=/google.com|googleapis.com|gstatic.com/0.0.0.0`): the kiosk is local-only, so block egress at the resolver. Stops the *leak*; the query still logs (answered 0.0.0.0, not forwarded) — eliminating the query is the browser policy's job. Production: do this at the boot-LAN firewall. - **NTP egress closed (done 2026-06-18):** the `*.pool.ntp.org` lookups were the OS (chrony), not Chromium. Now: a `chrony` NTP server in the serving stack (`serving/chrony/`), dnsmasq hands it via **DHCP option 42**, and the image's NM dispatcher (`files/etc/NetworkManager/dispatcher.d/10-chrony-dhcp-ntp`) feeds it into chrony's `sourcedir /run/chrony-dhcp`; the baked public pool is stripped from `/etc/chrony.conf`. Result: the workstation syncs time only from the lab server — no external NTP egress. BENCH-VERIFY: on a booted host `chronyc sources` shows the lab server (the DHCP one), not a public pool. - **Image size:** Chromium+Mesa add hundreds of MB to the *uncompressed* erofs — feeds the "shrink the rootfs" concern (and #7); reconsider whether a lighter browser/render path is viable. ### Original task Make the diskless workstation boot straight into a locked-down kiosk: a fullscreen Chromium showing a configured site (default: the boot HTTP server's `/kiosk/` test page). All baked at **build time** into the root tree (immutable erofs/verity root; runtime is the tmpfs overlay), fleet-generic (no per-host state). - **Graphical stack (keep it minimal):** add a Packer provisioner installing a lightweight kiosk compositor + browser. Preferred: `cage` (Wayland single-app kiosk compositor) + Mesa (software GL fallback for VMs/no-GPU); X11 + a minimal WM (matchbox/openbox) as the alternative. Chromium on Rocky 9 is in **EPEL** — enabling EPEL adds a repo/supply-chain surface (decision below). - **Autologin + session:** dedicated unprivileged `kiosk` user; launch the session via a `kiosk.service` running `cage -s -- chromium --kiosk …` (or `greetd` + cage). Stage units/configs under `files/` like the other provisioned config. - **Chromium flags:** `--kiosk --noerrdialogs --disable-infobars --incognito --no-first-run --disable-session-crashed-bubble `. Writable profile/cache land on the `systemd.volatile=overlay` tmpfs (RAM-backed, ephemeral) — fine, but watch RAM sizing; pin `--user-data-dir`/`--disk-cache-dir` to a tmpfs path if needed. - **Kiosk URL = Packer var** (e.g. `kiosk_url`, default the target site), baked into the launch command; keep it host-generic (per-env via DNS if ever needed). - **Verify in boot test:** graphical session comes up on real/VM display, Chromium fullscreen loads the URL over the booted network (works with the current `rd.dot1x=0` test path / open network). - **Decisions:** Wayland(cage) vs X11; accept EPEL for Chromium (vs another source); confirm the target URL. - **⚠ Conflict with item 2 (STIG):** passwordless autologin and a desktop browser are STIG findings and enlarge the attack surface/image. Reconcile — e.g. a tightly-constrained kiosk session + documented deviation — when hardening lands. ## [x] 5. Deploy & validate the end-to-end boot path (open-network test) ✅ **DONE (2026-06-15).** Full chain validated on QEMU/Proxmox: UEFI HTTP boot → initramfs → DHCP → `nvmf-connect.service` auto-connects the NVMe/TCP subsystem → two namespaces (data + verity hash) appear by pinned UUID → `systemd-veritysetup` maps `/dev/mapper/root` → uncompressed erofs mounts → `systemd.volatile=overlay` → multi-user. Booted with `rd.dot1x=0` (open network) and the IP discovery address. See bench findings below for the full list of fixes. Remaining cleanup to leave the lab-test posture (now unblocked): revert the hardcoded IP to the `nvme.fleet.lab` DNS name, and re-enable security (`DISABLE_DOT1X=false`) — the secure path needs the same connect trigger over macsec0 (see ⚠ note below). Original task: stand up `serving/` (dnsmasq + nginx + SPDK) per [serving/README.md](../serving/README.md), rebuild the UKI with `nvmet_traddr=nvme.fleet.lab` (now a Packer var — older UKIs have the literal `NVMET_TRADDR` placeholder and won't connect), then boot a UEFI HTTP-boot client (**Secure Boot OFF**, unsigned UKI; no TPM needed in `rd.dot1x=0` mode) on the boot LAN. - **Pass criteria:** UKI fetched over HTTP → initramfs → DHCP on NIC → NVMe/TCP namespace connected → dm-verity root mapped → `systemd.volatile=overlay` writable → multi-user. - This validation **gates** item #1 (network-manager port) and item #3 (errata → design v3). - **Bench findings (boot debug, 2026-06-14):** - Initramfs has no resolver (dracut writes `resolv.conf` only via its own DHCP path, not a manual `dhclient`), so the NVMe discovery address must be an **IP**, not a DNS name — build with `-var nvmet_traddr=192.168.50.10`. Old UKIs carrying `nvme.fleet.lab` can't connect early in boot. - NVMe controller indices are **not stable**: a transient discovery controller can take `nvme0`, pushing the I/O controller to `nvme1`. The hardcoded `/dev/nvme0n1` in the cmdline was a latent brick. **Fixed:** namespace UUID pinned on the target (`spdk-setup.sh` `add_ns -u f1ee7000-0000-4000-8000-000000000001`) and the cmdline now references `/dev/disk/by-id/nvme-uuid.` (`extract-artifacts.sh`). The two constants MUST stay in sync; make them version-specific for #6. - `nvme_tcp`/`nvme_fabrics` must be loaded before `/dev/nvme-fabrics` exists (dracut's `nvmf` module does this in a normal boot; at `rd.break` modprobe manually). - **Don't double-configure the NIC in test mode.** With `ip=dhcp` on the cmdline, dracut's network module brings the NIC up AND emits the "online" event that the `nvmf` module's `initqueue/online` hook waits on to run `nvme connect-all`. The `rd.dot1x=0` bypass used to *also* `ifup $NIC`, which (a) raced dracut → `Error: ipv4: Address already assigned`, and (b) didn't fire the nvmf online hook, so the namespace never auto-connected and boot spun on `/dev/mapper/root` forever (manual `connect-all` worked, which is what masked it). **Fixed:** the bypass branch no longer touches the NIC; `ip=dhcp` owns networking. (The secure path still addresses macsec0 itself — dracut doesn't know about that interface.) - **dracut's `nvmf` module never auto-connects on this distro.** Its cmdline hook (`92-parse-nvmf-boot-connections.sh`) parses `rd.nvmf.discover` into `/etc/nvme/discovery.conf` correctly, but `initqueue/online/` is **empty** — no autoconnect hook is registered, so `nvme connect-all` is never run (no attempt in dmesg) and the namespace never appears. Drivers load and `/dev/nvme-fabrics` exists; only the connect trigger is missing. A job placed on the *main* initqueue from a settled hook gets registered but is **never re-executed** once the loop blocks on udev events (verified: job present, manual `connect-all` works, namespace stays absent). An `initqueue/online` hook also does NOT work: this initramfs is **systemd- driven** ("A start job is running for /dev/mapper/root") and never sources dracut online hooks on link-up (verified: hook installs as `90-nvmf-connect.sh` but never fires — no log, no stamp). **Fixed:** the connect is now a **systemd unit** (`nvmf-connect.service`, `ConditionKernelCommandLine=rd.dot1x=0`, `Before=initrd-root-device.target`) whose ExecStart (`/usr/sbin/nvmf-connect`) polls for the network then runs `nvme connect-all` until the namespace appears. The dracut module installs the unit + script and enables it via an `initrd-root-device.target.wants` symlink; `diskless.pkr.hcl` hard-asserts the unit is in the initramfs. Gotcha: `$systemdsystemunitdir` is **empty** in this dracut's module-setup context — `inst_simple` then silently drops the unit (empty dest dir). Use a `${systemdsystemunitdir:-/usr/lib/systemd/system}` fallback and `return 1` if the unit didn't land, so the build fails loudly instead of shipping a UKI that can't connect its root. Second gotcha: the build assertion `lsinitrd … | grep -q 'nvmf-connect.service'` was a **false negative** — `grep -q` exits on first match, closes the pipe, `lsinitrd` dies of SIGPIPE, and under `pipefail` the pipeline reports failure even though the unit IS present. This masked the fact that the systemd unit install was working all along. Fix: `lsinitrd > file; grep -q … file` (no pipe). Never `lsinitrd | grep -q` under pipefail. Bonus: the survivor listing confirmed nvme-cli's native autoconnect bits ARE in the initramfs (`/usr/sbin/nvmf-autoconnect.sh`, `/etc/udev/rules.d/95-nvmf-initqueue.rules`) — so the fully-native `rd.nvmf.discover` autoconnect path is available as a fallback if the unit doesn't fire on boot. - **dm-verity: `hash-offset` is unsupported → two namespaces.** With NVMe connecting and the right by-id device present, verity still failed: `systemd-veritysetup` logged `unknown option 'hash-offset=…'` then `Failed to load verity superblock` — exactly the CLAUDE.md bench-verify item. The in-file (data+hash in one namespace, split by offset) layout is not supported by Rocky 9.8 systemd. **Fixed** via the documented fallback: `extract-artifacts.sh` now emits `rootfs.erofs` (data) + `rootfs.hash` (hash tree) as separate files, served as **two NVMe namespaces** with distinct pinned UUIDs (`f1ee7000-…-01` data, `…-02` hash); the cmdline uses `systemd.verity_root_data=`/`systemd.verity_root_hash=` by-id devices with no offset. Touches `extract-artifacts.sh`, `build-oci.sh` (artifact list + SHA256SUMS), `serving/spdk-setup.sh` (2nd bdev/namespace), and invariant #9. - **SELinux labels must survive into the erofs.** It booted to multi-user but `root`/`packer-build-only` login was accepted then the session was killed instantly. Cause: kickstart sets `selinux --enforcing`, but the build dropped `security.selinux` xattrs (`virt-tar-out` + plain `tar -xpf` don't carry them), so the enforcing target denied the login session on an unlabeled root — and a read-only erofs can't autorelabel. Confirmed by booting with `enforcing=0` (SMBIOS-injected) → login worked. **Fixed:** export via `guestfish tar-out … selinux:true xattrs:true acls:true`, restore with `tar --xattrs --xattrs-include='*' --acls --selinux`, and a `getfattr` build-time assertion that `/usr/bin/bash` is labeled before `mkfs.erofs` (mkfs.erofs preserves xattrs). Needs `attr` on the build host. The image now boots **enforcing**. - **`mkfs.erofs --all-root` was flattening all ownership to root:root.** Mode bits + (now) xattrs survive the export, but `--all-root` forced every inode to uid/gid 0 — discarding group ownership (`ssh_keys` on host keys → sshd "permissions 640 too open" → refused; `tty`/`crontab` setgid groups) and user ownership (home dirs). **Fixed:** dropped `--all-root` so real ownership is preserved from the tree. That now requires `extract-artifacts.sh` to run as **root** (added an `id -u` guard; `tar -x` needs CAP_CHOWN to restore ownership) — the post-processor runs it via `sudo` and `chown`s the output files back to the build user so `build-oci.sh` (rootless buildah + its own login) still works. Combined with the host-key strip above, sshd starts cleanly. **⚠ The secure path has the same dependency on the (absent) nvmf autoconnect** — when 802.1X is re-enabled, the macsec0 path must likewise drive `connect-all` (or the network-manager port #1 must wire a working autoconnect). Currently only the test path is fixed. - The target Rocky 9 kernel's erofs is built **without `CONFIG_EROFS_FS_ZIP`** — mounting a compressed image fails `erofs: lz4 algorithm isn't enabled`, which silently blocks the root mount. **Fixed:** `extract-artifacts.sh` now builds the erofs **uncompressed** (`mkfs.erofs` without `-z`). Tradeoff: larger rootfs namespace + more NVMe/TCP bandwidth per read. Re-enable compression only against a kernel verified to have `EROFS_FS_ZIP` (`grep CONFIG_EROFS /boot/config-$(uname -r)`). This is the CLAUDE.md bench-verify item ("erofs module availability on the target RHEL kernel"). Larger image size also feeds the "shrink the rootfs" angle of #7. - SPDK: add the listener to the discovery subsystem explicitly (`nqn.2014-08.org.nvmexpress.discovery`) to silence the deprecation warning. ## [ ] 6. Rolling fleet updates (multi-version serve → cohort reboots) Image-based, reboot-to-update across many live workstations. Hosts are immutable (read-only erofs/verity root over NVMe/TCP, ephemeral overlay), so updates are **atomic A/B reboots**, never in-place. The released OCI artifact + digest-pinned manifest + roothash coupling already give the atomic unit and instant rollback; this item builds the rollout machinery around it. - **Serve versions side-by-side** (must not overwrite a live rootfs): extend `serving/` so SPDK/nvmet exposes each release as its **own subsystem/NQN** (`nqn…:rootfs-`) and nginx serves multiple `fleet-.efi` (keep current + previous). Make the UKI's NQN version-specific so each signed UKI connects only its matching namespace (constant discovery DNS name). - **Version selection at boot**: dnsmasq **cohort tagging** (MAC / vendor class / assigned tag) hands each cohort a different option-67 UKI URL. Moving a cohort's URL = assigning the release. - **Control plane + in-image reboot agent**: a baked-in systemd timer polls a "desired version" endpoint and reboots **when idle / in a maintenance window** (or out-of-band via Redfish/BMC). Stagger + jitter reboots to avoid UKI-fetch / NVMe-connect storms. - **Staged rollout**: canary → waves (10/50/100%) with health gates (multi-user reached, NVMe connected, dm-verity mapped, kiosk URL rendered, prod: MACsec/802.1X up); auto-halt on threshold. - **Rollback**: repoint cohort URL to previous UKI + reboot (current+previous stay served); security rollback via dbx-revoke / TPM monotonic counter (design §6). - **Drain & retire** an old version only when SPDK/nvmet reports zero connected controllers on it. - **Availability**: run NVMe/TCP target replicas behind distinct discovery entries (design §7) — never serve the whole fleet from one unreplicated node, especially mid-rollout (N versions + reboot bursts). - Note: kiosk *content* changes are server-side (the website) and need no fleet update — only OS/UKI/rootfs changes require a reboot. ## [ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience) There is **no production-grade memory-safe NVMe-oF target** — the Rust NVMe work is all driver/initiator-side (vroom, RedLeaf, Rust-for-Linux); the only production targets are C (Linux kernel `nvmet`, SPDK). So this is **serving-node hardening, not a language swap**. dm-verity already removes the target from the clients' integrity TCB (verify-on-read rejects tampered blocks), so the risk is RCE/DoS on the node that holds the fleet's disk, not client integrity. The export is read-only + single-namespace = inherently small attack surface. - **Evaluate kernel `nvmet` vs SPDK** for the target: kernel `nvmet` keeps the data path in the hardened/fuzzed kernel with a tiny userspace footprint (arguably a better attack-surface story); SPDK is a userspace C daemon (current choice, needs hugepages + caps). Decide per environment. - **Confine the target** (whichever): unprivileged, drop caps, seccomp + SELinux/AppArmor profile, read-only container, minimal mounts; minimal immutable serving image, digest-pinned. - **Slim the SPDK image (multi-stage build)**: the current single-stage from-source image is ~600 MB (it keeps the whole toolchain — gcc, meson/ninja, kernel headers, the python venv — plus the SPDK/DPDK source and build trees). Convert `serving/spdk/Dockerfile` to a builder stage + a minimal runtime stage that copies only `nvmf_tgt` + its `ldd` shared libs (libnuma, libssl/crypto, libuuid, libaio…) + `rpc.py`/`python3`. Expect ~150–250 MB (debian-slim base or a static SPDK link trims further). Smaller attack surface + faster pulls, and it pairs with the confinement above. - **Network-isolate :4420** so only the boot VLAN reaches the target (compose with 802.1X/MACsec segmentation in prod). - **Replicas** behind distinct discovery entries (availability is the real risk — it's the fleet's disk). Ties into #6 (rolling updates) availability. - Revisit a memory-safe target only if/when an off-the-shelf one matures (e.g. a future Rust-for-Linux `nvmet`).