# TODO — diskless fleet boot system

Persistent backlog, kept in sync with the working session task tracker. Status:
`[ ]` pending · `[~]` in progress · `[x]` done.

---

## [ ] 1. Port the dracut module off `network-legacy` → `network-manager`

Durable replacement for the current `dhcp-client` stopgap. `dhclient` is
deprecated in RHEL 9 and will eventually leave the repos; the build only works
today because the provisioner installs `dhcp-client` so `network-legacy` passes
dracut's `check()`.

- Rewrite both network bring-up paths in
  [../files/dracut/95dot1x-macsec/dot1x-macsec-up.sh](../files/dracut/95dot1x-macsec/dot1x-macsec-up.sh):
  the secure path's `ifup macsec0` and the `rd.dot1x=0` bypass path's
  `ifup $NIC` — use `nmcli`/NetworkManager-in-initramfs (or `ip=` cmdline + the
  nm module) instead.
- Change `depends()` `network-legacy` → `network-manager` in
  [../files/dracut/95dot1x-macsec/module-setup.sh](../files/dracut/95dot1x-macsec/module-setup.sh).
- Update `add_dracutmodules` in
  [../files/etc/dracut.conf.d/90-diskless.conf](../files/etc/dracut.conf.d/90-diskless.conf).
- Drop `dhcp-client` from the provisioner `dnf install` in
  [../diskless.pkr.hcl](../diskless.pkr.hcl).
- Re-verify the `lsinitrd` assertion.
- **Gate:** full boot-chain smoke test (swtpm + hostapd + FreeRADIUS) before
  merge, per CLAUDE.md.

## [ ] 2. Apply RHEL9 STIG hardening to the built image

OpenSCAP + SCAP Security Guide (`xccdf_org.ssgproject.content_profile_stig`), run
as a Packer provisioner after `dnf install` and before the dracut/initramfs build
and cleanup.

- **Tailoring** (`autotailor`): unselect rules this architecture can't satisfy —
  separate-partition rules (single erofs root + tmpfs overlay; satisfy intent
  with `nodev,nosuid,noexec` tmpfs mounts), GRUB/kernel-cmdline rules (no GRUB),
  and AIDE/file-integrity rules (dm-verity is the documented compensating
  control).
- **Kernel params → UKI cmdline** in
  [../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh) before
  signing: `audit=1 audit_backlog_limit=8192 slub_debug=P page_poison=1
  vsyscall=none` (+ `fips=1` if FIPS).
- **Reproducibility guard:** strip non-deterministic remediation output (SSH host
  keys → regen at boot into overlay, AIDE db, timestamps, fresh UUIDs) so the
  verity hash stays stable.
- **Audit logging:** `audisp-remote`/rsyslog to a central collector addressed by
  DNS name (host-generic).
- **Optional FIPS:** `fips=1` cmdline + `01fips` dracut module + FIPS crypto
  policy.
- **CI:** run `oscap eval` (no `--remediate`) to publish an ARF+HTML STIG report
  as a build artifact; optionally hard-fail below a score threshold.
- **Decisions needed before implementing:** FIPS on/off · remote-audit collector
  DNS name (or ephemeral/none for lab) · CI fail-on-score threshold vs
  report-only.
- Note: orthogonal to the `rd.dot1x=0` test toggle, but a full STIG production
  posture wants 802.1X re-enabled.
- **Credentials (partly done):** root is now **locked** and access is via a
  baked sudo-enabled key-only admin user (`var.ssh_authorized_key` /
  `var.admin_user`, `wheel` + NOPASSWD sudo), with sshd drop-in
  `PermitRootLogin no` / `PasswordAuthentication no`. Two items for STIG to
  reconcile: **NOPASSWD sudo** is a finding (acceptable for a key-only account,
  but document/deviate), and the admin key/user are **fleet-wide baked** — for
  per-host/rotatable access without rebuilds, layer the signed HTTP/SMBIOS
  config-agent or an SSH CA (`TrustedUserCAKeys`) discussed in session.

## [ ] 3. Roll the errata into design doc v3

Once the boot path is validated and item 1 lands, fold
[design-v2-errata.md](design-v2-errata.md) into the source design document and
re-export a **v3** PDF (key change: `rootovl` → `systemd.volatile=overlay` in §3
step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source
(not in this repo).

## [ ] 4. Kiosk mode — autologin + fullscreen Chromium loading a URL

Make the diskless workstation boot straight into a locked-down kiosk: a
fullscreen Chromium showing a configured site (initial target: Reddit). All
baked at **build time** into the root tree (immutable erofs/verity root; runtime
is the tmpfs overlay), fleet-generic (no per-host state).

- **Graphical stack (keep it minimal):** add a Packer provisioner installing a
  lightweight kiosk compositor + browser. Preferred: `cage` (Wayland
  single-app kiosk compositor) + Mesa (software GL fallback for VMs/no-GPU);
  X11 + a minimal WM (matchbox/openbox) as the alternative. Chromium on Rocky 9
  is in **EPEL** — enabling EPEL adds a repo/supply-chain surface (decision
  below).
- **Autologin + session:** dedicated unprivileged `kiosk` user; launch the
  session via a `kiosk.service` running `cage -s -- chromium --kiosk …`
  (or `greetd` + cage). Stage units/configs under `files/` like the other
  provisioned config.
- **Chromium flags:** `--kiosk --noerrdialogs --disable-infobars --incognito
  --no-first-run --disable-session-crashed-bubble <URL>`. Writable profile/cache
  land on the `systemd.volatile=overlay` tmpfs (RAM-backed, ephemeral) — fine,
  but watch RAM sizing; pin `--user-data-dir`/`--disk-cache-dir` to a tmpfs path
  if needed.
- **Kiosk URL = Packer var** (e.g. `kiosk_url`, default the target site), baked
  into the launch command; keep it host-generic (per-env via DNS if ever needed).
- **Verify in boot test:** graphical session comes up on real/VM display,
  Chromium fullscreen loads the URL over the booted network (works with the
  current `rd.dot1x=0` test path / open network).
- **Decisions:** Wayland(cage) vs X11; accept EPEL for Chromium (vs another
  source); confirm the target URL.
- **⚠ Conflict with item 2 (STIG):** passwordless autologin and a desktop
  browser are STIG findings and enlarge the attack surface/image. Reconcile —
  e.g. a tightly-constrained kiosk session + documented deviation — when
  hardening lands.

## [x] 5. Deploy & validate the end-to-end boot path (open-network test)

✅ **DONE (2026-06-15).** Full chain validated on QEMU/Proxmox: UEFI HTTP boot →
initramfs → DHCP → `nvmf-connect.service` auto-connects the NVMe/TCP subsystem →
two namespaces (data + verity hash) appear by pinned UUID → `systemd-veritysetup`
maps `/dev/mapper/root` → uncompressed erofs mounts → `systemd.volatile=overlay`
→ multi-user. Booted with `rd.dot1x=0` (open network) and the IP discovery
address. See bench findings below for the full list of fixes. Remaining cleanup
to leave the lab-test posture (now unblocked): revert the hardcoded IP to the
`nvme.fleet.lab` DNS name, and re-enable security (`DISABLE_DOT1X=false`) — the
secure path needs the same connect trigger over macsec0 (see ⚠ note below).

Original task: stand up `serving/` (dnsmasq + nginx + SPDK) per
[serving/README.md](../serving/README.md), rebuild the UKI with
`nvmet_traddr=nvme.fleet.lab` (now a Packer var — older UKIs have the literal
`NVMET_TRADDR` placeholder and won't connect), then boot a UEFI HTTP-boot client
(**Secure Boot OFF**, unsigned UKI; no TPM needed in `rd.dot1x=0` mode) on the
boot LAN.

- **Pass criteria:** UKI fetched over HTTP → initramfs → DHCP on NIC → NVMe/TCP
  namespace connected → dm-verity root mapped → `systemd.volatile=overlay`
  writable → multi-user.
- This validation **gates** item #1 (network-manager port) and item #3
  (errata → design v3).
- **Bench findings (boot debug, 2026-06-14):**
  - Initramfs has no resolver (dracut writes `resolv.conf` only via its own
    DHCP path, not a manual `dhclient`), so the NVMe discovery address must be
    an **IP**, not a DNS name — build with `-var nvmet_traddr=192.168.50.10`.
    Old UKIs carrying `nvme.fleet.lab` can't connect early in boot.
  - NVMe controller indices are **not stable**: a transient discovery
    controller can take `nvme0`, pushing the I/O controller to `nvme1`. The
    hardcoded `/dev/nvme0n1` in the cmdline was a latent brick. **Fixed:**
    namespace UUID pinned on the target (`spdk-setup.sh` `add_ns -u
    f1ee7000-0000-4000-8000-000000000001`) and the cmdline now references
    `/dev/disk/by-id/nvme-uuid.<uuid>` (`extract-artifacts.sh`). The two
    constants MUST stay in sync; make them version-specific for #6.
  - `nvme_tcp`/`nvme_fabrics` must be loaded before `/dev/nvme-fabrics` exists
    (dracut's `nvmf` module does this in a normal boot; at `rd.break` modprobe
    manually).
  - **Don't double-configure the NIC in test mode.** With `ip=dhcp` on the
    cmdline, dracut's network module brings the NIC up AND emits the "online"
    event that the `nvmf` module's `initqueue/online` hook waits on to run
    `nvme connect-all`. The `rd.dot1x=0` bypass used to *also* `ifup $NIC`,
    which (a) raced dracut → `Error: ipv4: Address already assigned`, and (b)
    didn't fire the nvmf online hook, so the namespace never auto-connected and
    boot spun on `/dev/mapper/root` forever (manual `connect-all` worked, which
    is what masked it). **Fixed:** the bypass branch no longer touches the NIC;
    `ip=dhcp` owns networking. (The secure path still addresses macsec0 itself —
    dracut doesn't know about that interface.)
  - **dracut's `nvmf` module never auto-connects on this distro.** Its cmdline
    hook (`92-parse-nvmf-boot-connections.sh`) parses `rd.nvmf.discover` into
    `/etc/nvme/discovery.conf` correctly, but `initqueue/online/` is **empty** —
    no autoconnect hook is registered, so `nvme connect-all` is never run (no
    attempt in dmesg) and the namespace never appears. Drivers load and
    `/dev/nvme-fabrics` exists; only the connect trigger is missing. A job
    placed on the *main* initqueue from a settled hook gets registered but is
    **never re-executed** once the loop blocks on udev events (verified: job
    present, manual `connect-all` works, namespace stays absent). An
    `initqueue/online` hook also does NOT work: this initramfs is **systemd-
    driven** ("A start job is running for /dev/mapper/root") and never sources
    dracut online hooks on link-up (verified: hook installs as
    `90-nvmf-connect.sh` but never fires — no log, no stamp). **Fixed:** the
    connect is now a **systemd unit** (`nvmf-connect.service`,
    `ConditionKernelCommandLine=rd.dot1x=0`, `Before=initrd-root-device.target`)
    whose ExecStart (`/usr/sbin/nvmf-connect`) polls for the network then runs
    `nvme connect-all` until the namespace appears. The dracut module installs
    the unit + script and enables it via an `initrd-root-device.target.wants`
    symlink; `diskless.pkr.hcl` hard-asserts the unit is in the initramfs.
    Gotcha: `$systemdsystemunitdir` is **empty** in this dracut's module-setup
    context — `inst_simple` then silently drops the unit (empty dest dir). Use a
    `${systemdsystemunitdir:-/usr/lib/systemd/system}` fallback and `return 1`
    if the unit didn't land, so the build fails loudly instead of shipping a UKI
    that can't connect its root.
    Second gotcha: the build assertion `lsinitrd … | grep -q 'nvmf-connect.service'`
    was a **false negative** — `grep -q` exits on first match, closes the pipe,
    `lsinitrd` dies of SIGPIPE, and under `pipefail` the pipeline reports failure
    even though the unit IS present. This masked the fact that the systemd unit
    install was working all along. Fix: `lsinitrd > file; grep -q … file` (no
    pipe). Never `lsinitrd | grep -q` under pipefail.
    Bonus: the survivor listing confirmed nvme-cli's native autoconnect bits ARE
    in the initramfs (`/usr/sbin/nvmf-autoconnect.sh`,
    `/etc/udev/rules.d/95-nvmf-initqueue.rules`) — so the fully-native
    `rd.nvmf.discover` autoconnect path is available as a fallback if the unit
    doesn't fire on boot.
  - **dm-verity: `hash-offset` is unsupported → two namespaces.** With NVMe
    connecting and the right by-id device present, verity still failed:
    `systemd-veritysetup` logged `unknown option 'hash-offset=…'` then `Failed
    to load verity superblock` — exactly the CLAUDE.md bench-verify item. The
    in-file (data+hash in one namespace, split by offset) layout is not
    supported by Rocky 9.8 systemd. **Fixed** via the documented fallback:
    `extract-artifacts.sh` now emits `rootfs.erofs` (data) + `rootfs.hash` (hash
    tree) as separate files, served as **two NVMe namespaces** with distinct
    pinned UUIDs (`f1ee7000-…-01` data, `…-02` hash); the cmdline uses
    `systemd.verity_root_data=`/`systemd.verity_root_hash=` by-id devices with no
    offset. Touches `extract-artifacts.sh`, `build-oci.sh` (artifact list +
    SHA256SUMS), `serving/spdk-setup.sh` (2nd bdev/namespace), and invariant #9.
  - **SELinux labels must survive into the erofs.** It booted to multi-user but
    `root`/`packer-build-only` login was accepted then the session was killed
    instantly. Cause: kickstart sets `selinux --enforcing`, but the build dropped
    `security.selinux` xattrs (`virt-tar-out` + plain `tar -xpf` don't carry
    them), so the enforcing target denied the login session on an unlabeled root
    — and a read-only erofs can't autorelabel. Confirmed by booting with
    `enforcing=0` (SMBIOS-injected) → login worked. **Fixed:** export via
    `guestfish tar-out … selinux:true xattrs:true acls:true`, restore with
    `tar --xattrs --xattrs-include='*' --acls --selinux`, and a `getfattr`
    build-time assertion that `/usr/bin/bash` is labeled before `mkfs.erofs`
    (mkfs.erofs preserves xattrs). Needs `attr` on the build host. The image now
    boots **enforcing**.
    **⚠ The secure path has the same dependency on the
    (absent) nvmf autoconnect** — when 802.1X is re-enabled, the macsec0 path
    must likewise drive `connect-all` (or the network-manager port #1 must wire
    a working autoconnect). Currently only the test path is fixed.
  - The target Rocky 9 kernel's erofs is built **without
    `CONFIG_EROFS_FS_ZIP`** — mounting a compressed image fails `erofs: lz4
    algorithm isn't enabled`, which silently blocks the root mount. **Fixed:**
    `extract-artifacts.sh` now builds the erofs **uncompressed** (`mkfs.erofs`
    without `-z`). Tradeoff: larger rootfs namespace + more NVMe/TCP bandwidth
    per read. Re-enable compression only against a kernel verified to have
    `EROFS_FS_ZIP` (`grep CONFIG_EROFS /boot/config-$(uname -r)`). This is the
    CLAUDE.md bench-verify item ("erofs module availability on the target RHEL
    kernel"). Larger image size also feeds the "shrink the rootfs" angle of #7.
  - SPDK: add the listener to the discovery subsystem explicitly
    (`nqn.2014-08.org.nvmexpress.discovery`) to silence the deprecation warning.

## [ ] 6. Rolling fleet updates (multi-version serve → cohort reboots)

Image-based, reboot-to-update across many live workstations. Hosts are immutable
(read-only erofs/verity root over NVMe/TCP, ephemeral overlay), so updates are
**atomic A/B reboots**, never in-place. The released OCI artifact + digest-pinned
manifest + roothash coupling already give the atomic unit and instant rollback;
this item builds the rollout machinery around it.

- **Serve versions side-by-side** (must not overwrite a live rootfs): extend
  `serving/` so SPDK/nvmet exposes each release as its **own subsystem/NQN**
  (`nqn…:rootfs-<ver>`) and nginx serves multiple `fleet-<ver>.efi` (keep
  current + previous). Make the UKI's NQN version-specific so each signed UKI
  connects only its matching namespace (constant discovery DNS name).
- **Version selection at boot**: dnsmasq **cohort tagging** (MAC / vendor class /
  assigned tag) hands each cohort a different option-67 UKI URL. Moving a
  cohort's URL = assigning the release.
- **Control plane + in-image reboot agent**: a baked-in systemd timer polls a
  "desired version" endpoint and reboots **when idle / in a maintenance window**
  (or out-of-band via Redfish/BMC). Stagger + jitter reboots to avoid UKI-fetch
  / NVMe-connect storms.
- **Staged rollout**: canary → waves (10/50/100%) with health gates
  (multi-user reached, NVMe connected, dm-verity mapped, kiosk URL rendered,
  prod: MACsec/802.1X up); auto-halt on threshold.
- **Rollback**: repoint cohort URL to previous UKI + reboot (current+previous
  stay served); security rollback via dbx-revoke / TPM monotonic counter
  (design §6).
- **Drain & retire** an old version only when SPDK/nvmet reports zero connected
  controllers on it.
- **Availability**: run NVMe/TCP target replicas behind distinct discovery
  entries (design §7) — never serve the whole fleet from one unreplicated node,
  especially mid-rollout (N versions + reboot bursts).
- Note: kiosk *content* changes are server-side (the website) and need no fleet
  update — only OS/UKI/rootfs changes require a reboot.

## [ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience)

There is **no production-grade memory-safe NVMe-oF target** — the Rust NVMe work
is all driver/initiator-side (vroom, RedLeaf, Rust-for-Linux); the only
production targets are C (Linux kernel `nvmet`, SPDK). So this is **serving-node
hardening, not a language swap**. dm-verity already removes the target from the
clients' integrity TCB (verify-on-read rejects tampered blocks), so the risk is
RCE/DoS on the node that holds the fleet's disk, not client integrity. The
export is read-only + single-namespace = inherently small attack surface.

- **Evaluate kernel `nvmet` vs SPDK** for the target: kernel `nvmet` keeps the
  data path in the hardened/fuzzed kernel with a tiny userspace footprint
  (arguably a better attack-surface story); SPDK is a userspace C daemon
  (current choice, needs hugepages + caps). Decide per environment.
- **Confine the target** (whichever): unprivileged, drop caps, seccomp +
  SELinux/AppArmor profile, read-only container, minimal mounts; minimal
  immutable serving image, digest-pinned.
- **Slim the SPDK image (multi-stage build)**: the current single-stage
  from-source image is ~600 MB (it keeps the whole toolchain — gcc, meson/ninja,
  kernel headers, the python venv — plus the SPDK/DPDK source and build trees).
  Convert `serving/spdk/Dockerfile` to a builder stage + a minimal runtime stage
  that copies only `nvmf_tgt` + its `ldd` shared libs (libnuma, libssl/crypto,
  libuuid, libaio…) + `rpc.py`/`python3`. Expect ~150–250 MB (debian-slim base
  or a static SPDK link trims further). Smaller attack surface + faster pulls,
  and it pairs with the confinement above.
- **Network-isolate :4420** so only the boot VLAN reaches the target (compose
  with 802.1X/MACsec segmentation in prod).
- **Replicas** behind distinct discovery entries (availability is the real risk
  — it's the fleet's disk). Ties into #6 (rolling updates) availability.
- Revisit a memory-safe target only if/when an off-the-shelf one matures
  (e.g. a future Rust-for-Linux `nvmet`).