CLAUDE.md — Diskless Fleet Boot System

This repo builds the boot system for a fleet of diskless hosts: PXE/HTTP-booted, authenticating to the network with 802.1X EAP-TLS using a TPM-held key, running MACsec on the access link, and mounting an immutable erofs + dm-verity root over NVMe/TCP. Each release is a signed UKI plus root image packaged as a FROM-scratch OCI artifact image. The authoritative architecture document is docs/diskless-fleet-design-v2.pdf — read it before structural changes. docs/design-v2-errata.md records where the as-built RHEL/Rocky 9 implementation diverges from it (pending a consolidated v3).

Backlog (read at session start)

docs/TODO.md is the persistent backlog and the durable source of truth for outstanding work — it survives across sessions (the TodoWrite task tracker does not). At the start of a session, read it and mirror its open items into the TodoWrite tracker. Whenever items change (added, started, completed), update both docs/TODO.md and the tracker so they stay in sync.

Repo layout

diskless.pkr.hcl                          Packer template (qemu builder)
http/ks.cfg                               Kickstart for the throwaway build VM
files/dracut/95dot1x-macsec/              Custom dracut module (the heart of the system)
files/etc/dot1x/                          Supplicant config + RADIUS CA (placeholder)
files/etc/dracut.conf.d/90-diskless.conf  no-hostonly + drivers + modules
                                          (NM unmanaged split is now runtime:
                                          95dot1x-macsec/nm-unmanaged.sh, secure-mode only)
files/plymouth/themes/fleet/              Custom boot splash (scripted security
                                          checklist; driven by hook status tokens)
files/kiosk/                              weston+Chromium kiosk + chromium lockdown policy
scripts/extract-artifacts.sh              qcow2 -> kernel/initramfs/erofs+verity/cmdline
scripts/build-oci.sh                      ukify + sbsign + roothash gate + OCI image
scripts/build-ipxe-oci.sh                 sbsign snponly.efi -> FROM-scratch fleet-ipxe image
serving/                                  Lab serving stack: dnsmasq + nginx + SPDK
                                          (dnsmasq + fleet-ipxe + SPDK images are
                                          CI-built + pulled; see .gitea/workflows/)
serving/spdk/                             LONG-LIVED nvmf_tgt: spdk-target.sh
                                          (persistent entrypoint, never restarts
                                          for a version change) + spdk-release.sh
                                          (online add/remove/drain a version, 1
                                          subsystem/version) + spdk-ctl.sh (host
                                          load/unload/list for rolling updates).
                                          releases/<ver>/ = staged rootfs+verity.json.
                                          spdk-setup.sh (parent dir) is the legacy
                                          single-version entrypoint (deprecated).
serving/kiosk-app/                         Svelte+Tailwind spaceship-terminal SPA
                                          — GIT SUBMODULE (own repo + CI; builds
                                          the fleet-kiosk image the stack pulls).
                                          OS-side kiosk runtime stays in files/kiosk/
serving/dot1x-bench/                      Secure-path bench: step-ca + FreeRADIUS +
                                          hostapd authenticator + TPM enrolment
serving/siem/                             Wazuh SIEM overlay (audit analysis;
                                          agentless relay from audit-collector)
.gitea/workflows/                         Gitea CI: lint -> packer build -> push OCI
docs/                                     Design PDF and supporting notes

Build & test commands

packer init .
packer validate .                         # always before committing template changes
packer build -var image_version=X.Y.Z \
  -var nic_drivers="ice mlx5_core" \
  -var oci_image=<registry>/fleet-boot .  # add sb_key/sb_cert for signed builds
                                          # add disable_dot1x=true for an
                                          # open-port test UKI (rd.dot1x=0)
bash -n files/dracut/95dot1x-macsec/*.sh scripts/*.sh   # shell syntax gate
# .shellcheckrc disables the dracut false positives; CI passes them explicitly:
shellcheck --exclude=SC1091,SC2154 files/dracut/95dot1x-macsec/*.sh scripts/*.sh

Build-host deps: qemu-kvm, libguestfs-tools (+ a kernel for its supermin appliance), erofs-utils, cryptsetup, jq, systemd-ukify and systemd-boot-efi (the latter ships the linuxx64.efi.stub ukify needs), sbsigntools, buildah (or docker), attr (getfattr — extract-artifacts.sh asserts the SELinux labels survived the export; an unlabeled enforcing root denies login).

Architecture invariants — do not violate these

These encode failure modes that brick the fleet. Treat them as hard constraints; if a task appears to require breaking one, stop and flag it instead of working around it.

No per-host state in any artifact, ever. No hostnames, keys, certs, MACs, or per-host config in the initramfs, rootfs, UKI, or OCI image. All host identity derives from the TPM at runtime (key at persistent handle 0x81010001, cert at NV index 0x01800001, hostnqn derived from the cert serial). If a change needs per-host data, it belongs in the enrolment flow, not the image.
The hook ordering in dot1x-macsec-up.sh is load-bearing. NV-cert extraction → supplicant start → wait for Authorized + macsec0 → DHCP on macsec0 → hostnqn derivation. Never DHCP before auth completes (pre-auth leases are invalid after VLAN assignment); never connect NVMe before macsec0 is addressed.
The supplicant must survive switch-root. It is launched with exec -a @wpa_supplicant (systemd root-storage-daemon convention) and its state lives under /run. Do not “clean up” the @, move state off /run, or hand the physical NIC to NetworkManager — MKA times out in ~6 s without keepalives and the root filesystem link dies. The NM unmanaged split exists for this reason (see below); keep it in sync with interface naming.
NM owns IP/DHCP; wpa_supplicant owns the physical NIC. The initramfs uses the network-manager dracut module (not network-legacy) so NM renews the DHCP lease in the running system after switch-root — without this the one-shot lease expired, the address was dropped, and the NVMe/TCP root died (“keep alive timeout” / “no route to host”). The managed/unmanaged split is therefore load-bearing and mode-aware:
- Secure path: IP rides on macsec0, which must stay MANAGED by NM. The physical NIC (en*) must stay UNMANAGED (wpa_supplicant owns EAPOL+MKA on it). This is enforced at runtime by the cmdline-stage hook nm-unmanaged.sh, which writes /run/NetworkManager/conf.d/10-dot1x-unmanaged.conf before NM starts.
- Test path (rd.dot1x=0): no macsec0; the physical NIC carries IP and must be MANAGED so NM renews it — so the hook writes nothing. There is intentionally no static 99-dot1x-unmanaged.conf (it couldn't be mode-aware — it wrongly unmanaged en* in test mode, blocking renewal). Never add macsec0 back to an unmanaged list, and never statically unmanage en* for all modes.
roothash coupling is a hard gate. build-oci.sh refuses to build if the roothash= in the UKI cmdline ≠ verity.json. Never weaken, skip, or warn-instead-of-fail this check: a mismatched UKI/rootfs pair is an unbootable fleet.
The UKI cmdline is immutable once signed. Any cmdline change (including NVMET_TRADDR) requires a rebuild + re-sign. Prefer a per-environment DNS name for NVMe discovery over per-environment builds.
Signing keys never enter the build VM or this repo. sb_key / sb_cert are CI/HSM-provided paths on the build host. Never commit key material, never copy keys into provisioner steps, never echo them in logs. Unsigned builds are lab-only and must keep their loud warning.
Reproducibility of the root image is a security property. mkfs.erofs runs with fixed UUID and -T 0; the verity root hash must be a pure function of the input tree. Don’t introduce timestamps, random seeds, or build-host paths into the rootfs.
The initramfs is built --no-hostonly and contents are asserted. The build VM is virtio-only; real NIC drivers, macsec, nvme-tcp, erofs, and veritysetup are explicitly listed in 90-diskless.conf and checked via lsinitrd | grep in the template. When adding a runtime dependency to the hook, add it to the dracut module’s install() and extend the lsinitrd assertion.
The OCI artifact image stays FROM scratch. Only /artifacts/{fleet-<ver>.efi, rootfs.erofs, rootfs.hash, verity.json, SHA256SUMS}. rootfs.erofs (erofs data) and rootfs.hash (dm-verity hash tree) are separate namespaces because the as-built Rocky 9 systemd does not support the in-file hash-offset verity layout. No shell, no tools, no base image. Serving software (SPDK, nginx, dnsmasq) lives in separate, independently versioned images — never merge them into the artifact image. The signed iPXE bootloader (snponly.efi) is its own FROM-scratch fleet-ipxe image (scripts/build-ipxe-oci.sh, signed with db), mounted into dnsmasq at /ipxe — kept out of both the artifact image and the (generic, key-agnostic) dnsmasq image because its signature is per-Secure-Boot-environment. Don't bake snponly.efi into the dnsmasq image.
Deployments are pinned by digest. One commit per fleet release in the deployment manifest; that manifest is the single source coupling UKI ↔ namespace. Rollback = revert the commit.

Things that look like bugs but aren’t

identity="fleet-host" in the supplicant config is intentionally generic; RADIUS derives real identity from the certificate CN/SAN.
/etc/machine-id is truncated empty in the image; each host generates its own into the tmpfs overlay at boot.
/etc/fstab is effectively empty; root comes from netroot + verity.
The build VM has no TPM. Correct — nothing host-specific exists at build time. CI boot tests use swtpm.
files/etc/dot1x/radius-ca.pem is a placeholder; builds succeed with it but authentication cannot. Replace per environment, don’t commit real internal CA material unless the repo’s classification allows it.

Known bench-verify items (check before trusting a change in production)

TPM key reference method in wpa_supplicant: private_key="handle:..." via OpenSSL tpm2 provider vs tpm2-pkcs11 URI — depends on the distro build; the supplicant config carries both options.
erofs module availability on the target RHEL kernel (fallback: journal-less ext4 in extract-artifacts.sh).
STIG remediation: (a) datastream must match the distro. The provisioner maps /etc/os-release $ID → datastream (Rocky→ssg-rl9-ds.xml, RHEL→ssg-rhel9-ds.xml, al9/cs9/ol9). Do NOT run the RHEL9 datastream on Rocky — its CPE dictionary has no Rocky entry, so every rule returns notapplicable, the scan evaluates 0 rules, remediation does nothing, and the report is falsely clean. (Seen 2026-06-18: a build picked rhel9 via ls … | head -1 and produced 462 notapplicable + 1068 notselected, 0 pass/fail.) Two guards now exist: the provisioner aborts the build if 0 rules are evaluated, and stig-deviations.sh flags an INVALID SCAN + exits 2. Also confirm the ...content_profile_stig profile id exists in the chosen datastream; (b) remediation must not break boot/login (NM/NVMe, key-only sshd, kiosk, SELinux enforcing) — enable_stig=false is the escape hatch; (c) reproducibility: build twice and confirm identical root_hash in verity.json — if it drifts, oscap wrote something nondeterministic (timestamp/uuid/key); find + normalize it in 2d hygiene; (d) read the generated stig-deviations.md — every REVIEW row is a real finding to triage or justify in docs/stig-deviations.md. STIG-breaks-login gotchas already hit + fixed (2026-06-19) — don't regress: (i) fapolicyd blocks non-RPM executables → the kiosk's /usr/local/bin/kiosk-session must be in fapolicyd's trust (done in 2c-STIG-postfix); keep fapolicyd, don't waive it (it's the writable-overlay exec guard dm-verity can't provide). (ii) Admin SSH key must be ECDSA/RSA, NOT ed25519 — the STIG crypto policy drops ed25519 from sshd's accepted pubkey algorithms (server-sig-algs shows ecdsa+rsa-sha2 only), so an ed25519 key gets Permission denied (publickey) at the signature stage. (iii) passwordless/key-only appliance accounts (kiosk, fleetadmin) are exempted from STIG password-aging + faillock account phase in 2c-STIG-postfix (else locked out). (iv) the kiosk unit is WantedBy=multi-user.target (not graphical.target, which the host never reaches → unit dead at boot). Offline diagnosis without booting: mount the OCI image's rootfs.erofs read-only and inspect /etc/pam.d, fapolicyd, sshd config, etc. REVIEW-row triage (2026-06-20, scan app-9f7d8d1) added two more bench gates — don't regress: (v) USBGuard is ENABLED + enforcing (block target) with a hardware-independent HID/hub allowlist + AuditBackend=LinuxAudit (2c-STIG). The generated policy is built on the virtio build VM (no real USB), so without the HID/hub allow rules a physical console keyboard/mouse is blocked on real hardware — BENCH-VERIFY console input. (vi) admin sudo SELinux elevation: the admin is mapped to staff_u (semanage login) + sudoers TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL so selinux_context_elevation_for_sudo passes; staff_u (not sysadm_u) so SSH login isn't gated by ssh_sysadm_login. BENCH-VERIFY admin SSH login + sudo + privileged ops under confinement, and re-check reproducibility (semanage rebuilds the policy store → possible verity-hash drift). The triage waivers/decisions live in the tailoring + docs/stig-deviations.md #12; the generated report should show 0–1 REVIEW.
NTP via DHCP option 42 → chrony (no public-pool egress): confirm on a booted host chronyc sources lists the DHCP-handed lab server (not *.pool.ntp.org). Assumes (a) NM exports the lease's NTP servers to the dispatcher as $DHCP4_NTP_SERVERS, and (b) RHEL 9 /etc/chrony.conf honours sourcedir /run/chrony-dhcp (the provisioner appends it if absent). If sources are empty, the dispatcher didn't fire or the env var name differs — check journalctl -u NetworkManager and ls /run/chrony-dhcp/.
fleet Plymouth theme: confirm (a) the script plugin package actually installed (/usr/lib64/plymouth/script.so — else the build fell back to spinner, logged as a WARNING), (b) the scripted theme + a font got bundled into the initramfs (lsinitrd | grep -E 'fleet.script|script.so'), and (c) it renders (text via Image.Text needs the label plugin + a font in the early initramfs). If blank, the splash silently no-ops but boot continues; ESC shows the log. RENDER-VERIFIED 2026-06-18 (test-path build): the scripted theme draws and the handshake checklist updates from the hook status tokens. Still unverified on the secure path (the auth/macsec rows reaching done — see #8 bench items).
systemd.verity_root_options=hash-offset is NOT supported by Rocky 9.8 systemd-veritysetup (verified: "unknown option hash-offset=" → "Failed to load verity superblock"). Resolved by the documented fallback — data and hash are two separate NVMe namespaces (rootfs.erofs + rootfs.hash, distinct pinned UUIDs), so the cmdline uses plain systemd.verity_root_data= / systemd.verity_root_hash= by-id devices with no offset.
Network module: network-manager (ported 2026-06-17 from network-legacy). network-legacy's one-shot DHCP lease was never renewed after switch-root, so at lease expiry the address was dropped and the NVMe/TCP root died. The network-manager module hands its initramfs connections to the real-root NM (state under /run/NetworkManager), which renews the lease. No more dhcp-client/dhclient (deprecated). The hook drives macsec0 DHCP via nmcli (not ifup). See invariant on the managed/unmanaged split. Test path VERIFIED on the bench 2026-06-17 (lease renews past expiry; host stays up). STILL BENCH-VERIFY when disable_dot1x=false: macsec0 managed + en* unmanaged with MKA surviving switch-root (secure path).
Writable root overlay is systemd.volatile=overlay, not rootovl: RHEL/Rocky dracut ships no overlayfs/rootovl module. Needs systemd-volatile-root.service in the initramfs (bundled by recent dracut; install_items it if the root stays read-only). Misbehaves with NFS root — N/A here (NVMe block root).

RHEL 9.x build notes (learned — don't re-discover)

These bit us bringing the build up on Rocky 9.8; they are easy to regress.

ISO: default is a version-pinned Rocky 9.x minimal ISO on the internal mirror (iso_url + sha256 in iso_checksum; bump both together on a point release). http/ks.cfg installs only @^minimal-environment+sshd+chrony from the ISO; the diskless tooling is pulled online by the provisioner, so the build VM needs outbound repo access. Keeps the build ISO-variant-agnostic.
%{...} in provisioner shell (e.g. rpm --qf '%{VERSION}') must be escaped %%{...} — Packer's HCL2 engine parses a bare %{ as a template directive and packer validate fails otherwise.
File provisioner staging: pre-create the destination dir (mkdir -p /tmp/etc-stage) before a trailing-slash contents upload; Packer requires the dest to exist or the tree lands elsewhere and later install steps fail.
dracut module name in add_dracutmodules omits the NN dir-prefix: the directory is 95dot1x-macsec but the module is dot1x-macsec (cf. nvmf).
network-manager dracut module (was network-legacy+dhcp-client) and overlay → systemd.volatile=overlay: see the bench-verify items above.
Test toggle: disable_dot1x=true bakes rd.dot1x=0; the hook then skips 802.1X/MACsec and DHCPs the NIC directly for NVMe-oF on an open port. The auth stack stays in the image, dormant. The matching lab DHCP/HTTP/NVMe-oF serving stack lives in serving/ (docker compose). Re-enable security by dropping the var / setting DISABLE_DOT1X: "false" in the CI workflow.
STIG toggle: enable_stig=true (default) applies DISA STIG via OpenSCAP/SSG in a provisioner, tailored for the diskless/UKI/kiosk architecture (files/stig/diskless-tailoring.xml unselects the N/A or dangerous-to-remediate rules; every waiver is justified in docs/stig-deviations.md). STIG kernel params ride the UKI cmdline (extract-artifacts.sh, gated by the same flag) since there's no GRUB. FIPS is OFF and audit offload is LOCAL-only (decisions). CI is report-only: it publishes stig-report.html + stig-arf.xml + stig-deviations.md (generated by scripts/stig-deviations.sh) as artifacts and never fails on score. The scanner tooling is removed from the shipped image (2d hygiene). enable_stig=false is the escape hatch if hardening regresses the boot/secure-path bench. NEVER let a waiver hide a real finding — anything non-pass that isn't in the register shows as REVIEW in the report.
Boot splash toggle: boot_splash=true (default) bakes rhgb quiet for the Plymouth splash; false bakes plymouth.enable=0 for a verbose console boot — keep OFF while debugging the boot chain. Plymouth is always in the image + initramfs (the plymouth dracut module, which pulls drm); only the cmdline changes. Press ESC at boot to see details behind a splash. ks.cfg no longer excludes plymouth.
Splash theme = custom fleet (files/plymouth/themes/fleet/, script plugin, asset-free — drawn with Image.Text). It renders a security-handshake checklist (TPM → 802.1X → MACsec → Network → NVMe-oF) driven by best-effort plymouth update --status=<token> calls from the dracut hooks (dot1x-macsec-up.sh, nvmf-connect.sh). Tokens: mode-test (marks auth/MACsec skipped), tpm|dot1x|macsec|net|nvme-active|done. Rules: the emits are non-fatal (plymouth … || true) and must never reorder the load-bearing boot sequence; show only STATES, never per-host data (invariant #1). The provisioner installs the script plugin best-effort and falls back to the stock spinner theme if script.so is absent, so a plugin-package-name change can't break the build.

Testing changes

Full boot-chain smoke test runs in QEMU/Proxmox: swtpm (test key at the fleet handle, test cert in NV) + hostapd as wired 802.1X/MACsec authenticator + FreeRADIUS on a veth/bridge (bridge needs group_fwd_mask=8 or EAPOL is silently dropped). Pass criteria: port Authorized, macsec0 addressed, NVMe namespace connected, verity root mapped, overlay writable, supplicant alive after switch-root. This harness is scaffolded in serving/dot1x-bench/ (step-ca as the fleet CA on the serving host + FreeRADIUS EAP-TLS + a hostapd wired authenticator + tpm-provision.sh), with an isolated bridge per workstation (the flat VLAN can't authenticate — EAPOL is dropped and MACsec is point-to-point). See its README; several configs there are still BENCH-VERIFY.

Any change to the dracut module, supplicant config, or cmdline must pass this before merge. Template-only changes need packer validate + shellcheck at minimum.

Conventions

Shell: bash, set -euo pipefail in scripts (dracut hooks source dracut-lib instead and must stay POSIX-ish bash compatible with the initramfs environment — no GNU-only tools not installed by the module).
Keep comments that explain why (timing constraints, kernel behaviours); they encode bench findings that are expensive to rediscover.
Versioning: semver on image_version; the OCI tag matches; the git tag matches; one release == one digest == one manifest commit.

22 KiB Raw Blame History Unescape Escape