CLAUDE.md — Diskless Fleet Boot System

This repo builds the boot system for a fleet of diskless hosts: PXE/HTTP-booted, authenticating to the network with 802.1X EAP-TLS using a TPM-held key, running MACsec on the access link, and mounting an immutable erofs + dm-verity root over NVMe/TCP. Each release is a signed UKI plus root image packaged as a FROM-scratch OCI artifact image. The authoritative architecture document is docs/diskless-fleet-design-v2.pdf — read it before structural changes. docs/design-v2-errata.md records where the as-built RHEL/Rocky 9 implementation diverges from it (pending a consolidated v3).

Backlog (read at session start)

docs/TODO.md is the persistent backlog and the durable source of truth for outstanding work — it survives across sessions (the TodoWrite task tracker does not). At the start of a session, read it and mirror its open items into the TodoWrite tracker. Whenever items change (added, started, completed), update both docs/TODO.md and the tracker so they stay in sync.

Repo layout

diskless.pkr.hcl                          Packer template (qemu builder)
http/ks.cfg                               Kickstart for the throwaway build VM
files/dracut/95dot1x-macsec/              Custom dracut module (the heart of the system)
files/etc/dot1x/                          Supplicant config + RADIUS CA (placeholder)
files/etc/dracut.conf.d/90-diskless.conf  no-hostonly + drivers + modules
                                          (NM unmanaged split is now runtime:
                                          95dot1x-macsec/nm-unmanaged.sh, secure-mode only)
files/plymouth/themes/fleet/              Custom boot splash (scripted security
                                          checklist; driven by hook status tokens)
files/kiosk/                              weston+Chromium kiosk + chromium lockdown policy
scripts/extract-artifacts.sh              qcow2 -> kernel/initramfs/erofs+verity/cmdline
scripts/build-oci.sh                      ukify + sbsign + roothash gate + OCI image
serving/                                  Lab serving stack: dnsmasq + nginx + SPDK
.gitea/workflows/                         Gitea CI: lint -> packer build -> push OCI
docs/                                     Design PDF and supporting notes

Build & test commands

packer init .
packer validate .                         # always before committing template changes
packer build -var image_version=X.Y.Z \
  -var nic_drivers="ice mlx5_core" \
  -var oci_image=<registry>/fleet-boot .  # add sb_key/sb_cert for signed builds
                                          # add disable_dot1x=true for an
                                          # open-port test UKI (rd.dot1x=0)
bash -n files/dracut/95dot1x-macsec/*.sh scripts/*.sh   # shell syntax gate
# .shellcheckrc disables the dracut false positives; CI passes them explicitly:
shellcheck --exclude=SC1091,SC2154 files/dracut/95dot1x-macsec/*.sh scripts/*.sh

Build-host deps: qemu-kvm, libguestfs-tools (+ a kernel for its supermin appliance), erofs-utils, cryptsetup, jq, systemd-ukify and systemd-boot-efi (the latter ships the linuxx64.efi.stub ukify needs), sbsigntools, buildah (or docker), attr (getfattr — extract-artifacts.sh asserts the SELinux labels survived the export; an unlabeled enforcing root denies login).

Architecture invariants — do not violate these

These encode failure modes that brick the fleet. Treat them as hard constraints; if a task appears to require breaking one, stop and flag it instead of working around it.

No per-host state in any artifact, ever. No hostnames, keys, certs, MACs, or per-host config in the initramfs, rootfs, UKI, or OCI image. All host identity derives from the TPM at runtime (key at persistent handle 0x81010001, cert at NV index 0x01800001, hostnqn derived from the cert serial). If a change needs per-host data, it belongs in the enrolment flow, not the image.
The hook ordering in dot1x-macsec-up.sh is load-bearing. NV-cert extraction → supplicant start → wait for Authorized + macsec0 → DHCP on macsec0 → hostnqn derivation. Never DHCP before auth completes (pre-auth leases are invalid after VLAN assignment); never connect NVMe before macsec0 is addressed.
The supplicant must survive switch-root. It is launched with exec -a @wpa_supplicant (systemd root-storage-daemon convention) and its state lives under /run. Do not “clean up” the @, move state off /run, or hand the physical NIC to NetworkManager — MKA times out in ~6 s without keepalives and the root filesystem link dies. The NM unmanaged split exists for this reason (see below); keep it in sync with interface naming.
NM owns IP/DHCP; wpa_supplicant owns the physical NIC. The initramfs uses the network-manager dracut module (not network-legacy) so NM renews the DHCP lease in the running system after switch-root — without this the one-shot lease expired, the address was dropped, and the NVMe/TCP root died (“keep alive timeout” / “no route to host”). The managed/unmanaged split is therefore load-bearing and mode-aware:
- Secure path: IP rides on macsec0, which must stay MANAGED by NM. The physical NIC (en*) must stay UNMANAGED (wpa_supplicant owns EAPOL+MKA on it). This is enforced at runtime by the cmdline-stage hook nm-unmanaged.sh, which writes /run/NetworkManager/conf.d/10-dot1x-unmanaged.conf before NM starts.
- Test path (rd.dot1x=0): no macsec0; the physical NIC carries IP and must be MANAGED so NM renews it — so the hook writes nothing. There is intentionally no static 99-dot1x-unmanaged.conf (it couldn't be mode-aware — it wrongly unmanaged en* in test mode, blocking renewal). Never add macsec0 back to an unmanaged list, and never statically unmanage en* for all modes.
roothash coupling is a hard gate. build-oci.sh refuses to build if the roothash= in the UKI cmdline ≠ verity.json. Never weaken, skip, or warn-instead-of-fail this check: a mismatched UKI/rootfs pair is an unbootable fleet.
The UKI cmdline is immutable once signed. Any cmdline change (including NVMET_TRADDR) requires a rebuild + re-sign. Prefer a per-environment DNS name for NVMe discovery over per-environment builds.
Signing keys never enter the build VM or this repo. sb_key / sb_cert are CI/HSM-provided paths on the build host. Never commit key material, never copy keys into provisioner steps, never echo them in logs. Unsigned builds are lab-only and must keep their loud warning.
Reproducibility of the root image is a security property. mkfs.erofs runs with fixed UUID and -T 0; the verity root hash must be a pure function of the input tree. Don’t introduce timestamps, random seeds, or build-host paths into the rootfs.
The initramfs is built --no-hostonly and contents are asserted. The build VM is virtio-only; real NIC drivers, macsec, nvme-tcp, erofs, and veritysetup are explicitly listed in 90-diskless.conf and checked via lsinitrd | grep in the template. When adding a runtime dependency to the hook, add it to the dracut module’s install() and extend the lsinitrd assertion.
The OCI artifact image stays FROM scratch. Only /artifacts/{fleet-<ver>.efi, rootfs.erofs, rootfs.hash, verity.json, SHA256SUMS}. rootfs.erofs (erofs data) and rootfs.hash (dm-verity hash tree) are separate namespaces because the as-built Rocky 9 systemd does not support the in-file hash-offset verity layout. No shell, no tools, no base image. Serving software (SPDK, nginx) lives in separate, independently versioned images — never merge them into the artifact image.
Deployments are pinned by digest. One commit per fleet release in the deployment manifest; that manifest is the single source coupling UKI ↔ namespace. Rollback = revert the commit.

Things that look like bugs but aren’t

identity="fleet-host" in the supplicant config is intentionally generic; RADIUS derives real identity from the certificate CN/SAN.
/etc/machine-id is truncated empty in the image; each host generates its own into the tmpfs overlay at boot.
/etc/fstab is effectively empty; root comes from netroot + verity.
The build VM has no TPM. Correct — nothing host-specific exists at build time. CI boot tests use swtpm.
files/etc/dot1x/radius-ca.pem is a placeholder; builds succeed with it but authentication cannot. Replace per environment, don’t commit real internal CA material unless the repo’s classification allows it.

Known bench-verify items (check before trusting a change in production)

TPM key reference method in wpa_supplicant: private_key="handle:..." via OpenSSL tpm2 provider vs tpm2-pkcs11 URI — depends on the distro build; the supplicant config carries both options.
erofs module availability on the target RHEL kernel (fallback: journal-less ext4 in extract-artifacts.sh).
fleet Plymouth theme: confirm (a) the script plugin package actually installed (/usr/lib64/plymouth/script.so — else the build fell back to spinner, logged as a WARNING), (b) the scripted theme + a font got bundled into the initramfs (lsinitrd | grep -E 'fleet.script|script.so'), and (c) it renders (text via Image.Text needs the label plugin + a font in the early initramfs). If blank, the splash silently no-ops but boot continues; ESC shows the log. Untested in this workspace (needs a display).
systemd.verity_root_options=hash-offset is NOT supported by Rocky 9.8 systemd-veritysetup (verified: "unknown option hash-offset=" → "Failed to load verity superblock"). Resolved by the documented fallback — data and hash are two separate NVMe namespaces (rootfs.erofs + rootfs.hash, distinct pinned UUIDs), so the cmdline uses plain systemd.verity_root_data= / systemd.verity_root_hash= by-id devices with no offset.
Network module: network-manager (ported 2026-06-17 from network-legacy). network-legacy's one-shot DHCP lease was never renewed after switch-root, so at lease expiry the address was dropped and the NVMe/TCP root died. The network-manager module hands its initramfs connections to the real-root NM (state under /run/NetworkManager), which renews the lease. No more dhcp-client/dhclient (deprecated). The hook drives macsec0 DHCP via nmcli (not ifup). See invariant on the managed/unmanaged split. Test path VERIFIED on the bench 2026-06-17 (lease renews past expiry; host stays up). STILL BENCH-VERIFY when disable_dot1x=false: macsec0 managed + en* unmanaged with MKA surviving switch-root (secure path).
Writable root overlay is systemd.volatile=overlay, not rootovl: RHEL/Rocky dracut ships no overlayfs/rootovl module. Needs systemd-volatile-root.service in the initramfs (bundled by recent dracut; install_items it if the root stays read-only). Misbehaves with NFS root — N/A here (NVMe block root).

RHEL 9.x build notes (learned — don't re-discover)

These bit us bringing the build up on Rocky 9.8; they are easy to regress.

ISO: default is a version-pinned Rocky 9.x minimal ISO on the internal mirror (iso_url + sha256 in iso_checksum; bump both together on a point release). http/ks.cfg installs only @^minimal-environment+sshd+chrony from the ISO; the diskless tooling is pulled online by the provisioner, so the build VM needs outbound repo access. Keeps the build ISO-variant-agnostic.
%{...} in provisioner shell (e.g. rpm --qf '%{VERSION}') must be escaped %%{...} — Packer's HCL2 engine parses a bare %{ as a template directive and packer validate fails otherwise.
File provisioner staging: pre-create the destination dir (mkdir -p /tmp/etc-stage) before a trailing-slash contents upload; Packer requires the dest to exist or the tree lands elsewhere and later install steps fail.
dracut module name in add_dracutmodules omits the NN dir-prefix: the directory is 95dot1x-macsec but the module is dot1x-macsec (cf. nvmf).
network-manager dracut module (was network-legacy+dhcp-client) and overlay → systemd.volatile=overlay: see the bench-verify items above.
Test toggle: disable_dot1x=true bakes rd.dot1x=0; the hook then skips 802.1X/MACsec and DHCPs the NIC directly for NVMe-oF on an open port. The auth stack stays in the image, dormant. The matching lab DHCP/HTTP/NVMe-oF serving stack lives in serving/ (docker compose). Re-enable security by dropping the var / setting DISABLE_DOT1X: "false" in the CI workflow.
Boot splash toggle: boot_splash=true (default) bakes rhgb quiet for the Plymouth splash; false bakes plymouth.enable=0 for a verbose console boot — keep OFF while debugging the boot chain. Plymouth is always in the image + initramfs (the plymouth dracut module, which pulls drm); only the cmdline changes. Press ESC at boot to see details behind a splash. ks.cfg no longer excludes plymouth.
Splash theme = custom fleet (files/plymouth/themes/fleet/, script plugin, asset-free — drawn with Image.Text). It renders a security-handshake checklist (TPM → 802.1X → MACsec → Network → NVMe-oF) driven by best-effort plymouth update --status=<token> calls from the dracut hooks (dot1x-macsec-up.sh, nvmf-connect.sh). Tokens: mode-test (marks auth/MACsec skipped), tpm|dot1x|macsec|net|nvme-active|done. Rules: the emits are non-fatal (plymouth … || true) and must never reorder the load-bearing boot sequence; show only STATES, never per-host data (invariant #1). The provisioner installs the script plugin best-effort and falls back to the stock spinner theme if script.so is absent, so a plugin-package-name change can't break the build.

Testing changes

Full boot-chain smoke test runs in QEMU/Proxmox: swtpm (test key at the fleet handle, test cert in NV) + hostapd as wired 802.1X/MACsec authenticator + FreeRADIUS on a veth/bridge (bridge needs group_fwd_mask=8 or EAPOL is silently dropped). Pass criteria: port Authorized, macsec0 addressed, NVMe namespace connected, verity root mapped, overlay writable, supplicant alive after switch-root.

Any change to the dracut module, supplicant config, or cmdline must pass this before merge. Template-only changes need packer validate + shellcheck at minimum.

Conventions

Shell: bash, set -euo pipefail in scripts (dracut hooks source dracut-lib instead and must stay POSIX-ish bash compatible with the initramfs environment — no GNU-only tools not installed by the module).
Keep comments that explain why (timing constraints, kernel behaviours); they encode bench findings that are expensive to rediscover.
Versioning: semver on image_version; the OCI tag matches; the git tag matches; one release == one digest == one manifest commit.

15 KiB Raw Blame History Unescape Escape