CLAUDE.md — Diskless Fleet Boot System

This repo builds the boot system for a fleet of diskless hosts: PXE/HTTP-booted, authenticating to the network with 802.1X EAP-TLS using a TPM-held key, running MACsec on the access link, and mounting an immutable erofs + dm-verity root over NVMe/TCP. Each release is a signed UKI plus root image packaged as a FROM-scratch OCI artifact image. The authoritative architecture document is docs/diskless-fleet-design-v2.pdf — read it before structural changes. docs/design-v2-errata.md records where the as-built RHEL/Rocky 9 implementation diverges from it (pending a consolidated v3).

Backlog (read at session start)

docs/TODO.md is the persistent backlog and the durable source of truth for outstanding work — it survives across sessions (the TodoWrite task tracker does not). At the start of a session, read it and mirror its open items into the TodoWrite tracker. Whenever items change (added, started, completed), update both docs/TODO.md and the tracker so they stay in sync.

Repo layout

diskless.pkr.hcl                          Packer template (qemu builder)
http/ks.cfg                               Kickstart for the throwaway build VM
files/dracut/95dot1x-macsec/              Custom dracut module (the heart of the system)
files/etc/dot1x/                          Supplicant config + RADIUS CA (placeholder)
files/etc/dracut.conf.d/90-diskless.conf  no-hostonly + drivers + modules
files/etc/NetworkManager/                 NM unmanaged-device config
scripts/extract-artifacts.sh              qcow2 -> kernel/initramfs/erofs+verity/cmdline
scripts/build-oci.sh                      ukify + sbsign + roothash gate + OCI image
serving/                                  Lab serving stack: dnsmasq + nginx + SPDK
.gitea/workflows/                         Gitea CI: lint -> packer build -> push OCI
docs/                                     Design PDF and supporting notes

Build & test commands

packer init .
packer validate .                         # always before committing template changes
packer build -var image_version=X.Y.Z \
  -var nic_drivers="ice mlx5_core" \
  -var oci_image=<registry>/fleet-boot .  # add sb_key/sb_cert for signed builds
                                          # add disable_dot1x=true for an
                                          # open-port test UKI (rd.dot1x=0)
bash -n files/dracut/95dot1x-macsec/*.sh scripts/*.sh   # shell syntax gate
# .shellcheckrc disables the dracut false positives; CI passes them explicitly:
shellcheck --exclude=SC1091,SC2154 files/dracut/95dot1x-macsec/*.sh scripts/*.sh

Build-host deps: qemu-kvm, libguestfs-tools, erofs-utils, cryptsetup, jq, systemd-ukify, sbsigntools, buildah (or docker).

Architecture invariants — do not violate these

These encode failure modes that brick the fleet. Treat them as hard constraints; if a task appears to require breaking one, stop and flag it instead of working around it.

No per-host state in any artifact, ever. No hostnames, keys, certs, MACs, or per-host config in the initramfs, rootfs, UKI, or OCI image. All host identity derives from the TPM at runtime (key at persistent handle 0x81010001, cert at NV index 0x01800001, hostnqn derived from the cert serial). If a change needs per-host data, it belongs in the enrolment flow, not the image.
The hook ordering in dot1x-macsec-up.sh is load-bearing. NV-cert extraction → supplicant start → wait for Authorized + macsec0 → DHCP on macsec0 → hostnqn derivation. Never DHCP before auth completes (pre-auth leases are invalid after VLAN assignment); never connect NVMe before macsec0 is addressed.
The supplicant must survive switch-root. It is launched with exec -a @wpa_supplicant (systemd root-storage-daemon convention) and its state lives under /run. Do not “clean up” the @, move state off /run, or hand the interface to NetworkManager — MKA times out in ~6 s without keepalives and the root filesystem link dies. The NM unmanaged-devices config exists for this reason; keep it in sync with interface naming.
roothash coupling is a hard gate. build-oci.sh refuses to build if the roothash= in the UKI cmdline ≠ verity.json. Never weaken, skip, or warn-instead-of-fail this check: a mismatched UKI/rootfs pair is an unbootable fleet.
The UKI cmdline is immutable once signed. Any cmdline change (including NVMET_TRADDR) requires a rebuild + re-sign. Prefer a per-environment DNS name for NVMe discovery over per-environment builds.
Signing keys never enter the build VM or this repo. sb_key / sb_cert are CI/HSM-provided paths on the build host. Never commit key material, never copy keys into provisioner steps, never echo them in logs. Unsigned builds are lab-only and must keep their loud warning.
Reproducibility of the root image is a security property. mkfs.erofs runs with fixed UUID and -T 0; the verity root hash must be a pure function of the input tree. Don’t introduce timestamps, random seeds, or build-host paths into the rootfs.
The initramfs is built --no-hostonly and contents are asserted. The build VM is virtio-only; real NIC drivers, macsec, nvme-tcp, erofs, and veritysetup are explicitly listed in 90-diskless.conf and checked via lsinitrd | grep in the template. When adding a runtime dependency to the hook, add it to the dracut module’s install() and extend the lsinitrd assertion.
The OCI artifact image stays FROM scratch. Only /artifacts/{fleet-<ver>.efi, rootfs.erofs, verity.json, SHA256SUMS}. No shell, no tools, no base image. Serving software (SPDK, nginx) lives in separate, independently versioned images — never merge them into the artifact image.
Deployments are pinned by digest. One commit per fleet release in the deployment manifest; that manifest is the single source coupling UKI ↔ namespace. Rollback = revert the commit.

Things that look like bugs but aren’t

identity="fleet-host" in the supplicant config is intentionally generic; RADIUS derives real identity from the certificate CN/SAN.
/etc/machine-id is truncated empty in the image; each host generates its own into the tmpfs overlay at boot.
/etc/fstab is effectively empty; root comes from netroot + verity.
The build VM has no TPM. Correct — nothing host-specific exists at build time. CI boot tests use swtpm.
files/etc/dot1x/radius-ca.pem is a placeholder; builds succeed with it but authentication cannot. Replace per environment, don’t commit real internal CA material unless the repo’s classification allows it.

Known bench-verify items (check before trusting a change in production)

TPM key reference method in wpa_supplicant: private_key="handle:..." via OpenSSL tpm2 provider vs tpm2-pkcs11 URI — depends on the distro build; the supplicant config carries both options.
erofs module availability on the target RHEL kernel (fallback: journal-less ext4 in extract-artifacts.sh).
systemd.verity_root_options=hash-offset support in the shipped systemd (fallback: split data/hash into two namespaces).
network-legacy dracut module needs dhclient; RHEL 9 omits it from the minimal install, so the provisioner installs dhcp-client (without it dracut fails "module network-legacy can't be installed"). dhclient is deprecated — the durable fix is porting the hook to the network-manager dracut module.
Writable root overlay is systemd.volatile=overlay, not rootovl: RHEL/Rocky dracut ships no overlayfs/rootovl module. Needs systemd-volatile-root.service in the initramfs (bundled by recent dracut; install_items it if the root stays read-only). Misbehaves with NFS root — N/A here (NVMe block root).

RHEL 9.x build notes (learned — don't re-discover)

These bit us bringing the build up on Rocky 9.8; they are easy to regress.

ISO: default is a version-pinned Rocky 9.x minimal ISO on the internal mirror (iso_url + sha256 in iso_checksum; bump both together on a point release). http/ks.cfg installs only @^minimal-environment+sshd+chrony from the ISO; the diskless tooling is pulled online by the provisioner, so the build VM needs outbound repo access. Keeps the build ISO-variant-agnostic.
%{...} in provisioner shell (e.g. rpm --qf '%{VERSION}') must be escaped %%{...} — Packer's HCL2 engine parses a bare %{ as a template directive and packer validate fails otherwise.
File provisioner staging: pre-create the destination dir (mkdir -p /tmp/etc-stage) before a trailing-slash contents upload; Packer requires the dest to exist or the tree lands elsewhere and later install steps fail.
dracut module name in add_dracutmodules omits the NN dir-prefix: the directory is 95dot1x-macsec but the module is dot1x-macsec (cf. nvmf).
network-legacy → dhcp-client and overlay → systemd.volatile=overlay: see the bench-verify items above.
Test toggle: disable_dot1x=true bakes rd.dot1x=0; the hook then skips 802.1X/MACsec and DHCPs the NIC directly for NVMe-oF on an open port. The auth stack stays in the image, dormant. The matching lab DHCP/HTTP/NVMe-oF serving stack lives in serving/ (docker compose). Re-enable security by dropping the var / setting DISABLE_DOT1X: "false" in the CI workflow.

Testing changes

Full boot-chain smoke test runs in QEMU/Proxmox: swtpm (test key at the fleet handle, test cert in NV) + hostapd as wired 802.1X/MACsec authenticator + FreeRADIUS on a veth/bridge (bridge needs group_fwd_mask=8 or EAPOL is silently dropped). Pass criteria: port Authorized, macsec0 addressed, NVMe namespace connected, verity root mapped, overlay writable, supplicant alive after switch-root.

Any change to the dracut module, supplicant config, or cmdline must pass this before merge. Template-only changes need packer validate + shellcheck at minimum.

Conventions

Shell: bash, set -euo pipefail in scripts (dracut hooks source dracut-lib instead and must stay POSIX-ish bash compatible with the initramfs environment — no GNU-only tools not installed by the module).
Keep comments that explain why (timing constraints, kernel behaviours); they encode bench findings that are expensive to rediscover.
Versioning: semver on image_version; the OCI tag matches; the git tag matches; one release == one digest == one manifest commit.

10 KiB Raw Blame History Unescape Escape