CLAUDE.md — Diskless Fleet Boot System

This repo builds the boot system for a fleet of diskless hosts: PXE/HTTP-booted, authenticating to the network with 802.1X EAP-TLS using a TPM-held key, running MACsec on the access link, and mounting an immutable erofs + dm-verity root over NVMe/TCP. Each release is a signed UKI plus root image packaged as a FROM-scratch OCI artifact image. The authoritative architecture document is docs/diskless-fleet-design-v2.pdf — read it before structural changes. docs/design-v2-errata.md records where the as-built RHEL/Rocky 9 implementation diverges from it (pending a consolidated v3).

Backlog (read at session start)

docs/TODO.md is the persistent backlog and the durable source of truth for outstanding work — it survives across sessions (the TodoWrite task tracker does not). At the start of a session, read it and mirror its open items into the TodoWrite tracker. Whenever items change (added, started, completed), update both docs/TODO.md and the tracker so they stay in sync.

Repo layout

diskless.pkr.hcl                          Packer template (qemu builder)
http/ks.cfg                               Kickstart for the throwaway build VM
files/dracut/95dot1x-macsec/              Custom dracut module (the heart of the system)
files/etc/dot1x/                          Supplicant config + RADIUS CA (placeholder)
files/etc/dracut.conf.d/90-diskless.conf  no-hostonly + drivers + modules
                                          (NM unmanaged split is now runtime:
                                          95dot1x-macsec/nm-unmanaged.sh, secure-mode only)
files/plymouth/themes/fleet/              Custom boot splash (scripted security
                                          checklist; driven by hook status tokens)
files/kiosk/                              weston+Chromium kiosk + chromium lockdown policy
                                          (policy allows PNA to the host-info agent)
files/hostapi/                            Go host-info agent (fleet-hostapi): read-only,
                                          loopback :9099 /v1/host for the kiosk SPA
                                          (HOST-API.md contract). Built in 2b-hostapi,
                                          fapolicyd-trusted, golang removed in 2d.
files/ssh-hostcert/                       Go SSH host-cert client (fleet-ssh-hostcert):
                                          at boot signs this host's per-boot SSH host
                                          key with the fleet SSH CA via the enrol-broker
                                          host plane (TPM-identity mTLS), so admins trust
                                          via one @cert-authority line instead of per-boot
                                          TOFU (F-02/P-17). Built in 2b-ssh-hostcert,
                                          fapolicyd-trusted; enable_ssh_hostcert flag.
                                          docs/ssh-host-certificate-design.md.
scripts/extract-artifacts.sh              qcow2 -> kernel/initramfs/erofs+verity/cmdline
scripts/build-oci.sh                      ukify + sbsign + roothash gate + OCI image
scripts/build-ipxe-oci.sh                 sbsign snponly.efi -> FROM-scratch fleet-ipxe image
serving/                                  Lab serving stack: dnsmasq + nginx + SPDK
                                          (dnsmasq + fleet-ipxe + SPDK images are
                                          CI-built + pulled; see .gitea/workflows/)
serving/spdk/                             LONG-LIVED nvmf_tgt: spdk-target.sh
                                          (persistent entrypoint, never restarts
                                          for a version change) + spdk-release.sh
                                          (online add/remove/drain a version, 1
                                          subsystem/version) + spdk-ctl.sh (host
                                          load/unload/list for manual rolling
                                          updates) + spdk-loader.sh (DECLARATIVE:
                                          compose service that adds ${IMAGE_VERSION}
                                          online then auto-reaps drained old
                                          versions — bump tag + compose up -d).
                                          releases/<ver>/ = staged rootfs+verity.json.
                                          spdk-release parses RPC JSON via python3
                                          (SPDK image has no jq). spdk-setup.sh
                                          (parent dir) is the legacy single-version
                                          entrypoint (deprecated).
serving/kiosk-app/                         Svelte+Tailwind spaceship-terminal SPA
                                          — GIT SUBMODULE (own repo + CI; builds
                                          the fleet-kiosk image the stack pulls).
                                          OS-side kiosk runtime stays in files/kiosk/
serving/dot1x-bench/                      Secure-path bench: step-ca + FreeRADIUS +
                                          hostapd authenticator + TPM enrolment
                                          (+ enrol-broker service)
serving/enrol-broker/                     TPM identity enrolment broker (Go): the one
                                          unauth pre-auth endpoint (POST CSR -> OOB
                                          fingerprint -> admin mTLS approve -> step-ca
                                          sign). docs/tpm-enrolment-design.md
serving/siem/                             Wazuh SIEM overlay (audit analysis;
                                          agentless relay from audit-collector)
.gitea/workflows/                         Gitea CI: lint -> packer build -> push OCI
docs/                                     Design PDF and supporting notes

Build & test commands

packer init .
packer validate .                         # always before committing template changes
packer build -var image_version=X.Y.Z \
  -var nic_drivers="ice mlx5_core" \
  -var oci_image=<registry>/fleet-boot .  # add sb_key/sb_cert for signed builds
                                          # add disable_dot1x=true for an
                                          # open-port test UKI (rd.dot1x=0)
bash -n files/dracut/95dot1x-macsec/*.sh scripts/*.sh   # shell syntax gate
# .shellcheckrc disables the dracut false positives; CI passes them explicitly:
shellcheck --exclude=SC1091,SC2154 files/dracut/95dot1x-macsec/*.sh scripts/*.sh

Build-host deps: qemu-kvm, libguestfs-tools (+ a kernel for its supermin appliance), erofs-utils, cryptsetup, jq, systemd-ukify and systemd-boot-efi (the latter ships the linuxx64.efi.stub ukify needs), sbsigntools, buildah (or docker), attr (getfattr — extract-artifacts.sh asserts the SELinux labels survived the export; an unlabeled enforcing root denies login).

Architecture invariants — do not violate these

These encode failure modes that brick the fleet. Treat them as hard constraints; if a task appears to require breaking one, stop and flag it instead of working around it.

No per-host state in any artifact, ever. No hostnames, keys, certs, MACs, or per-host config in the initramfs, rootfs, UKI, or OCI image. All host identity derives from the TPM at runtime (signing key at persistent handle 0x81000100 — a non-reserved owner handle, deliberately NOT the EK slot 0x81010001 nor the SRK 0x81000001, which firmware/swtpm provisioning already occupy; cert at NV index 0x01800100 — also chosen to avoid OEM platform NV: a Dell 820 firmware squats 0x01800001–0x01800004
- 0x01810000 as platformcreate|policydelete indices that no owner/BIOS clear can remove, so the original 0x01800001 collided; hostnqn derived from the cert serial). If a change needs per-host data, it belongs in the enrolment flow, not the image. (Both are overridable: rd.dot1x.handle= / rd.dot1x.nvindex= / fleet-enrol -handle -nvindex.)
The hook ordering in dot1x-macsec-up.sh is load-bearing. NV-cert extraction → supplicant start → wait for Authorized + macsec0 → DHCP on macsec0 → hostnqn derivation → targeted NVMe/TCP connect (nvme connect -n $rd.nvmf.nqn, step 6). Never DHCP before auth completes (pre-auth leases are invalid after VLAN assignment); never connect NVMe before macsec0 is addressed. The connect is targeted to this version's subsystem (rd.nvmf.nqn, baked per-version into the signed cmdline) — never a discovery connect-all, which attaches every served version and defeats the per-version drain/reap gate (see TODO #6).
The supplicant must survive switch-root. It is launched with exec -a @wpa_supplicant (systemd root-storage-daemon convention) and its state lives under /run. Do not “clean up” the @, move state off /run, or hand the physical NIC to NetworkManager — MKA times out in ~6 s without keepalives and the root filesystem link dies. The NM unmanaged split exists for this reason (see below); keep it in sync with interface naming.
NM owns IP/DHCP; wpa_supplicant owns the physical NIC. The initramfs uses the network-manager dracut module (not network-legacy) so NM renews the DHCP lease in the running system after switch-root — without this the one-shot lease expired, the address was dropped, and the NVMe/TCP root died (“keep alive timeout” / “no route to host”). The managed/unmanaged split is therefore load-bearing and mode-aware:
- Secure path: IP rides on macsec0, which must stay MANAGED by NM. The physical NIC (en*) must stay UNMANAGED (wpa_supplicant owns EAPOL+MKA on it). This is enforced at runtime by the cmdline-stage hook nm-unmanaged.sh, which writes /run/NetworkManager/conf.d/10-dot1x-unmanaged.conf before NM starts.
- Test path (rd.dot1x=0): no macsec0; the physical NIC carries IP and must be MANAGED so NM renews it — so the hook writes nothing. There is intentionally no static 99-dot1x-unmanaged.conf (it couldn't be mode-aware — it wrongly unmanaged en* in test mode, blocking renewal). Never add macsec0 back to an unmanaged list, and never statically unmanage en* for all modes.
roothash coupling is a hard gate. build-oci.sh refuses to build if the roothash= in the UKI cmdline ≠ verity.json. Never weaken, skip, or warn-instead-of-fail this check: a mismatched UKI/rootfs pair is an unbootable fleet.
The UKI cmdline is immutable once signed. Any cmdline change (including NVMET_TRADDR) requires a rebuild + re-sign. Prefer a per-environment DNS name for NVMe discovery over per-environment builds.
Signing keys never enter the build VM or this repo. sb_key / sb_cert are CI/HSM-provided paths on the build host. Never commit key material, never copy keys into provisioner steps, never echo them in logs. Unsigned builds are local-lab-only (a bare packer build without keys) and must keep their loud warning; CI fails closed — the gitea-main-ci.yaml/ipxe-image.yaml "Production build gate" aborts unless SB_KEY/SB_CERT (and FLEET_CA for the UKI build) are set, so no unsigned or placeholder-CA artifact is producible via CI (POA&M P-03/P-05).
Reproducibility of the root image is a security property. mkfs.erofs runs with fixed UUID and -T 0; the verity root hash must be a pure function of the input tree. Don’t introduce timestamps, random seeds, or build-host paths into the rootfs.
The initramfs is built --no-hostonly and contents are asserted. The build VM is virtio-only; real NIC drivers, macsec, nvme-tcp, erofs, and veritysetup are explicitly listed in 90-diskless.conf and checked via lsinitrd | grep in the template. When adding a runtime dependency to the hook, add it to the dracut module’s install() and extend the lsinitrd assertion.
The OCI artifact image stays FROM scratch. Only /artifacts/{fleet-<ver>.efi, rootfs.erofs, rootfs.hash, verity.json, SHA256SUMS}. rootfs.erofs (erofs data) and rootfs.hash (dm-verity hash tree) are separate namespaces because the as-built Rocky 9 systemd does not support the in-file hash-offset verity layout. No shell, no tools, no base image. Serving software (SPDK, nginx, dnsmasq) lives in separate, independently versioned images — never merge them into the artifact image. The signed iPXE bootloader (snponly.efi) is its own FROM-scratch fleet-ipxe image (scripts/build-ipxe-oci.sh, signed with db), mounted into dnsmasq at /ipxe — kept out of both the artifact image and the (generic, key-agnostic) dnsmasq image because its signature is per-Secure-Boot-environment. Don't bake snponly.efi into the dnsmasq image.
Deployments are pinned by digest. One commit per fleet release in the deployment manifest; that manifest is the single source coupling UKI ↔ namespace. Rollback = revert the commit.

Things that look like bugs but aren’t

identity="fleet-host" in the supplicant config is intentionally generic; RADIUS derives real identity from the certificate CN/SAN.
/etc/machine-id is truncated empty in the image; each host generates its own into the tmpfs overlay at boot.
/etc/fstab is effectively empty; root comes from netroot + verity.
The build VM has no TPM. Correct — nothing host-specific exists at build time. CI boot tests use swtpm.
files/etc/dot1x/radius-ca.pem is a placeholder; builds succeed with it but authentication cannot. Replace per environment, don’t commit real internal CA material unless the repo’s classification allows it.

Known bench-verify items (check before trusting a change in production)

TPM key reference method in wpa_supplicant: private_key="handle:..." via OpenSSL tpm2 provider vs tpm2-pkcs11 URI — depends on the distro build; the supplicant config carries both options.
erofs module availability on the target RHEL kernel (fallback: journal-less ext4 in extract-artifacts.sh).
STIG remediation: (a) datastream must match the distro. The provisioner maps /etc/os-release $ID → datastream (Rocky→ssg-rl9-ds.xml, RHEL→ssg-rhel9-ds.xml, al9/cs9/ol9). Do NOT run the RHEL9 datastream on Rocky — its CPE dictionary has no Rocky entry, so every rule returns notapplicable, the scan evaluates 0 rules, remediation does nothing, and the report is falsely clean. (Seen 2026-06-18: a build picked rhel9 via ls … | head -1 and produced 462 notapplicable + 1068 notselected, 0 pass/fail.) Two guards now exist: the provisioner aborts the build if 0 rules are evaluated, and stig-deviations.sh flags an INVALID SCAN + exits 2. Also confirm the ...content_profile_stig profile id exists in the chosen datastream; (b) remediation must not break boot/login (NM/NVMe, key-only sshd, kiosk, SELinux enforcing) — waive the offending rule in the tailoring as a justified deviation (STIG is now enforced-true by Packer variable validation — enable_stig=false is no longer a build path; POA&M P-03); (c) reproducibility: build twice and confirm identical root_hash in verity.json — if it drifts, oscap wrote something nondeterministic (timestamp/uuid/key); find + normalize it in 2d hygiene; (d) read the generated stig-deviations.md — every REVIEW row is a real finding to triage or justify in docs/stig-deviations.md. STIG-breaks-login gotchas already hit + fixed (2026-06-19) — don't regress: (i) fapolicyd blocks non-RPM executables → the kiosk's /usr/local/bin/kiosk-session must be in fapolicyd's trust (done in 2c-STIG-postfix); keep fapolicyd, don't waive it (it's the writable-overlay exec guard dm-verity can't provide). (ii) Admin SSH key must be ECDSA/RSA, NOT ed25519 — the STIG crypto policy drops ed25519 from sshd's accepted pubkey algorithms (server-sig-algs shows ecdsa+rsa-sha2 only), so an ed25519 key gets Permission denied (publickey) at the signature stage. (iii) passwordless/key-only appliance accounts (kiosk, fleetadmin) are exempted from STIG password-aging + faillock account phase in 2c-STIG-postfix (else locked out). (iv) the kiosk unit is WantedBy=multi-user.target (not graphical.target, which the host never reaches → unit dead at boot). Offline diagnosis without booting: mount the OCI image's rootfs.erofs read-only and inspect /etc/pam.d, fapolicyd, sshd config, etc. REVIEW-row triage (2026-06-20, scan app-9f7d8d1) added two more bench gates — don't regress: (v) USBGuard is ENABLED + enforcing (block target) with a hardware-independent HID/hub allowlist + AuditBackend=LinuxAudit (2c-STIG). The generated policy is built on the virtio build VM (no real USB), so without the HID/hub allow rules a physical console keyboard/mouse is blocked on real hardware — BENCH-VERIFY console input. (vi) admin sudo SELinux elevation: the admin is mapped to staff_u (semanage login) + sudoers TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL so selinux_context_elevation_for_sudo passes; staff_u (not sysadm_u) so SSH login isn't gated by ssh_sysadm_login. BENCH-VERIFY admin SSH login + sudo + privileged ops under confinement, and re-check reproducibility (semanage rebuilds the policy store → possible verity-hash drift). The triage waivers/decisions live in the tailoring + docs/stig-deviations.md #12; the generated report should show 0–1 REVIEW.
NTP is secure_ntp=true by default (NTS, RFC 8915) — CA-authenticated time (closes F-03/P-18; docs/secure-ntp-design.md). No FQDN is baked: the NM dispatcher (10-chrony-dhcp-ntp, mode-aware) derives the server name at runtime as <ntp_prefix>.<DHCP-provided-domain> (default ntp.$DHCP4_DOMAIN_NAME, e.g. ntp.fleet.lab), writes server … iburst nts into sourcedir /run/chrony-dhcp, and chrony verifies the NTS-KE TLS cert against the fleet CA (/etc/dot1x/radius-ca.pem — reused; baked as ntstrustedcerts). Only the ntp label + the CA are baked, so the site domain can change without a rebuild (new DHCP domain + reissued NTS cert). A DHCP/NTP/DNS spoofer can only DoS (no fleet-CA-signed cert for the name it redirects to ⇒ NTS fails ⇒ no sync), never shift the clock. Serving needs: DHCP option 15 (domain-name) handed to hosts, a fleet-signed NTS cert with DNS SAN ntp.<domain> (serving/chrony/nts-cert.sh), and tcp/4460 (NTS-KE) open + udp/123. Confirm on a booted host: chronyc -N authdata shows the source in NTS mode, cookies
1. nocerttimecheck 1 lets a cold boot clock (no RTC) bootstrap the first sync. Toggle: secure_ntp=false reverts the dispatcher to the legacy unauthenticated DHCP option 42 path (plain server <ip> iburst).
fleet Plymouth theme: confirm (a) the script plugin package actually installed (/usr/lib64/plymouth/script.so — else the build fell back to spinner, logged as a WARNING), (b) the scripted theme + a font got bundled into the initramfs (lsinitrd | grep -E 'fleet.script|script.so'), and (c) it renders (text via Image.Text needs the label plugin + a font in the early initramfs). If blank, the splash silently no-ops but boot continues; ESC shows the log. RENDER-VERIFIED 2026-06-18 (test-path build): the scripted theme draws and the handshake checklist updates from the hook status tokens. Still unverified on the secure path (the auth/macsec rows reaching done — see #8 bench items).
systemd.verity_root_options=hash-offset is NOT supported by Rocky 9.8 systemd-veritysetup (verified: "unknown option hash-offset=" → "Failed to load verity superblock"). Resolved by the documented fallback — data and hash are two separate NVMe namespaces (rootfs.erofs + rootfs.hash, distinct pinned UUIDs), so the cmdline uses plain systemd.verity_root_data= / systemd.verity_root_hash= by-id devices with no offset.
Network module: network-manager (ported 2026-06-17 from network-legacy). network-legacy's one-shot DHCP lease was never renewed after switch-root, so at lease expiry the address was dropped and the NVMe/TCP root died. The network-manager module hands its initramfs connections to the real-root NM (state under /run/NetworkManager), which renews the lease. No more dhcp-client/dhclient (deprecated). The hook drives macsec0 DHCP via nmcli (not ifup). See invariant on the managed/unmanaged split. Test path VERIFIED on the bench 2026-06-17 (lease renews past expiry; host stays up). STILL BENCH-VERIFY when disable_dot1x=false: macsec0 managed + en* unmanaged with MKA surviving switch-root (secure path).
Writable root overlay is systemd.volatile=overlay, not rootovl: RHEL/Rocky dracut ships no overlayfs/rootovl module. Needs systemd-volatile-root.service in the initramfs (bundled by recent dracut; install_items it if the root stays read-only). Misbehaves with NFS root — N/A here (NVMe block root).

RHEL 9.x build notes (learned — don't re-discover)

These bit us bringing the build up on Rocky 9.8; they are easy to regress.

ISO: default is a version-pinned Rocky 9.x minimal ISO on the internal mirror (iso_url + sha256 in iso_checksum; bump both together on a point release). http/ks.cfg installs only @^minimal-environment+sshd+chrony from the ISO; the diskless tooling is pulled online by the provisioner, so the build VM needs outbound repo access. Keeps the build ISO-variant-agnostic.
%{...} in provisioner shell (e.g. rpm --qf '%{VERSION}') must be escaped %%{...} — Packer's HCL2 engine parses a bare %{ as a template directive and packer validate fails otherwise.
File provisioner staging: pre-create the destination dir (mkdir -p /tmp/etc-stage) before a trailing-slash contents upload; Packer requires the dest to exist or the tree lands elsewhere and later install steps fail.
dracut module name in add_dracutmodules omits the NN dir-prefix: the directory is 95dot1x-macsec but the module is dot1x-macsec (cf. nvmf).
network-manager dracut module (was network-legacy+dhcp-client) and overlay → systemd.volatile=overlay: see the bench-verify items above.
Test toggle: disable_dot1x=true bakes rd.dot1x=0; the hook then skips 802.1X/MACsec and DHCPs the NIC directly for NVMe-oF on an open port. The auth stack stays in the image, dormant. The matching lab DHCP/HTTP/NVMe-oF serving stack lives in serving/ (docker compose). Re-enable security by dropping the var / setting DISABLE_DOT1X: "false" in the CI workflow.
STIG toggle (now enforced-true, not just default): enable_stig=true applies DISA STIG via OpenSCAP/SSG in a provisioner, tailored for the diskless/UKI/kiosk architecture (files/stig/diskless-tailoring.xml unselects the N/A or dangerous-to-remediate rules; every waiver is justified in docs/stig-deviations.md). STIG kernel params ride the UKI cmdline (extract-artifacts.sh, gated by the same flag) since there's no GRUB. FIPS is OFF and audit offload is LOCAL-only (decisions). CI is report-only: it publishes stig-report.html + stig-arf.xml + stig-deviations.md (generated by scripts/stig-deviations.sh) as artifacts and never fails on score. The scanner tooling is removed from the shipped image (2d hygiene). enable_stig is enforced true by Packer variable validation (POA&M P-03) — a build with it false fails packer validate/build; there is no unhardened build path. If hardening regresses the boot/secure-path bench, waive the offending rule in the tailoring as a justified deviation (never by disabling STIG). NEVER let a waiver hide a real finding — anything non-pass that isn't in the register shows as REVIEW in the report.
Boot splash toggle: boot_splash=true (default) bakes rhgb quiet for the Plymouth splash; false bakes plymouth.enable=0 for a verbose console boot — keep OFF while debugging the boot chain. Plymouth is always in the image + initramfs (the plymouth dracut module, which pulls drm); only the cmdline changes. Press ESC at boot to see details behind a splash. ks.cfg no longer excludes plymouth.
Splash theme = custom fleet (files/plymouth/themes/fleet/, script plugin, asset-free — drawn with Image.Text). It renders a security-handshake checklist (TPM → 802.1X → MACsec → Network → NVMe-oF) driven by best-effort plymouth update --status=<token> calls from the dracut hooks (dot1x-macsec-up.sh, nvmf-connect.sh). Tokens: mode-test (marks auth/MACsec skipped), tpm|dot1x|macsec|net|nvme-active|done. Rules: the emits are non-fatal (plymouth … || true) and must never reorder the load-bearing boot sequence; show only STATES, never per-host data (invariant #1). The provisioner installs the script plugin best-effort and falls back to the stock spinner theme if script.so is absent, so a plugin-package-name change can't break the build.

Testing changes

Full boot-chain smoke test runs in QEMU/Proxmox: swtpm (test key at the fleet handle, test cert in NV) + hostapd as wired 802.1X/MACsec authenticator + FreeRADIUS on a veth/bridge (bridge needs group_fwd_mask=8 or EAPOL is silently dropped). Pass criteria: port Authorized, macsec0 addressed, NVMe namespace connected, verity root mapped, overlay writable, supplicant alive after switch-root. This harness is scaffolded in serving/dot1x-bench/ (step-ca as the fleet CA on the serving host + FreeRADIUS EAP-TLS + a hostapd wired authenticator + tpm-provision.sh), with an isolated bridge per workstation (the flat VLAN can't authenticate — EAPOL is dropped and MACsec is point-to-point). See its README; several configs there are still BENCH-VERIFY.

Any change to the dracut module, supplicant config, or cmdline must pass this before merge. Template-only changes need packer validate + shellcheck at minimum.

Conventions

Shell: bash, set -euo pipefail in scripts (dracut hooks source dracut-lib instead and must stay POSIX-ish bash compatible with the initramfs environment — no GNU-only tools not installed by the module).
Keep comments that explain why (timing constraints, kernel behaviours); they encode bench findings that are expensive to rediscover.
Versioning: semver on image_version; the OCI tag matches; the git tag matches; one release == one digest == one manifest commit.

26 KiB Raw Blame History Unescape Escape