# CLAUDE.md — Diskless Fleet Boot System This repo builds the boot system for a fleet of diskless hosts: PXE/HTTP-booted, authenticating to the network with **802.1X EAP-TLS using a TPM-held key**, running **MACsec** on the access link, and mounting an immutable **erofs + dm-verity root over NVMe/TCP**. Each release is a signed **UKI** plus root image packaged as a **FROM-scratch OCI artifact image**. The authoritative architecture document is `docs/diskless-fleet-design-v2.pdf` — read it before structural changes. `docs/design-v2-errata.md` records where the as-built RHEL/Rocky 9 implementation diverges from it (pending a consolidated v3). ## Backlog (read at session start) `docs/TODO.md` is the **persistent backlog** and the durable source of truth for outstanding work — it survives across sessions (the TodoWrite task tracker does not). At the start of a session, read it and mirror its open items into the TodoWrite tracker. Whenever items change (added, started, completed), update **both** `docs/TODO.md` and the tracker so they stay in sync. ## Repo layout ``` diskless.pkr.hcl Packer template (qemu builder) http/ks.cfg Kickstart for the throwaway build VM files/dracut/95dot1x-macsec/ Custom dracut module (the heart of the system) files/etc/dot1x/ Supplicant config + RADIUS CA (placeholder) files/etc/dracut.conf.d/90-diskless.conf no-hostonly + drivers + modules (NM unmanaged split is now runtime: 95dot1x-macsec/nm-unmanaged.sh, secure-mode only) files/plymouth/themes/fleet/ Custom boot splash (scripted security checklist; driven by hook status tokens) files/kiosk/ weston+Chromium kiosk + chromium lockdown policy (policy allows PNA to the host-info agent) files/hostapi/ Go host-info agent (fleet-hostapi): read-only, loopback :9099 /v1/host for the kiosk SPA (HOST-API.md contract). Built in 2b-hostapi, fapolicyd-trusted, golang removed in 2d. scripts/extract-artifacts.sh qcow2 -> kernel/initramfs/erofs+verity/cmdline scripts/build-oci.sh ukify + sbsign + roothash gate + OCI image scripts/build-ipxe-oci.sh sbsign snponly.efi -> FROM-scratch fleet-ipxe image serving/ Lab serving stack: dnsmasq + nginx + SPDK (dnsmasq + fleet-ipxe + SPDK images are CI-built + pulled; see .gitea/workflows/) serving/spdk/ LONG-LIVED nvmf_tgt: spdk-target.sh (persistent entrypoint, never restarts for a version change) + spdk-release.sh (online add/remove/drain a version, 1 subsystem/version) + spdk-ctl.sh (host load/unload/list for manual rolling updates) + spdk-loader.sh (DECLARATIVE: compose service that adds ${IMAGE_VERSION} online then auto-reaps drained old versions — bump tag + compose up -d). releases// = staged rootfs+verity.json. spdk-release parses RPC JSON via python3 (SPDK image has no jq). spdk-setup.sh (parent dir) is the legacy single-version entrypoint (deprecated). serving/kiosk-app/ Svelte+Tailwind spaceship-terminal SPA — GIT SUBMODULE (own repo + CI; builds the fleet-kiosk image the stack pulls). OS-side kiosk runtime stays in files/kiosk/ serving/dot1x-bench/ Secure-path bench: step-ca + FreeRADIUS + hostapd authenticator + TPM enrolment (+ enrol-broker service) serving/enrol-broker/ TPM identity enrolment broker (Go): the one unauth pre-auth endpoint (POST CSR -> OOB fingerprint -> admin mTLS approve -> step-ca sign). docs/tpm-enrolment-design.md serving/siem/ Wazuh SIEM overlay (audit analysis; agentless relay from audit-collector) .gitea/workflows/ Gitea CI: lint -> packer build -> push OCI docs/ Design PDF and supporting notes ``` ## Build & test commands ```bash packer init . packer validate . # always before committing template changes packer build -var image_version=X.Y.Z \ -var nic_drivers="ice mlx5_core" \ -var oci_image=/fleet-boot . # add sb_key/sb_cert for signed builds # add disable_dot1x=true for an # open-port test UKI (rd.dot1x=0) bash -n files/dracut/95dot1x-macsec/*.sh scripts/*.sh # shell syntax gate # .shellcheckrc disables the dracut false positives; CI passes them explicitly: shellcheck --exclude=SC1091,SC2154 files/dracut/95dot1x-macsec/*.sh scripts/*.sh ``` Build-host deps: qemu-kvm, libguestfs-tools (+ a kernel for its supermin appliance), erofs-utils, cryptsetup, jq, systemd-ukify **and systemd-boot-efi** (the latter ships the `linuxx64.efi.stub` ukify needs), sbsigntools, buildah (or docker), **attr** (`getfattr` — `extract-artifacts.sh` asserts the SELinux labels survived the export; an unlabeled enforcing root denies login). ## Architecture invariants — do not violate these These encode failure modes that brick the fleet. Treat them as hard constraints; if a task appears to require breaking one, stop and flag it instead of working around it. 1. **No per-host state in any artifact, ever.** No hostnames, keys, certs, MACs, or per-host config in the initramfs, rootfs, UKI, or OCI image. All host identity derives from the TPM at runtime (signing key at persistent handle `0x81000100` — a **non-reserved owner handle**, deliberately NOT the EK slot `0x81010001` nor the SRK `0x81000001`, which firmware/swtpm provisioning already occupy; cert at NV index `0x01800100` — also chosen to avoid **OEM platform NV**: a Dell 820 firmware squats `0x01800001`–`0x01800004` + `0x01810000` as `platformcreate|policydelete` indices that no owner/BIOS clear can remove, so the original `0x01800001` collided; hostnqn derived from the cert serial). If a change needs per-host data, it belongs in the enrolment flow, not the image. (Both are overridable: `rd.dot1x.handle=` / `rd.dot1x.nvindex=` / `fleet-enrol -handle -nvindex`.) 1. **The hook ordering in `dot1x-macsec-up.sh` is load-bearing.** NV-cert extraction → supplicant start → wait for `Authorized` + macsec0 → DHCP **on macsec0** → hostnqn derivation → **targeted NVMe/TCP connect** (`nvme connect -n $rd.nvmf.nqn`, step 6). Never DHCP before auth completes (pre-auth leases are invalid after VLAN assignment); never connect NVMe before macsec0 is addressed. The connect is **targeted to this version's subsystem** (`rd.nvmf.nqn`, baked per-version into the signed cmdline) — never a discovery `connect-all`, which attaches every served version and defeats the per-version drain/reap gate (see TODO #6). 1. **The supplicant must survive switch-root.** It is launched with `exec -a @wpa_supplicant` (systemd root-storage-daemon convention) and its state lives under `/run`. Do not “clean up” the `@`, move state off `/run`, or hand the **physical NIC** to NetworkManager — MKA times out in ~6 s without keepalives and the root filesystem link dies. The NM unmanaged split exists for this reason (see below); keep it in sync with interface naming. 1. **NM owns IP/DHCP; wpa_supplicant owns the physical NIC.** The initramfs uses the `network-manager` dracut module (not `network-legacy`) so NM **renews the DHCP lease** in the running system after switch-root — without this the one-shot lease expired, the address was dropped, and the NVMe/TCP root died (“keep alive timeout” / “no route to host”). The managed/unmanaged split is therefore load-bearing and **mode-aware**: - **Secure path:** IP rides on **`macsec0`**, which must stay **MANAGED** by NM. The **physical NIC (`en*`)** must stay **UNMANAGED** (wpa_supplicant owns EAPOL+MKA on it). This is enforced at runtime by the cmdline-stage hook `nm-unmanaged.sh`, which writes `/run/NetworkManager/conf.d/10-dot1x-unmanaged.conf` **before NM starts**. - **Test path (`rd.dot1x=0`):** no macsec0; the physical NIC carries IP and must be **MANAGED** so NM renews it — so the hook writes nothing. There is intentionally **no static `99-dot1x-unmanaged.conf`** (it couldn't be mode-aware — it wrongly unmanaged `en*` in test mode, blocking renewal). Never add macsec0 back to an unmanaged list, and never statically unmanage `en*` for all modes. 1. **roothash coupling is a hard gate.** `build-oci.sh` refuses to build if the `roothash=` in the UKI cmdline ≠ `verity.json`. Never weaken, skip, or warn-instead-of-fail this check: a mismatched UKI/rootfs pair is an unbootable fleet. 1. **The UKI cmdline is immutable once signed.** Any cmdline change (including `NVMET_TRADDR`) requires a rebuild + re-sign. Prefer a per-environment DNS name for NVMe discovery over per-environment builds. 1. **Signing keys never enter the build VM or this repo.** `sb_key` / `sb_cert` are CI/HSM-provided paths on the build host. Never commit key material, never copy keys into provisioner steps, never echo them in logs. Unsigned builds are lab-only and must keep their loud warning. 1. **Reproducibility of the root image is a security property.** `mkfs.erofs` runs with fixed UUID and `-T 0`; the verity root hash must be a pure function of the input tree. Don’t introduce timestamps, random seeds, or build-host paths into the rootfs. 1. **The initramfs is built `--no-hostonly` and contents are asserted.** The build VM is virtio-only; real NIC drivers, `macsec`, `nvme-tcp`, erofs, and veritysetup are explicitly listed in `90-diskless.conf` and checked via `lsinitrd | grep` in the template. When adding a runtime dependency to the hook, add it to the dracut module’s `install()` *and* extend the lsinitrd assertion. 1. **The OCI artifact image stays FROM scratch.** Only `/artifacts/{fleet-.efi, rootfs.erofs, rootfs.hash, verity.json, SHA256SUMS}`. `rootfs.erofs` (erofs data) and `rootfs.hash` (dm-verity hash tree) are **separate** namespaces because the as-built Rocky 9 systemd does not support the in-file `hash-offset` verity layout. No shell, no tools, no base image. Serving software (SPDK, nginx, dnsmasq) lives in separate, independently versioned images — never merge them into the artifact image. The signed iPXE bootloader (`snponly.efi`) is its own FROM-scratch **`fleet-ipxe`** image (`scripts/build-ipxe-oci.sh`, signed with `db`), mounted into dnsmasq at `/ipxe` — kept out of *both* the artifact image and the (generic, key-agnostic) dnsmasq image because its signature is per-Secure-Boot-environment. Don't bake `snponly.efi` into the dnsmasq image. 1. **Deployments are pinned by digest.** One commit per fleet release in the deployment manifest; that manifest is the single source coupling UKI ↔ namespace. Rollback = revert the commit. ## Things that look like bugs but aren’t - `identity="fleet-host"` in the supplicant config is intentionally generic; RADIUS derives real identity from the certificate CN/SAN. - `/etc/machine-id` is truncated empty in the image; each host generates its own into the tmpfs overlay at boot. - `/etc/fstab` is effectively empty; root comes from netroot + verity. - The build VM has no TPM. Correct — nothing host-specific exists at build time. CI boot tests use swtpm. - `files/etc/dot1x/radius-ca.pem` is a placeholder; builds succeed with it but authentication cannot. Replace per environment, don’t commit real internal CA material unless the repo’s classification allows it. ## Known bench-verify items (check before trusting a change in production) - TPM key reference method in wpa_supplicant: `private_key="handle:..."` via OpenSSL tpm2 provider vs tpm2-pkcs11 URI — depends on the distro build; the supplicant config carries both options. - erofs module availability on the target RHEL kernel (fallback: journal-less ext4 in `extract-artifacts.sh`). - STIG remediation: (a) datastream **must match the distro**. The provisioner maps `/etc/os-release` `$ID` → datastream (Rocky→`ssg-rl9-ds.xml`, RHEL→`ssg-rhel9-ds.xml`, al9/cs9/ol9). **Do NOT run the RHEL9 datastream on Rocky** — its CPE dictionary has no Rocky entry, so every rule returns `notapplicable`, the scan evaluates **0** rules, remediation does nothing, and the report is *falsely clean*. (Seen 2026-06-18: a build picked rhel9 via `ls … | head -1` and produced 462 notapplicable + 1068 notselected, **0 pass/fail**.) Two guards now exist: the provisioner aborts the build if 0 rules are evaluated, and `stig-deviations.sh` flags an INVALID SCAN + exits 2. Also confirm the `...content_profile_stig` profile id exists in the chosen datastream; (b) remediation must not break boot/login (NM/NVMe, key-only sshd, kiosk, SELinux enforcing) — `enable_stig=false` is the escape hatch; (c) **reproducibility**: build twice and confirm identical `root_hash` in verity.json — if it drifts, oscap wrote something nondeterministic (timestamp/uuid/key); find + normalize it in 2d hygiene; (d) read the generated `stig-deviations.md` — every REVIEW row is a real finding to triage or justify in `docs/stig-deviations.md`. **STIG-breaks-login gotchas already hit + fixed (2026-06-19) — don't regress:** (i) **fapolicyd** blocks non-RPM executables → the kiosk's `/usr/local/bin/kiosk-session` must be in fapolicyd's trust (done in 2c-STIG-postfix); keep fapolicyd, don't waive it (it's the writable-overlay exec guard dm-verity can't provide). (ii) **Admin SSH key must be ECDSA/RSA, NOT ed25519** — the STIG crypto policy drops ed25519 from sshd's accepted pubkey algorithms (`server-sig-algs` shows ecdsa+rsa-sha2 only), so an ed25519 key gets `Permission denied (publickey)` at the signature stage. (iii) **passwordless/key-only appliance accounts** (kiosk, fleetadmin) are exempted from STIG password-aging + faillock account phase in 2c-STIG-postfix (else locked out). (iv) the kiosk unit is `WantedBy=multi-user.target` (not graphical.target, which the host never reaches → unit dead at boot). Offline diagnosis without booting: mount the OCI image's `rootfs.erofs` read-only and inspect `/etc/pam.d`, fapolicyd, sshd config, etc. **REVIEW-row triage (2026-06-20, scan app-9f7d8d1) added two more bench gates — don't regress:** (v) **USBGuard is ENABLED + enforcing** (block target) with a hardware-independent HID/hub allowlist + `AuditBackend=LinuxAudit` (2c-STIG). The generated policy is built on the virtio build VM (no real USB), so without the HID/hub allow rules a **physical console keyboard/mouse is blocked** on real hardware — BENCH-VERIFY console input. (vi) **admin sudo SELinux elevation**: the admin is mapped to **`staff_u`** (`semanage login`) + sudoers `TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL` so `selinux_context_elevation_for_sudo` passes; `staff_u` (not `sysadm_u`) so SSH login isn't gated by `ssh_sysadm_login`. BENCH-VERIFY admin SSH login + sudo + privileged ops under confinement, and **re-check reproducibility** (semanage rebuilds the policy store → possible verity-hash drift). The triage waivers/decisions live in the tailoring + docs/stig-deviations.md #12; the generated report should show 0–1 REVIEW. - NTP via DHCP option 42 → chrony (no public-pool egress): confirm on a booted host `chronyc sources` lists the DHCP-handed lab server (not `*.pool.ntp.org`). Assumes (a) NM exports the lease's NTP servers to the dispatcher as `$DHCP4_NTP_SERVERS`, and (b) RHEL 9 `/etc/chrony.conf` honours `sourcedir /run/chrony-dhcp` (the provisioner appends it if absent). If sources are empty, the dispatcher didn't fire or the env var name differs — check `journalctl -u NetworkManager` and `ls /run/chrony-dhcp/`. - TLS audit offload (2f → `serving/audit-collector`, gtls/6514, since 2026-07-12): the plaintext-514 path was bench-verified; the TLS upgrade is NOT yet. On a booted host confirm the sender is ESTABLISHED to **:6514** (not the legacy :514), records land in `audit-collector/logs//`, and a wrong CA is rejected (sender spools + logs a cert-validation error — proves x509/name is enforced). Prereqs: collector cert issued (`audit-collector/issue-cert.sh`, SAN = `audit.fleet.lab`, same step-ca as the build's `fleet_ca`), 6514/tcp open on the boot NIC (host-networked — same ufw trap as 514), image built with the real fleet CA. The three `rsyslog_encrypt_offload_*` STIG rules are enforced (waivers dropped) and must PASS in the report. installed (`/usr/lib64/plymouth/script.so` — else the build fell back to `spinner`, logged as a WARNING), (b) the scripted theme + a font got bundled into the initramfs (`lsinitrd | grep -E 'fleet.script|script.so'`), and (c) it renders (text via `Image.Text` needs the label plugin + a font in the early initramfs). If blank, the splash silently no-ops but boot continues; ESC shows the log. **RENDER-VERIFIED 2026-06-18** (test-path build): the scripted theme draws and the handshake checklist updates from the hook status tokens. Still unverified on the **secure path** (the auth/macsec rows reaching `done` — see #8 bench items). - `systemd.verity_root_options=hash-offset` is **NOT supported** by Rocky 9.8 systemd-veritysetup (verified: "unknown option hash-offset=" → "Failed to load verity superblock"). Resolved by the documented fallback — data and hash are **two separate NVMe namespaces** (`rootfs.erofs` + `rootfs.hash`, distinct pinned UUIDs), so the cmdline uses plain `systemd.verity_root_data=` / `systemd.verity_root_hash=` by-id devices with no offset. - **Network module: `network-manager` (ported 2026-06-17 from `network-legacy`).** network-legacy's one-shot DHCP lease was never renewed after switch-root, so at lease expiry the address was dropped and the NVMe/TCP root died. The `network-manager` module hands its initramfs connections to the real-root NM (state under `/run/NetworkManager`), which renews the lease. No more `dhcp-client`/`dhclient` (deprecated). The hook drives macsec0 DHCP via `nmcli` (not `ifup`). See invariant on the managed/unmanaged split. Test path VERIFIED on the bench 2026-06-17 (lease renews past expiry; host stays up). STILL BENCH-VERIFY when `disable_dot1x=false`: macsec0 managed + `en*` unmanaged with MKA surviving switch-root (secure path). - Writable root overlay is `systemd.volatile=overlay`, **not** `rootovl`: RHEL/Rocky dracut ships no `overlayfs`/`rootovl` module. Needs `systemd-volatile-root.service` in the initramfs (bundled by recent dracut; `install_items` it if the root stays read-only). Misbehaves with NFS root — N/A here (NVMe block root). ## RHEL 9.x build notes (learned — don't re-discover) These bit us bringing the build up on Rocky 9.8; they are easy to regress. - **ISO**: default is a version-pinned Rocky 9.x **minimal** ISO on the internal mirror (`iso_url` + `sha256` in `iso_checksum`; bump both together on a point release). `http/ks.cfg` installs only `@^minimal-environment`+sshd+chrony from the ISO; the diskless tooling is pulled **online** by the provisioner, so the build VM needs outbound repo access. Keeps the build ISO-variant-agnostic. - **`%{...}` in provisioner shell** (e.g. `rpm --qf '%{VERSION}'`) must be escaped `%%{...}` — Packer's HCL2 engine parses a bare `%{` as a template directive and `packer validate` fails otherwise. - **File provisioner staging**: pre-create the destination dir (`mkdir -p /tmp/etc-stage`) before a trailing-slash *contents* upload; Packer requires the dest to exist or the tree lands elsewhere and later `install` steps fail. - **dracut module name** in `add_dracutmodules` omits the `NN` dir-prefix: the directory is `95dot1x-macsec` but the module is `dot1x-macsec` (cf. `nvmf`). - **`network-manager` dracut module** (was `network-legacy`+`dhcp-client`) and **overlay → `systemd.volatile=overlay`**: see the bench-verify items above. - **Test toggle**: `disable_dot1x=true` bakes `rd.dot1x=0`; the hook then skips 802.1X/MACsec and DHCPs the NIC directly for NVMe-oF on an open port. The auth stack stays in the image, dormant. The matching lab DHCP/HTTP/NVMe-oF serving stack lives in `serving/` (docker compose). Re-enable security by dropping the var / setting `DISABLE_DOT1X: "false"` in the CI workflow. - **STIG toggle**: `enable_stig=true` (default) applies DISA STIG via OpenSCAP/SSG in a provisioner, tailored for the diskless/UKI/kiosk architecture (`files/stig/diskless-tailoring.xml` unselects the N/A or dangerous-to-remediate rules; every waiver is justified in `docs/stig-deviations.md`). STIG kernel params ride the **UKI cmdline** (`extract-artifacts.sh`, gated by the same flag) since there's no GRUB. **FIPS is OFF** (decision); audit offload is **remote over syslog-TLS** (2f → `serving/audit-collector`, gtls/6514, fleet-CA x509/name; empty `audit_host` = local-only). CI is **report-only**: it publishes `stig-report.html` + `stig-arf.xml` + `stig-deviations.md` (generated by `scripts/stig-deviations.sh`) as artifacts and never fails on score. The scanner tooling is removed from the shipped image (2d hygiene). `enable_stig=false` is the escape hatch if hardening regresses the boot/secure-path bench. NEVER let a waiver hide a real finding — anything non-pass that isn't in the register shows as **REVIEW** in the report. - **Boot splash toggle**: `boot_splash=true` (default) bakes `rhgb quiet` for the Plymouth splash; `false` bakes `plymouth.enable=0` for a verbose console boot — keep OFF while debugging the boot chain. Plymouth is always in the image + initramfs (the `plymouth` dracut module, which pulls `drm`); only the cmdline changes. Press ESC at boot to see details behind a splash. ks.cfg no longer excludes `plymouth`. - **Splash theme = custom `fleet`** (`files/plymouth/themes/fleet/`, script plugin, asset-free — drawn with `Image.Text`). It renders a security-handshake checklist (TPM → 802.1X → MACsec → Network → NVMe-oF) driven by best-effort `plymouth update --status=` calls from the dracut hooks (`dot1x-macsec-up.sh`, `nvmf-connect.sh`). Tokens: `mode-test` (marks auth/MACsec skipped), `tpm|dot1x|macsec|net|nvme`-`active|done`. **Rules:** the emits are non-fatal (`plymouth … || true`) and must never reorder the load-bearing boot sequence; show only STATES, never per-host data (invariant #1). The provisioner installs the script plugin best-effort and **falls back to the stock `spinner` theme** if `script.so` is absent, so a plugin-package-name change can't break the build. ## Testing changes Full boot-chain smoke test runs in QEMU/Proxmox: swtpm (test key at the fleet handle, test cert in NV) + hostapd as wired 802.1X/MACsec authenticator + FreeRADIUS on a veth/bridge (bridge needs `group_fwd_mask=8` or EAPOL is silently dropped). Pass criteria: port `Authorized`, macsec0 addressed, NVMe namespace connected, verity root mapped, overlay writable, supplicant alive after switch-root. This harness is scaffolded in **`serving/dot1x-bench/`** (step-ca as the fleet CA on the serving host + FreeRADIUS EAP-TLS + a `hostapd` wired authenticator + `tpm-provision.sh`), with an **isolated bridge per workstation** (the flat VLAN can't authenticate — EAPOL is dropped and MACsec is point-to-point). See its README; several configs there are still BENCH-VERIFY. Any change to the dracut module, supplicant config, or cmdline must pass this before merge. Template-only changes need `packer validate` + shellcheck at minimum. ## Conventions - Shell: bash, `set -euo pipefail` in scripts (dracut hooks source dracut-lib instead and must stay POSIX-ish bash compatible with the initramfs environment — no GNU-only tools not installed by the module). - Keep comments that explain *why* (timing constraints, kernel behaviours); they encode bench findings that are expensive to rediscover. - Versioning: semver on `image_version`; the OCI tag matches; the git tag matches; one release == one digest == one manifest commit.