Files
diskless-workstation/CLAUDE.md
T
2026-06-27 10:44:42 +00:00

350 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CLAUDE.md — Diskless Fleet Boot System
This repo builds the boot system for a fleet of diskless hosts:
PXE/HTTP-booted, authenticating to the network with **802.1X EAP-TLS using a
TPM-held key**, running **MACsec** on the access link, and mounting an
immutable **erofs + dm-verity root over NVMe/TCP**. Each release is a signed
**UKI** plus root image packaged as a **FROM-scratch OCI artifact image**.
The authoritative architecture document is
`docs/diskless-fleet-design-v2.pdf` — read it before structural changes.
`docs/design-v2-errata.md` records where the as-built RHEL/Rocky 9
implementation diverges from it (pending a consolidated v3).
## Backlog (read at session start)
`docs/TODO.md` is the **persistent backlog** and the durable source of truth for
outstanding work — it survives across sessions (the TodoWrite task tracker does
not). At the start of a session, read it and mirror its open items into the
TodoWrite tracker. Whenever items change (added, started, completed), update
**both** `docs/TODO.md` and the tracker so they stay in sync.
## Repo layout
```
diskless.pkr.hcl Packer template (qemu builder)
http/ks.cfg Kickstart for the throwaway build VM
files/dracut/95dot1x-macsec/ Custom dracut module (the heart of the system)
files/etc/dot1x/ Supplicant config + RADIUS CA (placeholder)
files/etc/dracut.conf.d/90-diskless.conf no-hostonly + drivers + modules
(NM unmanaged split is now runtime:
95dot1x-macsec/nm-unmanaged.sh, secure-mode only)
files/plymouth/themes/fleet/ Custom boot splash (scripted security
checklist; driven by hook status tokens)
files/kiosk/ weston+Chromium kiosk + chromium lockdown policy
scripts/extract-artifacts.sh qcow2 -> kernel/initramfs/erofs+verity/cmdline
scripts/build-oci.sh ukify + sbsign + roothash gate + OCI image
scripts/build-ipxe-oci.sh sbsign snponly.efi -> FROM-scratch fleet-ipxe image
serving/ Lab serving stack: dnsmasq + nginx + SPDK
(dnsmasq + fleet-ipxe + SPDK images are
CI-built + pulled; see .gitea/workflows/)
serving/spdk/ LONG-LIVED nvmf_tgt: spdk-target.sh
(persistent entrypoint, never restarts
for a version change) + spdk-release.sh
(online add/remove/drain a version, 1
subsystem/version) + spdk-ctl.sh (host
load/unload/list for manual rolling
updates) + spdk-loader.sh (DECLARATIVE:
compose service that adds ${IMAGE_VERSION}
online then auto-reaps drained old
versions — bump tag + compose up -d).
releases/<ver>/ = staged rootfs+verity.json.
spdk-release parses RPC JSON via python3
(SPDK image has no jq). spdk-setup.sh
(parent dir) is the legacy single-version
entrypoint (deprecated).
serving/kiosk-app/ Svelte+Tailwind spaceship-terminal SPA
— GIT SUBMODULE (own repo + CI; builds
the fleet-kiosk image the stack pulls).
OS-side kiosk runtime stays in files/kiosk/
serving/dot1x-bench/ Secure-path bench: step-ca + FreeRADIUS +
hostapd authenticator + TPM enrolment
serving/siem/ Wazuh SIEM overlay (audit analysis;
agentless relay from audit-collector)
.gitea/workflows/ Gitea CI: lint -> packer build -> push OCI
docs/ Design PDF and supporting notes
```
## Build & test commands
```bash
packer init .
packer validate . # always before committing template changes
packer build -var image_version=X.Y.Z \
-var nic_drivers="ice mlx5_core" \
-var oci_image=<registry>/fleet-boot . # add sb_key/sb_cert for signed builds
# add disable_dot1x=true for an
# open-port test UKI (rd.dot1x=0)
bash -n files/dracut/95dot1x-macsec/*.sh scripts/*.sh # shell syntax gate
# .shellcheckrc disables the dracut false positives; CI passes them explicitly:
shellcheck --exclude=SC1091,SC2154 files/dracut/95dot1x-macsec/*.sh scripts/*.sh
```
Build-host deps: qemu-kvm, libguestfs-tools (+ a kernel for its supermin
appliance), erofs-utils, cryptsetup, jq, systemd-ukify **and systemd-boot-efi**
(the latter ships the `linuxx64.efi.stub` ukify needs), sbsigntools,
buildah (or docker), **attr** (`getfattr` — `extract-artifacts.sh` asserts the
SELinux labels survived the export; an unlabeled enforcing root denies login).
## Architecture invariants — do not violate these
These encode failure modes that brick the fleet. Treat them as hard
constraints; if a task appears to require breaking one, stop and flag it
instead of working around it.
1. **No per-host state in any artifact, ever.** No hostnames, keys, certs,
MACs, or per-host config in the initramfs, rootfs, UKI, or OCI image.
All host identity derives from the TPM at runtime (key at persistent
handle `0x81010001`, cert at NV index `0x01800001`, hostnqn derived from
the cert serial). If a change needs per-host data, it belongs in the
enrolment flow, not the image.
1. **The hook ordering in `dot1x-macsec-up.sh` is load-bearing.**
NV-cert extraction → supplicant start → wait for `Authorized` + macsec0 →
DHCP **on macsec0** → hostnqn derivation. Never DHCP before auth
completes (pre-auth leases are invalid after VLAN assignment); never
connect NVMe before macsec0 is addressed.
1. **The supplicant must survive switch-root.** It is launched with
`exec -a @wpa_supplicant` (systemd root-storage-daemon convention) and
its state lives under `/run`. Do not “clean up” the `@`, move state off
`/run`, or hand the **physical NIC** to NetworkManager — MKA times out in
~6 s without keepalives and the root filesystem link dies. The NM
unmanaged split exists for this reason (see below); keep it in sync with
interface naming.
1. **NM owns IP/DHCP; wpa_supplicant owns the physical NIC.** The initramfs
uses the `network-manager` dracut module (not `network-legacy`) so NM
**renews the DHCP lease** in the running system after switch-root — without
this the one-shot lease expired, the address was dropped, and the NVMe/TCP
root died (“keep alive timeout” / “no route to host”). The managed/unmanaged
split is therefore load-bearing and **mode-aware**:
- **Secure path:** IP rides on **`macsec0`**, which must stay **MANAGED** by
NM. The **physical NIC (`en*`)** must stay **UNMANAGED** (wpa_supplicant
owns EAPOL+MKA on it). This is enforced at runtime by the cmdline-stage
hook `nm-unmanaged.sh`, which writes
`/run/NetworkManager/conf.d/10-dot1x-unmanaged.conf` **before NM starts**.
- **Test path (`rd.dot1x=0`):** no macsec0; the physical NIC carries IP and
must be **MANAGED** so NM renews it — so the hook writes nothing.
There is intentionally **no static `99-dot1x-unmanaged.conf`** (it couldn't
be mode-aware — it wrongly unmanaged `en*` in test mode, blocking renewal).
Never add macsec0 back to an unmanaged list, and never statically unmanage
`en*` for all modes.
1. **roothash coupling is a hard gate.** `build-oci.sh` refuses to build if
the `roothash=` in the UKI cmdline ≠ `verity.json`. Never weaken, skip,
or warn-instead-of-fail this check: a mismatched UKI/rootfs pair is an
unbootable fleet.
1. **The UKI cmdline is immutable once signed.** Any cmdline change
(including `NVMET_TRADDR`) requires a rebuild + re-sign. Prefer a
per-environment DNS name for NVMe discovery over per-environment builds.
1. **Signing keys never enter the build VM or this repo.** `sb_key` /
`sb_cert` are CI/HSM-provided paths on the build host. Never commit key
material, never copy keys into provisioner steps, never echo them in
logs. Unsigned builds are lab-only and must keep their loud warning.
1. **Reproducibility of the root image is a security property.**
`mkfs.erofs` runs with fixed UUID and `-T 0`; the verity root hash must
be a pure function of the input tree. Don’t introduce timestamps,
random seeds, or build-host paths into the rootfs.
1. **The initramfs is built `--no-hostonly` and contents are asserted.**
The build VM is virtio-only; real NIC drivers, `macsec`, `nvme-tcp`,
erofs, and veritysetup are explicitly listed in
`90-diskless.conf` and checked via `lsinitrd | grep` in the template.
When adding a runtime dependency to the hook, add it to the dracut
module’s `install()` *and* extend the lsinitrd assertion.
1. **The OCI artifact image stays FROM scratch.** Only
`/artifacts/{fleet-<ver>.efi, rootfs.erofs, rootfs.hash, verity.json,
SHA256SUMS}`. `rootfs.erofs` (erofs data) and `rootfs.hash` (dm-verity hash
tree) are **separate** namespaces because the as-built Rocky 9 systemd does
not support the in-file `hash-offset` verity layout. No shell, no tools, no
base image. Serving software (SPDK, nginx, dnsmasq) lives in separate,
independently versioned images — never merge them into the artifact image.
The signed iPXE bootloader (`snponly.efi`) is its own FROM-scratch
**`fleet-ipxe`** image (`scripts/build-ipxe-oci.sh`, signed with `db`), mounted
into dnsmasq at `/ipxe` — kept out of *both* the artifact image and the
(generic, key-agnostic) dnsmasq image because its signature is
per-Secure-Boot-environment. Don't bake `snponly.efi` into the dnsmasq image.
1. **Deployments are pinned by digest.** One commit per fleet release in
the deployment manifest; that manifest is the single source coupling
UKI ↔ namespace. Rollback = revert the commit.
## Things that look like bugs but aren’t
- `identity="fleet-host"` in the supplicant config is intentionally generic;
RADIUS derives real identity from the certificate CN/SAN.
- `/etc/machine-id` is truncated empty in the image; each host generates its
own into the tmpfs overlay at boot.
- `/etc/fstab` is effectively empty; root comes from netroot + verity.
- The build VM has no TPM. Correct — nothing host-specific exists at build
time. CI boot tests use swtpm.
- `files/etc/dot1x/radius-ca.pem` is a placeholder; builds succeed with it
but authentication cannot. Replace per environment, don’t commit real
internal CA material unless the repo’s classification allows it.
## Known bench-verify items (check before trusting a change in production)
- TPM key reference method in wpa_supplicant: `private_key="handle:..."`
via OpenSSL tpm2 provider vs tpm2-pkcs11 URI — depends on the distro
build; the supplicant config carries both options.
- erofs module availability on the target RHEL kernel (fallback:
journal-less ext4 in `extract-artifacts.sh`).
- STIG remediation: (a) datastream **must match the distro**. The provisioner
maps `/etc/os-release` `$ID` → datastream (Rocky→`ssg-rl9-ds.xml`,
RHEL→`ssg-rhel9-ds.xml`, al9/cs9/ol9). **Do NOT run the RHEL9 datastream on
Rocky** — its CPE dictionary has no Rocky entry, so every rule returns
`notapplicable`, the scan evaluates **0** rules, remediation does nothing, and
the report is *falsely clean*. (Seen 2026-06-18: a build picked rhel9 via
`ls … | head -1` and produced 462 notapplicable + 1068 notselected, **0
pass/fail**.) Two guards now exist: the provisioner aborts the build if 0 rules
are evaluated, and `stig-deviations.sh` flags an INVALID SCAN + exits 2. Also
confirm the `...content_profile_stig` profile id exists in the chosen
datastream; (b) remediation must not break
boot/login (NM/NVMe, key-only sshd, kiosk, SELinux enforcing) — `enable_stig=false`
is the escape hatch; (c) **reproducibility**: build twice and confirm identical
`root_hash` in verity.json — if it drifts, oscap wrote something
nondeterministic (timestamp/uuid/key); find + normalize it in 2d hygiene;
(d) read the generated `stig-deviations.md` — every REVIEW row is a real finding
to triage or justify in `docs/stig-deviations.md`.
**STIG-breaks-login gotchas already hit + fixed (2026-06-19) — don't regress:**
(i) **fapolicyd** blocks non-RPM executables → the kiosk's
`/usr/local/bin/kiosk-session` must be in fapolicyd's trust (done in
2c-STIG-postfix); keep fapolicyd, don't waive it (it's the writable-overlay
exec guard dm-verity can't provide). (ii) **Admin SSH key must be ECDSA/RSA,
NOT ed25519** — the STIG crypto policy drops ed25519 from sshd's accepted
pubkey algorithms (`server-sig-algs` shows ecdsa+rsa-sha2 only), so an ed25519
key gets `Permission denied (publickey)` at the signature stage. (iii)
**passwordless/key-only appliance accounts** (kiosk, fleetadmin) are exempted
from STIG password-aging + faillock account phase in 2c-STIG-postfix (else
locked out). (iv) the kiosk unit is `WantedBy=multi-user.target` (not
graphical.target, which the host never reaches → unit dead at boot).
Offline diagnosis without booting: mount the OCI image's `rootfs.erofs`
read-only and inspect `/etc/pam.d`, fapolicyd, sshd config, etc.
**REVIEW-row triage (2026-06-20, scan app-9f7d8d1) added two more bench gates —
don't regress:** (v) **USBGuard is ENABLED + enforcing** (block target) with a
hardware-independent HID/hub allowlist + `AuditBackend=LinuxAudit` (2c-STIG). The
generated policy is built on the virtio build VM (no real USB), so without the
HID/hub allow rules a **physical console keyboard/mouse is blocked** on real
hardware — BENCH-VERIFY console input. (vi) **admin sudo SELinux elevation**:
the admin is mapped to **`staff_u`** (`semanage login`) + sudoers
`TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL` so `selinux_context_elevation_for_sudo`
passes; `staff_u` (not `sysadm_u`) so SSH login isn't gated by `ssh_sysadm_login`.
BENCH-VERIFY admin SSH login + sudo + privileged ops under confinement, and
**re-check reproducibility** (semanage rebuilds the policy store → possible
verity-hash drift). The triage waivers/decisions live in the tailoring +
docs/stig-deviations.md #12; the generated report should show 0–1 REVIEW.
- NTP via DHCP option 42 → chrony (no public-pool egress): confirm on a booted
host `chronyc sources` lists the DHCP-handed lab server (not `*.pool.ntp.org`).
Assumes (a) NM exports the lease's NTP servers to the dispatcher as
`$DHCP4_NTP_SERVERS`, and (b) RHEL 9 `/etc/chrony.conf` honours
`sourcedir /run/chrony-dhcp` (the provisioner appends it if absent). If sources
are empty, the dispatcher didn't fire or the env var name differs — check
`journalctl -u NetworkManager` and `ls /run/chrony-dhcp/`.
- `fleet` Plymouth theme: confirm (a) the script plugin package actually
installed (`/usr/lib64/plymouth/script.so` — else the build fell back to
`spinner`, logged as a WARNING), (b) the scripted theme + a font got bundled
into the initramfs (`lsinitrd | grep -E 'fleet.script|script.so'`), and (c) it
renders (text via `Image.Text` needs the label plugin + a font in the early
initramfs). If blank, the splash silently no-ops but boot continues; ESC shows
the log. **RENDER-VERIFIED 2026-06-18** (test-path build): the scripted theme
draws and the handshake checklist updates from the hook status tokens. Still
unverified on the **secure path** (the auth/macsec rows reaching `done` — see
#8 bench items).
- `systemd.verity_root_options=hash-offset` is **NOT supported** by Rocky 9.8
systemd-veritysetup (verified: "unknown option hash-offset=" → "Failed to load
verity superblock"). Resolved by the documented fallback — data and hash are
**two separate NVMe namespaces** (`rootfs.erofs` + `rootfs.hash`, distinct
pinned UUIDs), so the cmdline uses plain `systemd.verity_root_data=` /
`systemd.verity_root_hash=` by-id devices with no offset.
- **Network module: `network-manager` (ported 2026-06-17 from `network-legacy`).**
network-legacy's one-shot DHCP lease was never renewed after switch-root, so
at lease expiry the address was dropped and the NVMe/TCP root died. The
`network-manager` module hands its initramfs connections to the real-root NM
(state under `/run/NetworkManager`), which renews the lease. No more
`dhcp-client`/`dhclient` (deprecated). The hook drives macsec0 DHCP via `nmcli`
(not `ifup`). See invariant on the managed/unmanaged split. Test path
VERIFIED on the bench 2026-06-17 (lease renews past expiry; host stays up).
STILL BENCH-VERIFY when `disable_dot1x=false`: macsec0 managed + `en*`
unmanaged with MKA surviving switch-root (secure path).
- Writable root overlay is `systemd.volatile=overlay`, **not** `rootovl`:
RHEL/Rocky dracut ships no `overlayfs`/`rootovl` module. Needs
`systemd-volatile-root.service` in the initramfs (bundled by recent dracut;
`install_items` it if the root stays read-only). Misbehaves with NFS root —
N/A here (NVMe block root).
## RHEL 9.x build notes (learned — don't re-discover)
These bit us bringing the build up on Rocky 9.8; they are easy to regress.
- **ISO**: default is a version-pinned Rocky 9.x **minimal** ISO on the internal
mirror (`iso_url` + `sha256` in `iso_checksum`; bump both together on a point
release). `http/ks.cfg` installs only `@^minimal-environment`+sshd+chrony from
the ISO; the diskless tooling is pulled **online** by the provisioner, so the
build VM needs outbound repo access. Keeps the build ISO-variant-agnostic.
- **`%{...}` in provisioner shell** (e.g. `rpm --qf '%{VERSION}'`) must be
escaped `%%{...}` — Packer's HCL2 engine parses a bare `%{` as a template
directive and `packer validate` fails otherwise.
- **File provisioner staging**: pre-create the destination dir (`mkdir -p
/tmp/etc-stage`) before a trailing-slash *contents* upload; Packer requires
the dest to exist or the tree lands elsewhere and later `install` steps fail.
- **dracut module name** in `add_dracutmodules` omits the `NN` dir-prefix: the
directory is `95dot1x-macsec` but the module is `dot1x-macsec` (cf. `nvmf`).
- **`network-manager` dracut module** (was `network-legacy`+`dhcp-client`) and
**overlay → `systemd.volatile=overlay`**: see the bench-verify items above.
- **Test toggle**: `disable_dot1x=true` bakes `rd.dot1x=0`; the hook then skips
802.1X/MACsec and DHCPs the NIC directly for NVMe-oF on an open port. The auth
stack stays in the image, dormant. The matching lab DHCP/HTTP/NVMe-oF serving
stack lives in `serving/` (docker compose). Re-enable security by dropping the
var / setting `DISABLE_DOT1X: "false"` in the CI workflow.
- **STIG toggle**: `enable_stig=true` (default) applies DISA STIG via
OpenSCAP/SSG in a provisioner, tailored for the diskless/UKI/kiosk architecture
(`files/stig/diskless-tailoring.xml` unselects the N/A or
dangerous-to-remediate rules; every waiver is justified in
`docs/stig-deviations.md`). STIG kernel params ride the **UKI cmdline**
(`extract-artifacts.sh`, gated by the same flag) since there's no GRUB.
**FIPS is OFF and audit offload is LOCAL-only** (decisions). CI is
**report-only**: it publishes `stig-report.html` + `stig-arf.xml` +
`stig-deviations.md` (generated by `scripts/stig-deviations.sh`) as artifacts
and never fails on score. The scanner tooling is removed from the shipped image
(2d hygiene). `enable_stig=false` is the escape hatch if hardening regresses
the boot/secure-path bench. NEVER let a waiver hide a real finding — anything
non-pass that isn't in the register shows as **REVIEW** in the report.
- **Boot splash toggle**: `boot_splash=true` (default) bakes `rhgb quiet` for the
Plymouth splash; `false` bakes `plymouth.enable=0` for a verbose console boot —
keep OFF while debugging the boot chain. Plymouth is always in the image +
initramfs (the `plymouth` dracut module, which pulls `drm`); only the cmdline
changes. Press ESC at boot to see details behind a splash. ks.cfg no longer
excludes `plymouth`.
- **Splash theme = custom `fleet`** (`files/plymouth/themes/fleet/`, script
plugin, asset-free — drawn with `Image.Text`). It renders a security-handshake
checklist (TPM → 802.1X → MACsec → Network → NVMe-oF) driven by best-effort
`plymouth update --status=<token>` calls from the dracut hooks
(`dot1x-macsec-up.sh`, `nvmf-connect.sh`). Tokens: `mode-test` (marks
auth/MACsec skipped), `tpm|dot1x|macsec|net|nvme`-`active|done`. **Rules:** the
emits are non-fatal (`plymouth … || true`) and must never reorder the
load-bearing boot sequence; show only STATES, never per-host data (invariant
#1). The provisioner installs the script plugin best-effort and **falls back to
the stock `spinner` theme** if `script.so` is absent, so a plugin-package-name
change can't break the build.
## Testing changes
Full boot-chain smoke test runs in QEMU/Proxmox: swtpm (test key at the
fleet handle, test cert in NV) + hostapd as wired 802.1X/MACsec
authenticator + FreeRADIUS on a veth/bridge (bridge needs
`group_fwd_mask=8` or EAPOL is silently dropped). Pass criteria: port
`Authorized`, macsec0 addressed, NVMe namespace connected, verity root
mapped, overlay writable, supplicant alive after switch-root.
This harness is scaffolded in **`serving/dot1x-bench/`** (step-ca as the fleet
CA on the serving host + FreeRADIUS EAP-TLS + a `hostapd` wired authenticator +
`tpm-provision.sh`), with an **isolated bridge per workstation** (the flat VLAN
can't authenticate — EAPOL is dropped and MACsec is point-to-point). See its
README; several configs there are still BENCH-VERIFY.
Any change to the dracut module, supplicant config, or cmdline must pass
this before merge. Template-only changes need `packer validate` +
shellcheck at minimum.
## Conventions
- Shell: bash, `set -euo pipefail` in scripts (dracut hooks source
dracut-lib instead and must stay POSIX-ish bash compatible with the
initramfs environment — no GNU-only tools not installed by the module).
- Keep comments that explain *why* (timing constraints, kernel behaviours);
they encode bench findings that are expensive to rediscover.
- Versioning: semver on `image_version`; the OCI tag matches; the git tag
matches; one release == one digest == one manifest commit.