333 lines
21 KiB
Markdown
333 lines
21 KiB
Markdown
# CLAUDE.md — Diskless Fleet Boot System
|
||
|
||
This repo builds the boot system for a fleet of diskless hosts:
|
||
PXE/HTTP-booted, authenticating to the network with **802.1X EAP-TLS using a
|
||
TPM-held key**, running **MACsec** on the access link, and mounting an
|
||
immutable **erofs + dm-verity root over NVMe/TCP**. Each release is a signed
|
||
**UKI** plus root image packaged as a **FROM-scratch OCI artifact image**.
|
||
The authoritative architecture document is
|
||
`docs/diskless-fleet-design-v2.pdf` — read it before structural changes.
|
||
`docs/design-v2-errata.md` records where the as-built RHEL/Rocky 9
|
||
implementation diverges from it (pending a consolidated v3).
|
||
|
||
## Backlog (read at session start)
|
||
|
||
`docs/TODO.md` is the **persistent backlog** and the durable source of truth for
|
||
outstanding work — it survives across sessions (the TodoWrite task tracker does
|
||
not). At the start of a session, read it and mirror its open items into the
|
||
TodoWrite tracker. Whenever items change (added, started, completed), update
|
||
**both** `docs/TODO.md` and the tracker so they stay in sync.
|
||
|
||
## Repo layout
|
||
|
||
```
|
||
diskless.pkr.hcl Packer template (qemu builder)
|
||
http/ks.cfg Kickstart for the throwaway build VM
|
||
files/dracut/95dot1x-macsec/ Custom dracut module (the heart of the system)
|
||
files/etc/dot1x/ Supplicant config + RADIUS CA (placeholder)
|
||
files/etc/dracut.conf.d/90-diskless.conf no-hostonly + drivers + modules
|
||
(NM unmanaged split is now runtime:
|
||
95dot1x-macsec/nm-unmanaged.sh, secure-mode only)
|
||
files/plymouth/themes/fleet/ Custom boot splash (scripted security
|
||
checklist; driven by hook status tokens)
|
||
files/kiosk/ weston+Chromium kiosk + chromium lockdown policy
|
||
scripts/extract-artifacts.sh qcow2 -> kernel/initramfs/erofs+verity/cmdline
|
||
scripts/build-oci.sh ukify + sbsign + roothash gate + OCI image
|
||
scripts/build-ipxe-oci.sh sbsign snponly.efi -> FROM-scratch fleet-ipxe image
|
||
serving/ Lab serving stack: dnsmasq + nginx + SPDK
|
||
(dnsmasq + fleet-ipxe + SPDK images are
|
||
CI-built + pulled; see .gitea/workflows/)
|
||
serving/kiosk-app/ Svelte+Tailwind spaceship-terminal SPA
|
||
— GIT SUBMODULE (own repo + CI; builds
|
||
the fleet-kiosk image the stack pulls).
|
||
OS-side kiosk runtime stays in files/kiosk/
|
||
serving/dot1x-bench/ Secure-path bench: step-ca + FreeRADIUS +
|
||
hostapd authenticator + TPM enrolment
|
||
.gitea/workflows/ Gitea CI: lint -> packer build -> push OCI
|
||
docs/ Design PDF and supporting notes
|
||
```
|
||
|
||
## Build & test commands
|
||
|
||
```bash
|
||
packer init .
|
||
packer validate . # always before committing template changes
|
||
packer build -var image_version=X.Y.Z \
|
||
-var nic_drivers="ice mlx5_core" \
|
||
-var oci_image=<registry>/fleet-boot . # add sb_key/sb_cert for signed builds
|
||
# add disable_dot1x=true for an
|
||
# open-port test UKI (rd.dot1x=0)
|
||
bash -n files/dracut/95dot1x-macsec/*.sh scripts/*.sh # shell syntax gate
|
||
# .shellcheckrc disables the dracut false positives; CI passes them explicitly:
|
||
shellcheck --exclude=SC1091,SC2154 files/dracut/95dot1x-macsec/*.sh scripts/*.sh
|
||
```
|
||
|
||
Build-host deps: qemu-kvm, libguestfs-tools (+ a kernel for its supermin
|
||
appliance), erofs-utils, cryptsetup, jq, systemd-ukify **and systemd-boot-efi**
|
||
(the latter ships the `linuxx64.efi.stub` ukify needs), sbsigntools,
|
||
buildah (or docker), **attr** (`getfattr` — `extract-artifacts.sh` asserts the
|
||
SELinux labels survived the export; an unlabeled enforcing root denies login).
|
||
|
||
## Architecture invariants — do not violate these
|
||
|
||
These encode failure modes that brick the fleet. Treat them as hard
|
||
constraints; if a task appears to require breaking one, stop and flag it
|
||
instead of working around it.
|
||
|
||
1. **No per-host state in any artifact, ever.** No hostnames, keys, certs,
|
||
MACs, or per-host config in the initramfs, rootfs, UKI, or OCI image.
|
||
All host identity derives from the TPM at runtime (key at persistent
|
||
handle `0x81010001`, cert at NV index `0x01800001`, hostnqn derived from
|
||
the cert serial). If a change needs per-host data, it belongs in the
|
||
enrolment flow, not the image.
|
||
1. **The hook ordering in `dot1x-macsec-up.sh` is load-bearing.**
|
||
NV-cert extraction → supplicant start → wait for `Authorized` + macsec0 →
|
||
DHCP **on macsec0** → hostnqn derivation. Never DHCP before auth
|
||
completes (pre-auth leases are invalid after VLAN assignment); never
|
||
connect NVMe before macsec0 is addressed.
|
||
1. **The supplicant must survive switch-root.** It is launched with
|
||
`exec -a @wpa_supplicant` (systemd root-storage-daemon convention) and
|
||
its state lives under `/run`. Do not “clean up” the `@`, move state off
|
||
`/run`, or hand the **physical NIC** to NetworkManager — MKA times out in
|
||
~6 s without keepalives and the root filesystem link dies. The NM
|
||
unmanaged split exists for this reason (see below); keep it in sync with
|
||
interface naming.
|
||
1. **NM owns IP/DHCP; wpa_supplicant owns the physical NIC.** The initramfs
|
||
uses the `network-manager` dracut module (not `network-legacy`) so NM
|
||
**renews the DHCP lease** in the running system after switch-root — without
|
||
this the one-shot lease expired, the address was dropped, and the NVMe/TCP
|
||
root died (“keep alive timeout” / “no route to host”). The managed/unmanaged
|
||
split is therefore load-bearing and **mode-aware**:
|
||
- **Secure path:** IP rides on **`macsec0`**, which must stay **MANAGED** by
|
||
NM. The **physical NIC (`en*`)** must stay **UNMANAGED** (wpa_supplicant
|
||
owns EAPOL+MKA on it). This is enforced at runtime by the cmdline-stage
|
||
hook `nm-unmanaged.sh`, which writes
|
||
`/run/NetworkManager/conf.d/10-dot1x-unmanaged.conf` **before NM starts**.
|
||
- **Test path (`rd.dot1x=0`):** no macsec0; the physical NIC carries IP and
|
||
must be **MANAGED** so NM renews it — so the hook writes nothing.
|
||
There is intentionally **no static `99-dot1x-unmanaged.conf`** (it couldn't
|
||
be mode-aware — it wrongly unmanaged `en*` in test mode, blocking renewal).
|
||
Never add macsec0 back to an unmanaged list, and never statically unmanage
|
||
`en*` for all modes.
|
||
1. **roothash coupling is a hard gate.** `build-oci.sh` refuses to build if
|
||
the `roothash=` in the UKI cmdline ≠ `verity.json`. Never weaken, skip,
|
||
or warn-instead-of-fail this check: a mismatched UKI/rootfs pair is an
|
||
unbootable fleet.
|
||
1. **The UKI cmdline is immutable once signed.** Any cmdline change
|
||
(including `NVMET_TRADDR`) requires a rebuild + re-sign. Prefer a
|
||
per-environment DNS name for NVMe discovery over per-environment builds.
|
||
1. **Signing keys never enter the build VM or this repo.** `sb_key` /
|
||
`sb_cert` are CI/HSM-provided paths on the build host. Never commit key
|
||
material, never copy keys into provisioner steps, never echo them in
|
||
logs. Unsigned builds are lab-only and must keep their loud warning.
|
||
1. **Reproducibility of the root image is a security property.**
|
||
`mkfs.erofs` runs with fixed UUID and `-T 0`; the verity root hash must
|
||
be a pure function of the input tree. Don’t introduce timestamps,
|
||
random seeds, or build-host paths into the rootfs.
|
||
1. **The initramfs is built `--no-hostonly` and contents are asserted.**
|
||
The build VM is virtio-only; real NIC drivers, `macsec`, `nvme-tcp`,
|
||
erofs, and veritysetup are explicitly listed in
|
||
`90-diskless.conf` and checked via `lsinitrd | grep` in the template.
|
||
When adding a runtime dependency to the hook, add it to the dracut
|
||
module’s `install()` *and* extend the lsinitrd assertion.
|
||
1. **The OCI artifact image stays FROM scratch.** Only
|
||
`/artifacts/{fleet-<ver>.efi, rootfs.erofs, rootfs.hash, verity.json,
|
||
SHA256SUMS}`. `rootfs.erofs` (erofs data) and `rootfs.hash` (dm-verity hash
|
||
tree) are **separate** namespaces because the as-built Rocky 9 systemd does
|
||
not support the in-file `hash-offset` verity layout. No shell, no tools, no
|
||
base image. Serving software (SPDK, nginx, dnsmasq) lives in separate,
|
||
independently versioned images — never merge them into the artifact image.
|
||
The signed iPXE bootloader (`snponly.efi`) is its own FROM-scratch
|
||
**`fleet-ipxe`** image (`scripts/build-ipxe-oci.sh`, signed with `db`), mounted
|
||
into dnsmasq at `/ipxe` — kept out of *both* the artifact image and the
|
||
(generic, key-agnostic) dnsmasq image because its signature is
|
||
per-Secure-Boot-environment. Don't bake `snponly.efi` into the dnsmasq image.
|
||
1. **Deployments are pinned by digest.** One commit per fleet release in
|
||
the deployment manifest; that manifest is the single source coupling
|
||
UKI ↔ namespace. Rollback = revert the commit.
|
||
|
||
## Things that look like bugs but aren’t
|
||
|
||
- `identity="fleet-host"` in the supplicant config is intentionally generic;
|
||
RADIUS derives real identity from the certificate CN/SAN.
|
||
- `/etc/machine-id` is truncated empty in the image; each host generates its
|
||
own into the tmpfs overlay at boot.
|
||
- `/etc/fstab` is effectively empty; root comes from netroot + verity.
|
||
- The build VM has no TPM. Correct — nothing host-specific exists at build
|
||
time. CI boot tests use swtpm.
|
||
- `files/etc/dot1x/radius-ca.pem` is a placeholder; builds succeed with it
|
||
but authentication cannot. Replace per environment, don’t commit real
|
||
internal CA material unless the repo’s classification allows it.
|
||
|
||
## Known bench-verify items (check before trusting a change in production)
|
||
|
||
- TPM key reference method in wpa_supplicant: `private_key="handle:..."`
|
||
via OpenSSL tpm2 provider vs tpm2-pkcs11 URI — depends on the distro
|
||
build; the supplicant config carries both options.
|
||
- erofs module availability on the target RHEL kernel (fallback:
|
||
journal-less ext4 in `extract-artifacts.sh`).
|
||
- STIG remediation: (a) datastream **must match the distro**. The provisioner
|
||
maps `/etc/os-release` `$ID` → datastream (Rocky→`ssg-rl9-ds.xml`,
|
||
RHEL→`ssg-rhel9-ds.xml`, al9/cs9/ol9). **Do NOT run the RHEL9 datastream on
|
||
Rocky** — its CPE dictionary has no Rocky entry, so every rule returns
|
||
`notapplicable`, the scan evaluates **0** rules, remediation does nothing, and
|
||
the report is *falsely clean*. (Seen 2026-06-18: a build picked rhel9 via
|
||
`ls … | head -1` and produced 462 notapplicable + 1068 notselected, **0
|
||
pass/fail**.) Two guards now exist: the provisioner aborts the build if 0 rules
|
||
are evaluated, and `stig-deviations.sh` flags an INVALID SCAN + exits 2. Also
|
||
confirm the `...content_profile_stig` profile id exists in the chosen
|
||
datastream; (b) remediation must not break
|
||
boot/login (NM/NVMe, key-only sshd, kiosk, SELinux enforcing) — `enable_stig=false`
|
||
is the escape hatch; (c) **reproducibility**: build twice and confirm identical
|
||
`root_hash` in verity.json — if it drifts, oscap wrote something
|
||
nondeterministic (timestamp/uuid/key); find + normalize it in 2d hygiene;
|
||
(d) read the generated `stig-deviations.md` — every REVIEW row is a real finding
|
||
to triage or justify in `docs/stig-deviations.md`.
|
||
**STIG-breaks-login gotchas already hit + fixed (2026-06-19) — don't regress:**
|
||
(i) **fapolicyd** blocks non-RPM executables → the kiosk's
|
||
`/usr/local/bin/kiosk-session` must be in fapolicyd's trust (done in
|
||
2c-STIG-postfix); keep fapolicyd, don't waive it (it's the writable-overlay
|
||
exec guard dm-verity can't provide). (ii) **Admin SSH key must be ECDSA/RSA,
|
||
NOT ed25519** — the STIG crypto policy drops ed25519 from sshd's accepted
|
||
pubkey algorithms (`server-sig-algs` shows ecdsa+rsa-sha2 only), so an ed25519
|
||
key gets `Permission denied (publickey)` at the signature stage. (iii)
|
||
**passwordless/key-only appliance accounts** (kiosk, fleetadmin) are exempted
|
||
from STIG password-aging + faillock account phase in 2c-STIG-postfix (else
|
||
locked out). (iv) the kiosk unit is `WantedBy=multi-user.target` (not
|
||
graphical.target, which the host never reaches → unit dead at boot).
|
||
Offline diagnosis without booting: mount the OCI image's `rootfs.erofs`
|
||
read-only and inspect `/etc/pam.d`, fapolicyd, sshd config, etc.
|
||
**REVIEW-row triage (2026-06-20, scan app-9f7d8d1) added two more bench gates —
|
||
don't regress:** (v) **USBGuard is ENABLED + enforcing** (block target) with a
|
||
hardware-independent HID/hub allowlist + `AuditBackend=LinuxAudit` (2c-STIG). The
|
||
generated policy is built on the virtio build VM (no real USB), so without the
|
||
HID/hub allow rules a **physical console keyboard/mouse is blocked** on real
|
||
hardware — BENCH-VERIFY console input. (vi) **admin sudo SELinux elevation**:
|
||
the admin is mapped to **`staff_u`** (`semanage login`) + sudoers
|
||
`TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL` so `selinux_context_elevation_for_sudo`
|
||
passes; `staff_u` (not `sysadm_u`) so SSH login isn't gated by `ssh_sysadm_login`.
|
||
BENCH-VERIFY admin SSH login + sudo + privileged ops under confinement, and
|
||
**re-check reproducibility** (semanage rebuilds the policy store → possible
|
||
verity-hash drift). The triage waivers/decisions live in the tailoring +
|
||
docs/stig-deviations.md #12; the generated report should show 0–1 REVIEW.
|
||
- NTP via DHCP option 42 → chrony (no public-pool egress): confirm on a booted
|
||
host `chronyc sources` lists the DHCP-handed lab server (not `*.pool.ntp.org`).
|
||
Assumes (a) NM exports the lease's NTP servers to the dispatcher as
|
||
`$DHCP4_NTP_SERVERS`, and (b) RHEL 9 `/etc/chrony.conf` honours
|
||
`sourcedir /run/chrony-dhcp` (the provisioner appends it if absent). If sources
|
||
are empty, the dispatcher didn't fire or the env var name differs — check
|
||
`journalctl -u NetworkManager` and `ls /run/chrony-dhcp/`.
|
||
- `fleet` Plymouth theme: confirm (a) the script plugin package actually
|
||
installed (`/usr/lib64/plymouth/script.so` — else the build fell back to
|
||
`spinner`, logged as a WARNING), (b) the scripted theme + a font got bundled
|
||
into the initramfs (`lsinitrd | grep -E 'fleet.script|script.so'`), and (c) it
|
||
renders (text via `Image.Text` needs the label plugin + a font in the early
|
||
initramfs). If blank, the splash silently no-ops but boot continues; ESC shows
|
||
the log. **RENDER-VERIFIED 2026-06-18** (test-path build): the scripted theme
|
||
draws and the handshake checklist updates from the hook status tokens. Still
|
||
unverified on the **secure path** (the auth/macsec rows reaching `done` — see
|
||
#8 bench items).
|
||
- `systemd.verity_root_options=hash-offset` is **NOT supported** by Rocky 9.8
|
||
systemd-veritysetup (verified: "unknown option hash-offset=" → "Failed to load
|
||
verity superblock"). Resolved by the documented fallback — data and hash are
|
||
**two separate NVMe namespaces** (`rootfs.erofs` + `rootfs.hash`, distinct
|
||
pinned UUIDs), so the cmdline uses plain `systemd.verity_root_data=` /
|
||
`systemd.verity_root_hash=` by-id devices with no offset.
|
||
- **Network module: `network-manager` (ported 2026-06-17 from `network-legacy`).**
|
||
network-legacy's one-shot DHCP lease was never renewed after switch-root, so
|
||
at lease expiry the address was dropped and the NVMe/TCP root died. The
|
||
`network-manager` module hands its initramfs connections to the real-root NM
|
||
(state under `/run/NetworkManager`), which renews the lease. No more
|
||
`dhcp-client`/`dhclient` (deprecated). The hook drives macsec0 DHCP via `nmcli`
|
||
(not `ifup`). See invariant on the managed/unmanaged split. Test path
|
||
VERIFIED on the bench 2026-06-17 (lease renews past expiry; host stays up).
|
||
STILL BENCH-VERIFY when `disable_dot1x=false`: macsec0 managed + `en*`
|
||
unmanaged with MKA surviving switch-root (secure path).
|
||
- Writable root overlay is `systemd.volatile=overlay`, **not** `rootovl`:
|
||
RHEL/Rocky dracut ships no `overlayfs`/`rootovl` module. Needs
|
||
`systemd-volatile-root.service` in the initramfs (bundled by recent dracut;
|
||
`install_items` it if the root stays read-only). Misbehaves with NFS root —
|
||
N/A here (NVMe block root).
|
||
|
||
## RHEL 9.x build notes (learned — don't re-discover)
|
||
|
||
These bit us bringing the build up on Rocky 9.8; they are easy to regress.
|
||
|
||
- **ISO**: default is a version-pinned Rocky 9.x **minimal** ISO on the internal
|
||
mirror (`iso_url` + `sha256` in `iso_checksum`; bump both together on a point
|
||
release). `http/ks.cfg` installs only `@^minimal-environment`+sshd+chrony from
|
||
the ISO; the diskless tooling is pulled **online** by the provisioner, so the
|
||
build VM needs outbound repo access. Keeps the build ISO-variant-agnostic.
|
||
- **`%{...}` in provisioner shell** (e.g. `rpm --qf '%{VERSION}'`) must be
|
||
escaped `%%{...}` — Packer's HCL2 engine parses a bare `%{` as a template
|
||
directive and `packer validate` fails otherwise.
|
||
- **File provisioner staging**: pre-create the destination dir (`mkdir -p
|
||
/tmp/etc-stage`) before a trailing-slash *contents* upload; Packer requires
|
||
the dest to exist or the tree lands elsewhere and later `install` steps fail.
|
||
- **dracut module name** in `add_dracutmodules` omits the `NN` dir-prefix: the
|
||
directory is `95dot1x-macsec` but the module is `dot1x-macsec` (cf. `nvmf`).
|
||
- **`network-manager` dracut module** (was `network-legacy`+`dhcp-client`) and
|
||
**overlay → `systemd.volatile=overlay`**: see the bench-verify items above.
|
||
- **Test toggle**: `disable_dot1x=true` bakes `rd.dot1x=0`; the hook then skips
|
||
802.1X/MACsec and DHCPs the NIC directly for NVMe-oF on an open port. The auth
|
||
stack stays in the image, dormant. The matching lab DHCP/HTTP/NVMe-oF serving
|
||
stack lives in `serving/` (docker compose). Re-enable security by dropping the
|
||
var / setting `DISABLE_DOT1X: "false"` in the CI workflow.
|
||
- **STIG toggle**: `enable_stig=true` (default) applies DISA STIG via
|
||
OpenSCAP/SSG in a provisioner, tailored for the diskless/UKI/kiosk architecture
|
||
(`files/stig/diskless-tailoring.xml` unselects the N/A or
|
||
dangerous-to-remediate rules; every waiver is justified in
|
||
`docs/stig-deviations.md`). STIG kernel params ride the **UKI cmdline**
|
||
(`extract-artifacts.sh`, gated by the same flag) since there's no GRUB.
|
||
**FIPS is OFF and audit offload is LOCAL-only** (decisions). CI is
|
||
**report-only**: it publishes `stig-report.html` + `stig-arf.xml` +
|
||
`stig-deviations.md` (generated by `scripts/stig-deviations.sh`) as artifacts
|
||
and never fails on score. The scanner tooling is removed from the shipped image
|
||
(2d hygiene). `enable_stig=false` is the escape hatch if hardening regresses
|
||
the boot/secure-path bench. NEVER let a waiver hide a real finding — anything
|
||
non-pass that isn't in the register shows as **REVIEW** in the report.
|
||
- **Boot splash toggle**: `boot_splash=true` (default) bakes `rhgb quiet` for the
|
||
Plymouth splash; `false` bakes `plymouth.enable=0` for a verbose console boot —
|
||
keep OFF while debugging the boot chain. Plymouth is always in the image +
|
||
initramfs (the `plymouth` dracut module, which pulls `drm`); only the cmdline
|
||
changes. Press ESC at boot to see details behind a splash. ks.cfg no longer
|
||
excludes `plymouth`.
|
||
- **Splash theme = custom `fleet`** (`files/plymouth/themes/fleet/`, script
|
||
plugin, asset-free — drawn with `Image.Text`). It renders a security-handshake
|
||
checklist (TPM → 802.1X → MACsec → Network → NVMe-oF) driven by best-effort
|
||
`plymouth update --status=<token>` calls from the dracut hooks
|
||
(`dot1x-macsec-up.sh`, `nvmf-connect.sh`). Tokens: `mode-test` (marks
|
||
auth/MACsec skipped), `tpm|dot1x|macsec|net|nvme`-`active|done`. **Rules:** the
|
||
emits are non-fatal (`plymouth … || true`) and must never reorder the
|
||
load-bearing boot sequence; show only STATES, never per-host data (invariant
|
||
#1). The provisioner installs the script plugin best-effort and **falls back to
|
||
the stock `spinner` theme** if `script.so` is absent, so a plugin-package-name
|
||
change can't break the build.
|
||
|
||
## Testing changes
|
||
|
||
Full boot-chain smoke test runs in QEMU/Proxmox: swtpm (test key at the
|
||
fleet handle, test cert in NV) + hostapd as wired 802.1X/MACsec
|
||
authenticator + FreeRADIUS on a veth/bridge (bridge needs
|
||
`group_fwd_mask=8` or EAPOL is silently dropped). Pass criteria: port
|
||
`Authorized`, macsec0 addressed, NVMe namespace connected, verity root
|
||
mapped, overlay writable, supplicant alive after switch-root.
|
||
This harness is scaffolded in **`serving/dot1x-bench/`** (step-ca as the fleet
|
||
CA on the serving host + FreeRADIUS EAP-TLS + a `hostapd` wired authenticator +
|
||
`tpm-provision.sh`), with an **isolated bridge per workstation** (the flat VLAN
|
||
can't authenticate — EAPOL is dropped and MACsec is point-to-point). See its
|
||
README; several configs there are still BENCH-VERIFY.
|
||
|
||
Any change to the dracut module, supplicant config, or cmdline must pass
|
||
this before merge. Template-only changes need `packer validate` +
|
||
shellcheck at minimum.
|
||
|
||
## Conventions
|
||
|
||
- Shell: bash, `set -euo pipefail` in scripts (dracut hooks source
|
||
dracut-lib instead and must stay POSIX-ish bash compatible with the
|
||
initramfs environment — no GNU-only tools not installed by the module).
|
||
- Keep comments that explain *why* (timing constraints, kernel behaviours);
|
||
they encode bench findings that are expensive to rediscover.
|
||
- Versioning: semver on `image_version`; the OCI tag matches; the git tag
|
||
matches; one release == one digest == one manifest commit. |