# CLAUDE.md — Diskless Fleet Boot System

This repo builds the boot system for a fleet of diskless hosts:
PXE/HTTP-booted, authenticating to the network with **802.1X EAP-TLS using a
TPM-held key**, running **MACsec** on the access link, and mounting an
immutable **erofs + dm-verity root over NVMe/TCP**. Each release is a signed
**UKI** plus root image packaged as a **FROM-scratch OCI artifact image**.
The authoritative architecture document is
`docs/diskless-fleet-design-v2.pdf` — read it before structural changes.
`docs/design-v2-errata.md` records where the as-built RHEL/Rocky 9
implementation diverges from it (pending a consolidated v3).

## Backlog (read at session start)

`docs/TODO.md` is the **persistent backlog** and the durable source of truth for
outstanding work — it survives across sessions (the TodoWrite task tracker does
not). At the start of a session, read it and mirror its open items into the
TodoWrite tracker. Whenever items change (added, started, completed), update
**both** `docs/TODO.md` and the tracker so they stay in sync.

## Repo layout

```
diskless.pkr.hcl                          Packer template (qemu builder)
http/ks.cfg                               Kickstart for the throwaway build VM
files/dracut/95dot1x-macsec/              Custom dracut module (the heart of the system)
files/etc/dot1x/                          Supplicant config + RADIUS CA (placeholder)
files/etc/dracut.conf.d/90-diskless.conf  no-hostonly + drivers + modules
files/etc/NetworkManager/                 NM unmanaged-device config
scripts/extract-artifacts.sh              qcow2 -> kernel/initramfs/erofs+verity/cmdline
scripts/build-oci.sh                      ukify + sbsign + roothash gate + OCI image
serving/                                  Lab serving stack: dnsmasq + nginx + SPDK
.gitea/workflows/                         Gitea CI: lint -> packer build -> push OCI
docs/                                     Design PDF and supporting notes
```

## Build & test commands

```bash
packer init .
packer validate .                         # always before committing template changes
packer build -var image_version=X.Y.Z \
  -var nic_drivers="ice mlx5_core" \
  -var oci_image=<registry>/fleet-boot .  # add sb_key/sb_cert for signed builds
                                          # add disable_dot1x=true for an
                                          # open-port test UKI (rd.dot1x=0)
bash -n files/dracut/95dot1x-macsec/*.sh scripts/*.sh   # shell syntax gate
# .shellcheckrc disables the dracut false positives; CI passes them explicitly:
shellcheck --exclude=SC1091,SC2154 files/dracut/95dot1x-macsec/*.sh scripts/*.sh
```

Build-host deps: qemu-kvm, libguestfs-tools (+ a kernel for its supermin
appliance), erofs-utils, cryptsetup, jq, systemd-ukify **and systemd-boot-efi**
(the latter ships the `linuxx64.efi.stub` ukify needs), sbsigntools,
buildah (or docker).

## Architecture invariants — do not violate these

These encode failure modes that brick the fleet. Treat them as hard
constraints; if a task appears to require breaking one, stop and flag it
instead of working around it.

1. **No per-host state in any artifact, ever.** No hostnames, keys, certs,
   MACs, or per-host config in the initramfs, rootfs, UKI, or OCI image.
   All host identity derives from the TPM at runtime (key at persistent
   handle `0x81010001`, cert at NV index `0x01800001`, hostnqn derived from
   the cert serial). If a change needs per-host data, it belongs in the
   enrolment flow, not the image.
1. **The hook ordering in `dot1x-macsec-up.sh` is load-bearing.**
   NV-cert extraction → supplicant start → wait for `Authorized` + macsec0 →
   DHCP **on macsec0** → hostnqn derivation. Never DHCP before auth
   completes (pre-auth leases are invalid after VLAN assignment); never
   connect NVMe before macsec0 is addressed.
1. **The supplicant must survive switch-root.** It is launched with
   `exec -a @wpa_supplicant` (systemd root-storage-daemon convention) and
   its state lives under `/run`. Do not “clean up” the `@`, move state off
   `/run`, or hand the interface to NetworkManager — MKA times out in ~6 s
   without keepalives and the root filesystem link dies. The NM
   unmanaged-devices config exists for this reason; keep it in sync with
   interface naming.
1. **roothash coupling is a hard gate.** `build-oci.sh` refuses to build if
   the `roothash=` in the UKI cmdline ≠ `verity.json`. Never weaken, skip,
   or warn-instead-of-fail this check: a mismatched UKI/rootfs pair is an
   unbootable fleet.
1. **The UKI cmdline is immutable once signed.** Any cmdline change
   (including `NVMET_TRADDR`) requires a rebuild + re-sign. Prefer a
   per-environment DNS name for NVMe discovery over per-environment builds.
1. **Signing keys never enter the build VM or this repo.** `sb_key` /
   `sb_cert` are CI/HSM-provided paths on the build host. Never commit key
   material, never copy keys into provisioner steps, never echo them in
   logs. Unsigned builds are lab-only and must keep their loud warning.
1. **Reproducibility of the root image is a security property.**
   `mkfs.erofs` runs with fixed UUID and `-T 0`; the verity root hash must
   be a pure function of the input tree. Don’t introduce timestamps,
   random seeds, or build-host paths into the rootfs.
1. **The initramfs is built `--no-hostonly` and contents are asserted.**
   The build VM is virtio-only; real NIC drivers, `macsec`, `nvme-tcp`,
   erofs, and veritysetup are explicitly listed in
   `90-diskless.conf` and checked via `lsinitrd | grep` in the template.
   When adding a runtime dependency to the hook, add it to the dracut
   module’s `install()` *and* extend the lsinitrd assertion.
1. **The OCI artifact image stays FROM scratch.** Only
   `/artifacts/{fleet-<ver>.efi, rootfs.erofs, verity.json, SHA256SUMS}`.
   No shell, no tools, no base image. Serving software (SPDK, nginx) lives
   in separate, independently versioned images — never merge them into the
   artifact image.
1. **Deployments are pinned by digest.** One commit per fleet release in
   the deployment manifest; that manifest is the single source coupling
   UKI ↔ namespace. Rollback = revert the commit.

## Things that look like bugs but aren’t

- `identity="fleet-host"` in the supplicant config is intentionally generic;
  RADIUS derives real identity from the certificate CN/SAN.
- `/etc/machine-id` is truncated empty in the image; each host generates its
  own into the tmpfs overlay at boot.
- `/etc/fstab` is effectively empty; root comes from netroot + verity.
- The build VM has no TPM. Correct — nothing host-specific exists at build
  time. CI boot tests use swtpm.
- `files/etc/dot1x/radius-ca.pem` is a placeholder; builds succeed with it
  but authentication cannot. Replace per environment, don’t commit real
  internal CA material unless the repo’s classification allows it.

## Known bench-verify items (check before trusting a change in production)

- TPM key reference method in wpa_supplicant: `private_key="handle:..."`
  via OpenSSL tpm2 provider vs tpm2-pkcs11 URI — depends on the distro
  build; the supplicant config carries both options.
- erofs module availability on the target RHEL kernel (fallback:
  journal-less ext4 in `extract-artifacts.sh`).
- `systemd.verity_root_options=hash-offset` support in the shipped systemd
  (fallback: split data/hash into two namespaces).
- `network-legacy` dracut module needs `dhclient`; RHEL 9 omits it from the
  minimal install, so the provisioner installs `dhcp-client` (without it dracut
  fails "module network-legacy can't be installed"). dhclient is deprecated —
  the durable fix is porting the hook to the `network-manager` dracut module.
- Writable root overlay is `systemd.volatile=overlay`, **not** `rootovl`:
  RHEL/Rocky dracut ships no `overlayfs`/`rootovl` module. Needs
  `systemd-volatile-root.service` in the initramfs (bundled by recent dracut;
  `install_items` it if the root stays read-only). Misbehaves with NFS root —
  N/A here (NVMe block root).

## RHEL 9.x build notes (learned — don't re-discover)

These bit us bringing the build up on Rocky 9.8; they are easy to regress.

- **ISO**: default is a version-pinned Rocky 9.x **minimal** ISO on the internal
  mirror (`iso_url` + `sha256` in `iso_checksum`; bump both together on a point
  release). `http/ks.cfg` installs only `@^minimal-environment`+sshd+chrony from
  the ISO; the diskless tooling is pulled **online** by the provisioner, so the
  build VM needs outbound repo access. Keeps the build ISO-variant-agnostic.
- **`%{...}` in provisioner shell** (e.g. `rpm --qf '%{VERSION}'`) must be
  escaped `%%{...}` — Packer's HCL2 engine parses a bare `%{` as a template
  directive and `packer validate` fails otherwise.
- **File provisioner staging**: pre-create the destination dir (`mkdir -p
  /tmp/etc-stage`) before a trailing-slash *contents* upload; Packer requires
  the dest to exist or the tree lands elsewhere and later `install` steps fail.
- **dracut module name** in `add_dracutmodules` omits the `NN` dir-prefix: the
  directory is `95dot1x-macsec` but the module is `dot1x-macsec` (cf. `nvmf`).
- **`network-legacy` → `dhcp-client`** and **overlay → `systemd.volatile=overlay`**:
  see the bench-verify items above.
- **Test toggle**: `disable_dot1x=true` bakes `rd.dot1x=0`; the hook then skips
  802.1X/MACsec and DHCPs the NIC directly for NVMe-oF on an open port. The auth
  stack stays in the image, dormant. The matching lab DHCP/HTTP/NVMe-oF serving
  stack lives in `serving/` (docker compose). Re-enable security by dropping the
  var / setting `DISABLE_DOT1X: "false"` in the CI workflow.

## Testing changes

Full boot-chain smoke test runs in QEMU/Proxmox: swtpm (test key at the
fleet handle, test cert in NV) + hostapd as wired 802.1X/MACsec
authenticator + FreeRADIUS on a veth/bridge (bridge needs
`group_fwd_mask=8` or EAPOL is silently dropped). Pass criteria: port
`Authorized`, macsec0 addressed, NVMe namespace connected, verity root
mapped, overlay writable, supplicant alive after switch-root.

Any change to the dracut module, supplicant config, or cmdline must pass
this before merge. Template-only changes need `packer validate` +
shellcheck at minimum.

## Conventions

- Shell: bash, `set -euo pipefail` in scripts (dracut hooks source
  dracut-lib instead and must stay POSIX-ish bash compatible with the
  initramfs environment — no GNU-only tools not installed by the module).
- Keep comments that explain *why* (timing constraints, kernel behaviours);
  they encode bench findings that are expensive to rediscover.
- Versioning: semver on `image_version`; the OCI tag matches; the git tag
  matches; one release == one digest == one manifest commit.