diskless-workstation/docs/design-v2-errata.md

Design v2 — implementation errata (RHEL/Rocky 9)

Records where the as-built implementation diverges from diskless-fleet-design-v2.pdf. The PDF remains authoritative for the architecture; this note captures deltas discovered bringing the build up on Rocky 9.8 until they are folded into a consolidated v3. Last updated 2026-06-22.

The architecture is unchanged: 802.1X EAP-TLS with a TPM-held key, MACsec on the access link, dm-verity-protected erofs root over NVMe/TCP, signed UKI, and the FROM-scratch OCI release artifact. Only the items below differ.

1. Design-level change — writable root overlay

PDF says (§3 step 8, §5): runtime state lives on a tmpfs overlay enabled by the kernel cmdline option rootovl.

As built: the cmdline uses systemd.volatile=overlay instead.

Why: rootovl/overlayfs is a Debian/third-party dracut module; RHEL/Rocky dracut does not ship it (dracut: module 'overlayfs' cannot be found). systemd.volatile=overlay is the RHEL-native, systemd-supported mechanism and is the recommended overlay for dm-verity-protected read-only roots. Behaviour is equivalent: the verity erofs root mounts read-only and a tmpfs overlay carries runtime writes; per-host machine-id is still generated into the overlay.

Caveat: requires systemd-volatile-root.service in the initramfs (bundled by recent dracut; install_items it if the root stays read-only at boot). The known failure mode is NFS root — not applicable here (NVMe block root). Set in ../scripts/extract-artifacts.sh; module list in ../files/etc/dracut.conf.d/90-diskless.conf.

1b. Design-level change — dm-verity uses two namespaces, not in-file offset

PDF says (§5): the erofs data and the dm-verity hash tree share a single NVMe/TCP namespace, appended in one file and split on the host with systemd.verity_root_options=hash-offset=.

As built: data and hash are two separate namespaces — rootfs.erofs (data) and rootfs.hash (hash tree) — each a file in the OCI artifact, served as its own NVMe namespace with a distinct pinned UUID. The cmdline uses systemd.verity_root_data= / systemd.verity_root_hash= by-id devices with no offset.

Why: Rocky 9.8 systemd-veritysetup does not support hash-offset= — it logs Encountered unknown option 'hash-offset=…', ignoring then Failed to load verity superblock: invalid argument, so the single-namespace layout never maps /dev/mapper/root. Two namespaces is the design's own documented fallback and uses systemd's native verity (no custom activation code; keeps root-hash signature support for production). Devices are referenced by stable /dev/disk/by-id/nvme-uuid.<uuid> (not /dev/nvmeNnM) because controller indices shift with the transient discovery controller.

Touches: ../scripts/extract-artifacts.sh (emits rootfs.hash, two UUIDs, offset-free cmdline), ../scripts/build-oci.sh (artifact list + SHA256SUMS), ../serving/spdk-setup.sh (2nd bdev/namespace), and CLAUDE.md invariant #9.

1c. Design-level addition — pre-auth boot path & per-host identity enrolment

Two design points the PDF leaves implicit; both are load-bearing for the secure model and are recorded here pending v3.

The boot path is inherently pre-auth, so a pre-auth permit-set is mandatory. The firmware (UEFI HTTP boot / PXE→iPXE) fetches the UKI before any OS or supplicant exists — there is no firmware 802.1X supplicant — so the UKI/initramfs fetch always happens in the unauthenticated state. The secure model therefore requires the authenticator to permit a scoped set of services pre-auth (DHCP/DNS + TFTP + HTTP to the serving host) and only complete 802.1X once the OS is running; RADIUS returns the secure VLAN on Access-Accept, after which DHCP-for-the-running-system and the NVMe/TCP root ride macsec0 (post-auth, encrypted). Mechanism on a real switch: low-impact 802.1X / a pre-auth dACL / boot→secure dynamic VLAN. Consequences:

• Pre-auth traffic has no MACsec (plaintext at L2). Integrity of the boot therefore comes from Secure Boot (the signed UKI + signed fleet-ipxe), not the network — this is why signing matters: the boot transport is untrusted.
• hostapd wired (the bench authenticator) is strictly port-based and can't model a pre-auth per-flow permit — the in-place pre-auth ACL is a real-switch feature, validated on the 820's switch, not the VM rig (see serving/dot1x-bench/).

Per-host 802.1X identity is created by an enrolment flow, never baked in (invariant #1). Chosen pattern: trust-on-first-use with out-of-band fingerprint approval — on an empty TPM the host generates a key in the TPM (0x81000100), submits a CSR to an enrolment broker (reachable via the same pre-auth permit-set, e.g. an /enrol/ location on the serving nginx → broker → step-ca signer), and shows a fingerprint on the Plymouth splash; an admin cross-checks it and approves; the signed cert is written to NV 0x01800001 and authentication proceeds. Enrol triggers on empty-NV only (expired cert → automatic renew; auth-failure-with-cert → alert, not re-enrol). The fingerprint authorises which device; TPM attestation (step-ca ACME device-attest-01) proves key residency — use both for full assurance. Runs in a provisioning UKI/mode, not the production initramfs. Full design + decisions: TODO #11 (the network-identity counterpart to #10, which enrols Secure Boot PK/KEK/db; both share the provisioning-UKI vehicle).

2. Implementation notes (not architectural)

These don't contradict the PDF (it doesn't specify them) but are worth recording.

• Network backend. The dracut hook uses the network-manager dracut module (ported 2026-06-17 from network-legacy/ifup/dhclient). NM hands its initramfs connections to the real-root NetworkManager, which renews the DHCP lease in the running system — network-legacy's one-shot lease was never renewed, so the address dropped at lease expiry and the NVMe/TCP root died. Managed/unmanaged is mode-aware: secure path → macsec0 managed (NM DHCPs + renews it), physical en* unmanaged (wpa_supplicant owns EAPOL/MKA), enforced by the nm-unmanaged.sh cmdline hook writing a /run/NetworkManager/conf.d drop-in before NM starts; test path → boot NIC managed, hook is a no-op. No static unmanaged file (it couldn't be mode-aware). Awaiting boot-bench verify.
• Build ISO. Defaults to a version-pinned Rocky 9.x minimal ISO on the internal mirror (iso_url + sha256 iso_checksum). http/ks.cfg installs only core+sshd+chrony from the ISO; the diskless tooling is pulled online by the provisioner, so the build VM needs outbound repo access.
• Reference serving stack. §7's SPDK + nginx + DHCP/DNS serving stack has a concrete docker-compose implementation in ../serving/, incl. Docker image-volume mounting of the OCI artifact image.
• STIG hardening. DISA STIG (SSG) is applied with a diskless/UKI/kiosk tailoring; several rule families are deviated because the architecture makes them N/A (no partitions, no GRUB, AIDE→dm-verity) or by lab decision (FIPS off, firewall off, audit local-only). Full rationale + compensating controls: stig-deviations.md. STIG kernel params live in the UKI cmdline (the no-GRUB compensating control).
• Assorted build-time fixes (HCL %%{ escaping, file-provisioner staging, dracut module naming) are documented in ../CLAUDE.md → "RHEL 9.x build notes".

3. Temporary deviation — must be reverted before production

• 802.1X/MACsec test bypass. disable_dot1x=true bakes rd.dot1x=0 into the cmdline so the host skips authentication/MACsec and DHCPs the NIC directly, to validate the UKI/rootfs/NVMe-oF path on an open switch port. The auth stack stays in the image (dormant). This is lab-only; a production build must drop the flag (DISABLE_DOT1X: "false" in CI) to restore the secure boot chain described throughout the PDF.