Audiobuzz
6fe623c727
The Dell 820 kept "finding a cert" and bypassing enrolment despite repeated BIOS TPM clears. tpm2_nvreadpublic revealed 0x01800001 on that hardware is a FIRMWARE platform index (platformcreate|policydelete|writelocked, 70 bytes — not our cert; siblings at 0x01800003/4 + 0x01810000). platformcreate survives TPM2_Clear and policydelete needs the platform's secret policy, so it's permanent and unusable by us — the blind check false-positived on it, and enrolment could never write there anyway. Same class of bug as the EK-handle collision. Move the NV cert index default 0x01800001 -> 0x01800100 (free on the 820, owner range, overridable via rd.dot1x.nvindex): cmdline (extract-artifacts.sh), fleet-enrol -nvindex, the dracut hooks, bench scripts, CLAUDE.md invariant #1, and design/SSP docs. The signing-key handle (0x81000100) is unchanged. Harden WriteNVCert: refuse a platformcreate index with a clear "pick a free owner index via rd.dot1x.nvindex" error instead of a confusing undefine failure. Regression simtest/TestWriteNVCertRefusesPlatformIndex; NV roundtrip + ReadNVCert tests moved to 0x01800100. All go tests + packer validate pass.
136 lines
8.1 KiB
Markdown
136 lines
8.1 KiB
Markdown
# Design v2 — implementation errata (RHEL/Rocky 9)
|
|
|
|
Records where the as-built implementation diverges from
|
|
`diskless-fleet-design-v2.pdf`. The PDF remains authoritative for the
|
|
architecture; this note captures deltas discovered bringing the build up on
|
|
**Rocky 9.8** until they are folded into a consolidated **v3**. Last updated
|
|
2026-06-22.
|
|
|
|
The architecture is unchanged: 802.1X EAP-TLS with a TPM-held key, MACsec on the
|
|
access link, dm-verity-protected erofs root over NVMe/TCP, signed UKI, and the
|
|
FROM-scratch OCI release artifact. Only the items below differ.
|
|
|
|
## 1. Design-level change — writable root overlay
|
|
|
|
**PDF says (§3 step 8, §5):** runtime state lives on a tmpfs overlay enabled by
|
|
the kernel cmdline option **`rootovl`**.
|
|
|
|
**As built:** the cmdline uses **`systemd.volatile=overlay`** instead.
|
|
|
|
**Why:** `rootovl`/`overlayfs` is a Debian/third-party dracut module; RHEL/Rocky
|
|
dracut does not ship it (`dracut: module 'overlayfs' cannot be found`).
|
|
`systemd.volatile=overlay` is the RHEL-native, systemd-supported mechanism and
|
|
is the recommended overlay for dm-verity-protected read-only roots. Behaviour is
|
|
equivalent: the verity erofs root mounts read-only and a tmpfs overlay carries
|
|
runtime writes; per-host `machine-id` is still generated into the overlay.
|
|
|
|
**Caveat:** requires `systemd-volatile-root.service` in the initramfs (bundled by
|
|
recent dracut; `install_items` it if the root stays read-only at boot). The known
|
|
failure mode is NFS root — not applicable here (NVMe block root). Set in
|
|
[../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh); module list
|
|
in [../files/etc/dracut.conf.d/90-diskless.conf](../files/etc/dracut.conf.d/90-diskless.conf).
|
|
|
|
## 1b. Design-level change — dm-verity uses two namespaces, not in-file offset
|
|
|
|
**PDF says (§5):** the erofs data and the dm-verity hash tree share a single
|
|
NVMe/TCP namespace, appended in one file and split on the host with
|
|
`systemd.verity_root_options=hash-offset=`.
|
|
|
|
**As built:** data and hash are **two separate namespaces** — `rootfs.erofs`
|
|
(data) and `rootfs.hash` (hash tree) — each a file in the OCI artifact, served as
|
|
its own NVMe namespace with a distinct pinned UUID. The cmdline uses
|
|
`systemd.verity_root_data=` / `systemd.verity_root_hash=` by-id devices with no
|
|
offset.
|
|
|
|
**Why:** Rocky 9.8 `systemd-veritysetup` does not support `hash-offset=` — it
|
|
logs `Encountered unknown option 'hash-offset=…', ignoring` then `Failed to load
|
|
verity superblock: invalid argument`, so the single-namespace layout never maps
|
|
`/dev/mapper/root`. Two namespaces is the design's own documented fallback and
|
|
uses systemd's native verity (no custom activation code; keeps root-hash
|
|
signature support for production). Devices are referenced by stable
|
|
`/dev/disk/by-id/nvme-uuid.<uuid>` (not `/dev/nvmeNnM`) because controller
|
|
indices shift with the transient discovery controller.
|
|
|
|
**Touches:** [../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh)
|
|
(emits `rootfs.hash`, two UUIDs, offset-free cmdline),
|
|
[../scripts/build-oci.sh](../scripts/build-oci.sh) (artifact list + SHA256SUMS),
|
|
[../serving/spdk-setup.sh](../serving/spdk-setup.sh) (2nd bdev/namespace), and
|
|
CLAUDE.md invariant #9.
|
|
|
|
## 1c. Design-level addition — pre-auth boot path & per-host identity enrolment
|
|
|
|
Two design points the PDF leaves implicit; both are load-bearing for the secure
|
|
model and are recorded here pending v3.
|
|
|
|
**The boot path is inherently pre-auth, so a pre-auth permit-set is mandatory.**
|
|
The firmware (UEFI HTTP boot / PXE→iPXE) fetches the UKI **before** any OS or
|
|
supplicant exists — there is no firmware 802.1X supplicant — so the UKI/initramfs
|
|
fetch always happens in the unauthenticated state. The secure model therefore
|
|
requires the authenticator to permit a *scoped* set of services pre-auth
|
|
(DHCP/DNS + TFTP + HTTP to the serving host) and only complete 802.1X once the
|
|
**OS** is running; RADIUS returns the secure VLAN on Access-Accept, after which
|
|
DHCP-for-the-running-system and the NVMe/TCP root ride **`macsec0`** (post-auth,
|
|
encrypted). Mechanism on a real switch: low-impact 802.1X / a pre-auth dACL /
|
|
boot→secure dynamic VLAN. Consequences:
|
|
- Pre-auth traffic has **no MACsec** (plaintext at L2). Integrity of the boot
|
|
therefore comes from **Secure Boot** (the signed UKI + signed `fleet-ipxe`), not
|
|
the network — this is *why* signing matters: the boot transport is untrusted.
|
|
- `hostapd` wired (the bench authenticator) is strictly port-based and can't model
|
|
a pre-auth per-flow permit — the in-place pre-auth ACL is a real-switch feature,
|
|
validated on the 820's switch, not the VM rig (see `serving/dot1x-bench/`).
|
|
|
|
**Per-host 802.1X identity is created by an enrolment flow, never baked in**
|
|
(invariant #1). Chosen pattern: **trust-on-first-use with out-of-band fingerprint
|
|
approval** — on an empty TPM the host generates a key *in* the TPM
|
|
(`0x81000100`), submits a CSR to an enrolment **broker** (reachable via the same
|
|
pre-auth permit-set, e.g. an `/enrol/` location on the serving nginx → broker →
|
|
step-ca signer), and shows a fingerprint on the Plymouth splash; an admin
|
|
cross-checks it and approves; the signed cert is written to NV `0x01800100` and
|
|
authentication proceeds. Enrol triggers on **empty-NV only** (expired cert →
|
|
automatic renew; auth-failure-with-cert → alert, not re-enrol). The fingerprint
|
|
authorises *which device*; **TPM attestation** (step-ca ACME `device-attest-01`)
|
|
proves *key residency* — use both for full assurance. Runs in a provisioning
|
|
UKI/mode, not the production initramfs. Full design + decisions: **TODO #11** (the
|
|
network-identity counterpart to **#10**, which enrols Secure Boot PK/KEK/db; both
|
|
share the provisioning-UKI vehicle).
|
|
|
|
## 2. Implementation notes (not architectural)
|
|
|
|
These don't contradict the PDF (it doesn't specify them) but are worth recording.
|
|
|
|
- **Network backend.** The dracut hook uses the `network-manager` dracut module
|
|
(ported 2026-06-17 from `network-legacy`/`ifup`/`dhclient`). NM hands its
|
|
initramfs connections to the real-root NetworkManager, which **renews the DHCP
|
|
lease** in the running system — network-legacy's one-shot lease was never
|
|
renewed, so the address dropped at lease expiry and the NVMe/TCP root died.
|
|
Managed/unmanaged is mode-aware: secure path → `macsec0` managed (NM DHCPs +
|
|
renews it), physical `en*` unmanaged (wpa_supplicant owns EAPOL/MKA), enforced
|
|
by the `nm-unmanaged.sh` cmdline hook writing a `/run/NetworkManager/conf.d`
|
|
drop-in before NM starts; test path → boot NIC managed, hook is a no-op. No
|
|
static unmanaged file (it couldn't be mode-aware). Awaiting boot-bench verify.
|
|
- **Build ISO.** Defaults to a version-pinned Rocky 9.x **minimal** ISO on the
|
|
internal mirror (`iso_url` + `sha256` `iso_checksum`). `http/ks.cfg` installs
|
|
only core+sshd+chrony from the ISO; the diskless tooling is pulled online by
|
|
the provisioner, so the build VM needs outbound repo access.
|
|
- **Reference serving stack.** §7's SPDK + nginx + DHCP/DNS serving stack has a
|
|
concrete docker-compose implementation in [../serving/](../serving/), incl.
|
|
Docker image-volume mounting of the OCI artifact image.
|
|
- **STIG hardening.** DISA STIG (SSG) is applied with a diskless/UKI/kiosk
|
|
tailoring; several rule families are deviated because the architecture makes
|
|
them N/A (no partitions, no GRUB, AIDE→dm-verity) or by lab decision (FIPS off,
|
|
firewall off, audit local-only). Full rationale + compensating controls:
|
|
[stig-deviations.md](stig-deviations.md). STIG kernel params live in the UKI
|
|
cmdline (the no-GRUB compensating control).
|
|
- Assorted build-time fixes (HCL `%%{` escaping, file-provisioner staging,
|
|
dracut module naming) are documented in
|
|
[../CLAUDE.md](../CLAUDE.md) → "RHEL 9.x build notes".
|
|
|
|
## 3. Temporary deviation — must be reverted before production
|
|
|
|
- **802.1X/MACsec test bypass.** `disable_dot1x=true` bakes `rd.dot1x=0` into the
|
|
cmdline so the host skips authentication/MACsec and DHCPs the NIC directly, to
|
|
validate the UKI/rootfs/NVMe-oF path on an open switch port. The auth stack
|
|
stays in the image (dormant). This is **lab-only**; a production build must
|
|
drop the flag (`DISABLE_DOT1X: "false"` in CI) to restore the secure boot
|
|
chain described throughout the PDF.
|