diff --git a/CLAUDE.md b/CLAUDE.md index 124a6e5..b3ec231 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,6 +19,8 @@ files/etc/dracut.conf.d/90-diskless.conf no-hostonly + drivers + modules files/etc/NetworkManager/ NM unmanaged-device config scripts/extract-artifacts.sh qcow2 -> kernel/initramfs/erofs+verity/cmdline scripts/build-oci.sh ukify + sbsign + roothash gate + OCI image +serving/ Lab serving stack: dnsmasq + nginx + SPDK +.gitea/workflows/ Gitea CI: lint -> packer build -> push OCI docs/ Design PDF and supporting notes ``` @@ -30,8 +32,11 @@ packer validate . # always before committing template ch packer build -var image_version=X.Y.Z \ -var nic_drivers="ice mlx5_core" \ -var oci_image=/fleet-boot . # add sb_key/sb_cert for signed builds + # add disable_dot1x=true for an + # open-port test UKI (rd.dot1x=0) bash -n files/dracut/95dot1x-macsec/*.sh scripts/*.sh # shell syntax gate -shellcheck files/dracut/95dot1x-macsec/*.sh scripts/*.sh +# .shellcheckrc disables the dracut false positives; CI passes them explicitly: +shellcheck --exclude=SC1091,SC2154 files/dracut/95dot1x-macsec/*.sh scripts/*.sh ``` Build-host deps: qemu-kvm, libguestfs-tools, erofs-utils, cryptsetup, jq, @@ -113,8 +118,40 @@ instead of working around it. journal-less ext4 in `extract-artifacts.sh`). - `systemd.verity_root_options=hash-offset` support in the shipped systemd (fallback: split data/hash into two namespaces). -- `network-legacy` dracut module availability (the hook’s `ifup` step needs - porting if a future release drops it). +- `network-legacy` dracut module needs `dhclient`; RHEL 9 omits it from the + minimal install, so the provisioner installs `dhcp-client` (without it dracut + fails "module network-legacy can't be installed"). dhclient is deprecated — + the durable fix is porting the hook to the `network-manager` dracut module. +- Writable root overlay is `systemd.volatile=overlay`, **not** `rootovl`: + RHEL/Rocky dracut ships no `overlayfs`/`rootovl` module. Needs + `systemd-volatile-root.service` in the initramfs (bundled by recent dracut; + `install_items` it if the root stays read-only). Misbehaves with NFS root — + N/A here (NVMe block root). + +## RHEL 9.x build notes (learned — don't re-discover) + +These bit us bringing the build up on Rocky 9.8; they are easy to regress. + +- **ISO**: default is a version-pinned Rocky 9.x **minimal** ISO on the internal + mirror (`iso_url` + `sha256` in `iso_checksum`; bump both together on a point + release). `http/ks.cfg` installs only `@^minimal-environment`+sshd+chrony from + the ISO; the diskless tooling is pulled **online** by the provisioner, so the + build VM needs outbound repo access. Keeps the build ISO-variant-agnostic. +- **`%{...}` in provisioner shell** (e.g. `rpm --qf '%{VERSION}'`) must be + escaped `%%{...}` — Packer's HCL2 engine parses a bare `%{` as a template + directive and `packer validate` fails otherwise. +- **File provisioner staging**: pre-create the destination dir (`mkdir -p + /tmp/etc-stage`) before a trailing-slash *contents* upload; Packer requires + the dest to exist or the tree lands elsewhere and later `install` steps fail. +- **dracut module name** in `add_dracutmodules` omits the `NN` dir-prefix: the + directory is `95dot1x-macsec` but the module is `dot1x-macsec` (cf. `nvmf`). +- **`network-legacy` → `dhcp-client`** and **overlay → `systemd.volatile=overlay`**: + see the bench-verify items above. +- **Test toggle**: `disable_dot1x=true` bakes `rd.dot1x=0`; the hook then skips + 802.1X/MACsec and DHCPs the NIC directly for NVMe-oF on an open port. The auth + stack stays in the image, dormant. The matching lab DHCP/HTTP/NVMe-oF serving + stack lives in `serving/` (docker compose). Re-enable security by dropping the + var / setting `DISABLE_DOT1X: "false"` in the CI workflow. ## Testing changes diff --git a/README.md b/README.md index de560c4..a60e6d5 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,11 @@ files/etc/dracut.conf.d/90-diskless.conf no-hostonly + nvmf/verity/erofs + driv files/etc/NetworkManager/ marks NIC/macsec0 unmanaged post-pivot scripts/extract-artifacts.sh qcow2 -> vmlinuz + initramfs.img + rootfs.erofs (+verity) + cmdline +scripts/build-oci.sh ukify (+sbsign) + roothash gate -> OCI image +serving/ Lab serving stack (docker compose): + dnsmasq DHCP/DNS + nginx (UKI/HTTP) + SPDK +.gitea/workflows/ Gitea CI: lint -> packer build -> push OCI +docs/diskless-fleet-design-v2.pdf Architecture design (authoritative) ``` ## Build host requirements @@ -33,6 +38,12 @@ scripts/extract-artifacts.sh qcow2 -> vmlinuz + initramfs.img - **systemd-ukify** (+ **sbsigntools** when signing) — UKI assembly - **buildah** (or docker) — OCI release-artifact image build +The default `iso_url`/`iso_checksum` point at a version-pinned **Rocky 9.x +minimal** ISO on the internal mirror. `http/ks.cfg` installs only core+sshd from +that ISO; the diskless tooling is pulled online by the provisioner, so the build +VM needs **outbound network to the distro repos** (and a fast mirror helps). +Bump the URL and the pinned sha256 together when moving to a new point release. + ## Build ```bash @@ -58,6 +69,13 @@ Substitute the real NVMe target address for `NVMET_TRADDR` in `scripts/extract-artifacts.sh` (or post-edit `cmdline.txt`) **before** the OCI step: the cmdline is immutable once signed into the UKI. +**Test mode (auth disabled).** Add `-var disable_dot1x=true` to bake +`rd.dot1x=0` into the cmdline. At boot the hook then skips 802.1X/MACsec and +DHCPs the physical NIC directly to reach the NVMe/TCP target — for validating +the UKI/rootfs/NVMe-oF path on an **open switch port**. The auth stack stays in +the image (dormant); drop the var to restore the secure boot. See `serving/` +for the matching DHCP/HTTP/NVMe-oF lab stack. + Outputs: `artifacts//` gains `fleet-.efi` (the signed UKI) alongside `rootfs.erofs`/`verity.json`, and the build produces the release-artifact OCI image `:` containing only @@ -88,7 +106,7 @@ release only changes which artifact image feeds them: ## Kernel command line (per build — emitted in cmdline.txt) ``` -rd.neednet=1 rootovl \ +rd.neednet=1 systemd.volatile=overlay \ rd.dot1x.nvindex=0x01800001 rd.dot1x.timeout=60 \ rd.nvmf.discover=tcp,NVMET_TRADDR,4420 \ root=/dev/mapper/root rootfstype=erofs rootflags=ro \ @@ -98,9 +116,12 @@ systemd.verity_root_hash=/dev/nvme0n1 \ systemd.verity_root_options=hash-offset= ``` -Single-file layout: data and hash tree share one namespace, split by -`hash-offset`. `rd.dot1x.if=` may pin the NIC; otherwise the first -non-loopback interface is used. +`systemd.volatile=overlay` gives the writable tmpfs overlay over the read-only +verity root — RHEL/Rocky dracut has **no** `rootovl`/`overlayfs` module (that is +a Debian/third-party feature), and systemd's volatile overlay is the supported +mechanism for dm-verity roots. Single-file layout: data and hash tree share one +namespace, split by `hash-offset`. `rd.dot1x.if=` may pin the NIC; otherwise the +first non-loopback interface is used. Test builds prepend `rd.dot1x=0`. ## Bench-verify before fleet rollout @@ -118,14 +139,23 @@ non-loopback interface is used. 5. **Pivot survival**: `@wpa_supplicant` survives switch-root; MKA continuity across it. NVMe/TCP keepalives must also ride out the pivot (they will, if the link never drops — that is the point of the @-daemon). -6. **dracut module choice**: skeleton uses `network-legacy`; if removed in - your release, port the hook's ifup step to the available backend. +6. **network backend**: the hook uses `network-legacy`/`ifup`, which requires + `dhclient`. RHEL 9 deprecated it and the minimal ISO omits it, so the + provisioner installs `dhcp-client` explicitly (without it dracut fails with + "module network-legacy can't be installed"). dhclient is on its way out; the + durable fix is to port the hook to the `network-manager` dracut module. 7. **hostnqn allow-listing**: if the nvmet target filters hosts, the allowed-NQN list must be fed from the enrolment pipeline (NQNs are derived from certificate serials — see the hook). 8. **CI smoke test**: boot artifacts in qemu/Proxmox with swtpm against hostapd-as-authenticator + FreeRADIUS (remember bridge group_fwd_mask=8 for EAPOL); assert macsec0 up, dm-verity root mapped, overlay rw. +9. **writable root overlay**: provided by `systemd.volatile=overlay`, which + needs `systemd-volatile-root.service` in the initramfs (bundled by recent + dracut). If the root stays read-only at boot, add `install_items+=" + /usr/lib/systemd/systemd-volatile-root + /usr/lib/systemd/system/systemd-volatile-root.service "` to + `90-diskless.conf`. Known to misbehave with NFS root — N/A here (NVMe block). ## What is deliberately NOT in the image