# Diskless Fleet Image Build (Packer) Builds the fleet-generic boot artifacts for PXE/HTTP-booted diskless hosts that authenticate with 802.1X EAP-TLS (TPM-held key, certificate in TPM NV) and run MACsec on the access link. The root filesystem is an immutable, reproducible **erofs image with an appended dm-verity hash tree**, served as a single shared **read-only NVMe/TCP namespace**, with a tmpfs overlay for runtime state. Companion to `docs/diskless-fleet-design-v2.pdf`. ## Layout ``` diskless.pkr.hcl Packer template (qemu builder) http/ks.cfg Kickstart for the throwaway build VM files/dracut/95dot1x-macsec/ Custom dracut module module-setup.sh module definition dot1x-macsec-up.sh initqueue hook: NV cert -> @wpa_supplicant -> wait authorized + macsec0 -> DHCP -> hostnqn derived from cert files/etc/dot1x/wpa_supplicant-dot1x.conf Fleet-generic supplicant config files/etc/dot1x/radius-ca.pem PLACEHOLDER - replace with real CA chain files/etc/dracut.conf.d/90-diskless.conf no-hostonly + nvmf/verity/erofs + drivers files/etc/NetworkManager/ marks NIC/macsec0 unmanaged post-pivot scripts/extract-artifacts.sh qcow2 -> vmlinuz + initramfs.img + rootfs.erofs (+verity) + cmdline scripts/build-oci.sh ukify (+sbsign) + roothash gate -> OCI image serving/ Lab serving stack (docker compose): dnsmasq DHCP/DNS + nginx (UKI/HTTP) + SPDK .gitea/workflows/ Gitea CI: lint -> packer build -> push OCI docs/diskless-fleet-design-v2.pdf Architecture design (authoritative) ``` ## Build host requirements - Packer >= 1.10, qemu plugin (`packer init .`) - qemu-kvm with KVM available - libguestfs-tools, **erofs-utils**, **cryptsetup**, jq (artifact extraction) - **systemd-ukify** (+ **sbsigntools** when signing) — UKI assembly - **buildah** (or docker) — OCI release-artifact image build The default `iso_url`/`iso_checksum` point at a version-pinned **Rocky 9.x minimal** ISO on the internal mirror. `http/ks.cfg` installs only core+sshd from that ISO; the diskless tooling is pulled online by the provisioner, so the build VM needs **outbound network to the distro repos** (and a fast mirror helps). Bump the URL and the pinned sha256 together when moving to a new point release. ## Build ```bash # 1. Replace the CA placeholder cp /path/to/radius-ca-chain.pem files/etc/dot1x/radius-ca.pem # 2. Set the NIC drivers for your real fleet hardware (default covers # common Intel + Mellanox); the build VM only has virtio, so anything # not listed here will not be in the initramfs. packer init . packer build \ -var image_version=1.0.0 \ -var nic_drivers="ice mlx5_core" \ -var oci_image=registry.example.org/fleet-boot \ -var oci_push=true \ -var sb_key=/secure/ci/fleet-sb.key \ -var sb_cert=/secure/ci/fleet-sb.crt \ . ``` Omit `sb_key`/`sb_cert` for unsigned lab builds (the pipeline warns). Substitute the real NVMe target address for `NVMET_TRADDR` in `scripts/extract-artifacts.sh` (or post-edit `cmdline.txt`) **before** the OCI step: the cmdline is immutable once signed into the UKI. **Test mode (auth disabled).** Add `-var disable_dot1x=true` to bake `rd.dot1x=0` into the cmdline. At boot the hook then skips 802.1X/MACsec and DHCPs the physical NIC directly to reach the NVMe/TCP target — for validating the UKI/rootfs/NVMe-oF path on an **open switch port**. The auth stack stays in the image (dormant); drop the var to restore the secure boot. See `serving/` for the matching DHCP/HTTP/NVMe-oF lab stack. Outputs: `artifacts//` gains `fleet-.efi` (the signed UKI) alongside `rootfs.erofs`/`verity.json`, and the build produces the release-artifact OCI image `:` containing only `/artifacts/{fleet-.efi, rootfs.erofs, verity.json, SHA256SUMS}` with the roothash recorded as an image label. The build hard-fails if the roothash baked into the UKI cmdline does not match `verity.json`. ## Publishing a version The OCI image **is** the release. The serving stack stays constant (separate, independently-patched SPDK and nginx/caddy images); each fleet release only changes which artifact image feeds them: 1. Push `:` (or let `oci_push=true` do it) and record the **digest**. 2. In the fleet-boot deployment manifest (k8s Deployment or compose), bump the artifact initContainer / image-volume reference to that digest — one commit per fleet release; this manifest is the single place coupling UKI <-> root namespace. 3. The initContainer copies `/artifacts` into the shared volume; nginx serves `fleet-.efi` over HTTPS (firmware validates against the TlsCaCertificate CA, then Secure Boot verifies the UKI signature); SPDK exposes `rootfs.erofs` as a **read-only** NVMe/TCP namespace. 4. Rollout = digest bump; rollback = revert the commit. cosign the image so the deploy pipeline verifies provenance (sbsign is for firmware, cosign is for your cluster). ## Kernel command line (per build — emitted in cmdline.txt) ``` rd.neednet=1 systemd.volatile=overlay \ rd.dot1x.nvindex=0x01800001 rd.dot1x.timeout=60 \ rd.nvmf.discover=tcp,NVMET_TRADDR,4420 \ root=/dev/mapper/root rootfstype=erofs rootflags=ro \ systemd.verity=yes roothash= \ systemd.verity_root_data=/dev/nvme0n1 \ systemd.verity_root_hash=/dev/nvme0n1 \ systemd.verity_root_options=hash-offset= ``` `systemd.volatile=overlay` gives the writable tmpfs overlay over the read-only verity root — RHEL/Rocky dracut has **no** `rootovl`/`overlayfs` module (that is a Debian/third-party feature), and systemd's volatile overlay is the supported mechanism for dm-verity roots. Single-file layout: data and hash tree share one namespace, split by `hash-offset`. `rd.dot1x.if=` may pin the NIC; otherwise the first non-loopback interface is used. Test builds prepend `rd.dot1x=0`. ## Bench-verify before fleet rollout 1. **erofs module present** in the target RHEL kernel (`modprobe erofs`) — enabled in later RHEL 9 minors for image mode; if absent, fall back to journal-less ext4 (`mkfs.ext4 -O ^has_journal -d `) in `extract-artifacts.sh`. 2. **systemd.verity_root_options support** in the shipped systemd/dracut (needs a reasonably recent systemd; otherwise publish data and hash as two separate namespaces and drop the hash-offset option). 3. **TPM key reference**: confirm wpa_supplicant accepts `private_key="handle:0x81010001"` via the OpenSSL tpm2 provider; if not, switch to the tpm2-pkcs11 option in the supplicant config. 4. **CONFIG_MACSEC** in the shipped wpa_supplicant build. 5. **Pivot survival**: `@wpa_supplicant` survives switch-root; MKA continuity across it. NVMe/TCP keepalives must also ride out the pivot (they will, if the link never drops — that is the point of the @-daemon). 6. **network backend**: the hook uses `network-legacy`/`ifup`, which requires `dhclient`. RHEL 9 deprecated it and the minimal ISO omits it, so the provisioner installs `dhcp-client` explicitly (without it dracut fails with "module network-legacy can't be installed"). dhclient is on its way out; the durable fix is to port the hook to the `network-manager` dracut module. 7. **hostnqn allow-listing**: if the nvmet target filters hosts, the allowed-NQN list must be fed from the enrolment pipeline (NQNs are derived from certificate serials — see the hook). 8. **CI smoke test**: boot artifacts in qemu/Proxmox with swtpm against hostapd-as-authenticator + FreeRADIUS (remember bridge group_fwd_mask=8 for EAPOL); assert macsec0 up, dm-verity root mapped, overlay rw. 9. **writable root overlay**: provided by `systemd.volatile=overlay`, which needs `systemd-volatile-root.service` in the initramfs (bundled by recent dracut). If the root stays read-only at boot, add `install_items+=" /usr/lib/systemd/systemd-volatile-root /usr/lib/systemd/system/systemd-volatile-root.service "` to `90-diskless.conf`. Known to misbehave with NFS root — N/A here (NVMe block). ## What is deliberately NOT in the image - Private keys, certificates, host names, per-host config of any kind. All identity lives in each host's TPM (persistent handle + NV index); the NVMe hostnqn is derived from the certificate at boot.