Diskless Fleet Image Build (Packer)

Builds the fleet-generic boot artifacts for PXE/HTTP-booted diskless hosts that authenticate with 802.1X EAP-TLS (TPM-held key, certificate in TPM NV) and run MACsec on the access link. The root filesystem is an immutable, reproducible erofs image with an appended dm-verity hash tree, served as a single shared read-only NVMe/TCP namespace, with a tmpfs overlay for runtime state. Companion to docs/diskless-fleet-design-v2.pdf (see docs/design-v2-errata.md for as-built deltas on RHEL/Rocky 9).

Layout

diskless.pkr.hcl                          Packer template (qemu builder)
http/ks.cfg                               Kickstart for the throwaway build VM
files/dracut/95dot1x-macsec/              Custom dracut module
  module-setup.sh                           module definition
  dot1x-macsec-up.sh                        initqueue hook: NV cert -> @wpa_supplicant
                                            -> wait authorized + macsec0 -> DHCP
                                            -> hostnqn derived from cert
files/etc/dot1x/wpa_supplicant-dot1x.conf Fleet-generic supplicant config
files/etc/dot1x/radius-ca.pem             PLACEHOLDER - replace with real CA chain
files/etc/dracut.conf.d/90-diskless.conf  no-hostonly + nvmf/verity/erofs + drivers
files/etc/NetworkManager/                 marks NIC/macsec0 unmanaged post-pivot
scripts/extract-artifacts.sh              qcow2 -> vmlinuz + initramfs.img
                                                 + rootfs.erofs (+verity) + cmdline
scripts/build-oci.sh                      ukify (+sbsign) + roothash gate -> OCI image
serving/                                  Lab serving stack (docker compose):
                                          dnsmasq DHCP/DNS + nginx (UKI/HTTP) + SPDK
.gitea/workflows/                         Gitea CI: lint -> packer build -> push OCI
docs/diskless-fleet-design-v2.pdf         Architecture design (authoritative)

Build host requirements

• Packer >= 1.10, qemu plugin (packer init .)
• qemu-kvm with KVM available
• libguestfs-tools, erofs-utils, cryptsetup, jq (artifact extraction)
• systemd-ukify + systemd-boot-efi (the EFI stub ukify wraps; it's only a recommends of ukify) (+ sbsigntools when signing) — UKI assembly
• buildah (or docker) — OCI release-artifact image build

The default iso_url/iso_checksum point at a version-pinned Rocky 9.x minimal ISO on the internal mirror. http/ks.cfg installs only core+sshd from that ISO; the diskless tooling is pulled online by the provisioner, so the build VM needs outbound network to the distro repos (and a fast mirror helps). Bump the URL and the pinned sha256 together when moving to a new point release.

Build

# 1. Replace the CA placeholder
cp /path/to/radius-ca-chain.pem files/etc/dot1x/radius-ca.pem

# 2. Set the NIC drivers for your real fleet hardware (default covers
#    common Intel + Mellanox); the build VM only has virtio, so anything
#    not listed here will not be in the initramfs.
packer init .
packer build \
  -var image_version=1.0.0 \
  -var nic_drivers="ice mlx5_core" \
  -var oci_image=registry.example.org/fleet-boot \
  -var oci_push=true \
  -var sb_key=/secure/ci/fleet-sb.key \
  -var sb_cert=/secure/ci/fleet-sb.crt \
  .

Omit sb_key/sb_cert for unsigned lab builds (the pipeline warns). Set the NVMe discovery address with -var nvmet_traddr=<dns-name> (default nvme.fleet.lab, which the serving/ stack's dnsmasq resolves); it is baked into the cmdline and immutable once signed, so prefer a per-environment DNS name over a raw IP. Deploy/boot-test instructions live in serving/README.md.

Test mode (auth disabled). Add -var disable_dot1x=true to bake rd.dot1x=0 into the cmdline. At boot the hook then skips 802.1X/MACsec and DHCPs the physical NIC directly to reach the NVMe/TCP target — for validating the UKI/rootfs/NVMe-oF path on an open switch port. The auth stack stays in the image (dormant); drop the var to restore the secure boot. See serving/ for the matching DHCP/HTTP/NVMe-oF lab stack.

Outputs: artifacts/<version>/ gains fleet-<version>.efi (the signed UKI) alongside rootfs.erofs/verity.json, and the build produces the release-artifact OCI image <oci_image>:<version> containing only /artifacts/{fleet-<version>.efi, rootfs.erofs, verity.json, SHA256SUMS} with the roothash recorded as an image label. The build hard-fails if the roothash baked into the UKI cmdline does not match verity.json.

Publishing a version

The OCI image is the release. The serving stack stays constant (separate, independently-patched SPDK and nginx/caddy images); each fleet release only changes which artifact image feeds them:

1. Push <oci_image>:<version> (or let oci_push=true do it) and record the digest.
2. In the fleet-boot deployment manifest (k8s Deployment or compose), bump the artifact initContainer / image-volume reference to that digest — one commit per fleet release; this manifest is the single place coupling UKI <-> root namespace.
3. The initContainer copies /artifacts into the shared volume; nginx serves fleet-<version>.efi over HTTPS (firmware validates against the TlsCaCertificate CA, then Secure Boot verifies the UKI signature); SPDK exposes rootfs.erofs as a read-only NVMe/TCP namespace.
4. Rollout = digest bump; rollback = revert the commit. cosign the image so the deploy pipeline verifies provenance (sbsign is for firmware, cosign is for your cluster).

Kernel command line (per build — emitted in cmdline.txt)

rd.neednet=1 systemd.volatile=overlay \
rd.dot1x.nvindex=0x01800100 rd.dot1x.timeout=60 \
rd.nvmf.discover=tcp,NVMET_TRADDR,4420 \
root=/dev/mapper/root rootfstype=erofs rootflags=ro \
systemd.verity=yes roothash=<from verity.json> \
systemd.verity_root_data=/dev/nvme0n1 \
systemd.verity_root_hash=/dev/nvme0n1 \
systemd.verity_root_options=hash-offset=<from verity.json>

systemd.volatile=overlay gives the writable tmpfs overlay over the read-only verity root — RHEL/Rocky dracut has no rootovl/overlayfs module (that is a Debian/third-party feature), and systemd's volatile overlay is the supported mechanism for dm-verity roots. Single-file layout: data and hash tree share one namespace, split by hash-offset. rd.dot1x.if= may pin the NIC; otherwise the first non-loopback interface is used. Test builds prepend rd.dot1x=0.

Bench-verify before fleet rollout

1. erofs module present in the target RHEL kernel (modprobe erofs) — enabled in later RHEL 9 minors for image mode; if absent, fall back to journal-less ext4 (mkfs.ext4 -O ^has_journal -d <tree>) in extract-artifacts.sh.
2. systemd.verity_root_options support in the shipped systemd/dracut (needs a reasonably recent systemd; otherwise publish data and hash as two separate namespaces and drop the hash-offset option).
3. TPM key reference: confirm wpa_supplicant accepts private_key="handle:0x81000100" via the OpenSSL tpm2 provider; if not, switch to the tpm2-pkcs11 option in the supplicant config.
4. CONFIG_MACSEC in the shipped wpa_supplicant build.
5. Pivot survival: @wpa_supplicant survives switch-root; MKA continuity across it. NVMe/TCP keepalives must also ride out the pivot (they will, if the link never drops — that is the point of the @-daemon).
6. network backend: the hook uses network-legacy/ifup, which requires dhclient. RHEL 9 deprecated it and the minimal ISO omits it, so the provisioner installs dhcp-client explicitly (without it dracut fails with "module network-legacy can't be installed"). dhclient is on its way out; the durable fix is to port the hook to the network-manager dracut module.
7. hostnqn allow-listing: if the nvmet target filters hosts, the allowed-NQN list must be fed from the enrolment pipeline (NQNs are derived from certificate serials — see the hook).
8. CI smoke test: boot artifacts in qemu/Proxmox with swtpm against hostapd-as-authenticator + FreeRADIUS (remember bridge group_fwd_mask=8 for EAPOL); assert macsec0 up, dm-verity root mapped, overlay rw.
9. writable root overlay: provided by systemd.volatile=overlay, which needs systemd-volatile-root.service in the initramfs (bundled by recent dracut). If the root stays read-only at boot, add install_items+=" /usr/lib/systemd/systemd-volatile-root /usr/lib/systemd/system/systemd-volatile-root.service " to 90-diskless.conf. Known to misbehave with NFS root — N/A here (NVMe block).

What is deliberately NOT in the image

• Private keys, certificates, host names, per-host config of any kind. All identity lives in each host's TPM (persistent handle + NV index); the NVMe hostnqn is derived from the certificate at boot.