2026-06-12 01:46:29 +00:00
2026-06-12 01:46:29 +00:00
2026-06-12 01:46:29 +00:00
2026-06-13 02:15:04 +00:00
2026-06-12 01:46:29 +00:00
2026-06-12 01:46:29 +00:00
2026-06-12 02:01:42 +00:00
2026-06-12 01:46:29 +00:00
2026-06-12 01:46:29 +00:00
2026-06-12 01:46:29 +00:00
2026-06-12 01:46:29 +00:00

Diskless Fleet Image Build (Packer)

Builds the fleet-generic boot artifacts for PXE/HTTP-booted diskless hosts that authenticate with 802.1X EAP-TLS (TPM-held key, certificate in TPM NV) and run MACsec on the access link. The root filesystem is an immutable, reproducible erofs image with an appended dm-verity hash tree, served as a single shared read-only NVMe/TCP namespace, with a tmpfs overlay for runtime state. Companion to docs/diskless-fleet-design-v2.pdf.

Layout

diskless.pkr.hcl                          Packer template (qemu builder)
http/ks.cfg                               Kickstart for the throwaway build VM
files/dracut/95dot1x-macsec/              Custom dracut module
  module-setup.sh                           module definition
  dot1x-macsec-up.sh                        initqueue hook: NV cert -> @wpa_supplicant
                                            -> wait authorized + macsec0 -> DHCP
                                            -> hostnqn derived from cert
files/etc/dot1x/wpa_supplicant-dot1x.conf Fleet-generic supplicant config
files/etc/dot1x/radius-ca.pem             PLACEHOLDER - replace with real CA chain
files/etc/dracut.conf.d/90-diskless.conf  no-hostonly + nvmf/verity/erofs + drivers
files/etc/NetworkManager/                 marks NIC/macsec0 unmanaged post-pivot
scripts/extract-artifacts.sh              qcow2 -> vmlinuz + initramfs.img
                                                 + rootfs.erofs (+verity) + cmdline

Build host requirements

  • Packer >= 1.10, qemu plugin (packer init .)
  • qemu-kvm with KVM available
  • libguestfs-tools, erofs-utils, cryptsetup, jq (artifact extraction)
  • systemd-ukify (+ sbsigntools when signing) — UKI assembly
  • buildah (or docker) — OCI release-artifact image build

Build

# 1. Replace the CA placeholder
cp /path/to/radius-ca-chain.pem files/etc/dot1x/radius-ca.pem

# 2. Set the NIC drivers for your real fleet hardware (default covers
#    common Intel + Mellanox); the build VM only has virtio, so anything
#    not listed here will not be in the initramfs.
packer init .
packer build \
  -var image_version=1.0.0 \
  -var nic_drivers="ice mlx5_core" \
  -var oci_image=registry.example.org/fleet-boot \
  -var oci_push=true \
  -var sb_key=/secure/ci/fleet-sb.key \
  -var sb_cert=/secure/ci/fleet-sb.crt \
  .

Omit sb_key/sb_cert for unsigned lab builds (the pipeline warns). Substitute the real NVMe target address for NVMET_TRADDR in scripts/extract-artifacts.sh (or post-edit cmdline.txt) before the OCI step: the cmdline is immutable once signed into the UKI.

Outputs: artifacts/<version>/ gains fleet-<version>.efi (the signed UKI) alongside rootfs.erofs/verity.json, and the build produces the release-artifact OCI image <oci_image>:<version> containing only /artifacts/{fleet-<version>.efi, rootfs.erofs, verity.json, SHA256SUMS} with the roothash recorded as an image label. The build hard-fails if the roothash baked into the UKI cmdline does not match verity.json.

Publishing a version

The OCI image is the release. The serving stack stays constant (separate, independently-patched SPDK and nginx/caddy images); each fleet release only changes which artifact image feeds them:

  1. Push <oci_image>:<version> (or let oci_push=true do it) and record the digest.
  2. In the fleet-boot deployment manifest (k8s Deployment or compose), bump the artifact initContainer / image-volume reference to that digest — one commit per fleet release; this manifest is the single place coupling UKI <-> root namespace.
  3. The initContainer copies /artifacts into the shared volume; nginx serves fleet-<version>.efi over HTTPS (firmware validates against the TlsCaCertificate CA, then Secure Boot verifies the UKI signature); SPDK exposes rootfs.erofs as a read-only NVMe/TCP namespace.
  4. Rollout = digest bump; rollback = revert the commit. cosign the image so the deploy pipeline verifies provenance (sbsign is for firmware, cosign is for your cluster).

Kernel command line (per build — emitted in cmdline.txt)

rd.neednet=1 rootovl \
rd.dot1x.nvindex=0x01800001 rd.dot1x.timeout=60 \
rd.nvmf.discover=tcp,NVMET_TRADDR,4420 \
root=/dev/mapper/root rootfstype=erofs rootflags=ro \
systemd.verity=yes roothash=<from verity.json> \
systemd.verity_root_data=/dev/nvme0n1 \
systemd.verity_root_hash=/dev/nvme0n1 \
systemd.verity_root_options=hash-offset=<from verity.json>

Single-file layout: data and hash tree share one namespace, split by hash-offset. rd.dot1x.if= may pin the NIC; otherwise the first non-loopback interface is used.

Bench-verify before fleet rollout

  1. erofs module present in the target RHEL kernel (modprobe erofs) — enabled in later RHEL 9 minors for image mode; if absent, fall back to journal-less ext4 (mkfs.ext4 -O ^has_journal -d <tree>) in extract-artifacts.sh.
  2. systemd.verity_root_options support in the shipped systemd/dracut (needs a reasonably recent systemd; otherwise publish data and hash as two separate namespaces and drop the hash-offset option).
  3. TPM key reference: confirm wpa_supplicant accepts private_key="handle:0x81010001" via the OpenSSL tpm2 provider; if not, switch to the tpm2-pkcs11 option in the supplicant config.
  4. CONFIG_MACSEC in the shipped wpa_supplicant build.
  5. Pivot survival: @wpa_supplicant survives switch-root; MKA continuity across it. NVMe/TCP keepalives must also ride out the pivot (they will, if the link never drops — that is the point of the @-daemon).
  6. dracut module choice: skeleton uses network-legacy; if removed in your release, port the hook's ifup step to the available backend.
  7. hostnqn allow-listing: if the nvmet target filters hosts, the allowed-NQN list must be fed from the enrolment pipeline (NQNs are derived from certificate serials — see the hook).
  8. CI smoke test: boot artifacts in qemu/Proxmox with swtpm against hostapd-as-authenticator + FreeRADIUS (remember bridge group_fwd_mask=8 for EAPOL); assert macsec0 up, dm-verity root mapped, overlay rw.

What is deliberately NOT in the image

  • Private keys, certificates, host names, per-host config of any kind. All identity lives in each host's TPM (persistent handle + NV index); the NVMe hostnqn is derived from the certificate at boot.
S
Description
No description provided
Readme MIT
6.6 MiB
Languages
Go 47.2%
Shell 27.7%
HCL 15.5%
Svelte 4.5%
JavaScript 2%
Other 3%