169 lines
8.8 KiB
Markdown
169 lines
8.8 KiB
Markdown
# Diskless Fleet Image Build (Packer)
|
|
|
|
Builds the fleet-generic boot artifacts for PXE/HTTP-booted diskless hosts that
|
|
authenticate with 802.1X EAP-TLS (TPM-held key, certificate in TPM NV) and run
|
|
MACsec on the access link. The root filesystem is an immutable, reproducible
|
|
**erofs image with an appended dm-verity hash tree**, served as a single
|
|
shared **read-only NVMe/TCP namespace**, with a tmpfs overlay for runtime
|
|
state. Companion to `docs/diskless-fleet-design-v2.pdf` (see
|
|
`docs/design-v2-errata.md` for as-built deltas on RHEL/Rocky 9).
|
|
|
|
## Layout
|
|
|
|
```
|
|
diskless.pkr.hcl Packer template (qemu builder)
|
|
http/ks.cfg Kickstart for the throwaway build VM
|
|
files/dracut/95dot1x-macsec/ Custom dracut module
|
|
module-setup.sh module definition
|
|
dot1x-macsec-up.sh initqueue hook: NV cert -> @wpa_supplicant
|
|
-> wait authorized + macsec0 -> DHCP
|
|
-> hostnqn derived from cert
|
|
files/etc/dot1x/wpa_supplicant-dot1x.conf Fleet-generic supplicant config
|
|
files/etc/dot1x/radius-ca.pem PLACEHOLDER - replace with real CA chain
|
|
files/etc/dracut.conf.d/90-diskless.conf no-hostonly + nvmf/verity/erofs + drivers
|
|
files/etc/NetworkManager/ marks NIC/macsec0 unmanaged post-pivot
|
|
scripts/extract-artifacts.sh qcow2 -> vmlinuz + initramfs.img
|
|
+ rootfs.erofs (+verity) + cmdline
|
|
scripts/build-oci.sh ukify (+sbsign) + roothash gate -> OCI image
|
|
serving/ Lab serving stack (docker compose):
|
|
dnsmasq DHCP/DNS + nginx (UKI/HTTP) + SPDK
|
|
.gitea/workflows/ Gitea CI: lint -> packer build -> push OCI
|
|
docs/diskless-fleet-design-v2.pdf Architecture design (authoritative)
|
|
```
|
|
|
|
## Build host requirements
|
|
|
|
- Packer >= 1.10, qemu plugin (`packer init .`)
|
|
- qemu-kvm with KVM available
|
|
- libguestfs-tools, **erofs-utils**, **cryptsetup**, jq (artifact extraction)
|
|
- **systemd-ukify** + **systemd-boot-efi** (the EFI stub `ukify` wraps;
|
|
it's only a *recommends* of ukify) (+ **sbsigntools** when signing) — UKI assembly
|
|
- **buildah** (or docker) — OCI release-artifact image build
|
|
|
|
The default `iso_url`/`iso_checksum` point at a version-pinned **Rocky 9.x
|
|
minimal** ISO on the internal mirror. `http/ks.cfg` installs only core+sshd from
|
|
that ISO; the diskless tooling is pulled online by the provisioner, so the build
|
|
VM needs **outbound network to the distro repos** (and a fast mirror helps).
|
|
Bump the URL and the pinned sha256 together when moving to a new point release.
|
|
|
|
## Build
|
|
|
|
```bash
|
|
# 1. Replace the CA placeholder
|
|
cp /path/to/radius-ca-chain.pem files/etc/dot1x/radius-ca.pem
|
|
|
|
# 2. Set the NIC drivers for your real fleet hardware (default covers
|
|
# common Intel + Mellanox); the build VM only has virtio, so anything
|
|
# not listed here will not be in the initramfs.
|
|
packer init .
|
|
packer build \
|
|
-var image_version=1.0.0 \
|
|
-var nic_drivers="ice mlx5_core" \
|
|
-var oci_image=registry.example.org/fleet-boot \
|
|
-var oci_push=true \
|
|
-var sb_key=/secure/ci/fleet-sb.key \
|
|
-var sb_cert=/secure/ci/fleet-sb.crt \
|
|
.
|
|
```
|
|
|
|
Omit `sb_key`/`sb_cert` for unsigned lab builds (the pipeline warns).
|
|
Set the NVMe discovery address with `-var nvmet_traddr=<dns-name>` (default
|
|
`nvme.fleet.lab`, which the `serving/` stack's dnsmasq resolves); it is baked
|
|
into the cmdline and **immutable once signed**, so prefer a per-environment DNS
|
|
name over a raw IP. Deploy/boot-test instructions live in
|
|
[serving/README.md](serving/README.md).
|
|
|
|
**Test mode (auth disabled).** Add `-var disable_dot1x=true` to bake
|
|
`rd.dot1x=0` into the cmdline. At boot the hook then skips 802.1X/MACsec and
|
|
DHCPs the physical NIC directly to reach the NVMe/TCP target — for validating
|
|
the UKI/rootfs/NVMe-oF path on an **open switch port**. The auth stack stays in
|
|
the image (dormant); drop the var to restore the secure boot. See `serving/`
|
|
for the matching DHCP/HTTP/NVMe-oF lab stack.
|
|
|
|
Outputs: `artifacts/<version>/` gains `fleet-<version>.efi` (the signed
|
|
UKI) alongside `rootfs.erofs`/`verity.json`, and the build produces the
|
|
release-artifact OCI image `<oci_image>:<version>` containing only
|
|
`/artifacts/{fleet-<version>.efi, rootfs.erofs, verity.json, SHA256SUMS}`
|
|
with the roothash recorded as an image label. The build hard-fails if the
|
|
roothash baked into the UKI cmdline does not match `verity.json`.
|
|
|
|
## Publishing a version
|
|
|
|
The OCI image **is** the release. The serving stack stays constant
|
|
(separate, independently-patched SPDK and nginx/caddy images); each fleet
|
|
release only changes which artifact image feeds them:
|
|
|
|
1. Push `<oci_image>:<version>` (or let `oci_push=true` do it) and record
|
|
the **digest**.
|
|
2. In the fleet-boot deployment manifest (k8s Deployment or compose),
|
|
bump the artifact initContainer / image-volume reference to that digest
|
|
— one commit per fleet release; this manifest is the single place
|
|
coupling UKI <-> root namespace.
|
|
3. The initContainer copies `/artifacts` into the shared volume; nginx
|
|
serves `fleet-<version>.efi` over HTTPS (firmware validates against
|
|
the TlsCaCertificate CA, then Secure Boot verifies the UKI signature);
|
|
SPDK exposes `rootfs.erofs` as a **read-only** NVMe/TCP namespace.
|
|
4. Rollout = digest bump; rollback = revert the commit. cosign the image
|
|
so the deploy pipeline verifies provenance (sbsign is for firmware,
|
|
cosign is for your cluster).
|
|
|
|
## Kernel command line (per build — emitted in cmdline.txt)
|
|
|
|
```
|
|
rd.neednet=1 systemd.volatile=overlay \
|
|
rd.dot1x.nvindex=0x01800001 rd.dot1x.timeout=60 \
|
|
rd.nvmf.discover=tcp,NVMET_TRADDR,4420 \
|
|
root=/dev/mapper/root rootfstype=erofs rootflags=ro \
|
|
systemd.verity=yes roothash=<from verity.json> \
|
|
systemd.verity_root_data=/dev/nvme0n1 \
|
|
systemd.verity_root_hash=/dev/nvme0n1 \
|
|
systemd.verity_root_options=hash-offset=<from verity.json>
|
|
```
|
|
|
|
`systemd.volatile=overlay` gives the writable tmpfs overlay over the read-only
|
|
verity root — RHEL/Rocky dracut has **no** `rootovl`/`overlayfs` module (that is
|
|
a Debian/third-party feature), and systemd's volatile overlay is the supported
|
|
mechanism for dm-verity roots. Single-file layout: data and hash tree share one
|
|
namespace, split by `hash-offset`. `rd.dot1x.if=` may pin the NIC; otherwise the
|
|
first non-loopback interface is used. Test builds prepend `rd.dot1x=0`.
|
|
|
|
## Bench-verify before fleet rollout
|
|
|
|
1. **erofs module present** in the target RHEL kernel (`modprobe erofs`) —
|
|
enabled in later RHEL 9 minors for image mode; if absent, fall back to
|
|
journal-less ext4 (`mkfs.ext4 -O ^has_journal -d <tree>`) in
|
|
`extract-artifacts.sh`.
|
|
2. **systemd.verity_root_options support** in the shipped systemd/dracut
|
|
(needs a reasonably recent systemd; otherwise publish data and hash as
|
|
two separate namespaces and drop the hash-offset option).
|
|
3. **TPM key reference**: confirm wpa_supplicant accepts
|
|
`private_key="handle:0x81010001"` via the OpenSSL tpm2 provider; if not,
|
|
switch to the tpm2-pkcs11 option in the supplicant config.
|
|
4. **CONFIG_MACSEC** in the shipped wpa_supplicant build.
|
|
5. **Pivot survival**: `@wpa_supplicant` survives switch-root; MKA continuity
|
|
across it. NVMe/TCP keepalives must also ride out the pivot (they will,
|
|
if the link never drops — that is the point of the @-daemon).
|
|
6. **network backend**: the hook uses `network-legacy`/`ifup`, which requires
|
|
`dhclient`. RHEL 9 deprecated it and the minimal ISO omits it, so the
|
|
provisioner installs `dhcp-client` explicitly (without it dracut fails with
|
|
"module network-legacy can't be installed"). dhclient is on its way out; the
|
|
durable fix is to port the hook to the `network-manager` dracut module.
|
|
7. **hostnqn allow-listing**: if the nvmet target filters hosts, the
|
|
allowed-NQN list must be fed from the enrolment pipeline (NQNs are
|
|
derived from certificate serials — see the hook).
|
|
8. **CI smoke test**: boot artifacts in qemu/Proxmox with swtpm against
|
|
hostapd-as-authenticator + FreeRADIUS (remember bridge group_fwd_mask=8
|
|
for EAPOL); assert macsec0 up, dm-verity root mapped, overlay rw.
|
|
9. **writable root overlay**: provided by `systemd.volatile=overlay`, which
|
|
needs `systemd-volatile-root.service` in the initramfs (bundled by recent
|
|
dracut). If the root stays read-only at boot, add `install_items+="
|
|
/usr/lib/systemd/systemd-volatile-root
|
|
/usr/lib/systemd/system/systemd-volatile-root.service "` to
|
|
`90-diskless.conf`. Known to misbehave with NFS root — N/A here (NVMe block).
|
|
|
|
## What is deliberately NOT in the image
|
|
|
|
- Private keys, certificates, host names, per-host config of any kind.
|
|
All identity lives in each host's TPM (persistent handle + NV index);
|
|
the NVMe hostnqn is derived from the certificate at boot.
|