Initial commit
This commit is contained in:
Audiobuzz
@@ -0,0 +1,134 @@
|
||||
# Diskless Fleet Image Build (Packer)
|
||||
|
||||
Builds the fleet-generic boot artifacts for PXE/HTTP-booted diskless hosts that
|
||||
authenticate with 802.1X EAP-TLS (TPM-held key, certificate in TPM NV) and run
|
||||
MACsec on the access link. The root filesystem is an immutable, reproducible
|
||||
**erofs image with an appended dm-verity hash tree**, served as a single
|
||||
shared **read-only NVMe/TCP namespace**, with a tmpfs overlay for runtime
|
||||
state. Companion to `docs/diskless-fleet-design-v2.pdf`.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
diskless.pkr.hcl Packer template (qemu builder)
|
||||
http/ks.cfg Kickstart for the throwaway build VM
|
||||
files/dracut/95dot1x-macsec/ Custom dracut module
|
||||
module-setup.sh module definition
|
||||
dot1x-macsec-up.sh initqueue hook: NV cert -> @wpa_supplicant
|
||||
-> wait authorized + macsec0 -> DHCP
|
||||
-> hostnqn derived from cert
|
||||
files/etc/dot1x/wpa_supplicant-dot1x.conf Fleet-generic supplicant config
|
||||
files/etc/dot1x/radius-ca.pem PLACEHOLDER - replace with real CA chain
|
||||
files/etc/dracut.conf.d/90-diskless.conf no-hostonly + nvmf/verity/erofs + drivers
|
||||
files/etc/NetworkManager/ marks NIC/macsec0 unmanaged post-pivot
|
||||
scripts/extract-artifacts.sh qcow2 -> vmlinuz + initramfs.img
|
||||
+ rootfs.erofs (+verity) + cmdline
|
||||
```
|
||||
|
||||
## Build host requirements
|
||||
|
||||
- Packer >= 1.10, qemu plugin (`packer init .`)
|
||||
- qemu-kvm with KVM available
|
||||
- libguestfs-tools, **erofs-utils**, **cryptsetup**, jq (artifact extraction)
|
||||
- **systemd-ukify** (+ **sbsigntools** when signing) — UKI assembly
|
||||
- **buildah** (or docker) — OCI release-artifact image build
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
# 1. Replace the CA placeholder
|
||||
cp /path/to/radius-ca-chain.pem files/etc/dot1x/radius-ca.pem
|
||||
|
||||
# 2. Set the NIC drivers for your real fleet hardware (default covers
|
||||
# common Intel + Mellanox); the build VM only has virtio, so anything
|
||||
# not listed here will not be in the initramfs.
|
||||
packer init .
|
||||
packer build \
|
||||
-var image_version=1.0.0 \
|
||||
-var nic_drivers="ice mlx5_core" \
|
||||
-var oci_image=registry.example.org/fleet-boot \
|
||||
-var oci_push=true \
|
||||
-var sb_key=/secure/ci/fleet-sb.key \
|
||||
-var sb_cert=/secure/ci/fleet-sb.crt \
|
||||
.
|
||||
```
|
||||
|
||||
Omit `sb_key`/`sb_cert` for unsigned lab builds (the pipeline warns).
|
||||
Substitute the real NVMe target address for `NVMET_TRADDR` in
|
||||
`scripts/extract-artifacts.sh` (or post-edit `cmdline.txt`) **before** the
|
||||
OCI step: the cmdline is immutable once signed into the UKI.
|
||||
|
||||
Outputs: `artifacts/<version>/` gains `fleet-<version>.efi` (the signed
|
||||
UKI) alongside `rootfs.erofs`/`verity.json`, and the build produces the
|
||||
release-artifact OCI image `<oci_image>:<version>` containing only
|
||||
`/artifacts/{fleet-<version>.efi, rootfs.erofs, verity.json, SHA256SUMS}`
|
||||
with the roothash recorded as an image label. The build hard-fails if the
|
||||
roothash baked into the UKI cmdline does not match `verity.json`.
|
||||
|
||||
## Publishing a version
|
||||
|
||||
The OCI image **is** the release. The serving stack stays constant
|
||||
(separate, independently-patched SPDK and nginx/caddy images); each fleet
|
||||
release only changes which artifact image feeds them:
|
||||
|
||||
1. Push `<oci_image>:<version>` (or let `oci_push=true` do it) and record
|
||||
the **digest**.
|
||||
2. In the fleet-boot deployment manifest (k8s Deployment or compose),
|
||||
bump the artifact initContainer / image-volume reference to that digest
|
||||
— one commit per fleet release; this manifest is the single place
|
||||
coupling UKI <-> root namespace.
|
||||
3. The initContainer copies `/artifacts` into the shared volume; nginx
|
||||
serves `fleet-<version>.efi` over HTTPS (firmware validates against
|
||||
the TlsCaCertificate CA, then Secure Boot verifies the UKI signature);
|
||||
SPDK exposes `rootfs.erofs` as a **read-only** NVMe/TCP namespace.
|
||||
4. Rollout = digest bump; rollback = revert the commit. cosign the image
|
||||
so the deploy pipeline verifies provenance (sbsign is for firmware,
|
||||
cosign is for your cluster).
|
||||
|
||||
## Kernel command line (per build — emitted in cmdline.txt)
|
||||
|
||||
```
|
||||
rd.neednet=1 rootovl \
|
||||
rd.dot1x.nvindex=0x01800001 rd.dot1x.timeout=60 \
|
||||
rd.nvmf.discover=tcp,NVMET_TRADDR,4420 \
|
||||
root=/dev/mapper/root rootfstype=erofs rootflags=ro \
|
||||
systemd.verity=yes roothash=<from verity.json> \
|
||||
systemd.verity_root_data=/dev/nvme0n1 \
|
||||
systemd.verity_root_hash=/dev/nvme0n1 \
|
||||
systemd.verity_root_options=hash-offset=<from verity.json>
|
||||
```
|
||||
|
||||
Single-file layout: data and hash tree share one namespace, split by
|
||||
`hash-offset`. `rd.dot1x.if=` may pin the NIC; otherwise the first
|
||||
non-loopback interface is used.
|
||||
|
||||
## Bench-verify before fleet rollout
|
||||
|
||||
1. **erofs module present** in the target RHEL kernel (`modprobe erofs`) —
|
||||
enabled in later RHEL 9 minors for image mode; if absent, fall back to
|
||||
journal-less ext4 (`mkfs.ext4 -O ^has_journal -d <tree>`) in
|
||||
`extract-artifacts.sh`.
|
||||
2. **systemd.verity_root_options support** in the shipped systemd/dracut
|
||||
(needs a reasonably recent systemd; otherwise publish data and hash as
|
||||
two separate namespaces and drop the hash-offset option).
|
||||
3. **TPM key reference**: confirm wpa_supplicant accepts
|
||||
`private_key="handle:0x81010001"` via the OpenSSL tpm2 provider; if not,
|
||||
switch to the tpm2-pkcs11 option in the supplicant config.
|
||||
4. **CONFIG_MACSEC** in the shipped wpa_supplicant build.
|
||||
5. **Pivot survival**: `@wpa_supplicant` survives switch-root; MKA continuity
|
||||
across it. NVMe/TCP keepalives must also ride out the pivot (they will,
|
||||
if the link never drops — that is the point of the @-daemon).
|
||||
6. **dracut module choice**: skeleton uses `network-legacy`; if removed in
|
||||
your release, port the hook's ifup step to the available backend.
|
||||
7. **hostnqn allow-listing**: if the nvmet target filters hosts, the
|
||||
allowed-NQN list must be fed from the enrolment pipeline (NQNs are
|
||||
derived from certificate serials — see the hook).
|
||||
8. **CI smoke test**: boot artifacts in qemu/Proxmox with swtpm against
|
||||
hostapd-as-authenticator + FreeRADIUS (remember bridge group_fwd_mask=8
|
||||
for EAPOL); assert macsec0 up, dm-verity root mapped, overlay rw.
|
||||
|
||||
## What is deliberately NOT in the image
|
||||
|
||||
- Private keys, certificates, host names, per-host config of any kind.
|
||||
All identity lives in each host's TPM (persistent handle + NV index);
|
||||
the NVMe hostnqn is derived from the certificate at boot.
|
||||
Reference in New Issue
Block a user