Add SPDDK build

2026-06-14 05:58:55 +00:00
parent e98d8813cf
commit 62e997297c
9 changed files with 273 additions and 6 deletions
@@ -0,0 +1,115 @@
+# Serving stack — deploy & test (lab)
+
+Stands up DHCP/DNS + HTTP(UKI) + NVMe/TCP(rootfs) so a diskless client can
+network-boot the release built by the pipeline, and walks through testing it.
+This targets the **open-network test path** (`rd.dot1x=0`, unsigned UKI) — see
+the repo README for the secure path.
+
+## 0. Prerequisites
+
+- A serving host **on the same L2 as the test client** (the boot VLAN / test
+  bridge). Note its interface (`BOOT_IFACE`) and address (`SERVER_IP`).
+- **Docker Engine ≥ 28** (for the `type: image` artifact volume; otherwise use
+  the commented bind-mount fallback in `docker-compose.yml`).
+- Hugepages for SPDK: `sudo sysctl vm.nr_hugepages=1024`.
+- Registry creds (`docker login git.bitasylum.org`) — pulls both the artifact
+  image and the SPDK target image.
+- The **SPDK image is prebuilt by CI** (`.gitea/workflows/spdk-image.yaml` →
+  `${SPDK_IMAGE}:${SPDK_VERSION}`); the serving host just pulls it. To roll the
+  SPDK version, bump `SPDK_VERSION` (or run that workflow manually with a tag)
+  so the image exists in the registry, then update `.env`. (`dnsmasq` is the
+  only service still built locally — a tiny alpine image.)
+
+## 1. Build with a real NVMe discovery address  ⚠ required
+
+The UKI's `rd.nvmf.discover` address is **baked in at build time** (immutable
+once signed). The default is the DNS name `nvme.fleet.lab`, which this stack's
+dnsmasq resolves to `SERVER_IP`. If you built before `nvmet_traddr` existed, the
+UKI still contains the literal `NVMET_TRADDR` placeholder and **will not connect
+NVMe** — rebuild:
+
+```bash
+packer build \
+  -var image_version=<ver> \
+  -var disable_dot1x=true \
+  -var nvmet_traddr=nvme.fleet.lab \
+  -var oci_image=git.bitasylum.org/audiobuzz/fleet-boot -var oci_push=true \
+  .
+```
+
+Keep `nvmet_traddr` (UKI) == `NVME_HOST` (this stack's `.env`).
+
+## 2. Deploy the serving stack
+
+```bash
+cd serving
+cp .env.example .env      # edit IMAGE_VERSION, BOOT_IFACE, SERVER_IP, DHCP range
+docker login git.bitasylum.org
+docker compose up -d --build
+docker compose ps
+```
+
+`IMAGE_VERSION` must match the built tag (e.g. a branch build is
+`0.0.0-<short-sha>`; a tag push is the clean version).
+
+**Verify each service:**
+
+```bash
+# HTTP: UKI is served
+curl -sI http://${SERVER_IP}/fleet-${IMAGE_VERSION}.efi | head -1   # 200 OK
+
+# NVMe/TCP: target is listening + discoverable
+ss -lntp | grep 4420
+sudo nvme discover -t tcp -a ${SERVER_IP} -s 4420                   # lists the subsystem
+
+# DNS: discovery name resolves to the serving host
+dig +short @${SERVER_IP} nvme.fleet.lab                             # -> ${SERVER_IP}
+
+# DHCP/boot: watch leases + bootfile handout
+docker compose logs -f dnsmasq
+```
+
+## 3. Boot a test client
+
+Create a diskless client VM (or bare metal) on the boot LAN:
+
+- **UEFI firmware (OVMF)** with **HTTP Boot** enabled.
+- **Secure Boot DISABLED** — the lab UKI is **unsigned**; firmware will reject it
+  otherwise. (Re-enable Secure Boot only once you ship a signed UKI.)
+- **No disk** (root comes from NVMe/TCP), NIC on the boot bridge.
+- No TPM/swtpm needed in this mode (`rd.dot1x=0` skips 802.1X/MACsec).
+
+qemu example (adjust bridge/OVMF paths):
+
+```bash
+qemu-system-x86_64 -machine q35 -accel kvm -m 4096 \
+  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
+  -drive if=pflash,format=raw,file=OVMF_VARS.fd \
+  -netdev bridge,id=n0,br=<boot-bridge> -device virtio-net-pci,netdev=n0 \
+  -nographic
+```
+(In the OVMF menu enable HTTP Boot / pick the NIC if it doesn't auto-boot.)
+
+## 4. Expected sequence & success criteria
+
+1. DHCP lease from dnsmasq; firmware gets option 67 → `http://${SERVER_IP}/fleet-<ver>.efi`.
+2. Firmware fetches the UKI over HTTP and boots it (no GRUB/shim).
+3. initramfs: `rd.dot1x=0` bypass → DHCP on the NIC.
+4. `nvme discover` to `nvme.fleet.lab:4420` (→ `SERVER_IP`) → connect namespace.
+5. dm-verity maps `/dev/mapper/root`; erofs root mounts read-only.
+6. `systemd.volatile=overlay` adds the tmpfs overlay; systemd reaches multi-user.
+
+**Pass:** the client reaches a login/multi-user state with the root served over
+NVMe/TCP. Confirm on the client console: `findmnt /` (erofs, ro),
+`nvme list` (the connected namespace), `lsblk` (dm-verity `root`).
+
+## 5. Troubleshooting
+
+| Symptom | Likely cause |
+|---|---|
+| Firmware won't load the UKI | Secure Boot still enabled (UKI is unsigned), or wrong boot file URL |
+| No DHCP / no boot menu | client not on the same L2 as `BOOT_IFACE`; dnsmasq not bound to that iface |
+| Boots, hangs at NVMe | `nvmet_traddr` ≠ `NVME_HOST`, or UKI still has the `NVMET_TRADDR` placeholder (rebuild) |
+| `nvme discover` empty | SPDK not listening (check `docker compose logs nvmeof`; hugepages allocated?) |
+| SPDK won't start | hugepages not allocated, or `bdev_aio_create -r` unsupported on the pinned SPDK tag |
+| verity mapping fails | rootfs.erofs / verity.json mismatch — must be the same release as the UKI (they are, from one build) |