Add SPDDK build

2026-06-14 05:58:55 +00:00
parent e98d8813cf
commit 62e997297c
9 changed files with 273 additions and 6 deletions
@@ -12,6 +12,11 @@ OCI_IMAGE=git.bitasylum.org/audiobuzz/fleet-boot
 # commented bind-mounts in docker-compose.yml. Ignored when image volumes work.
 ARTIFACT_SRC=../artifacts/0.1.0

+# SPDK NVMe/TCP target image — prebuilt + pushed by CI
+# (.gitea/workflows/spdk-image.yaml); the serving host just pulls it.
+SPDK_IMAGE=git.bitasylum.org/audiobuzz/spdk-nvmf
+SPDK_VERSION=v24.09
+
 # --- This serving host on the boot LAN -------------------------------------
 BOOT_IFACE=eth0            # interface facing the booting client / test bridge
 SERVER_IP=192.168.50.10    # this host's address on BOOT_IFACE (DHCP/DNS/HTTP/NVMe)
@@ -0,0 +1,115 @@
+# Serving stack — deploy & test (lab)
+
+Stands up DHCP/DNS + HTTP(UKI) + NVMe/TCP(rootfs) so a diskless client can
+network-boot the release built by the pipeline, and walks through testing it.
+This targets the **open-network test path** (`rd.dot1x=0`, unsigned UKI) — see
+the repo README for the secure path.
+
+## 0. Prerequisites
+
+- A serving host **on the same L2 as the test client** (the boot VLAN / test
+  bridge). Note its interface (`BOOT_IFACE`) and address (`SERVER_IP`).
+- **Docker Engine ≥ 28** (for the `type: image` artifact volume; otherwise use
+  the commented bind-mount fallback in `docker-compose.yml`).
+- Hugepages for SPDK: `sudo sysctl vm.nr_hugepages=1024`.
+- Registry creds (`docker login git.bitasylum.org`) — pulls both the artifact
+  image and the SPDK target image.
+- The **SPDK image is prebuilt by CI** (`.gitea/workflows/spdk-image.yaml` →
+  `${SPDK_IMAGE}:${SPDK_VERSION}`); the serving host just pulls it. To roll the
+  SPDK version, bump `SPDK_VERSION` (or run that workflow manually with a tag)
+  so the image exists in the registry, then update `.env`. (`dnsmasq` is the
+  only service still built locally — a tiny alpine image.)
+
+## 1. Build with a real NVMe discovery address  ⚠ required
+
+The UKI's `rd.nvmf.discover` address is **baked in at build time** (immutable
+once signed). The default is the DNS name `nvme.fleet.lab`, which this stack's
+dnsmasq resolves to `SERVER_IP`. If you built before `nvmet_traddr` existed, the
+UKI still contains the literal `NVMET_TRADDR` placeholder and **will not connect
+NVMe** — rebuild:
+
+```bash
+packer build \
+  -var image_version=<ver> \
+  -var disable_dot1x=true \
+  -var nvmet_traddr=nvme.fleet.lab \
+  -var oci_image=git.bitasylum.org/audiobuzz/fleet-boot -var oci_push=true \
+  .
+```
+
+Keep `nvmet_traddr` (UKI) == `NVME_HOST` (this stack's `.env`).
+
+## 2. Deploy the serving stack
+
+```bash
+cd serving
+cp .env.example .env      # edit IMAGE_VERSION, BOOT_IFACE, SERVER_IP, DHCP range
+docker login git.bitasylum.org
+docker compose up -d --build
+docker compose ps
+```
+
+`IMAGE_VERSION` must match the built tag (e.g. a branch build is
+`0.0.0-<short-sha>`; a tag push is the clean version).
+
+**Verify each service:**
+
+```bash
+# HTTP: UKI is served
+curl -sI http://${SERVER_IP}/fleet-${IMAGE_VERSION}.efi | head -1   # 200 OK
+
+# NVMe/TCP: target is listening + discoverable
+ss -lntp | grep 4420
+sudo nvme discover -t tcp -a ${SERVER_IP} -s 4420                   # lists the subsystem
+
+# DNS: discovery name resolves to the serving host
+dig +short @${SERVER_IP} nvme.fleet.lab                             # -> ${SERVER_IP}
+
+# DHCP/boot: watch leases + bootfile handout
+docker compose logs -f dnsmasq
+```
+
+## 3. Boot a test client
+
+Create a diskless client VM (or bare metal) on the boot LAN:
+
+- **UEFI firmware (OVMF)** with **HTTP Boot** enabled.
+- **Secure Boot DISABLED** — the lab UKI is **unsigned**; firmware will reject it
+  otherwise. (Re-enable Secure Boot only once you ship a signed UKI.)
+- **No disk** (root comes from NVMe/TCP), NIC on the boot bridge.
+- No TPM/swtpm needed in this mode (`rd.dot1x=0` skips 802.1X/MACsec).
+
+qemu example (adjust bridge/OVMF paths):
+
+```bash
+qemu-system-x86_64 -machine q35 -accel kvm -m 4096 \
+  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
+  -drive if=pflash,format=raw,file=OVMF_VARS.fd \
+  -netdev bridge,id=n0,br=<boot-bridge> -device virtio-net-pci,netdev=n0 \
+  -nographic
+```
+(In the OVMF menu enable HTTP Boot / pick the NIC if it doesn't auto-boot.)
+
+## 4. Expected sequence & success criteria
+
+1. DHCP lease from dnsmasq; firmware gets option 67 → `http://${SERVER_IP}/fleet-<ver>.efi`.
+2. Firmware fetches the UKI over HTTP and boots it (no GRUB/shim).
+3. initramfs: `rd.dot1x=0` bypass → DHCP on the NIC.
+4. `nvme discover` to `nvme.fleet.lab:4420` (→ `SERVER_IP`) → connect namespace.
+5. dm-verity maps `/dev/mapper/root`; erofs root mounts read-only.
+6. `systemd.volatile=overlay` adds the tmpfs overlay; systemd reaches multi-user.
+
+**Pass:** the client reaches a login/multi-user state with the root served over
+NVMe/TCP. Confirm on the client console: `findmnt /` (erofs, ro),
+`nvme list` (the connected namespace), `lsblk` (dm-verity `root`).
+
+## 5. Troubleshooting
+
+| Symptom | Likely cause |
+|---|---|
+| Firmware won't load the UKI | Secure Boot still enabled (UKI is unsigned), or wrong boot file URL |
+| No DHCP / no boot menu | client not on the same L2 as `BOOT_IFACE`; dnsmasq not bound to that iface |
+| Boots, hangs at NVMe | `nvmet_traddr` ≠ `NVME_HOST`, or UKI still has the `NVMET_TRADDR` placeholder (rebuild) |
+| `nvme discover` empty | SPDK not listening (check `docker compose logs nvmeof`; hugepages allocated?) |
+| SPDK won't start | hugepages not allocated, or `bdev_aio_create -r` unsupported on the pinned SPDK tag |
+| verity mapping fails | rootfs.erofs / verity.json mismatch — must be the same release as the UKI (they are, from one build) |
@@ -90,7 +90,10 @@ services:

  # --- NVMe/TCP target (serves rootfs.erofs, read-only) ----------------------
  nvmeof:
-    image: spdk/spdk:v24.09   # pin to a tag you've tested
+    # Prebuilt by CI (.gitea/workflows/spdk-image.yaml) and pulled here — no
+    # official spdk/spdk image exists, so we compile our own once in the
+    # pipeline. Local-dev alternative: replace this line with `build: ./spdk`.
+    image: ${SPDK_IMAGE}:${SPDK_VERSION}
    network_mode: host        # NVMe/TCP listener on ${TARGET_PORT}
    privileged: true          # hugepages + IPC_LOCK/SYS_ADMIN for SPDK
    restart: unless-stopped
@@ -0,0 +1,39 @@
+# Build SPDK's nvmf_tgt from source.
+#
+# There is no reliable official `spdk/spdk` image to pull (SPDK's guidance is to
+# build your own — https://spdk.io/doc/containers.html), so we compile a pinned
+# release here. Single-stage: the image is large and the FIRST build is slow
+# (~10-20 min — it pulls submodules incl. DPDK and compiles them), but it's
+# self-contained, reproducible, and cached thereafter. Bump SPDK_VERSION to a
+# tag from https://github.com/spdk/spdk/releases.
+FROM ubuntu:24.04
+
+ARG SPDK_VERSION=v24.09
+ARG SPDK_REPO=https://github.com/spdk/spdk
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends git ca-certificates \
+ && rm -rf /var/lib/apt/lists/*
+
+# Shallow-clone the pinned release; submodules at their pinned commits.
+RUN git clone --branch "${SPDK_VERSION}" --depth 1 "${SPDK_REPO}" /spdk \
+ && git -C /spdk submodule update --init --recursive
+
+# SPDK's own dependency installer (kernel headers, build tools, DPDK deps,
+# python for rpc.py, …). Then configure (TCP nvmf transport is in the default
+# build; trim the test artifacts) and build.
+RUN /spdk/scripts/pkgdep.sh \
+ && cd /spdk \
+ && ./configure --disable-tests --disable-unit-tests \
+ && make -j"$(nproc)"
+
+# nvmf_tgt lives in build/bin; rpc.py in scripts/ (kept ON PATH so its relative
+# imports resolve from the source tree). spdk-setup.sh locates both via `command
+# -v`. LD_LIBRARY_PATH is defensive in case of a shared-lib build.
+ENV PATH=/spdk/build/bin:/spdk/scripts:$PATH \
+    LD_LIBRARY_PATH=/spdk/build/lib:/spdk/dpdk/build/lib
+
+# spdk-setup.sh is bind-mounted at /spdk-setup.sh and set as the entrypoint by
+# docker-compose.yml.