diskless-workstation/serving

Serving stack — deploy & test (lab)

Stands up DHCP/DNS + HTTP(UKI) + NVMe/TCP(rootfs) so a diskless client can network-boot the release built by the pipeline, and walks through testing it. This targets the open-network test path (rd.dot1x=0, unsigned UKI) — see the repo README for the secure path.

0. Prerequisites

• A serving host on the same L2 as the test client (the boot VLAN / test bridge). Note its interface (BOOT_IFACE) and address (SERVER_IP).
• Docker Engine ≥ 28 (for the type: image artifact volume; otherwise use the commented bind-mount fallback in docker-compose.yml).
• Hugepages for SPDK: sudo sysctl vm.nr_hugepages=1024.
• Registry creds (docker login git.bitasylum.org) — pulls both the artifact image and the SPDK target image.
• The SPDK image is prebuilt by CI (.gitea/workflows/spdk-image.yaml → ${SPDK_IMAGE}:${SPDK_VERSION}); the serving host just pulls it. To roll the SPDK version, bump SPDK_VERSION (or run that workflow manually with a tag) so the image exists in the registry, then update .env. (dnsmasq is the only service still built locally — a tiny alpine image.)

1. Build with a real NVMe discovery address ⚠ required

The UKI's rd.nvmf.discover address is baked in at build time (immutable once signed). The default is the DNS name nvme.fleet.lab, which this stack's dnsmasq resolves to SERVER_IP. If you built before nvmet_traddr existed, the UKI still contains the literal NVMET_TRADDR placeholder and will not connect NVMe — rebuild:

packer build \
  -var image_version=<ver> \
  -var disable_dot1x=true \
  -var nvmet_traddr=nvme.fleet.lab \
  -var oci_image=git.bitasylum.org/audiobuzz/fleet-boot -var oci_push=true \
  .

Keep nvmet_traddr (UKI) == NVME_HOST (this stack's .env).

2. Deploy the serving stack

cd serving
cp .env.example .env      # edit IMAGE_VERSION, BOOT_IFACE, SERVER_IP, DHCP range
docker login git.bitasylum.org
docker compose up -d --build
docker compose ps

IMAGE_VERSION must match the built tag (e.g. a branch build is 0.0.0-<short-sha>; a tag push is the clean version).

Verify each service:

# HTTP: UKI is served
curl -sI http://${SERVER_IP}/fleet-${IMAGE_VERSION}.efi | head -1   # 200 OK

# HTTP: kiosk SPA is served (reverse-proxied to the kiosk-app container —
# the spaceship terminal the diskless kiosk loads by default)
curl -sI http://${SERVER_IP}/kiosk/ | head -1                       # 200 OK

# NTP: lab time server answers (diskless hosts get it via DHCP option 42)
chronyc -h ${SERVER_IP} tracking 2>/dev/null || ntpdate -q ${SERVER_IP}

# NVMe/TCP: target is listening + discoverable
ss -lntp | grep 4420
sudo nvme discover -t tcp -a ${SERVER_IP} -s 4420                   # lists the subsystem

# DNS: discovery name resolves to the serving host
dig +short @${SERVER_IP} nvme.fleet.lab                             # -> ${SERVER_IP}

# Audit collector: rsyslog receiver is listening on :514 (booted hosts offload
# auditd here; records land in audit-collector/logs/<sender-ip>/)
ss -lntp | grep ':514'
logger -n ${SERVER_IP} -P 514 -T -t audispd "collector smoke test"  # -> logs/<ip>/audit.log

# DHCP/boot: watch leases + bootfile handout
docker compose logs -f dnsmasq

3. Boot a test client

Create a diskless client VM (or bare metal) on the boot LAN:

• UEFI firmware (OVMF) with HTTP Boot enabled.
• Secure Boot DISABLED — the lab UKI is unsigned; firmware will reject it otherwise. (Re-enable Secure Boot only once you ship a signed UKI.)
• No disk (root comes from NVMe/TCP), NIC on the boot bridge.
• No TPM/swtpm needed in this mode (rd.dot1x=0 skips 802.1X/MACsec).

qemu example (adjust bridge/OVMF paths):

qemu-system-x86_64 -machine q35 -accel kvm -m 4096 \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=OVMF_VARS.fd \
  -netdev bridge,id=n0,br=<boot-bridge> -device virtio-net-pci,netdev=n0 \
  -nographic

(In the OVMF menu enable HTTP Boot / pick the NIC if it doesn't auto-boot.)

4. Expected sequence & success criteria

1. DHCP lease from dnsmasq; firmware gets option 67 → http://${SERVER_IP}/fleet-<ver>.efi.
2. Firmware fetches the UKI over HTTP and boots it (no GRUB/shim).
3. initramfs: rd.dot1x=0 bypass → DHCP on the NIC.
4. nvme discover to nvme.fleet.lab:4420 (→ SERVER_IP) → connect namespace.
5. dm-verity maps /dev/mapper/root; erofs root mounts read-only.
6. systemd.volatile=overlay adds the tmpfs overlay; systemd reaches multi-user.

Pass: the client reaches a login/multi-user state with the root served over NVMe/TCP. Confirm on the client console: findmnt / (erofs, ro), nvme list (the connected namespace), lsblk (dm-verity root).

5. Troubleshooting

Symptom	Likely cause
Firmware won't load the UKI	Secure Boot still enabled (UKI is unsigned), or wrong boot file URL
No DHCP / no boot menu	client not on the same L2 as BOOT_IFACE; dnsmasq not bound to that iface
Boots, hangs at NVMe	nvmet_traddr ≠ NVME_HOST, or UKI still has the NVMET_TRADDR placeholder (rebuild)
nvme discover empty	SPDK not listening (check docker compose logs nvmeof; hugepages allocated?)
SPDK won't start	hugepages not allocated, or bdev_aio_create -r unsupported on the pinned SPDK tag
verity mapping fails	rootfs.erofs / verity.json mismatch — must be the same release as the UKI (they are, from one build)