diskless-workstation/serving/README.md

# Serving stack — deploy & test (lab)

Stands up DHCP/DNS + HTTP(UKI) + NVMe/TCP(rootfs) so a diskless client can
network-boot the release built by the pipeline, and walks through testing it.
This targets the **open-network test path** (`rd.dot1x=0`, unsigned UKI) — see
the repo README for the secure path.

## 0. Prerequisites

- A serving host **on the same L2 as the test client** (the boot VLAN / test
  bridge). Note its interface (`BOOT_IFACE`) and address (`SERVER_IP`).
- **Docker Engine ≥ 28** (for the `type: image` artifact volume; otherwise use
  the commented bind-mount fallback in `docker-compose.yml`).
- Hugepages for SPDK: `sudo sysctl vm.nr_hugepages=1024`.
- Registry creds (`docker login git.bitasylum.org`) — pulls both the artifact
  image and the SPDK target image.
- The **SPDK image is prebuilt by CI** (`.gitea/workflows/spdk-image.yaml` →
  `${SPDK_IMAGE}:${SPDK_VERSION}`); the serving host just pulls it. To roll the
  SPDK version, bump `SPDK_VERSION` (or run that workflow manually with a tag)
  so the image exists in the registry, then update `.env`. (`dnsmasq` is the
  only service still built locally — a tiny alpine image.)

## 1. Build with a real NVMe discovery address  ⚠ required

The UKI's `rd.nvmf.discover` address is **baked in at build time** (immutable
once signed). The default is the DNS name `nvme.fleet.lab`, which this stack's
dnsmasq resolves to `SERVER_IP`. If you built before `nvmet_traddr` existed, the
UKI still contains the literal `NVMET_TRADDR` placeholder and **will not connect
NVMe** — rebuild:

```bash
packer build \
  -var image_version=<ver> \
  -var disable_dot1x=true \
  -var nvmet_traddr=nvme.fleet.lab \
  -var oci_image=git.bitasylum.org/audiobuzz/fleet-boot -var oci_push=true \
  .
```

Keep `nvmet_traddr` (UKI) == `NVME_HOST` (this stack's `.env`).

## 2. Deploy the serving stack

```bash
cd serving
# config = committed defaults.env + your .env overrides, layered (or use `make`):
export COMPOSE_ENV_FILES=defaults.env,.env
printf 'IMAGE_VERSION=<ver>\nBOOT_IFACE=<iface>\nSERVER_IP=<ip>\n' > .env   # + DHCP range
docker login git.bitasylum.org
docker compose up -d --build      # or simply: make up
docker compose ps                 # or: make ps
```

`IMAGE_VERSION` must match the built tag (e.g. a branch build is
`0.0.0-<short-sha>`; a tag push is the clean version).

**Verify each service:**

```bash
# HTTP: UKI is served
curl -sI http://${SERVER_IP}/fleet-${IMAGE_VERSION}.efi | head -1   # 200 OK

# HTTP: kiosk SPA is served (reverse-proxied to the kiosk-app container —
# the spaceship terminal the diskless kiosk loads by default)
curl -sI http://${SERVER_IP}/kiosk/ | head -1                       # 200 OK

# NTP: lab time server answers (diskless hosts get it via DHCP option 42)
chronyc -h ${SERVER_IP} tracking 2>/dev/null || ntpdate -q ${SERVER_IP}

# NVMe/TCP: target is listening + discoverable
ss -lntp | grep 4420
sudo nvme discover -t tcp -a ${SERVER_IP} -s 4420                   # lists the subsystem

# DNS: discovery name resolves to the serving host
dig +short @${SERVER_IP} nvme.fleet.lab                             # -> ${SERVER_IP}

# Audit collector: syslog-TLS receiver on :6514 (booted hosts offload auditd
# here; records land in audit-collector/logs/<sender-ip>/). Needs the server
# cert issued first: audit-collector/issue-cert.sh (fleet step-ca).
ss -lntp | grep ':6514'
openssl s_client -connect ${SERVER_IP}:6514 -CAfile audit-collector/tls/ca.crt </dev/null  # Verify return code: 0
# End-to-end record via the LEGACY plaintext :514 intake (kept for pre-TLS images)
logger -n ${SERVER_IP} -P 514 -T -t audispd "collector smoke test"  # -> logs/<ip>/audit.log

# DHCP/boot: watch leases + bootfile handout
docker compose logs -f dnsmasq
```

## 3. Boot a test client

Create a diskless client VM (or bare metal) on the boot LAN:

- **UEFI firmware (OVMF)** with **HTTP Boot** enabled.
- **Secure Boot DISABLED** — the lab UKI is **unsigned**; firmware will reject it
  otherwise. (Re-enable Secure Boot only once you ship a signed UKI.)
- **No disk** (root comes from NVMe/TCP), NIC on the boot bridge.
- No TPM/swtpm needed in this mode (`rd.dot1x=0` skips 802.1X/MACsec).

qemu example (adjust bridge/OVMF paths):

```bash
qemu-system-x86_64 -machine q35 -accel kvm -m 4096 \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=OVMF_VARS.fd \
  -netdev bridge,id=n0,br=<boot-bridge> -device virtio-net-pci,netdev=n0 \
  -nographic
```
(In the OVMF menu enable HTTP Boot / pick the NIC if it doesn't auto-boot.)

## 4. Expected sequence & success criteria

1. DHCP lease from dnsmasq; firmware gets option 67 → `http://${SERVER_IP}/fleet-<ver>.efi`.
2. Firmware fetches the UKI over HTTP and boots it (no GRUB/shim).
3. initramfs: `rd.dot1x=0` bypass → DHCP on the NIC.
4. `nvme discover` to `nvme.fleet.lab:4420` (→ `SERVER_IP`) → connect namespace.
5. dm-verity maps `/dev/mapper/root`; erofs root mounts read-only.
6. `systemd.volatile=overlay` adds the tmpfs overlay; systemd reaches multi-user.

**Pass:** the client reaches a login/multi-user state with the root served over
NVMe/TCP. Confirm on the client console: `findmnt /` (erofs, ro),
`nvme list` (the connected namespace), `lsblk` (dm-verity `root`).

## 5. Troubleshooting

| Symptom | Likely cause |
|---|---|
| Firmware won't load the UKI | Secure Boot still enabled (UKI is unsigned), or wrong boot file URL |
| No DHCP / no boot menu | client not on the same L2 as `BOOT_IFACE`; dnsmasq not bound to that iface |
| Boots, hangs at NVMe | `nvmet_traddr` ≠ `NVME_HOST`, or UKI still has the `NVMET_TRADDR` placeholder (rebuild) |
| `nvme discover` empty | SPDK not listening (check `docker compose logs nvmeof`; hugepages allocated?) |
| SPDK won't start | hugepages not allocated, or `bdev_aio_create -r` unsupported on the pinned SPDK tag |
| verity mapping fails | rootfs.erofs / verity.json mismatch — must be the same release as the UKI (they are, from one build) |