diskless-workstation/serving/spdk/README.md

SPDK NVMe/TCP target — long-lived, multi-version

This is the fleet's root disk: a userspace SPDK nvmf_tgt exporting each rootfs release as two read-only NVMe/TCP namespaces (erofs DATA + dm-verity HASH — the distro's systemd-veritysetup can't do the in-file hash-offset layout, so they're separate namespaces with distinct UUIDs).

Why the target is long-lived

nvmf_tgt sits in the live data path of every running host's root fs. Restarting the process tears down every NVMe/TCP association at once (fleet-wide, not version-scoped) — a host whose root link drops past ctrl_loss_tmo gets root I/O errors and must reboot. So the process must never restart for a routine change.

A host changes root version only by rebooting (no live swap). Therefore: version transitions are reboots; SPDK changes are additive/online; the target process stays up. Routine rollouts never restart SPDK.

How it works

Piece	Role
spdk-target.sh	Persistent entrypoint. Starts nvmf_tgt once, creates the TCP transport + discovery listener, then reconciles every staged release. Never restarted to change versions. On (re)start it re-adds everything under ./releases/, so a serving-host reboot restores all served versions.
spdk-release.sh	In-container control (JSON-RPC). add/remove/list/drain-status/reconcile a version live. One subsystem per version (distinct NQN) so adding/removing one never disturbs the subsystem other hosts are attached to. Removal is drain-gated.
spdk-ctl.sh	Host-side ops. Stages a release's artifacts from its OCI image into ./releases/<ver>/ and drives the live target via docker compose exec.
./releases/<version>/	Per-version rootfs.erofs + rootfs.hash + verity.json. verity.json is the single source of truth for that version's namespace UUIDs + subsystem NQN (written by scripts/extract-artifacts.sh), so the host's by-id verity refs and the target's namespace UUIDs always agree.

Per-version namespace UUIDs are required: two versions served at once must have globally-distinct namespace UUIDs or they collide on the same /dev/disk/by-id/nvme-uuid.<uuid> symlink on the host. extract-artifacts.sh derives them deterministically (UUIDv5 of the version) and bakes them into both the immutable UKI cmdline and verity.json.

Rolling update (add new, drain old, clean up)

cd serving/spdk

./spdk-ctl.sh load 1.3.0            # serve 1.3.0 ALONGSIDE the running 1.2.0 (live, no restart)
# ... bump the deployment manifest digest; reboot hosts → they come up on 1.3.0 ...
./spdk-ctl.sh list                  # watch 1.2.0 CONNS drop toward 0
#   VERSION                CONNS  NQN
#   rootfs-1.3.0           7      nqn.2026-06.org.example.fleet:rootfs-1.3.0
#   rootfs-1.2.0           0      nqn.2026-06.org.example.fleet:rootfs-1.2.0
./spdk-ctl.sh unload 1.2.0          # drain-gated: refuses while CONNS > 0

unload refuses to remove a version that still has connected controllers (an assigned-but-offline host could still return to it). Drain those hosts first, or ./spdk-ctl.sh unload 1.2.0 --force to evict them (they lose root I/O — only for a known-dead version).

In-container equivalents (what spdk-ctl.sh calls):

docker compose exec nvmeof spdk-release list
docker compose exec nvmeof spdk-release add 1.3.0
docker compose exec nvmeof spdk-release drain-status 1.2.0
docker compose exec nvmeof spdk-release remove 1.2.0 [--force]

Capacity & availability

• Each concurrently-served version costs target memory/hugepages + a file-backed bdev. Plan for N live versions (current + previous + canary) during long rollouts; raise HUGEPAGES accordingly.
• The target is still a single-node SPOF. Even with online add/remove, a node failure or a planned nvmf_tgt restart drops every connection. The structural fix is HA + NVMe-oF multipath (ANA): ≥2 targets serving byte-identical, same-UUID, read-only namespaces (replication is coherent precisely because the root is read-only + content-addressed). Cheap host-side mitigation: generous ctrl_loss_tmo/reconnect_delay. See docs/TODO.md #6/#7.
• There is no zero-impact single-node restart — only drain/failover.

Bootstrap (simple single-version)

docker compose up -d still works with one version: the OCI image volume at /artifacts is seeded into ./releases/${IMAGE_VERSION}/ and served automatically. Use spdk-ctl.sh load to add more versions on top.

Legacy

../spdk-setup.sh is the original single-version, configure-at-start entrypoint (changing versions = container restart = all connections dropped). Superseded by the long-lived target; kept for reference.