This is the fleet's root disk: a userspace SPDK nvmf_tgt exporting each rootfs release as two read-only NVMe/TCP namespaces (erofs DATA + dm-verity HASH — the distro's systemd-veritysetup can't do the in-file hash-offset layout, so they're separate namespaces with distinct UUIDs).

Why the target is long-lived

nvmf_tgt sits in the live data path of every running host's root fs. Restarting the process tears down every NVMe/TCP association at once (fleet-wide, not version-scoped) — a host whose root link drops past ctrl_loss_tmo gets root I/O errors and must reboot. So the process must never restart for a routine change.

A host changes root version only by rebooting (no live swap). Therefore: version transitions are reboots; SPDK changes are additive/online; the target process stays up. Routine rollouts never restart SPDK.

How it works

Piece	Role
`spdk-target.sh`	Persistent entrypoint. Starts `nvmf_tgt` once, creates the TCP transport + discovery listener, then reconciles every staged release. Never restarted to change versions. On (re)start it re-adds everything under `./releases/`, so a serving-host reboot restores all served versions.
`spdk-release.sh`	In-container control (JSON-RPC). `add`/`remove`/`list`/`drain-status`/`reconcile` a version live. One subsystem per version (distinct NQN) so adding/removing one never disturbs the subsystem other hosts are attached to. Removal is drain-gated.
`spdk-ctl.sh`	Host-side ops (manual / cleanup). Stages a release's artifacts from its OCI image into `./releases/<ver>/` and drives the live target via `docker compose exec`.
`spdk-loader.sh`	Declarative rolling updates (compose service `spdk-loader`, the "init container that stays up"). On `compose up` after an `IMAGE_VERSION` bump it stages + adds the new rootfs online, then auto-reaps old versions once they drain.
`./releases/<version>/`	Per-version `rootfs.erofs` + `rootfs.hash` + `verity.json`. `verity.json` is the single source of truth for that version's namespace UUIDs + subsystem NQN (written by `scripts/extract-artifacts.sh`), so the host's by-id verity refs and the target's namespace UUIDs always agree.

Per-version namespace UUIDs are required: two versions served at once must have globally-distinct namespace UUIDs or they collide on the same /dev/disk/by-id/nvme-uuid.<uuid> symlink on the host. extract-artifacts.sh derives them deterministically (UUIDv5 of the version) and bakes them into both the immutable UKI cmdline and verity.json.

Rolling update — the declarative way (recommended)

Bump one tag and bring the stack up:

# in serving/.env
IMAGE_VERSION=1.3.0

docker compose up -d

Compose recreates the spdk-loader and the http/dnsmasq UKI servers (their image volumes changed), but not the nvmeof target (its definition is unchanged) — so no connections drop. The loader adds 1.3.0 to the live target, serves the new UKI to all hosts, and then auto-reaps 1.2.0 once every host has rebooted onto 1.3.0 and 1.2.0 has been drained for REAP_GRACE_SECONDS. Rollback = set IMAGE_VERSION back + up -d. Set REAP_OLD=false to keep old versions and reap them yourself.

Why auto-reap is safe here: there is a single served UKI, so a powered-off host that returns boots the current UKI — nothing can come back to an old rootfs once its live connections (CONNS) hit 0. Removal is still drain-gated and debounced by the grace window (ride out transient nvme-tcp reconnects — keep REAP_GRACE_SECONDS above the host ctrl_loss_tmo).

Rolling update — the manual way (`spdk-ctl.sh`)

cd serving/spdk

./spdk-ctl.sh load 1.3.0            # serve 1.3.0 ALONGSIDE the running 1.2.0 (live, no restart)
# ... bump the deployment manifest digest; reboot hosts → they come up on 1.3.0 ...
./spdk-ctl.sh list                  # watch 1.2.0 CONNS drop toward 0
#   VERSION                CONNS  NQN
#   rootfs-1.3.0           7      nqn.2026-06.org.example.fleet:rootfs-1.3.0
#   rootfs-1.2.0           0      nqn.2026-06.org.example.fleet:rootfs-1.2.0
./spdk-ctl.sh unload 1.2.0          # drain-gated: refuses while CONNS > 0

unload refuses to remove a version that still has connected controllers (an assigned-but-offline host could still return to it). Drain those hosts first, or ./spdk-ctl.sh unload 1.2.0 --force to evict them (they lose root I/O — only for a known-dead version).

In-container equivalents (what spdk-ctl.sh calls):

docker compose exec nvmeof spdk-release list
docker compose exec nvmeof spdk-release add 1.3.0
docker compose exec nvmeof spdk-release drain-status 1.2.0
docker compose exec nvmeof spdk-release remove 1.2.0 [--force]

Capacity & availability

Each concurrently-served version costs target memory/hugepages + a file-backed bdev. Plan for N live versions (current + previous + canary) during long rollouts; raise HUGEPAGES accordingly.
The target is still a single-node SPOF. Even with online add/remove, a node failure or a planned nvmf_tgt restart drops every connection. The structural fix is HA + NVMe-oF multipath (ANA): ≥2 targets serving byte-identical, same-UUID, read-only namespaces (replication is coherent precisely because the root is read-only + content-addressed). Cheap host-side mitigation: generous ctrl_loss_tmo/reconnect_delay. See docs/TODO.md #6/#7.
There is no zero-impact single-node restart — only drain/failover.

Bootstrap (simple single-version)

docker compose up -d still works with one version: the OCI image volume at /artifacts is seeded into ./releases/${IMAGE_VERSION}/ and served automatically. Use spdk-ctl.sh load to add more versions on top.

Legacy

../spdk-setup.sh is the original single-version, configure-at-start entrypoint (changing versions = container restart = all connections dropped). Superseded by the long-lived target; kept for reference.

README.md

SPDK NVMe/TCP target — long-lived, multi-version