Permanent SPDK trial

docs/TODO.md

@@ -645,15 +645,40 @@ must be built around **never restarting the process that holds live connections.
 - **Planned SPDK restarts only via drain/failover** (multipath fail-over, or a
   maintenance window with generous `ctrl_loss_tmo` so hosts recover not die). There
   is **no zero-impact single-node SPDK restart.**
-- **Implementation gap (current lab):** SPDK runs as a container with
-  `spdk-setup.sh` as entrypoint + the artifact image mounted → effectively
-  **single-version, configured at container start**, so changing versions =
-  container restart = drop all connections. The work here is a **long-lived target
-  + an RPC-driven control path** ("add release / remove release / drain status")
-  decoupled from container lifecycle, with per-version subsystems.
+- **[x] Long-lived target + online release control — IMPLEMENTED 2026-06-27.**
+  The old gap (SPDK ran `spdk-setup.sh` as entrypoint, single-version,
+  configured-at-start → changing versions = container restart = drop ALL
+  connections) is closed. Now in `serving/spdk/`:
+  - **`spdk-target.sh`** — persistent entrypoint: starts `nvmf_tgt` ONCE
+    (release-agnostic), creates the TCP transport + discovery listener, then
+    reconciles every staged release. Never restarted to change versions; on
+    (re)start re-adds everything under `./releases/` (a serving-host reboot
+    restores all served versions).
+  - **`spdk-release.sh`** — JSON-RPC control: `add`/`remove`/`list`/
+    `drain-status`/`reconcile`, **one subsystem per version** (distinct NQN, so
+    adding/removing one never disturbs others' connections), **drain-gated
+    removal** (refuses while a version has connected controllers unless
+    `--force`).
+  - **`spdk-ctl.sh`** — host ops: `load`/`unload`/`list`; stages a release's
+    artifacts from its OCI image into `./releases/<ver>/` and drives the live
+    target. Rolling update = `load` new → reboot hosts → `unload` old once
+    drained.
+  - **Per-version namespace UUIDs** (`scripts/extract-artifacts.sh`): derived
+    deterministically (UUIDv5 of `image_version`), baked into the UKI cmdline +
+    `verity.json` (now also carries `subnqn`); the serving side reads them back
+    from each release's `verity.json`. Required so two versions don't collide on
+    the same `/dev/disk/by-id/nvme-uuid.<uuid>` host symlink.
+  - **Bootstrap unchanged:** `docker compose up` with one version still works —
+    the `/artifacts` image volume is seeded into `./releases/${IMAGE_VERSION}`.
+  - **BENCH-VERIFY:** (a) `load` a 2nd version while a host is connected to the
+    1st → 1st's connection survives (no AER/reset on the running host); (b)
+    `unload` drain-gate actually blocks while CONNS>0; (c) serving-host reboot
+    reconciles all staged versions; (d) per-version UUID build is reproducible
+    (two builds, same version → identical `data_ns_uuid`/`hash_ns_uuid` +
+    `root_hash`). See `serving/spdk/README.md`.
 - **Capacity:** each concurrently-served version costs target memory/hugepages +
   file-backed bdev overhead — plan for N live versions (current + previous +
-  canary) during long rollouts.
+  canary) during long rollouts (raise `HUGEPAGES`).
 - **Attach gating + discovery:** with several subsystems advertised, the dracut
   connect + per-subsystem hostNQN allow-list must ensure a vX host attaches **only**
   vX's namespace (the by-UUID verity ref protects *what mounts*, but gate the