Permanent SPDK trial
This commit is contained in:
Audiobuzz
@@ -0,0 +1,86 @@
|
||||
# SPDK NVMe/TCP target — long-lived, multi-version
|
||||
|
||||
This is the fleet's **root disk**: a userspace SPDK `nvmf_tgt` exporting each
|
||||
rootfs release as **two read-only NVMe/TCP namespaces** (erofs DATA + dm-verity
|
||||
HASH — the distro's systemd-veritysetup can't do the in-file hash-offset layout,
|
||||
so they're separate namespaces with distinct UUIDs).
|
||||
|
||||
## Why the target is long-lived
|
||||
|
||||
`nvmf_tgt` sits in the **live data path of every running host's root fs**.
|
||||
Restarting the process tears down **every** NVMe/TCP association at once
|
||||
(fleet-wide, not version-scoped) — a host whose root link drops past
|
||||
`ctrl_loss_tmo` gets root I/O errors and must reboot. So **the process must never
|
||||
restart for a routine change.**
|
||||
|
||||
A host changes root version **only by rebooting** (no live swap). Therefore:
|
||||
**version transitions are reboots; SPDK changes are additive/online; the target
|
||||
process stays up.** Routine rollouts never restart SPDK.
|
||||
|
||||
## How it works
|
||||
|
||||
| Piece | Role |
|
||||
|---|---|
|
||||
| [`spdk-target.sh`](spdk-target.sh) | **Persistent entrypoint.** Starts `nvmf_tgt` once, creates the TCP transport + discovery listener, then reconciles every staged release. Never restarted to change versions. On (re)start it re-adds everything under `./releases/`, so a serving-host reboot restores all served versions. |
|
||||
| [`spdk-release.sh`](spdk-release.sh) | **In-container control** (JSON-RPC). `add`/`remove`/`list`/`drain-status`/`reconcile` a version **live**. One **subsystem per version** (distinct NQN) so adding/removing one never disturbs the subsystem other hosts are attached to. Removal is **drain-gated**. |
|
||||
| [`spdk-ctl.sh`](spdk-ctl.sh) | **Host-side ops.** Stages a release's artifacts from its OCI image into `./releases/<ver>/` and drives the live target via `docker compose exec`. |
|
||||
| `./releases/<version>/` | Per-version `rootfs.erofs` + `rootfs.hash` + `verity.json`. `verity.json` is the **single source of truth** for that version's namespace UUIDs + subsystem NQN (written by `scripts/extract-artifacts.sh`), so the host's by-id verity refs and the target's namespace UUIDs always agree. |
|
||||
|
||||
Per-version namespace UUIDs are required: two versions served at once must have
|
||||
globally-distinct namespace UUIDs or they collide on the same
|
||||
`/dev/disk/by-id/nvme-uuid.<uuid>` symlink on the host. `extract-artifacts.sh`
|
||||
derives them deterministically (UUIDv5 of the version) and bakes them into both
|
||||
the immutable UKI cmdline and `verity.json`.
|
||||
|
||||
## Rolling update (add new, drain old, clean up)
|
||||
|
||||
```bash
|
||||
cd serving/spdk
|
||||
|
||||
./spdk-ctl.sh load 1.3.0 # serve 1.3.0 ALONGSIDE the running 1.2.0 (live, no restart)
|
||||
# ... bump the deployment manifest digest; reboot hosts → they come up on 1.3.0 ...
|
||||
./spdk-ctl.sh list # watch 1.2.0 CONNS drop toward 0
|
||||
# VERSION CONNS NQN
|
||||
# rootfs-1.3.0 7 nqn.2026-06.org.example.fleet:rootfs-1.3.0
|
||||
# rootfs-1.2.0 0 nqn.2026-06.org.example.fleet:rootfs-1.2.0
|
||||
./spdk-ctl.sh unload 1.2.0 # drain-gated: refuses while CONNS > 0
|
||||
```
|
||||
|
||||
`unload` **refuses to remove a version that still has connected controllers**
|
||||
(an assigned-but-offline host could still return to it). Drain those hosts
|
||||
first, or `./spdk-ctl.sh unload 1.2.0 --force` to **evict** them (they lose root
|
||||
I/O — only for a known-dead version).
|
||||
|
||||
In-container equivalents (what `spdk-ctl.sh` calls):
|
||||
|
||||
```bash
|
||||
docker compose exec nvmeof spdk-release list
|
||||
docker compose exec nvmeof spdk-release add 1.3.0
|
||||
docker compose exec nvmeof spdk-release drain-status 1.2.0
|
||||
docker compose exec nvmeof spdk-release remove 1.2.0 [--force]
|
||||
```
|
||||
|
||||
## Capacity & availability
|
||||
|
||||
- Each concurrently-served version costs target memory/hugepages + a file-backed
|
||||
bdev. Plan for N live versions (current + previous + canary) during long
|
||||
rollouts; raise `HUGEPAGES` accordingly.
|
||||
- The target is still a **single-node SPOF**. Even with online add/remove, a node
|
||||
failure or a planned `nvmf_tgt` restart drops every connection. The structural
|
||||
fix is **HA + NVMe-oF multipath (ANA)**: ≥2 targets serving byte-identical,
|
||||
same-UUID, read-only namespaces (replication is coherent precisely because the
|
||||
root is read-only + content-addressed). Cheap host-side mitigation: generous
|
||||
`ctrl_loss_tmo`/`reconnect_delay`. See `docs/TODO.md` #6/#7.
|
||||
- There is **no zero-impact single-node restart** — only drain/failover.
|
||||
|
||||
## Bootstrap (simple single-version)
|
||||
|
||||
`docker compose up -d` still works with one version: the OCI image volume at
|
||||
`/artifacts` is **seeded** into `./releases/${IMAGE_VERSION}/` and served
|
||||
automatically. Use `spdk-ctl.sh load` to add more versions on top.
|
||||
|
||||
## Legacy
|
||||
|
||||
[`../spdk-setup.sh`](../spdk-setup.sh) is the original **single-version,
|
||||
configure-at-start** entrypoint (changing versions = container restart = all
|
||||
connections dropped). Superseded by the long-lived target; kept for reference.
|
||||
Reference in New Issue
Block a user