113 lines
6.1 KiB
Markdown
113 lines
6.1 KiB
Markdown
# SPDK NVMe/TCP target — long-lived, multi-version
|
|
|
|
This is the fleet's **root disk**: a userspace SPDK `nvmf_tgt` exporting each
|
|
rootfs release as **two read-only NVMe/TCP namespaces** (erofs DATA + dm-verity
|
|
HASH — the distro's systemd-veritysetup can't do the in-file hash-offset layout,
|
|
so they're separate namespaces with distinct UUIDs).
|
|
|
|
## Why the target is long-lived
|
|
|
|
`nvmf_tgt` sits in the **live data path of every running host's root fs**.
|
|
Restarting the process tears down **every** NVMe/TCP association at once
|
|
(fleet-wide, not version-scoped) — a host whose root link drops past
|
|
`ctrl_loss_tmo` gets root I/O errors and must reboot. So **the process must never
|
|
restart for a routine change.**
|
|
|
|
A host changes root version **only by rebooting** (no live swap). Therefore:
|
|
**version transitions are reboots; SPDK changes are additive/online; the target
|
|
process stays up.** Routine rollouts never restart SPDK.
|
|
|
|
## How it works
|
|
|
|
| Piece | Role |
|
|
|---|---|
|
|
| [`spdk-target.sh`](spdk-target.sh) | **Persistent entrypoint.** Starts `nvmf_tgt` once, creates the TCP transport + discovery listener, then reconciles every staged release. Never restarted to change versions. On (re)start it re-adds everything under `./releases/`, so a serving-host reboot restores all served versions. |
|
|
| [`spdk-release.sh`](spdk-release.sh) | **In-container control** (JSON-RPC). `add`/`remove`/`list`/`drain-status`/`reconcile` a version **live**. One **subsystem per version** (distinct NQN) so adding/removing one never disturbs the subsystem other hosts are attached to. Removal is **drain-gated**. |
|
|
| [`spdk-ctl.sh`](spdk-ctl.sh) | **Host-side ops** (manual / cleanup). Stages a release's artifacts from its OCI image into `./releases/<ver>/` and drives the live target via `docker compose exec`. |
|
|
| [`spdk-loader.sh`](spdk-loader.sh) | **Declarative rolling updates** (compose service `spdk-loader`, the "init container that stays up"). On `compose up` after an `IMAGE_VERSION` bump it stages + adds the new rootfs online, then auto-reaps old versions once they drain. |
|
|
| `./releases/<version>/` | Per-version `rootfs.erofs` + `rootfs.hash` + `verity.json`. `verity.json` is the **single source of truth** for that version's namespace UUIDs + subsystem NQN (written by `scripts/extract-artifacts.sh`), so the host's by-id verity refs and the target's namespace UUIDs always agree. |
|
|
|
|
Per-version namespace UUIDs are required: two versions served at once must have
|
|
globally-distinct namespace UUIDs or they collide on the same
|
|
`/dev/disk/by-id/nvme-uuid.<uuid>` symlink on the host. `extract-artifacts.sh`
|
|
derives them deterministically (UUIDv5 of the version) and bakes them into both
|
|
the immutable UKI cmdline and `verity.json`.
|
|
|
|
## Rolling update — the declarative way (recommended)
|
|
|
|
Bump one tag and bring the stack up:
|
|
|
|
```bash
|
|
# in serving/.env
|
|
IMAGE_VERSION=1.3.0
|
|
|
|
docker compose up -d
|
|
```
|
|
|
|
Compose recreates the **`spdk-loader`** and the **http/dnsmasq** UKI servers (their
|
|
image volumes changed), but **not** the `nvmeof` target (its definition is
|
|
unchanged) — so no connections drop. The loader adds 1.3.0 to the live target,
|
|
serves the new UKI to all hosts, and then **auto-reaps** 1.2.0 once every host
|
|
has rebooted onto 1.3.0 and 1.2.0 has been drained for `REAP_GRACE_SECONDS`.
|
|
Rollback = set `IMAGE_VERSION` back + `up -d`. Set `REAP_OLD=false` to keep old
|
|
versions and reap them yourself.
|
|
|
|
Why auto-reap is safe here: there is a **single served UKI**, so a powered-off
|
|
host that returns boots the *current* UKI — nothing can come back to an old
|
|
rootfs once its live connections (CONNS) hit 0. Removal is still drain-gated and
|
|
debounced by the grace window (ride out transient nvme-tcp reconnects — keep
|
|
`REAP_GRACE_SECONDS` above the host `ctrl_loss_tmo`).
|
|
|
|
## Rolling update — the manual way (`spdk-ctl.sh`)
|
|
|
|
```bash
|
|
cd serving/spdk
|
|
|
|
./spdk-ctl.sh load 1.3.0 # serve 1.3.0 ALONGSIDE the running 1.2.0 (live, no restart)
|
|
# ... bump the deployment manifest digest; reboot hosts → they come up on 1.3.0 ...
|
|
./spdk-ctl.sh list # watch 1.2.0 CONNS drop toward 0
|
|
# VERSION CONNS NQN
|
|
# rootfs-1.3.0 7 nqn.2026-06.org.example.fleet:rootfs-1.3.0
|
|
# rootfs-1.2.0 0 nqn.2026-06.org.example.fleet:rootfs-1.2.0
|
|
./spdk-ctl.sh unload 1.2.0 # drain-gated: refuses while CONNS > 0
|
|
```
|
|
|
|
`unload` **refuses to remove a version that still has connected controllers**
|
|
(an assigned-but-offline host could still return to it). Drain those hosts
|
|
first, or `./spdk-ctl.sh unload 1.2.0 --force` to **evict** them (they lose root
|
|
I/O — only for a known-dead version).
|
|
|
|
In-container equivalents (what `spdk-ctl.sh` calls):
|
|
|
|
```bash
|
|
docker compose exec nvmeof spdk-release list
|
|
docker compose exec nvmeof spdk-release add 1.3.0
|
|
docker compose exec nvmeof spdk-release drain-status 1.2.0
|
|
docker compose exec nvmeof spdk-release remove 1.2.0 [--force]
|
|
```
|
|
|
|
## Capacity & availability
|
|
|
|
- Each concurrently-served version costs target memory/hugepages + a file-backed
|
|
bdev. Plan for N live versions (current + previous + canary) during long
|
|
rollouts; raise `HUGEPAGES` accordingly.
|
|
- The target is still a **single-node SPOF**. Even with online add/remove, a node
|
|
failure or a planned `nvmf_tgt` restart drops every connection. The structural
|
|
fix is **HA + NVMe-oF multipath (ANA)**: ≥2 targets serving byte-identical,
|
|
same-UUID, read-only namespaces (replication is coherent precisely because the
|
|
root is read-only + content-addressed). Cheap host-side mitigation: generous
|
|
`ctrl_loss_tmo`/`reconnect_delay`. See `docs/TODO.md` #6/#7.
|
|
- There is **no zero-impact single-node restart** — only drain/failover.
|
|
|
|
## Bootstrap (simple single-version)
|
|
|
|
`docker compose up -d` still works with one version: the OCI image volume at
|
|
`/artifacts` is **seeded** into `./releases/${IMAGE_VERSION}/` and served
|
|
automatically. Use `spdk-ctl.sh load` to add more versions on top.
|
|
|
|
## Legacy
|
|
|
|
[`../spdk-setup.sh`](../spdk-setup.sh) is the original **single-version,
|
|
configure-at-start** entrypoint (changing versions = container restart = all
|
|
connections dropped). Superseded by the long-lived target; kept for reference.
|