nvmf: targeted per-version connect (rd.nvmf.nqn) instead of connect-all
Hosts connected with 'nvme connect-all', which attaches EVERY subsystem in the discovery log. Each host then mounts only its own by-UUID verity namespaces but leaves a live controller on every other served version's subsystem. Observed via spdk-release list: more controllers than workstations, each box on every version (1/3/3/1/3 across 5 versions for ~3 boxes). Beyond wasted connections (O(hosts x versions)), it defeated the rolling-update drain/reap: no version ever reached 0 connections while any host was up, so spdk-loader's auto-reap could never fire and old versions accumulated without bound. Fix — targeted connect to this build's subsystem only: - extract-artifacts.sh bakes the per-version SUBNQN (already in verity.json .subnqn, matched by the serving side) into the signed UKI cmdline as rd.nvmf.nqn=. NQN is version identity, not per-host state, and the cmdline is already per-version + immutable-per-signed-UKI, so invariant #1 holds. - nvmf-connect.sh (test path) uses 'nvme connect -n <nqn>'; legacy UKIs without rd.nvmf.nqn fall back to connect-all with a warning so they still boot. - dot1x-macsec-up.sh (secure path) gains an explicit step-6 targeted connect over macsec0 (after it's addressed, per invariant #2), replacing the delegation to the dracut nvmf module's discovery connect-all. Now each host = 1 controller on its own subsystem; versions drain to 0 when their hosts migrate, so auto-reap works and connections scale O(hosts). Requires a rebuild + re-sign (cmdline change, invariant #6). BENCH-VERIFY both paths; confirm the dracut nvmf module isn't also connect-all-ing on the secure path. CLAUDE.md invariant #2 + TODO #6 updated.
This commit is contained in:
@@ -675,6 +675,33 @@ must be built around **never restarting the process that holds live connections.
|
||||
the current UKI), so auto-reap is safe; removal is still drain-gated +
|
||||
grace-debounced. The loader + target share the RPC socket on a named volume
|
||||
(`nvmf_tgt -r`); the loader reuses the SPDK image (has python3 + rpc.py).
|
||||
- **BENCH FINDING 2026-07-03 (FIXED) — hosts did `connect-all`, so NO version
|
||||
ever drained → auto-reap could never fire.** Observed: `spdk-release list`
|
||||
showed more controllers than workstations, with every box appearing on *every*
|
||||
served version's subsystem (e.g. 1/3/3/1/3 across 5 versions for ~3 boxes).
|
||||
Root cause: the initramfs connected with **`nvme connect-all`** (test path
|
||||
`nvmf-connect.sh`; secure path delegated to the dracut nvmf module's discovery
|
||||
autoconnect), which attaches EVERY subsystem in the discovery log — the host
|
||||
then mounts only its own by-UUID verity namespaces but leaves a live controller
|
||||
on all the others. So each running host pinned every version above 0 conns, and
|
||||
`spdk-loader`'s "auto-reap once drained (0 conns)" **could never trigger** while
|
||||
the fleet was up — the served-version list grew without bound. This defeats the
|
||||
whole drain/reap model, not just cosmetics. **Fix:** targeted connect. The
|
||||
per-version subsystem NQN (`SUBNQN`, already in `verity.json .subnqn`) is now
|
||||
also baked into the signed UKI cmdline as **`rd.nvmf.nqn=`** (`extract-artifacts.sh`),
|
||||
and both connect paths use **`nvme connect -n <nqn>`** — `nvmf-connect.sh` (test)
|
||||
and a new explicit step 6 in `dot1x-macsec-up.sh` (secure, replacing the module
|
||||
delegation; runs after macsec0 is addressed per invariant #2). Legacy UKIs
|
||||
without `rd.nvmf.nqn` fall back to `connect-all` (with a warning) so they still
|
||||
boot. Now each host = 1 controller on its own subsystem → versions drain to 0
|
||||
when their hosts move off → auto-reap works; connections scale O(hosts) not
|
||||
O(hosts × versions). Requires a rebuild + re-sign (cmdline change, invariant #6).
|
||||
**BENCH-VERIFY:** (a) test path — after rebuild, `spdk-release list` shows each
|
||||
box on ONLY its version; old versions reap after hosts migrate; (b) secure path
|
||||
— the new step-6 connect brings up the NVMe root over macsec0 (this path was not
|
||||
yet bench-confirmed end-to-end); (c) confirm the dracut nvmf module isn't ALSO
|
||||
connect-all-ing on the secure path (if it is, suppress it — the targeted connect
|
||||
is meant to be the sole connector).
|
||||
- **BENCH FINDING 2026-07-03 (FIXED) — the target WAS still recreated on an
|
||||
`IMAGE_VERSION` bump.** Despite the above, the `nvmeof` (spdk-target) service
|
||||
still carried a versioned `type: image` bootstrap mount
|
||||
|
||||
Reference in New Issue
Block a user