nvmf: targeted per-version connect (rd.nvmf.nqn) instead of connect-all
Hosts connected with 'nvme connect-all', which attaches EVERY subsystem in the discovery log. Each host then mounts only its own by-UUID verity namespaces but leaves a live controller on every other served version's subsystem. Observed via spdk-release list: more controllers than workstations, each box on every version (1/3/3/1/3 across 5 versions for ~3 boxes). Beyond wasted connections (O(hosts x versions)), it defeated the rolling-update drain/reap: no version ever reached 0 connections while any host was up, so spdk-loader's auto-reap could never fire and old versions accumulated without bound. Fix — targeted connect to this build's subsystem only: - extract-artifacts.sh bakes the per-version SUBNQN (already in verity.json .subnqn, matched by the serving side) into the signed UKI cmdline as rd.nvmf.nqn=. NQN is version identity, not per-host state, and the cmdline is already per-version + immutable-per-signed-UKI, so invariant #1 holds. - nvmf-connect.sh (test path) uses 'nvme connect -n <nqn>'; legacy UKIs without rd.nvmf.nqn fall back to connect-all with a warning so they still boot. - dot1x-macsec-up.sh (secure path) gains an explicit step-6 targeted connect over macsec0 (after it's addressed, per invariant #2), replacing the delegation to the dracut nvmf module's discovery connect-all. Now each host = 1 controller on its own subsystem; versions drain to 0 when their hosts migrate, so auto-reap works and connections scale O(hosts). Requires a rebuild + re-sign (cmdline change, invariant #6). BENCH-VERIFY both paths; confirm the dracut nvmf module isn't also connect-all-ing on the secure path. CLAUDE.md invariant #2 + TODO #6 updated.
This commit is contained in:
@@ -73,9 +73,17 @@ ROOT_DATA_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_DATA_UUID}"
|
||||
ROOT_HASH_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_HASH_UUID}"
|
||||
# Per-version subsystem NQN (distinct NQN per release) — lets a host attach to
|
||||
# exactly one version's subsystem and gives clean per-release drain/allow-listing
|
||||
# on the long-lived target. Written to verity.json for the serving side; NOT in
|
||||
# the cmdline (host still discovers + connects; the by-UUID verity refs select
|
||||
# what mounts). An explicit SUBNQN env wins.
|
||||
# on the long-lived target. Written to verity.json for the serving side AND baked
|
||||
# into the UKI cmdline as rd.nvmf.nqn= (below), so the host does a TARGETED
|
||||
# `nvme connect -n <nqn>` instead of a discovery `connect-all`. connect-all
|
||||
# attached EVERY served version's subsystem to EVERY host, which inflated target
|
||||
# controllers to O(hosts x versions) and — worse — kept every old version pinned
|
||||
# above zero connections so the per-version drain/reap gate could never fire
|
||||
# (spdk-loader never reaps; see TODO #6). The NQN is version identity, not per-host
|
||||
# state, and the cmdline is already per-version + immutable-per-signed-UKI
|
||||
# (roothash, by-id UUIDs), so baking it does NOT violate invariant #1. It MUST
|
||||
# match verity.json .subnqn (same var here) so host and target agree. Explicit
|
||||
# SUBNQN env wins.
|
||||
SUBNQN_BASE=${SUBNQN_BASE:-nqn.2026-06.org.example.fleet:rootfs}
|
||||
SUBNQN=${SUBNQN:-${SUBNQN_BASE}-${IMAGE_VERSION}}
|
||||
|
||||
@@ -226,6 +234,7 @@ printf '%s' \
|
||||
"rd.dot1x.nvindex=0x01800100 rd.dot1x.handle=0x81000100 rd.dot1x.timeout=60 " \
|
||||
"rd.fleet.enrol_url=${ENROL_URL} " \
|
||||
"rd.nvmf.discover=tcp,${NVMET_TRADDR},4420 " \
|
||||
"rd.nvmf.nqn=${SUBNQN} " \
|
||||
"root=/dev/mapper/root rootfstype=erofs rootflags=ro " \
|
||||
"systemd.verity=yes roothash=${ROOT_HASH} " \
|
||||
"systemd.verity_root_data=${ROOT_DATA_DEV} " \
|
||||
|
||||
Reference in New Issue
Block a user