nvmf: targeted per-version connect (rd.nvmf.nqn) instead of connect-all

Hosts connected with 'nvme connect-all', which attaches EVERY subsystem in the
discovery log. Each host then mounts only its own by-UUID verity namespaces but
leaves a live controller on every other served version's subsystem. Observed via
spdk-release list: more controllers than workstations, each box on every version
(1/3/3/1/3 across 5 versions for ~3 boxes). Beyond wasted connections
(O(hosts x versions)), it defeated the rolling-update drain/reap: no version ever
reached 0 connections while any host was up, so spdk-loader's auto-reap could
never fire and old versions accumulated without bound.

Fix — targeted connect to this build's subsystem only:
- extract-artifacts.sh bakes the per-version SUBNQN (already in verity.json
  .subnqn, matched by the serving side) into the signed UKI cmdline as
  rd.nvmf.nqn=. NQN is version identity, not per-host state, and the cmdline is
  already per-version + immutable-per-signed-UKI, so invariant #1 holds.
- nvmf-connect.sh (test path) uses 'nvme connect -n <nqn>'; legacy UKIs without
  rd.nvmf.nqn fall back to connect-all with a warning so they still boot.
- dot1x-macsec-up.sh (secure path) gains an explicit step-6 targeted connect over
  macsec0 (after it's addressed, per invariant #2), replacing the delegation to
  the dracut nvmf module's discovery connect-all.

Now each host = 1 controller on its own subsystem; versions drain to 0 when their
hosts migrate, so auto-reap works and connections scale O(hosts). Requires a
rebuild + re-sign (cmdline change, invariant #6). BENCH-VERIFY both paths;
confirm the dracut nvmf module isn't also connect-all-ing on the secure path.
CLAUDE.md invariant #2 + TODO #6 updated.
This commit is contained in:
Audiobuzz
2026-07-03 06:14:08 +00:00
parent 81c10b6e7a
commit 0bfe872828
5 changed files with 107 additions and 10 deletions
+20 -3
View File
@@ -26,6 +26,15 @@ _rest=${_disc#*,}
_traddr=${_rest%%,*}
_trsvcid=${_rest##*,}
# Targeted connect: rd.nvmf.nqn pins THIS build's subsystem (baked per-version
# into the signed cmdline by extract-artifacts.sh, matching verity.json .subnqn
# on the target). This replaces a discovery `connect-all`, which attached EVERY
# served version's subsystem to every host — inflating target controllers and
# keeping old versions pinned above zero connections so they never drained/reaped
# (see TODO #6). If rd.nvmf.nqn is absent (a legacy UKI), fall back to connect-all
# so the host still boots — with a warning.
_nqn=$(getarg rd.nvmf.nqn=)
# nvme_tcp is usually auto-loaded; ensure it so /dev/nvme-fabrics exists.
modprobe nvme_tcp 2>/dev/null
@@ -50,11 +59,19 @@ _ctrl_loss_tmo=-1
_reconnect_delay=10
# Networking (ip=dhcp) is brought up in parallel, so poll-and-connect until a
# fabrics namespace appears or we time out. connect-all is idempotent.
# fabrics namespace appears or we time out. Both connect forms are idempotent
# (re-connecting an already-attached subsystem just errors, harmlessly).
[ -n "$_nqn" ] || echo "nvmf-connect: rd.nvmf.nqn not set — falling back to connect-all (legacy UKI; attaches every served version)" >&2
_i=0
while [ "$_i" -lt 60 ]; do
nvme connect-all -t "$_trtype" -a "$_traddr" -s "$_trsvcid" \
-k "$_kato" -l "$_ctrl_loss_tmo" -c "$_reconnect_delay" 2>/dev/null
if [ -n "$_nqn" ]; then
# TARGETED: attach only this build's subsystem (its data + hash namespaces).
nvme connect -t "$_trtype" -a "$_traddr" -s "$_trsvcid" -n "$_nqn" \
-k "$_kato" -l "$_ctrl_loss_tmo" -c "$_reconnect_delay" 2>/dev/null
else
nvme connect-all -t "$_trtype" -a "$_traddr" -s "$_trsvcid" \
-k "$_kato" -l "$_ctrl_loss_tmo" -c "$_reconnect_delay" 2>/dev/null
fi
for _ns in /dev/nvme[0-9]*n[0-9]*; do
if [ -b "$_ns" ]; then
mkdir -p /run/dot1x