nvmf: targeted per-version connect (rd.nvmf.nqn) instead of connect-all
Hosts connected with 'nvme connect-all', which attaches EVERY subsystem in the discovery log. Each host then mounts only its own by-UUID verity namespaces but leaves a live controller on every other served version's subsystem. Observed via spdk-release list: more controllers than workstations, each box on every version (1/3/3/1/3 across 5 versions for ~3 boxes). Beyond wasted connections (O(hosts x versions)), it defeated the rolling-update drain/reap: no version ever reached 0 connections while any host was up, so spdk-loader's auto-reap could never fire and old versions accumulated without bound. Fix — targeted connect to this build's subsystem only: - extract-artifacts.sh bakes the per-version SUBNQN (already in verity.json .subnqn, matched by the serving side) into the signed UKI cmdline as rd.nvmf.nqn=. NQN is version identity, not per-host state, and the cmdline is already per-version + immutable-per-signed-UKI, so invariant #1 holds. - nvmf-connect.sh (test path) uses 'nvme connect -n <nqn>'; legacy UKIs without rd.nvmf.nqn fall back to connect-all with a warning so they still boot. - dot1x-macsec-up.sh (secure path) gains an explicit step-6 targeted connect over macsec0 (after it's addressed, per invariant #2), replacing the delegation to the dracut nvmf module's discovery connect-all. Now each host = 1 controller on its own subsystem; versions drain to 0 when their hosts migrate, so auto-reap works and connections scale O(hosts). Requires a rebuild + re-sign (cmdline change, invariant #6). BENCH-VERIFY both paths; confirm the dracut nvmf module isn't also connect-all-ing on the secure path. CLAUDE.md invariant #2 + TODO #6 updated.
This commit is contained in:
Audiobuzz
@@ -115,9 +115,13 @@ instead of working around it.
|
||||
`rd.dot1x.nvindex=` / `fleet-enrol -handle -nvindex`.)
|
||||
1. **The hook ordering in `dot1x-macsec-up.sh` is load-bearing.**
|
||||
NV-cert extraction → supplicant start → wait for `Authorized` + macsec0 →
|
||||
DHCP **on macsec0** → hostnqn derivation. Never DHCP before auth
|
||||
DHCP **on macsec0** → hostnqn derivation → **targeted NVMe/TCP connect**
|
||||
(`nvme connect -n $rd.nvmf.nqn`, step 6). Never DHCP before auth
|
||||
completes (pre-auth leases are invalid after VLAN assignment); never
|
||||
connect NVMe before macsec0 is addressed.
|
||||
connect NVMe before macsec0 is addressed. The connect is **targeted to this
|
||||
version's subsystem** (`rd.nvmf.nqn`, baked per-version into the signed
|
||||
cmdline) — never a discovery `connect-all`, which attaches every served
|
||||
version and defeats the per-version drain/reap gate (see TODO #6).
|
||||
1. **The supplicant must survive switch-root.** It is launched with
|
||||
`exec -a @wpa_supplicant` (systemd root-storage-daemon convention) and
|
||||
its state lives under `/run`. Do not “clean up” the `@`, move state off
|
||||
|
||||
@@ -675,6 +675,33 @@ must be built around **never restarting the process that holds live connections.
|
||||
the current UKI), so auto-reap is safe; removal is still drain-gated +
|
||||
grace-debounced. The loader + target share the RPC socket on a named volume
|
||||
(`nvmf_tgt -r`); the loader reuses the SPDK image (has python3 + rpc.py).
|
||||
- **BENCH FINDING 2026-07-03 (FIXED) — hosts did `connect-all`, so NO version
|
||||
ever drained → auto-reap could never fire.** Observed: `spdk-release list`
|
||||
showed more controllers than workstations, with every box appearing on *every*
|
||||
served version's subsystem (e.g. 1/3/3/1/3 across 5 versions for ~3 boxes).
|
||||
Root cause: the initramfs connected with **`nvme connect-all`** (test path
|
||||
`nvmf-connect.sh`; secure path delegated to the dracut nvmf module's discovery
|
||||
autoconnect), which attaches EVERY subsystem in the discovery log — the host
|
||||
then mounts only its own by-UUID verity namespaces but leaves a live controller
|
||||
on all the others. So each running host pinned every version above 0 conns, and
|
||||
`spdk-loader`'s "auto-reap once drained (0 conns)" **could never trigger** while
|
||||
the fleet was up — the served-version list grew without bound. This defeats the
|
||||
whole drain/reap model, not just cosmetics. **Fix:** targeted connect. The
|
||||
per-version subsystem NQN (`SUBNQN`, already in `verity.json .subnqn`) is now
|
||||
also baked into the signed UKI cmdline as **`rd.nvmf.nqn=`** (`extract-artifacts.sh`),
|
||||
and both connect paths use **`nvme connect -n <nqn>`** — `nvmf-connect.sh` (test)
|
||||
and a new explicit step 6 in `dot1x-macsec-up.sh` (secure, replacing the module
|
||||
delegation; runs after macsec0 is addressed per invariant #2). Legacy UKIs
|
||||
without `rd.nvmf.nqn` fall back to `connect-all` (with a warning) so they still
|
||||
boot. Now each host = 1 controller on its own subsystem → versions drain to 0
|
||||
when their hosts move off → auto-reap works; connections scale O(hosts) not
|
||||
O(hosts × versions). Requires a rebuild + re-sign (cmdline change, invariant #6).
|
||||
**BENCH-VERIFY:** (a) test path — after rebuild, `spdk-release list` shows each
|
||||
box on ONLY its version; old versions reap after hosts migrate; (b) secure path
|
||||
— the new step-6 connect brings up the NVMe root over macsec0 (this path was not
|
||||
yet bench-confirmed end-to-end); (c) confirm the dracut nvmf module isn't ALSO
|
||||
connect-all-ing on the secure path (if it is, suppress it — the targeted connect
|
||||
is meant to be the sole connector).
|
||||
- **BENCH FINDING 2026-07-03 (FIXED) — the target WAS still recreated on an
|
||||
`IMAGE_VERSION` bump.** Despite the above, the `nvmeof` (spdk-target) service
|
||||
still carried a versioned `type: image` bootstrap mount
|
||||
|
||||
@@ -5,8 +5,9 @@
|
||||
# 2. start wpa_supplicant (argv[0]=@wpa_supplicant => survives switch-root)
|
||||
# 3. wait for EAP success + macsec0
|
||||
# 4. DHCP on macsec0 (the pre-auth lease, if any, is invalid post-auth/VLAN)
|
||||
# 5. derive hostnqn/hostid from the cert so dracut's nvmf module can
|
||||
# connect the NVMe/TCP root namespace over macsec0
|
||||
# 5. derive hostnqn/hostid from the cert
|
||||
# 6. TARGETED NVMe/TCP connect over macsec0 (rd.nvmf.nqn -> this version's
|
||||
# subsystem only, not a discovery connect-all of every served version)
|
||||
# Idempotent: guarded by a stamp file because initqueue may re-run hooks.
|
||||
|
||||
type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh
|
||||
@@ -159,5 +160,44 @@ if [ ! -s /etc/nvme/hostnqn ]; then
|
||||
info "dot1x-macsec: hostnqn $(cat /etc/nvme/hostnqn)"
|
||||
fi
|
||||
|
||||
# --- 6. connect the NVMe/TCP root over macsec0 (TARGETED to this version) ----
|
||||
# Explicit targeted connect. This hook used to only set /etc/nvme/hostnqn and
|
||||
# leave the connect to the dracut nvmf module's discovery mechanism — a
|
||||
# `connect-all` that attaches EVERY served version's subsystem to this host,
|
||||
# inflating target controllers to O(hosts x versions) and keeping old versions
|
||||
# pinned above zero connections so the per-version drain/reap gate never fires
|
||||
# (see TODO #6). rd.nvmf.nqn pins THIS build's subsystem (baked per-version into
|
||||
# the signed cmdline by extract-artifacts.sh, matching verity.json .subnqn on the
|
||||
# target); nvme uses the hostnqn/hostid written in step 5. Load-bearing order
|
||||
# (invariant #2): this runs AFTER macsec0 is addressed — never connect NVMe
|
||||
# before macsec0 has an IP. Keep-alive tuning matches the test path (-k 30 long
|
||||
# KATO, -l -1 retry forever, -c 10) so the root link survives kiosk CPU pressure
|
||||
# and blips. BENCH-VERIFY: the secure path is not yet confirmed end-to-end.
|
||||
_disc=$(getarg rd.nvmf.discover=)
|
||||
_nqn=$(getarg rd.nvmf.nqn=)
|
||||
if [ -n "$_disc" ] && [ -n "$_nqn" ] && [ ! -e /run/dot1x/.nvmf-connected ]; then
|
||||
_trtype=${_disc%%,*}; _rest=${_disc#*,}
|
||||
_traddr=${_rest%%,*}; _trsvcid=${_rest##*,}
|
||||
modprobe nvme_tcp 2>/dev/null
|
||||
ply nvme-active
|
||||
i=0
|
||||
while [ "$i" -lt "$TIMEOUT" ]; do
|
||||
nvme connect -t "$_trtype" -a "$_traddr" -s "$_trsvcid" -n "$_nqn" \
|
||||
-k 30 -l -1 -c 10 2>/dev/null
|
||||
for _ns in /dev/nvme[0-9]*n[0-9]*; do
|
||||
if [ -b "$_ns" ]; then
|
||||
: > /run/dot1x/.nvmf-connected
|
||||
ply nvme-done
|
||||
info "dot1x-macsec: NVMe root connected over macsec0 ($_ns)"
|
||||
break 2
|
||||
fi
|
||||
done
|
||||
sleep 1; i=$((i + 1))
|
||||
done
|
||||
[ -e /run/dot1x/.nvmf-connected ] || warn "dot1x-macsec: NVMe/TCP connect to $_nqn timed out after ${TIMEOUT}s"
|
||||
elif [ -z "$_nqn" ]; then
|
||||
warn "dot1x-macsec: rd.nvmf.nqn not set (legacy UKI) — leaving NVMe connect to the dracut nvmf module (connect-all)"
|
||||
fi
|
||||
|
||||
: > "$STAMP"
|
||||
exit 0
|
||||
|
||||
@@ -26,6 +26,15 @@ _rest=${_disc#*,}
|
||||
_traddr=${_rest%%,*}
|
||||
_trsvcid=${_rest##*,}
|
||||
|
||||
# Targeted connect: rd.nvmf.nqn pins THIS build's subsystem (baked per-version
|
||||
# into the signed cmdline by extract-artifacts.sh, matching verity.json .subnqn
|
||||
# on the target). This replaces a discovery `connect-all`, which attached EVERY
|
||||
# served version's subsystem to every host — inflating target controllers and
|
||||
# keeping old versions pinned above zero connections so they never drained/reaped
|
||||
# (see TODO #6). If rd.nvmf.nqn is absent (a legacy UKI), fall back to connect-all
|
||||
# so the host still boots — with a warning.
|
||||
_nqn=$(getarg rd.nvmf.nqn=)
|
||||
|
||||
# nvme_tcp is usually auto-loaded; ensure it so /dev/nvme-fabrics exists.
|
||||
modprobe nvme_tcp 2>/dev/null
|
||||
|
||||
@@ -50,11 +59,19 @@ _ctrl_loss_tmo=-1
|
||||
_reconnect_delay=10
|
||||
|
||||
# Networking (ip=dhcp) is brought up in parallel, so poll-and-connect until a
|
||||
# fabrics namespace appears or we time out. connect-all is idempotent.
|
||||
# fabrics namespace appears or we time out. Both connect forms are idempotent
|
||||
# (re-connecting an already-attached subsystem just errors, harmlessly).
|
||||
[ -n "$_nqn" ] || echo "nvmf-connect: rd.nvmf.nqn not set — falling back to connect-all (legacy UKI; attaches every served version)" >&2
|
||||
_i=0
|
||||
while [ "$_i" -lt 60 ]; do
|
||||
nvme connect-all -t "$_trtype" -a "$_traddr" -s "$_trsvcid" \
|
||||
-k "$_kato" -l "$_ctrl_loss_tmo" -c "$_reconnect_delay" 2>/dev/null
|
||||
if [ -n "$_nqn" ]; then
|
||||
# TARGETED: attach only this build's subsystem (its data + hash namespaces).
|
||||
nvme connect -t "$_trtype" -a "$_traddr" -s "$_trsvcid" -n "$_nqn" \
|
||||
-k "$_kato" -l "$_ctrl_loss_tmo" -c "$_reconnect_delay" 2>/dev/null
|
||||
else
|
||||
nvme connect-all -t "$_trtype" -a "$_traddr" -s "$_trsvcid" \
|
||||
-k "$_kato" -l "$_ctrl_loss_tmo" -c "$_reconnect_delay" 2>/dev/null
|
||||
fi
|
||||
for _ns in /dev/nvme[0-9]*n[0-9]*; do
|
||||
if [ -b "$_ns" ]; then
|
||||
mkdir -p /run/dot1x
|
||||
|
||||
@@ -73,9 +73,17 @@ ROOT_DATA_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_DATA_UUID}"
|
||||
ROOT_HASH_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_HASH_UUID}"
|
||||
# Per-version subsystem NQN (distinct NQN per release) — lets a host attach to
|
||||
# exactly one version's subsystem and gives clean per-release drain/allow-listing
|
||||
# on the long-lived target. Written to verity.json for the serving side; NOT in
|
||||
# the cmdline (host still discovers + connects; the by-UUID verity refs select
|
||||
# what mounts). An explicit SUBNQN env wins.
|
||||
# on the long-lived target. Written to verity.json for the serving side AND baked
|
||||
# into the UKI cmdline as rd.nvmf.nqn= (below), so the host does a TARGETED
|
||||
# `nvme connect -n <nqn>` instead of a discovery `connect-all`. connect-all
|
||||
# attached EVERY served version's subsystem to EVERY host, which inflated target
|
||||
# controllers to O(hosts x versions) and — worse — kept every old version pinned
|
||||
# above zero connections so the per-version drain/reap gate could never fire
|
||||
# (spdk-loader never reaps; see TODO #6). The NQN is version identity, not per-host
|
||||
# state, and the cmdline is already per-version + immutable-per-signed-UKI
|
||||
# (roothash, by-id UUIDs), so baking it does NOT violate invariant #1. It MUST
|
||||
# match verity.json .subnqn (same var here) so host and target agree. Explicit
|
||||
# SUBNQN env wins.
|
||||
SUBNQN_BASE=${SUBNQN_BASE:-nqn.2026-06.org.example.fleet:rootfs}
|
||||
SUBNQN=${SUBNQN:-${SUBNQN_BASE}-${IMAGE_VERSION}}
|
||||
|
||||
@@ -226,6 +234,7 @@ printf '%s' \
|
||||
"rd.dot1x.nvindex=0x01800100 rd.dot1x.handle=0x81000100 rd.dot1x.timeout=60 " \
|
||||
"rd.fleet.enrol_url=${ENROL_URL} " \
|
||||
"rd.nvmf.discover=tcp,${NVMET_TRADDR},4420 " \
|
||||
"rd.nvmf.nqn=${SUBNQN} " \
|
||||
"root=/dev/mapper/root rootfstype=erofs rootflags=ro " \
|
||||
"systemd.verity=yes roothash=${ROOT_HASH} " \
|
||||
"systemd.verity_root_data=${ROOT_DATA_DEV} " \
|
||||
|
||||
Reference in New Issue
Block a user