Audiobuzz
0bfe872828
Hosts connected with 'nvme connect-all', which attaches EVERY subsystem in the discovery log. Each host then mounts only its own by-UUID verity namespaces but leaves a live controller on every other served version's subsystem. Observed via spdk-release list: more controllers than workstations, each box on every version (1/3/3/1/3 across 5 versions for ~3 boxes). Beyond wasted connections (O(hosts x versions)), it defeated the rolling-update drain/reap: no version ever reached 0 connections while any host was up, so spdk-loader's auto-reap could never fire and old versions accumulated without bound. Fix — targeted connect to this build's subsystem only: - extract-artifacts.sh bakes the per-version SUBNQN (already in verity.json .subnqn, matched by the serving side) into the signed UKI cmdline as rd.nvmf.nqn=. NQN is version identity, not per-host state, and the cmdline is already per-version + immutable-per-signed-UKI, so invariant #1 holds. - nvmf-connect.sh (test path) uses 'nvme connect -n <nqn>'; legacy UKIs without rd.nvmf.nqn fall back to connect-all with a warning so they still boot. - dot1x-macsec-up.sh (secure path) gains an explicit step-6 targeted connect over macsec0 (after it's addressed, per invariant #2), replacing the delegation to the dracut nvmf module's discovery connect-all. Now each host = 1 controller on its own subsystem; versions drain to 0 when their hosts migrate, so auto-reap works and connections scale O(hosts). Requires a rebuild + re-sign (cmdline change, invariant #6). BENCH-VERIFY both paths; confirm the dracut nvmf module isn't also connect-all-ing on the secure path. CLAUDE.md invariant #2 + TODO #6 updated.
89 lines
4.2 KiB
Bash
89 lines
4.2 KiB
Bash
#!/bin/bash
|
|
# nvmf-connect — ExecStart for nvmf-connect.service (initramfs, TEST MODE).
|
|
#
|
|
# Why a systemd unit and not a dracut hook: this initramfs is systemd-driven and
|
|
# does NOT source dracut initqueue/online hooks when the ip=dhcp link comes up
|
|
# (verified on the bench — the online hook installs but never fires: no log, no
|
|
# connect). dracut's nvmf module also registers no autoconnect, and a job placed
|
|
# on the main initqueue is never re-executed once the loop blocks on events. A
|
|
# oneshot unit ordered Before=initrd-root-device.target, with its own
|
|
# wait-for-network + connect loop, deterministically connects the NVMe-oF root
|
|
# namespace so the by-id device appears and systemd-veritysetup can map
|
|
# /dev/mapper/root.
|
|
#
|
|
# rd.dot1x=0 (open-network test) ONLY — the unit's ConditionKernelCommandLine
|
|
# already gates this, but re-check so the script is safe to run directly. The
|
|
# secure path connects over macsec0 from dot1x-macsec-up.sh.
|
|
type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh
|
|
|
|
[ "$(getarg rd.dot1x=)" = "0" ] || exit 0
|
|
[ -e /run/dot1x/.nvmf-connected ] && exit 0
|
|
|
|
_disc=$(getarg rd.nvmf.discover=) # e.g. tcp,192.168.50.10,4420
|
|
[ -n "$_disc" ] || { echo "nvmf-connect: rd.nvmf.discover not set" >&2; exit 1; }
|
|
_trtype=${_disc%%,*}
|
|
_rest=${_disc#*,}
|
|
_traddr=${_rest%%,*}
|
|
_trsvcid=${_rest##*,}
|
|
|
|
# Targeted connect: rd.nvmf.nqn pins THIS build's subsystem (baked per-version
|
|
# into the signed cmdline by extract-artifacts.sh, matching verity.json .subnqn
|
|
# on the target). This replaces a discovery `connect-all`, which attached EVERY
|
|
# served version's subsystem to every host — inflating target controllers and
|
|
# keeping old versions pinned above zero connections so they never drained/reaped
|
|
# (see TODO #6). If rd.nvmf.nqn is absent (a legacy UKI), fall back to connect-all
|
|
# so the host still boots — with a warning.
|
|
_nqn=$(getarg rd.nvmf.nqn=)
|
|
|
|
# nvme_tcp is usually auto-loaded; ensure it so /dev/nvme-fabrics exists.
|
|
modprobe nvme_tcp 2>/dev/null
|
|
|
|
# Best-effort boot-splash status for the 'fleet' Plymouth theme (never fatal). By
|
|
# the time this runs the ip=dhcp link is up, so mark Network done and NVMe active.
|
|
ply() { plymouth update --status="$1" 2>/dev/null || true; }
|
|
ply net-done
|
|
ply nvme-active
|
|
|
|
# Keep-alive / reconnect tuning for the ROOT device. These persist across
|
|
# switch-root (the in-kernel nvme-tcp controller and its opts survive).
|
|
# -k 30 keep-alive timeout (s). The 5s default means a KA every 2.5s; under
|
|
# kiosk CPU/memory pressure the nvme-tcp keep-alive work can be delayed
|
|
# past 5s, the target sees no KA and disconnects ("keep alive timeout").
|
|
# 30s gives a 15s emit interval and far more slack.
|
|
# -l -1 ctrl-loss-tmo = retry reconnection forever. This is the rootfs: the
|
|
# 600s default makes the host GIVE UP after a blip, after which root I/O
|
|
# fails permanently and the box wedges — strictly worse than retrying.
|
|
# -c 10 wait 10s between reconnect attempts.
|
|
_kato=30
|
|
_ctrl_loss_tmo=-1
|
|
_reconnect_delay=10
|
|
|
|
# Networking (ip=dhcp) is brought up in parallel, so poll-and-connect until a
|
|
# fabrics namespace appears or we time out. Both connect forms are idempotent
|
|
# (re-connecting an already-attached subsystem just errors, harmlessly).
|
|
[ -n "$_nqn" ] || echo "nvmf-connect: rd.nvmf.nqn not set — falling back to connect-all (legacy UKI; attaches every served version)" >&2
|
|
_i=0
|
|
while [ "$_i" -lt 60 ]; do
|
|
if [ -n "$_nqn" ]; then
|
|
# TARGETED: attach only this build's subsystem (its data + hash namespaces).
|
|
nvme connect -t "$_trtype" -a "$_traddr" -s "$_trsvcid" -n "$_nqn" \
|
|
-k "$_kato" -l "$_ctrl_loss_tmo" -c "$_reconnect_delay" 2>/dev/null
|
|
else
|
|
nvme connect-all -t "$_trtype" -a "$_traddr" -s "$_trsvcid" \
|
|
-k "$_kato" -l "$_ctrl_loss_tmo" -c "$_reconnect_delay" 2>/dev/null
|
|
fi
|
|
for _ns in /dev/nvme[0-9]*n[0-9]*; do
|
|
if [ -b "$_ns" ]; then
|
|
mkdir -p /run/dot1x
|
|
: > /run/dot1x/.nvmf-connected
|
|
ply nvme-done
|
|
echo "nvmf-connect: connected ${_trtype}/${_traddr}/${_trsvcid} ($_ns)"
|
|
exit 0
|
|
fi
|
|
done
|
|
sleep 1
|
|
_i=$((_i + 1))
|
|
done
|
|
echo "nvmf-connect: timed out connecting ${_trtype}/${_traddr}/${_trsvcid}" >&2
|
|
exit 1
|