Hosts connected with 'nvme connect-all', which attaches EVERY subsystem in the
discovery log. Each host then mounts only its own by-UUID verity namespaces but
leaves a live controller on every other served version's subsystem. Observed via
spdk-release list: more controllers than workstations, each box on every version
(1/3/3/1/3 across 5 versions for ~3 boxes). Beyond wasted connections
(O(hosts x versions)), it defeated the rolling-update drain/reap: no version ever
reached 0 connections while any host was up, so spdk-loader's auto-reap could
never fire and old versions accumulated without bound.
Fix — targeted connect to this build's subsystem only:
- extract-artifacts.sh bakes the per-version SUBNQN (already in verity.json
.subnqn, matched by the serving side) into the signed UKI cmdline as
rd.nvmf.nqn=. NQN is version identity, not per-host state, and the cmdline is
already per-version + immutable-per-signed-UKI, so invariant #1 holds.
- nvmf-connect.sh (test path) uses 'nvme connect -n <nqn>'; legacy UKIs without
rd.nvmf.nqn fall back to connect-all with a warning so they still boot.
- dot1x-macsec-up.sh (secure path) gains an explicit step-6 targeted connect over
macsec0 (after it's addressed, per invariant #2), replacing the delegation to
the dracut nvmf module's discovery connect-all.
Now each host = 1 controller on its own subsystem; versions drain to 0 when their
hosts migrate, so auto-reap works and connections scale O(hosts). Requires a
rebuild + re-sign (cmdline change, invariant #6). BENCH-VERIFY both paths;
confirm the dracut nvmf module isn't also connect-all-ing on the secure path.
CLAUDE.md invariant #2 + TODO #6 updated.
The Dell 820 kept "finding a cert" and bypassing enrolment despite repeated BIOS
TPM clears. tpm2_nvreadpublic revealed 0x01800001 on that hardware is a FIRMWARE
platform index (platformcreate|policydelete|writelocked, 70 bytes — not our
cert; siblings at 0x01800003/4 + 0x01810000). platformcreate survives TPM2_Clear
and policydelete needs the platform's secret policy, so it's permanent and
unusable by us — the blind check false-positived on it, and enrolment could
never write there anyway. Same class of bug as the EK-handle collision.
Move the NV cert index default 0x01800001 -> 0x01800100 (free on the 820, owner
range, overridable via rd.dot1x.nvindex): cmdline (extract-artifacts.sh),
fleet-enrol -nvindex, the dracut hooks, bench scripts, CLAUDE.md invariant #1,
and design/SSP docs. The signing-key handle (0x81000100) is unchanged.
Harden WriteNVCert: refuse a platformcreate index with a clear "pick a free
owner index via rd.dot1x.nvindex" error instead of a confusing undefine
failure. Regression simtest/TestWriteNVCertRefusesPlatformIndex; NV roundtrip +
ReadNVCert tests moved to 0x01800100. All go tests + packer validate pass.
Bench (Proxmox vTPM, host cygnus-cruiser-99) confirmed via tpm2_readpublic that
0x81010001 — the handle our design used for the fleet signing key — is the
TCG-standard RSA Endorsement Key slot: the object there is the swtpm-provisioned
EK (restricted|decrypt|adminwithpolicy, RSA-2048/AES128CFB, canonical EK
authPolicy 837197…, EK cert at NV 0x1c00002). Most firmware/swtpm TPMs occupy
that slot, so EnsureKey could never create our key there — it errored reading
the EK as ECC and enrolment died before the CSR/POST.
Move the fleet signing key to 0x81000100, a non-reserved owner persistent
handle (clear of EK 0x81010001/2 and SRK 0x81000001/2). Threaded through:
- fleet-enrol -handle default + new rd.dot1x.handle= cmdline knob (baked by
extract-artifacts.sh, overridable via SMBIOS)
- files/etc/dot1x/wpa_supplicant-dot1x.conf private_key handle
- the dracut hook, bench tpm-provision.sh / tpm-selftest.sh
- CLAUDE.md invariant #1 + design/SSP docs + TODO #11
NV index 0x01800001 (owner range) is unchanged.
EnsureKey hardening: refuse to evict any restricted-decrypt object (EK/SRK) as
a safety net — never clobber the TPM endorsement/storage identity (also
protects the future EK/AK attestation path). Still self-heals a genuine stale
signing key. New regression TestEnsureKeyRefusesToEvictProtectedKey; the
stale-key regression and handle constants moved to 0x81000100.
Design note records the IDevID (0x81020001) option for the later EK/AK
attestation work. Follow-up items logged: content-aware NV presence check
(verify the cert chains to the fleet CA, so leftover/foreign NV can't suppress
enrolment) and a de-enrol/TPM-reset procedure.
getfattr --only-values emits the NUL-terminated security.selinux value;
bash's $(...) then warns "ignored null byte in input". Pipe through
tr -d '\0' to silence the cosmetic warning. The label assertion above is
unaffected.
Builds the host side of TPM identity enrolment — the "inline single-UKI"
vehicle from docs/tpm-enrolment-design.md. A production (secure) UKI now
auto-enrols a fresh host with no client cert, then reboots into the normal
secure path.
New dracut hook files/dracut/95dot1x-macsec/tpm-enrol.sh (initqueue/settled
prio 25, BEFORE dot1x-macsec-up at 30):
- no-op when a cert is already present in NV 0x01800001 (normal boot) or
under rd.dot1x=0 (test path has no trustworthy identity to enrol);
- on EMPTY NV: generate the signing key in the TPM @0x81010001 (openssl tpm2
provider builds the CSR; key never leaves the TPM), pre-auth DHCP on the
PHYSICAL NIC (nmcli re-manages en*, overriding the secure-mode unmanaged
drop-in), POST CSR+context to rd.fleet.enrol_url, show the broker
fingerprint on Plymouth, poll /status, write the issued leaf cert (DER) to
NV, and reboot to converge.
- Empty-NV and the secure path are mutually exclusive, so this never reorders
the load-bearing boot sequence (invariant #2).
Wiring:
- module-setup.sh: require+install the tpm2 key-gen/NV tools, curl, awk, and
the openssl tpm2 PROVIDER (ossl-modules/tpm2.so, asserted present — tpm2-tss
alone can't sign with a handle: key); install the hook.
- diskless.pkr.hcl: install tpm2-openssl + curl; assert hook/tools/provider in
the initramfs (invariant #9). New enrol_url var threaded to extract-artifacts.
- extract-artifacts.sh: bake rd.fleet.enrol_url into the cmdline.
- fleet.script: enrol-active/enrol-wait/enrol-done banner; fingerprint on the
message line (public — states only, invariant #1).
Shell JSON build/parse/cert-extract pipeline (no jq in the initramfs) verified
end-to-end against a live broker. BENCH-VERIFY list in docs/TODO.md #11 (tpm2
provider key-ref on real/swtpm, empty-NV detect, pre-auth DHCP override, Plymouth
render, NV write + reboot convergence, prod HTTPS).
Audiobuzz