Update SSP

docs/TODO.md

@@ -572,13 +572,67 @@ this item builds the rollout machinery around it.
   stay served); security rollback via dbx-revoke / TPM monotonic counter
   (design §6).
 - **Drain & retire** an old version only when SPDK/nvmet reports zero connected
-  controllers on it.
+  controllers on it. **Drain accuracy:** account for hosts powered off mid-rollout
+  — an assigned-but-offline host will reconnect to a version that may be gone, so
+  don't remove a version while any host that could return is still assigned to it.
 - **Availability**: run NVMe/TCP target replicas behind distinct discovery
   entries (design §7) — never serve the whole fleet from one unreplicated node,
   especially mid-rollout (N versions + reboot bursts).
 - Note: kiosk *content* changes are server-side (the website) and need no fleet
   update — only OS/UKI/rootfs changes require a reboot.
 
+### SPDK restart / availability is the load-bearing constraint (analysis 2026-06-25)
+
+SPDK sits in the **live data path of every running host's root fs**, so a process
+restart isn't like restarting a stateless service — it tears down **every**
+NVMe/TCP association at once (fleet-wide, NOT version-scoped). The rollout design
+must be built around **never restarting the process that holds live connections.**
+
+- **Core principle:** a host changes root version *only by rebooting* (no live
+  swap). So: **version transitions = reboots; SPDK changes = additive/online; the
+  process stays up.** Routine rollouts never restart SPDK.
+- **Online add/remove via JSON-RPC, no restart.** `nvmf_tgt` is fully
+  runtime-configurable (`bdev_aio_create`, `nvmf_create_subsystem`,
+  `nvmf_subsystem_add_ns`, `..._add_listener`). Add vY's namespaces live (existing
+  vX connections untouched); remove vX live once drained. **Per-version subsystem
+  (distinct NQN)** matters here: adding vY as a *new subsystem* never disturbs the
+  subsystem vX hosts are attached to — adding a *namespace* to a shared subsystem
+  fires a namespace-change AER + rescan at running hosts. Per-version subsystem
+  also gives clean per-release drain + per-release hostNQN allow-listing.
+- **What a connection loss does to a running host:** root is **read-only erofs +
+  tmpfs overlay**. tmpfs writes (RAM) + page-cached reads keep working; **cold
+  reads block**. The nvme-tcp host driver reconnects (`reconnect_delay`) until
+  `ctrl_loss_tmo`. **Fast SPDK restart** (same NQN/UUID/addr within the window) →
+  host **survives with a stall**; **long/changed/expired** → root I/O errors, host
+  needs a reboot ("keep alive timeout / no route to host"). Read-only +
+  content-addressed makes reconnect unusually clean (no write/reservation state).
+- **Cheap host-side mitigation:** tune `ctrl_loss_tmo` (even `-1` = retry forever)
+  + `reconnect_delay` at `nvme connect` time (dracut nvmf step). For a diskless box
+  "hang and recover" beats "die" — there's nothing better to fail to.
+- **Structural answer — HA + NVMe-oF multipath (ANA):** run ≥2 targets serving
+  **byte-identical, same-UUID** namespaces; hosts multipath across them, so one
+  target can be restarted/upgraded while the other serves (no root loss). The
+  **read-only, content-addressed root makes replication trivially coherent** (no
+  write-coherence problem) — this architecture is ideally suited to it. Wrinkle:
+  the **immutable UKI cmdline** points at one discovery address, so multipath needs
+  discovery to return multiple paths (or a DNS name resolving to several targets).
+- **Planned SPDK restarts only via drain/failover** (multipath fail-over, or a
+  maintenance window with generous `ctrl_loss_tmo` so hosts recover not die). There
+  is **no zero-impact single-node SPDK restart.**
+- **Implementation gap (current lab):** SPDK runs as a container with
+  `spdk-setup.sh` as entrypoint + the artifact image mounted → effectively
+  **single-version, configured at container start**, so changing versions =
+  container restart = drop all connections. The work here is a **long-lived target
+  + an RPC-driven control path** ("add release / remove release / drain status")
+  decoupled from container lifecycle, with per-version subsystems.
+- **Capacity:** each concurrently-served version costs target memory/hugepages +
+  file-backed bdev overhead — plan for N live versions (current + previous +
+  canary) during long rollouts.
+- **Attach gating + discovery:** with several subsystems advertised, the dracut
+  connect + per-subsystem hostNQN allow-list must ensure a vX host attaches **only**
+  vX's namespace (the by-UUID verity ref protects *what mounts*, but gate the
+  *attach* too); `rd.nvmf.discover` selection must be deterministic, not connect-all.
+
 ## [ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience)
 
 There is **no production-grade memory-safe NVMe-oF target** — the Rust NVMe work
@@ -607,7 +661,13 @@ export is read-only + single-namespace = inherently small attack surface.
 - **Network-isolate :4420** so only the boot VLAN reaches the target (compose
   with 802.1X/MACsec segmentation in prod).
 - **Replicas** behind distinct discovery entries (availability is the real risk
-  — it's the fleet's disk). Ties into #6 (rolling updates) availability.
+  — it's the fleet's disk). The target is a **fleet-wide SPOF in every running
+  host's root data path**; a process restart drops *all* NVMe/TCP associations at
+  once. Structural fix = **HA + NVMe-oF multipath (ANA)** over byte-identical,
+  same-UUID, read-only namespaces (replication is coherent precisely because the
+  root is read-only + content-addressed); cheap mitigation = generous host
+  `ctrl_loss_tmo`/`reconnect_delay`. See the **"SPDK restart / availability"**
+  analysis under #6 (rolling updates) — same constraint, fuller treatment.
 - Revisit a memory-safe target only if/when an off-the-shelf one matures
   (e.g. a future Rust-for-Linux `nvmet`).
 
@@ -631,6 +691,13 @@ smoke test.
     documented upgrade on the same CA. **BENCH-VERIFY** items flagged in the files:
     hostapd MACsec build, openssl tpm2 provider vs pkcs11 key ref, EAP-TLS version.
     PKI product decision (greenfield): start step-ca; OpenBao/EJBCA/Dogtag later.
+  - **First step (DONE 2026-06-22): `enrol/tpm-selftest.sh`** — non-destructive TPM
+    capability check (run on the booted open-port image, no `disable_dot1x` needed)
+    to de-risk the TPM half before the full path. Likely first finding: no openssl
+    TPM2 **provider** in the image (`tpm2-tss` ≠ the provider) — add `tpm2-openssl`.
+  - **Enrolment of the per-host identity** (key@`0x81010001` + cert@NV) is its own
+    design — see **#11** (TOFU + OOB fingerprint broker, pre-auth permit-set,
+    provisioning UKI). `tpm-provision.sh` is the manual bench stand-in for it.
 - **NVMe discovery by DNS:** `nvmet_traddr=nvme.fleet.lab` (CI), resolved at boot
   by the serving dnsmasq — drop the hardcoded `192.168.50.10`.
 - **Signing — SIGNED-UKI + SECURE-BOOT-ON PATH BENCH-VERIFIED on the Proxmox VM
@@ -645,7 +712,9 @@ smoke test.
     EFI attribute prefix in a bash variable (`ATTR=$'\x27\x00\x00\x00'`). Bash
     strings can't hold NULs → only `0x27` survived → efivarfs read a corrupt
     attribute u32 → every write failed with `EINVAL` ("dd: Invalid argument").
-    Now emitted directly with `printf` at write time. (Red herring on the way:
+    Now emitted directly with `printf` at write time. **Baked fix VERIFIED on the
+    820 (2026-06-22): `sudo fleet-sb-enroll` enrolled db→KEK→PK cleanly, no inline
+    workaround.** (Red herring on the way:
     Proxmox EFI disks default to `pre-enrolled-keys=1`, so `mokutil --db` shows
     the MS cert even in Setup Mode — looked like a non-empty-db rejection but was
     the NUL bug all along; it failed identically on a clean `pre-enrolled-keys=0`
@@ -657,12 +726,15 @@ smoke test.
     (`.gitea/workflows/dnsmasq-image.yaml`) — no longer built at deploy time, and
     carries no keys/binary. (Decision: separate `fleet-ipxe` image, not baked into
     dnsmasq, so the daemon image stays key-agnostic across SB environments.)
+  - **820 / PXE signed chain — BENCH-VERIFIED 2026-06-22.** Real hardware (HP
+    EliteBook 820 G4), Secure Boot **enforcing** (`mokutil --sb-state` = enabled,
+    Setup Mode no, db/KEK/PK enrolled): firmware verifies the signed `snponly.efi`
+    (fleet-ipxe image) → signed UKI and boots. Confirms the whole `fleet-ipxe`
+    signing pipeline + the PXE→iPXE→UKI chain under SB on real silicon (the VM only
+    covered the HTTP path, which has no iPXE stage).
   - **REMAINING for production:** (a) **secure path** — re-verify signed boot with
-    `disable_dot1x=false` (802.1X/MACsec), not just the test path; (b) **820 / PXE
-    path** — BENCH-VERIFY the signed `fleet-ipxe` chain on the 820 (the VM's HTTP
-    path has no iPXE stage, so the signed iPXE binary is still unverified on real
-    PXE hardware); (c) confirm signing is reproducible (verity hash stable across
-    two signed builds).
+    `disable_dot1x=false` (802.1X/MACsec), not just the test path; (b) confirm
+    signing is reproducible (verity hash stable across two signed builds).
 - **PXE-only iPXE fallback — BENCH-VERIFIED on the EliteBook 820 G4 (2026-06-20).**
   Firmware without UEFI HTTP boot chainloads `serving/tftp/snponly.efi` → renders
   `autoexec.ipxe` (ifopen + DHCP-retry + `imgargs <real cmdline>`) → UKI over HTTP
@@ -757,7 +829,9 @@ full ~17-min Packer build.
 Speed up bringing up new workstations: bake the **public** PK/KEK/db certs into an
 image, detect Secure Boot **Setup Mode** at boot, and auto-enroll them so a
 fresh/reset box adopts the fleet trust root without a manual firmware-setup visit.
-Pairs with #8 (signing) + [secure-boot-keys.md](secure-boot-keys.md).
+Pairs with #8 (signing) + [secure-boot-keys.md](secure-boot-keys.md). Shares the
+provisioning-UKI + isolated-environment vehicle with **#11** (which enrols the
+*network* identity); build them together.
 
 **Tooling gap (found 2026-06-20): `efitools` is NOT packaged for EL9** — no
 `efi-updatevar`/`efi-readvar`/`KeyTool`. Resolved for the OS-side path by writing
@@ -817,3 +891,70 @@ enrolls.
 reasonable for a controlled fleet; the highest-assurance path is attended/firmware
 or manufacturer pre-provisioning. Default recommendation: **separate provisioning
 UKI + isolated VLAN + read-back verify.**
+
+## [ ] 11. (DESIGN) TPM 802.1X identity enrolment (TOFU + OOB fingerprint)
+
+How a host gets its **802.1X EAP-TLS client identity** the first time — the key at
+TPM handle `0x81010001` + cert at NV `0x01800001` that invariant #1 says must come
+from enrolment, never the image. Distinct from #10 (that enrols *Secure Boot*
+PK/KEK/db); this enrols the *network identity*. They share the provisioning UKI +
+isolated-environment vehicle. Pairs with #8 + the CA decision (start **step-ca**,
+[serving/dot1x-bench/](../serving/dot1x-bench/); OpenBao/EJBCA/Dogtag later).
+
+**Chosen pattern — trust-on-first-use with out-of-band fingerprint approval:**
+on an empty TPM the host generates a key *in* the TPM, builds a CSR, submits it to
+an enrolment **broker**, and shows a **fingerprint** on the Plymouth splash; an
+admin visually cross-checks that fingerprint against the broker's pending-list and
+approves; the broker has step-ca sign the CSR and returns the cert; the host writes
+it to NV `0x01800001` and proceeds to authenticate. Sound, recognised pattern
+(SSH/WireGuard/Signal-style OOB verification); good fit for invariant #1.
+
+**Key design decisions (settled 2026-06-22):**
+
+- **The pre-auth permit-set already exists — enrolment rides in it.** The firmware
+  can't do 802.1X, so the UKI fetch (DHCP/DNS/TFTP/HTTP to the serving host) is
+  **always pre-auth**; the secure model therefore *requires* a pre-auth permit-set
+  (low-impact 802.1X / pre-auth dACL / boot→secure dynamic VLAN; RADIUS returns the
+  secure VLAN on Access-Accept). The enrolment endpoint is just **another entry in
+  that set** — expose it as an `/enrol/` location on the **existing serving nginx**,
+  reverse-proxied to the broker. No new network mechanism.
+- **The broker is the only new component.** step-ca signs but has no human-approval
+  queue: broker = `POST CSR + context → PENDING + fingerprint → admin approves →
+  step-ca signs → host polls → cert`. Exposed to *unauthenticated* devices by
+  design → harden: OOB-fingerprint gate (no auto-sign), rate-limit, time-boxed
+  pending, HTTPS (image trusts the serving CA), pre-auth permit scoped to only
+  these services + the serving host.
+- **Trigger on EMPTY-NV only — split the three events:** empty NV → enrol (human);
+  valid key + expired/near-expiry cert → **renew** automatically (no human, it
+  already holds a trusted key); auth fails with a present cert → **alert + retry,
+  do NOT re-enrol** (auto-reenrol on failure would mask revocation and churn keys
+  on a transient RADIUS outage). Re-enrol after the first time = explicit admin
+  action (clear/flag the host).
+- **Fingerprint: what it proves + entropy.** It proves "this CSR came from *this
+  physical device*" (approver reads the screen) — it does NOT prove TPM residency.
+  Pair it with **TPM attestation** (step-ca ACME `device-attest-01`, EK/AK) for the
+  residency proof: fingerprint = human authorises the device, attestation = the key
+  is really in a TPM (together → strong SSP R-01 story). Size for entropy — a short
+  hex fp is grindable; use a word-list/BIP39-style encoding (reads short, more
+  bits) + rate-limiting + a server nonce. Show device context (asset/port/location)
+  so the admin knows *what* they're approving.
+- **Pre-auth traffic has no MACsec (plaintext at L2).** Fine for enrol (CSR/cert
+  are public; OOB fingerprint gives authenticity) — and it's exactly **why the UKI
+  must be Secure-Boot-signed** (#8): integrity of the boot comes from the signature,
+  not the network. Enrolment design + Secure Boot are two halves of the same
+  "pre-auth path is hostile" assumption.
+- **Run it in a provisioning UKI/mode, not the production initramfs** — the boot
+  hook ordering is load-bearing (invariant #2); don't park it on a multi-minute
+  human-approval wait. Shares the #10 provisioning-UKI vehicle.
+
+**Bench caveat:** `hostapd` wired is strictly port-based (no pre-auth per-flow
+ACL) — before EAP success it passes nothing, so the bench can't reproduce
+"boot + enrol pre-auth on the same port." Model the pre-auth state with a separate
+provisioning bridge/VLAN on the rig; the in-place low-impact pre-auth permit is a
+real-switch feature to validate on the 820's switch.
+
+**First step DONE (2026-06-22):** `serving/dot1x-bench/enrol/tpm-selftest.sh` — a
+non-destructive TPM capability check (device, key persist, NV roundtrip, openssl
+TPM2 provider, CSR signed by the TPM key) to de-risk the TPM half before any of
+this. Expected gap: the image installs `tpm2-tss` but no openssl TPM2 **provider**
+(`tpm2-openssl`) — add it to the build once the self-test confirms.