docs/TODO #14: on-host agent telemetry BENCH-VERIFIED on real hardware

Record the 2026-07-10 end-to-end verification (Dell 820 comet-beacon-54, real
TPM): TPM-identity mTLS -> enrolled -> streaming health + receiving desired-state,
manager logs "agent connected". Add the "deploying the agent plane" gotcha list
(placeholder CA, root+intermediate client CA, distroless PKI perms, split-horizon
DNS, out-of-band inventory seeding) — all automated by commit a89c3d2 — so the
next deploy doesn't rediscover the debug chain.

docs/TODO.md

@@ -1422,11 +1422,43 @@ the design forks resolved:
   dracut/initramfs changes (post-switch-root only). **STILL TO BUILD (deferred):**
   the **reconcile actuators** — apply kiosk_url, reboot_at/REBOOT, drain — wired to
   be received but not acted on; land after the telemetry half is bench-verified.
-  **BENCH-VERIFY:** on an enrolled booted host, `systemctl status fleet-agent`
+  **BENCH-VERIFIED END-TO-END ON REAL HARDWARE 2026-07-10** (Dell 820
-  active; manager logs `agent connected: <serial>`; SPA Monitor shows the host
+  `comet-beacon-54`, real TPM, image `fleet-manager-254ec79`): the agent loads its
-  online + health; InfluxDB `fleet_health` accrues points. Requires the host
+  reused TPM identity (serial derived on-device via go-tpm), completes mTLS to the
-  already enrolled (broker approve birthed its inventory row) — else the manager
+  manager, streams health, AND receives desired-state (logged, not applied). Manager
-  returns `PermissionDenied` (the correct "not enrolled yet" signal).
+  logs `agent connected: fed27077… (comet-beacon-54)`; the host stays connected (no
+  reconnect loop). This closes the TPM-identity path (802.1X = NVMe hostnqn = agent
+  mTLS, "one identity, three stages") on hardware.
+
+  **Deploying the agent plane — gotchas hit + FIXED in commit `a89c3d2` (don't
+  rediscover):** the serving-side bring-up cost a long debug chain, each now
+  automated:
+  1. **Host CA was the placeholder.** `/etc/dot1x/radius-ca.pem` shipped a
+     non-CA placeholder → `client TLS: no certificates parsed from fleet CA`.
+     FIX: `fleet_ca` build var bakes the real step-ca root (CI secret → file →
+     `-var fleet_ca`; default = placeholder, lab-only). The agent's `FLEET_AGENT_CA`
+     points here; only the ROOT is needed host-side (the manager sends its chain).
+  2. **`tls: certificate required`.** step-ca signs leaf certs with an
+     INTERMEDIATE and enrolment stores the leaf-only cert in NV, so a root-only
+     client CA makes the manager advertise only the root in its CertificateRequest →
+     the Go client sends NO cert. FIX: `server-pki.sh` installs the agent CLIENT CA
+     (`FLEET_MGR_AGENT_CLIENT_CA`) as **root+intermediate**; `bootstrap.sh` exports
+     `intermediate_ca.crt`.
+  3. **Distroless perms.** The nonroot manager (uid 65532) can't read `pki/agent/*`
+     if created `0600` by the host user → `open …/server.crt: permission denied`.
+     `chown 65532` the agent PKI (as with the fleet-data volume).
+  4. **Wrong manager IP / split-horizon DNS.** `diskless-server.bitasylum.org`
+     resolved to the serving host's OUTBOUND IP (`no route to host` from the fleet
+     LAN). Bench workaround: `FLEET_AGENT_MANAGER=<fleet-lan-ip>:9600` +
+     `FLEET_AGENT_SERVER_NAME=<dns-name>` (dial the IP, verify the cert's SAN).
+     PROD FIX (not code): fleet dnsmasq `--address=/<name>/<fleet-lan-ip>` so the
+     baked default resolves on the host VLAN.
+  5. **Host not in the manager's inventory** → `PermissionDenied` (surfaced as a
+     bare `EOF` on the agent before the fix). A host enrolled OUT OF BAND (bench
+     broker / migrated fleet) has no inventory row. FIX: `fleet-manager
+     register-host <serial> -name <starship>` (writes through the real store to
+     `FLEET_MGR_INVENTORY`). The manager caches inventory in memory — any edit needs
+     a `--force-recreate`. Also: the agent now logs the real gRPC status, not `EOF`.
 - **STILL TO BUILD:** (b) **per-host boot-version assignment** — `target_version` is
   stored + pushed but does not change what a host *boots*; that needs dnsmasq
   cohort steering + multi-version UKI serving (**#6**). Kiosk URL / reboot / drain