From 0068a1591791fe49de59c374f3b33d3b41395fcd Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 10 Jul 2026 06:08:48 +0000 Subject: [PATCH] docs/TODO #14: on-host agent telemetry BENCH-VERIFIED on real hardware MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the 2026-07-10 end-to-end verification (Dell 820 comet-beacon-54, real TPM): TPM-identity mTLS -> enrolled -> streaming health + receiving desired-state, manager logs "agent connected". Add the "deploying the agent plane" gotcha list (placeholder CA, root+intermediate client CA, distroless PKI perms, split-horizon DNS, out-of-band inventory seeding) — all automated by commit a89c3d2 — so the next deploy doesn't rediscover the debug chain. --- docs/TODO.md | 42 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 5 deletions(-) diff --git a/docs/TODO.md b/docs/TODO.md index d91776b..eb7d908 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1422,11 +1422,43 @@ the design forks resolved: dracut/initramfs changes (post-switch-root only). **STILL TO BUILD (deferred):** the **reconcile actuators** — apply kiosk_url, reboot_at/REBOOT, drain — wired to be received but not acted on; land after the telemetry half is bench-verified. - **BENCH-VERIFY:** on an enrolled booted host, `systemctl status fleet-agent` - active; manager logs `agent connected: `; SPA Monitor shows the host - online + health; InfluxDB `fleet_health` accrues points. Requires the host - already enrolled (broker approve birthed its inventory row) — else the manager - returns `PermissionDenied` (the correct "not enrolled yet" signal). + **BENCH-VERIFIED END-TO-END ON REAL HARDWARE 2026-07-10** (Dell 820 + `comet-beacon-54`, real TPM, image `fleet-manager-254ec79`): the agent loads its + reused TPM identity (serial derived on-device via go-tpm), completes mTLS to the + manager, streams health, AND receives desired-state (logged, not applied). Manager + logs `agent connected: fed27077… (comet-beacon-54)`; the host stays connected (no + reconnect loop). This closes the TPM-identity path (802.1X = NVMe hostnqn = agent + mTLS, "one identity, three stages") on hardware. + + **Deploying the agent plane — gotchas hit + FIXED in commit `a89c3d2` (don't + rediscover):** the serving-side bring-up cost a long debug chain, each now + automated: + 1. **Host CA was the placeholder.** `/etc/dot1x/radius-ca.pem` shipped a + non-CA placeholder → `client TLS: no certificates parsed from fleet CA`. + FIX: `fleet_ca` build var bakes the real step-ca root (CI secret → file → + `-var fleet_ca`; default = placeholder, lab-only). The agent's `FLEET_AGENT_CA` + points here; only the ROOT is needed host-side (the manager sends its chain). + 2. **`tls: certificate required`.** step-ca signs leaf certs with an + INTERMEDIATE and enrolment stores the leaf-only cert in NV, so a root-only + client CA makes the manager advertise only the root in its CertificateRequest → + the Go client sends NO cert. FIX: `server-pki.sh` installs the agent CLIENT CA + (`FLEET_MGR_AGENT_CLIENT_CA`) as **root+intermediate**; `bootstrap.sh` exports + `intermediate_ca.crt`. + 3. **Distroless perms.** The nonroot manager (uid 65532) can't read `pki/agent/*` + if created `0600` by the host user → `open …/server.crt: permission denied`. + `chown 65532` the agent PKI (as with the fleet-data volume). + 4. **Wrong manager IP / split-horizon DNS.** `diskless-server.bitasylum.org` + resolved to the serving host's OUTBOUND IP (`no route to host` from the fleet + LAN). Bench workaround: `FLEET_AGENT_MANAGER=:9600` + + `FLEET_AGENT_SERVER_NAME=` (dial the IP, verify the cert's SAN). + PROD FIX (not code): fleet dnsmasq `--address=//` so the + baked default resolves on the host VLAN. + 5. **Host not in the manager's inventory** → `PermissionDenied` (surfaced as a + bare `EOF` on the agent before the fix). A host enrolled OUT OF BAND (bench + broker / migrated fleet) has no inventory row. FIX: `fleet-manager + register-host -name ` (writes through the real store to + `FLEET_MGR_INVENTORY`). The manager caches inventory in memory — any edit needs + a `--force-recreate`. Also: the agent now logs the real gRPC status, not `EOF`. - **STILL TO BUILD:** (b) **per-host boot-version assignment** — `target_version` is stored + pushed but does not change what a host *boots*; that needs dnsmasq cohort steering + multi-version UKI serving (**#6**). Kiosk URL / reboot / drain