docs: reflect chrony/uptime metrics, agent-cert renewer, and the chrony-access findings

Recent work added telemetry fields and a cert path not yet in the docs: - files/agent/README.md: add uptime to the health list; document that chrony is read from /var/log/chrony/tracking.log (not the command socket) and needs BOTH the chrony supplementary group (DAC) and the fleet_agent_chronylog SELinux allow (TE), plus a reachable lab NTP (123/udp firewall); note the chrony group on the unit row. - serving/fleet-manager/README.md: add uptime + chrony detail to the health list; note the cert-auto agent-cert-renewer sidecar + GetCertificate hot-reload (certreload.go) in the bring-up and checkpoint 7. - docs/TODO.md #14: probe list gains uptime + chrony-from-tracking.log; new dated block records uptime/reboot-pending, the 4-layer chrony-access chase (all silent-failing; inventory.json is the isolator), and the agent-plane cert auto-renewal, all bench-verified 2026-07-10. - docs/fleet-management-design.md: health record gains uptime + chrony source/stratum/offset detail.
2026-07-10 22:45:00 +00:00
parent a46e5edf69
commit 3284a2713b
4 changed files with 62 additions and 15 deletions
@@ -1412,8 +1412,9 @@ the design forks resolved:
  the host **TPM identity** for mTLS (signing key at `0x81000100` via
  `tpmkey.LoadSigner` — a read-only copy of the enrol `tpmkey`; cert from NV
  `0x01800100`), opens the long-lived `Session` stream, sends `Hello` + periodic
-  `HealthSnapshot` (load/cpu/temp/mem/tmpfs/nvme/auth/selinux/chrony/kiosk probed
-  read-only from `/proc`+`/sys` + hostapi facts), and — **telemetry-only v1** —
+  `HealthSnapshot` (load/cpu/temp/mem/tmpfs/**uptime**/nvme/auth/selinux/kiosk from
+  `/proc`+`/sys` + hostapi facts, plus **chrony** clock health — state/source/
+  stratum/offset — read from `/var/log/chrony/tracking.log`), and — **telemetry-only v1** —
  **receives `DesiredState`/`Command` but only LOGS them** (zero host mutation).
  Static `CGO_ENABLED=0` binary built out-of-VM in CI (like enrol/hostapi), staged
  by the `2b-agent` provisioner (`enable_agent`, default on), fapolicyd-trusted,
@@ -1464,6 +1465,32 @@ the design forks resolved:
     register-host <serial> -name <starship>` (writes through the real store to
     `FLEET_MGR_INVENTORY`). The manager caches inventory in memory — any edit needs
     a `--force-recreate`. Also: the agent now logs the real gRPC status, not `EOF`.
+
+  **Metrics + cert auto-renewal added 2026-07-10 (`fleet-manager` branch):**
+  - **uptime + reboot-pending**: the agent sends `uptime_seconds`; the manager/UI
+    derive "reboot pending in X / rebooted" from boot_time (= last_seen − uptime) vs
+    `reboot_at`, all in the manager clock frame (no host↔manager clock sync needed).
+    Monitor metric graphs auto-refresh (10 s poll).
+  - **chrony clock health** (sync state + NTP source/stratum/offset). The agent reads
+    `/var/log/chrony/tracking.log` (image bakes `log tracking`), NOT chronyd's command
+    socket (owner-write-only unix socket; STIG `cmdport 0` closes the UDP port).
+    **Reaching that log from the hardened DynamicUser agent took a 4-layer chase, every
+    layer silent-failing** (memory `fleet-agent-hardened-probe-access`): (i) lab NTP
+    reachable — a `123/udp` rule on the serving-host firewall (host-networked ntp
+    container; same trap as audit `514`); (ii) DAC — `chrony` supplementary group on the
+    unit to traverse `/var/log/chrony` (0750); (iii) SELinux TE —
+    `files/stig/fleet-agent-chronylog.te` allow for `chronyd_var_log_t` (permissive
+    masks this); (iv) the MANAGER image must be ≥ the chrony commit — one with
+    uptime-but-not-chrony silently drops the fields (manager & workstation are SEPARATE
+    images/deploys). Isolator: read the manager `inventory.json` off the `fleet-data`
+    volume via a throwaway busybox (distroless manager, no shell). BENCH-VERIFIED on
+    `comet-beacon-54` 2026-07-10.
+  - **agent-plane server-cert auto-renewal**: the `agent-cert-renewer` compose sidecar
+    (profile `cert-auto`) issues + `step ca renew --daemon` renews the cert from a
+    co-located step-ca; the manager hot-reloads via `GetCertificate` (`certreload.go`),
+    no restart. Replaces the manual `server-pki.sh` + `docker exec … step` dance and
+    the 90-day re-mint. The server cert SANs MUST include `manager.fleet.lab` (the
+    fleet-DNS name the agent dials) — a missing SAN fails the handshake.
 - **STILL TO BUILD:** (b) **per-host boot-version assignment** — `target_version` is
  stored + pushed but does not change what a host *boots*; that needs dnsmasq
  cohort steering + multi-version UKI serving (**#6**). Kiosk URL / reboot / drain
@@ -125,8 +125,9 @@ Reported facts:

 - **Health:** CPU/load, GPU act-freq, temps, **tmpfs overlay usage** (the
  RAM-exhaustion → NVMe-keepalive risk — the single most important kiosk health
-  signal), NVMe-root link state, 802.1X/MACsec `Authorized`, SELinux enforcing,
-  chrony sync, `kiosk.service` active.
+  signal), host **uptime** (server derives reboot-pending vs `reboot_at`),
+  NVMe-root link state, 802.1X/MACsec `Authorized`, SELinux enforcing, chrony
+  clock health (sync state + NTP source/stratum/offset), `kiosk.service` active.
 - **Version:** running image version (`/etc/fleet-image`), rootfs digest, kernel.
 - **Usage:** kiosk URL, session uptime, **Chromium reload/crash count** (from the
  `kiosk-session` restart loop — the exact signal behind the GPU-bug diagnosis),
@@ -186,8 +187,9 @@ UKI).
 Host (by cert-serial / hostnqn)
  ├─ identity:   cert serial, hostnqn, MAC, starship-name (dnsmasq)
  ├─ version:    image_version, rootfs digest, kernel, desired_version
-  ├─ health:     cpu/load, gpu_freq, temps, tmpfs_used%, nvme_root, auth/macsec,
-  │              selinux, chrony, kiosk_active, last_seen
+  ├─ health:     cpu/load, gpu_freq, temps, tmpfs_used%, uptime, nvme_root,
+  │              auth/macsec, selinux, chrony (sync + source/stratum/offset),
+  │              kiosk_active, last_seen
  ├─ usage:      kiosk_url, session_uptime, chromium_crashes, input_active, display
  └─ desired:    target_version, kiosk_url, reboot_at, drain   (server-authoritative)
 ```
@@ -22,9 +22,12 @@ build VM** by CI (deps proxy-fetched, `go.sum`-pinned) and file-provisioned in.
 3. **Sends `Hello`** (image version, kernel, kiosk URL, root hash, starship name —
   the fleet-generic facts, pulled from `fleet-hostapi` on `127.0.0.1:9099`).
 4. **Sends a `HealthSnapshot` every `FLEET_AGENT_INTERVAL`** — load/cpu/temp/mem/
-   tmpfs, nvme-root, 802.1X-macsec, selinux, chrony, kiosk-active — probed
-   read-only from `/proc`+`/sys` (each probe degrades gracefully; a failure never
-   kills the stream).
+   tmpfs, **uptime**, nvme-root, 802.1X-macsec, selinux, kiosk-active — probed
+   read-only from `/proc`+`/sys`, plus **chrony** clock health (sync state +
+   source/stratum/offset relative to the NTP source; see note). Each probe
+   degrades gracefully; a failure never kills the stream. `uptime` lets the
+   manager derive **reboot-pending** (boot_time = last_seen − uptime vs the
+   requested `reboot_at`, all in the manager's clock frame).
 5. **Receives `DesiredState`/`Command` and LOGS them.** This build does **not**
   act on them. The reconcile actuators (apply kiosk URL, reboot, drain) are a
   deliberate follow-up (see `docs/TODO.md` #14) — the receive path is wired now so
@@ -33,6 +36,17 @@ build VM** by CI (deps proxy-fetched, `go.sum`-pinned) and file-provisioned in.
 A dropped stream is retried with jittered backoff; the manager treats stream loss
 as the host going **offline**.

+> **Note — the chrony probe reads a file, not the command socket.** chronyd's unix
+> command socket is owner-write-only and STIG sets `cmdport 0` (no UDP command
+> port), so the sandboxed agent can't query it. Instead the image enables
+> `log tracking`, and the agent reads the last line of
+> `/var/log/chrony/tracking.log`. That read needs **two** access layers, both wired
+> in: the **`chrony` supplementary group** (DAC — `/var/log/chrony` is `0750`) and
+> a minimal **SELinux allow** for the `chronyd_var_log_t` label
+> (`files/stig/fleet-agent-chronylog.te`). The lab NTP server must also be
+> reachable (a `123/udp` host-firewall rule on the serving host) or the log only
+> carries `unsynced` lines. See `docs/stig-deviations.md` #16.
+
 ## Prerequisite: the host must already be enrolled

 The manager rejects the stream (`PermissionDenied`) until an inventory record
@@ -56,7 +70,7 @@ identity, mirror `rd.dot1x.*`), `FLEET_AGENT_INTERVAL`, `FLEET_AGENT_HOSTAPI`.
 | `snapshot.go` | read-only `/proc`+`/sys` health probes + hostapi facts fetch |
 | `tpmkey/` | **copy** of `files/enrol/tpmkey` + read-only `LoadSigner` (see its header) |
 | `proto/` | **copy** of the generated `fleet.agent.v1` package (one `.proto` contract, two peers) |
-| `fleet-agent.service` | hardened unit (tss group for the TPM, INET egress) |
+| `fleet-agent.service` | hardened unit (`tss` group for the TPM, `chrony` group to read chrony's tracking.log, INET egress) |

 ## Build / test

@@ -69,7 +69,7 @@ Host (cert_serial)
  ├─ binding  : asset_tag, switch_port, location        (recorded at approval)
  ├─ enrolled : approver, at, fingerprint
  ├─ version  : image_version, rootfs_digest, kernel     (agent-reported)
-  ├─ health   : load, cpu, temp, tmpfs%, nvme_root, auth, selinux, chrony, kiosk  (latest snapshot; series in InfluxDB)
+  ├─ health   : load, cpu, temp, tmpfs%, uptime, nvme_root, auth, selinux, chrony (state+source/stratum/offset), kiosk  (latest snapshot; series in InfluxDB)
  ├─ usage    : kiosk_url, session_uptime, chromium_crashes, display
  └─ desired  : target_version, kiosk_url, reboot_at, drain   (SERVER-AUTHORITATIVE)
 ```
@@ -141,7 +141,9 @@ inventory for offline delivery. Metrics writing is behind a `MetricsSink`
 Full bring-up (PKI, Let's Encrypt DNS-01, `register-admin`, compose) is in
 [`DEPLOY.md`](DEPLOY.md). Config is layered: **`defaults.env`** (committed) +
 **`.env`** (overrides/secrets), via `COMPOSE_ENV_FILES=defaults.env,.env`. In
-short: set that + write your secrets to `.env` → `./server-pki.sh` +
+short: set that + write your secrets to `.env` → `./server-pki.sh` (or, where
+step-ca is co-located, the `cert-auto` **`agent-cert-renewer`** sidecar that
+issues + auto-renews the agent-plane server cert and hot-reloads it — DEPLOY §2a) +
 `../enrol-broker/admin-pki.sh … fleet-manager` → `docker compose up -d influxdb`
 → `docker compose run --rm fleet-manager register-admin <you>` → `docker compose
 up -d fleet-manager traefik`. InfluxDB is private (published to loopback only);
@@ -187,9 +189,11 @@ curl -s localhost:9500/api/hosts | jq '.hosts[] | {name:.identity.starship_name,
 - **Checkpoint 7 (done):** deployment — three-stage `Dockerfile` (npm→dist→
  `go build -tags embedspa`→distroless), `docker-compose.yml` (private InfluxDB +
  host-networked manager), `defaults.env`, `server-pki.sh` (agent-plane fleet-CA
-  server cert), and [`DEPLOY.md`](DEPLOY.md) (PKI, LE DNS-01, `register-admin`
-  bootstrap). Design docs updated (manager marked built; resolved transport/auth/
-  metrics forks; per-host-versioning gap recorded).
+  server cert) **+ the `cert-auto` `agent-cert-renewer` sidecar** (auto-issues +
+  auto-renews that cert from a co-located step-ca; the manager hot-reloads it via
+  `GetCertificate` in `certreload.go` — no restart), and [`DEPLOY.md`](DEPLOY.md)
+  (PKI, LE DNS-01, `register-admin` bootstrap). Design docs updated (manager marked
+  built; resolved transport/auth/metrics forks; per-host-versioning gap recorded).
 - **Checkpoint 8 (done):** hardening + ingress — CI image build
  (`fleet-manager-image.yaml`, pulled by tag), `backup.sh`; a **behind-proxy admin
  mode** (passkey over plaintext behind a TLS-terminating proxy) fronted by a