diff --git a/docs/TODO.md b/docs/TODO.md index 9eb1a99..aab7d0e 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1412,8 +1412,9 @@ the design forks resolved: the host **TPM identity** for mTLS (signing key at `0x81000100` via `tpmkey.LoadSigner` — a read-only copy of the enrol `tpmkey`; cert from NV `0x01800100`), opens the long-lived `Session` stream, sends `Hello` + periodic - `HealthSnapshot` (load/cpu/temp/mem/tmpfs/nvme/auth/selinux/chrony/kiosk probed - read-only from `/proc`+`/sys` + hostapi facts), and — **telemetry-only v1** — + `HealthSnapshot` (load/cpu/temp/mem/tmpfs/**uptime**/nvme/auth/selinux/kiosk from + `/proc`+`/sys` + hostapi facts, plus **chrony** clock health — state/source/ + stratum/offset — read from `/var/log/chrony/tracking.log`), and — **telemetry-only v1** — **receives `DesiredState`/`Command` but only LOGS them** (zero host mutation). Static `CGO_ENABLED=0` binary built out-of-VM in CI (like enrol/hostapi), staged by the `2b-agent` provisioner (`enable_agent`, default on), fapolicyd-trusted, @@ -1464,6 +1465,32 @@ the design forks resolved: register-host -name ` (writes through the real store to `FLEET_MGR_INVENTORY`). The manager caches inventory in memory — any edit needs a `--force-recreate`. Also: the agent now logs the real gRPC status, not `EOF`. + + **Metrics + cert auto-renewal added 2026-07-10 (`fleet-manager` branch):** + - **uptime + reboot-pending**: the agent sends `uptime_seconds`; the manager/UI + derive "reboot pending in X / rebooted" from boot_time (= last_seen − uptime) vs + `reboot_at`, all in the manager clock frame (no host↔manager clock sync needed). + Monitor metric graphs auto-refresh (10 s poll). + - **chrony clock health** (sync state + NTP source/stratum/offset). The agent reads + `/var/log/chrony/tracking.log` (image bakes `log tracking`), NOT chronyd's command + socket (owner-write-only unix socket; STIG `cmdport 0` closes the UDP port). + **Reaching that log from the hardened DynamicUser agent took a 4-layer chase, every + layer silent-failing** (memory `fleet-agent-hardened-probe-access`): (i) lab NTP + reachable — a `123/udp` rule on the serving-host firewall (host-networked ntp + container; same trap as audit `514`); (ii) DAC — `chrony` supplementary group on the + unit to traverse `/var/log/chrony` (0750); (iii) SELinux TE — + `files/stig/fleet-agent-chronylog.te` allow for `chronyd_var_log_t` (permissive + masks this); (iv) the MANAGER image must be ≥ the chrony commit — one with + uptime-but-not-chrony silently drops the fields (manager & workstation are SEPARATE + images/deploys). Isolator: read the manager `inventory.json` off the `fleet-data` + volume via a throwaway busybox (distroless manager, no shell). BENCH-VERIFIED on + `comet-beacon-54` 2026-07-10. + - **agent-plane server-cert auto-renewal**: the `agent-cert-renewer` compose sidecar + (profile `cert-auto`) issues + `step ca renew --daemon` renews the cert from a + co-located step-ca; the manager hot-reloads via `GetCertificate` (`certreload.go`), + no restart. Replaces the manual `server-pki.sh` + `docker exec … step` dance and + the 90-day re-mint. The server cert SANs MUST include `manager.fleet.lab` (the + fleet-DNS name the agent dials) — a missing SAN fails the handshake. - **STILL TO BUILD:** (b) **per-host boot-version assignment** — `target_version` is stored + pushed but does not change what a host *boots*; that needs dnsmasq cohort steering + multi-version UKI serving (**#6**). Kiosk URL / reboot / drain diff --git a/docs/fleet-management-design.md b/docs/fleet-management-design.md index 2e46aaa..28a4493 100644 --- a/docs/fleet-management-design.md +++ b/docs/fleet-management-design.md @@ -125,8 +125,9 @@ Reported facts: - **Health:** CPU/load, GPU act-freq, temps, **tmpfs overlay usage** (the RAM-exhaustion → NVMe-keepalive risk — the single most important kiosk health - signal), NVMe-root link state, 802.1X/MACsec `Authorized`, SELinux enforcing, - chrony sync, `kiosk.service` active. + signal), host **uptime** (server derives reboot-pending vs `reboot_at`), + NVMe-root link state, 802.1X/MACsec `Authorized`, SELinux enforcing, chrony + clock health (sync state + NTP source/stratum/offset), `kiosk.service` active. - **Version:** running image version (`/etc/fleet-image`), rootfs digest, kernel. - **Usage:** kiosk URL, session uptime, **Chromium reload/crash count** (from the `kiosk-session` restart loop — the exact signal behind the GPU-bug diagnosis), @@ -186,8 +187,9 @@ UKI). Host (by cert-serial / hostnqn) ├─ identity: cert serial, hostnqn, MAC, starship-name (dnsmasq) ├─ version: image_version, rootfs digest, kernel, desired_version - ├─ health: cpu/load, gpu_freq, temps, tmpfs_used%, nvme_root, auth/macsec, - │ selinux, chrony, kiosk_active, last_seen + ├─ health: cpu/load, gpu_freq, temps, tmpfs_used%, uptime, nvme_root, + │ auth/macsec, selinux, chrony (sync + source/stratum/offset), + │ kiosk_active, last_seen ├─ usage: kiosk_url, session_uptime, chromium_crashes, input_active, display └─ desired: target_version, kiosk_url, reboot_at, drain (server-authoritative) ``` diff --git a/files/agent/README.md b/files/agent/README.md index 3b96469..4b330a9 100644 --- a/files/agent/README.md +++ b/files/agent/README.md @@ -22,9 +22,12 @@ build VM** by CI (deps proxy-fetched, `go.sum`-pinned) and file-provisioned in. 3. **Sends `Hello`** (image version, kernel, kiosk URL, root hash, starship name — the fleet-generic facts, pulled from `fleet-hostapi` on `127.0.0.1:9099`). 4. **Sends a `HealthSnapshot` every `FLEET_AGENT_INTERVAL`** — load/cpu/temp/mem/ - tmpfs, nvme-root, 802.1X-macsec, selinux, chrony, kiosk-active — probed - read-only from `/proc`+`/sys` (each probe degrades gracefully; a failure never - kills the stream). + tmpfs, **uptime**, nvme-root, 802.1X-macsec, selinux, kiosk-active — probed + read-only from `/proc`+`/sys`, plus **chrony** clock health (sync state + + source/stratum/offset relative to the NTP source; see note). Each probe + degrades gracefully; a failure never kills the stream. `uptime` lets the + manager derive **reboot-pending** (boot_time = last_seen − uptime vs the + requested `reboot_at`, all in the manager's clock frame). 5. **Receives `DesiredState`/`Command` and LOGS them.** This build does **not** act on them. The reconcile actuators (apply kiosk URL, reboot, drain) are a deliberate follow-up (see `docs/TODO.md` #14) — the receive path is wired now so @@ -33,6 +36,17 @@ build VM** by CI (deps proxy-fetched, `go.sum`-pinned) and file-provisioned in. A dropped stream is retried with jittered backoff; the manager treats stream loss as the host going **offline**. +> **Note — the chrony probe reads a file, not the command socket.** chronyd's unix +> command socket is owner-write-only and STIG sets `cmdport 0` (no UDP command +> port), so the sandboxed agent can't query it. Instead the image enables +> `log tracking`, and the agent reads the last line of +> `/var/log/chrony/tracking.log`. That read needs **two** access layers, both wired +> in: the **`chrony` supplementary group** (DAC — `/var/log/chrony` is `0750`) and +> a minimal **SELinux allow** for the `chronyd_var_log_t` label +> (`files/stig/fleet-agent-chronylog.te`). The lab NTP server must also be +> reachable (a `123/udp` host-firewall rule on the serving host) or the log only +> carries `unsynced` lines. See `docs/stig-deviations.md` #16. + ## Prerequisite: the host must already be enrolled The manager rejects the stream (`PermissionDenied`) until an inventory record @@ -56,7 +70,7 @@ identity, mirror `rd.dot1x.*`), `FLEET_AGENT_INTERVAL`, `FLEET_AGENT_HOSTAPI`. | `snapshot.go` | read-only `/proc`+`/sys` health probes + hostapi facts fetch | | `tpmkey/` | **copy** of `files/enrol/tpmkey` + read-only `LoadSigner` (see its header) | | `proto/` | **copy** of the generated `fleet.agent.v1` package (one `.proto` contract, two peers) | -| `fleet-agent.service` | hardened unit (tss group for the TPM, INET egress) | +| `fleet-agent.service` | hardened unit (`tss` group for the TPM, `chrony` group to read chrony's tracking.log, INET egress) | ## Build / test diff --git a/serving/fleet-manager/README.md b/serving/fleet-manager/README.md index 9b8f91a..89d98b0 100644 --- a/serving/fleet-manager/README.md +++ b/serving/fleet-manager/README.md @@ -69,7 +69,7 @@ Host (cert_serial) ├─ binding : asset_tag, switch_port, location (recorded at approval) ├─ enrolled : approver, at, fingerprint ├─ version : image_version, rootfs_digest, kernel (agent-reported) - ├─ health : load, cpu, temp, tmpfs%, nvme_root, auth, selinux, chrony, kiosk (latest snapshot; series in InfluxDB) + ├─ health : load, cpu, temp, tmpfs%, uptime, nvme_root, auth, selinux, chrony (state+source/stratum/offset), kiosk (latest snapshot; series in InfluxDB) ├─ usage : kiosk_url, session_uptime, chromium_crashes, display └─ desired : target_version, kiosk_url, reboot_at, drain (SERVER-AUTHORITATIVE) ``` @@ -141,7 +141,9 @@ inventory for offline delivery. Metrics writing is behind a `MetricsSink` Full bring-up (PKI, Let's Encrypt DNS-01, `register-admin`, compose) is in [`DEPLOY.md`](DEPLOY.md). Config is layered: **`defaults.env`** (committed) + **`.env`** (overrides/secrets), via `COMPOSE_ENV_FILES=defaults.env,.env`. In -short: set that + write your secrets to `.env` → `./server-pki.sh` + +short: set that + write your secrets to `.env` → `./server-pki.sh` (or, where +step-ca is co-located, the `cert-auto` **`agent-cert-renewer`** sidecar that +issues + auto-renews the agent-plane server cert and hot-reloads it — DEPLOY §2a) + `../enrol-broker/admin-pki.sh … fleet-manager` → `docker compose up -d influxdb` → `docker compose run --rm fleet-manager register-admin ` → `docker compose up -d fleet-manager traefik`. InfluxDB is private (published to loopback only); @@ -187,9 +189,11 @@ curl -s localhost:9500/api/hosts | jq '.hosts[] | {name:.identity.starship_name, - **Checkpoint 7 (done):** deployment — three-stage `Dockerfile` (npm→dist→ `go build -tags embedspa`→distroless), `docker-compose.yml` (private InfluxDB + host-networked manager), `defaults.env`, `server-pki.sh` (agent-plane fleet-CA - server cert), and [`DEPLOY.md`](DEPLOY.md) (PKI, LE DNS-01, `register-admin` - bootstrap). Design docs updated (manager marked built; resolved transport/auth/ - metrics forks; per-host-versioning gap recorded). + server cert) **+ the `cert-auto` `agent-cert-renewer` sidecar** (auto-issues + + auto-renews that cert from a co-located step-ca; the manager hot-reloads it via + `GetCertificate` in `certreload.go` — no restart), and [`DEPLOY.md`](DEPLOY.md) + (PKI, LE DNS-01, `register-admin` bootstrap). Design docs updated (manager marked + built; resolved transport/auth/metrics forks; per-host-versioning gap recorded). - **Checkpoint 8 (done):** hardening + ingress — CI image build (`fleet-manager-image.yaml`, pulled by tag), `backup.sh`; a **behind-proxy admin mode** (passkey over plaintext behind a TLS-terminating proxy) fronted by a