docs: reflect chrony/uptime metrics, agent-cert renewer, and the chrony-access findings
Recent work added telemetry fields and a cert path not yet in the docs: - files/agent/README.md: add uptime to the health list; document that chrony is read from /var/log/chrony/tracking.log (not the command socket) and needs BOTH the chrony supplementary group (DAC) and the fleet_agent_chronylog SELinux allow (TE), plus a reachable lab NTP (123/udp firewall); note the chrony group on the unit row. - serving/fleet-manager/README.md: add uptime + chrony detail to the health list; note the cert-auto agent-cert-renewer sidecar + GetCertificate hot-reload (certreload.go) in the bring-up and checkpoint 7. - docs/TODO.md #14: probe list gains uptime + chrony-from-tracking.log; new dated block records uptime/reboot-pending, the 4-layer chrony-access chase (all silent-failing; inventory.json is the isolator), and the agent-plane cert auto-renewal, all bench-verified 2026-07-10. - docs/fleet-management-design.md: health record gains uptime + chrony source/stratum/offset detail.
This commit is contained in:
+29
-2
@@ -1412,8 +1412,9 @@ the design forks resolved:
|
|||||||
the host **TPM identity** for mTLS (signing key at `0x81000100` via
|
the host **TPM identity** for mTLS (signing key at `0x81000100` via
|
||||||
`tpmkey.LoadSigner` — a read-only copy of the enrol `tpmkey`; cert from NV
|
`tpmkey.LoadSigner` — a read-only copy of the enrol `tpmkey`; cert from NV
|
||||||
`0x01800100`), opens the long-lived `Session` stream, sends `Hello` + periodic
|
`0x01800100`), opens the long-lived `Session` stream, sends `Hello` + periodic
|
||||||
`HealthSnapshot` (load/cpu/temp/mem/tmpfs/nvme/auth/selinux/chrony/kiosk probed
|
`HealthSnapshot` (load/cpu/temp/mem/tmpfs/**uptime**/nvme/auth/selinux/kiosk from
|
||||||
read-only from `/proc`+`/sys` + hostapi facts), and — **telemetry-only v1** —
|
`/proc`+`/sys` + hostapi facts, plus **chrony** clock health — state/source/
|
||||||
|
stratum/offset — read from `/var/log/chrony/tracking.log`), and — **telemetry-only v1** —
|
||||||
**receives `DesiredState`/`Command` but only LOGS them** (zero host mutation).
|
**receives `DesiredState`/`Command` but only LOGS them** (zero host mutation).
|
||||||
Static `CGO_ENABLED=0` binary built out-of-VM in CI (like enrol/hostapi), staged
|
Static `CGO_ENABLED=0` binary built out-of-VM in CI (like enrol/hostapi), staged
|
||||||
by the `2b-agent` provisioner (`enable_agent`, default on), fapolicyd-trusted,
|
by the `2b-agent` provisioner (`enable_agent`, default on), fapolicyd-trusted,
|
||||||
@@ -1464,6 +1465,32 @@ the design forks resolved:
|
|||||||
register-host <serial> -name <starship>` (writes through the real store to
|
register-host <serial> -name <starship>` (writes through the real store to
|
||||||
`FLEET_MGR_INVENTORY`). The manager caches inventory in memory — any edit needs
|
`FLEET_MGR_INVENTORY`). The manager caches inventory in memory — any edit needs
|
||||||
a `--force-recreate`. Also: the agent now logs the real gRPC status, not `EOF`.
|
a `--force-recreate`. Also: the agent now logs the real gRPC status, not `EOF`.
|
||||||
|
|
||||||
|
**Metrics + cert auto-renewal added 2026-07-10 (`fleet-manager` branch):**
|
||||||
|
- **uptime + reboot-pending**: the agent sends `uptime_seconds`; the manager/UI
|
||||||
|
derive "reboot pending in X / rebooted" from boot_time (= last_seen − uptime) vs
|
||||||
|
`reboot_at`, all in the manager clock frame (no host↔manager clock sync needed).
|
||||||
|
Monitor metric graphs auto-refresh (10 s poll).
|
||||||
|
- **chrony clock health** (sync state + NTP source/stratum/offset). The agent reads
|
||||||
|
`/var/log/chrony/tracking.log` (image bakes `log tracking`), NOT chronyd's command
|
||||||
|
socket (owner-write-only unix socket; STIG `cmdport 0` closes the UDP port).
|
||||||
|
**Reaching that log from the hardened DynamicUser agent took a 4-layer chase, every
|
||||||
|
layer silent-failing** (memory `fleet-agent-hardened-probe-access`): (i) lab NTP
|
||||||
|
reachable — a `123/udp` rule on the serving-host firewall (host-networked ntp
|
||||||
|
container; same trap as audit `514`); (ii) DAC — `chrony` supplementary group on the
|
||||||
|
unit to traverse `/var/log/chrony` (0750); (iii) SELinux TE —
|
||||||
|
`files/stig/fleet-agent-chronylog.te` allow for `chronyd_var_log_t` (permissive
|
||||||
|
masks this); (iv) the MANAGER image must be ≥ the chrony commit — one with
|
||||||
|
uptime-but-not-chrony silently drops the fields (manager & workstation are SEPARATE
|
||||||
|
images/deploys). Isolator: read the manager `inventory.json` off the `fleet-data`
|
||||||
|
volume via a throwaway busybox (distroless manager, no shell). BENCH-VERIFIED on
|
||||||
|
`comet-beacon-54` 2026-07-10.
|
||||||
|
- **agent-plane server-cert auto-renewal**: the `agent-cert-renewer` compose sidecar
|
||||||
|
(profile `cert-auto`) issues + `step ca renew --daemon` renews the cert from a
|
||||||
|
co-located step-ca; the manager hot-reloads via `GetCertificate` (`certreload.go`),
|
||||||
|
no restart. Replaces the manual `server-pki.sh` + `docker exec … step` dance and
|
||||||
|
the 90-day re-mint. The server cert SANs MUST include `manager.fleet.lab` (the
|
||||||
|
fleet-DNS name the agent dials) — a missing SAN fails the handshake.
|
||||||
- **STILL TO BUILD:** (b) **per-host boot-version assignment** — `target_version` is
|
- **STILL TO BUILD:** (b) **per-host boot-version assignment** — `target_version` is
|
||||||
stored + pushed but does not change what a host *boots*; that needs dnsmasq
|
stored + pushed but does not change what a host *boots*; that needs dnsmasq
|
||||||
cohort steering + multi-version UKI serving (**#6**). Kiosk URL / reboot / drain
|
cohort steering + multi-version UKI serving (**#6**). Kiosk URL / reboot / drain
|
||||||
|
|||||||
@@ -125,8 +125,9 @@ Reported facts:
|
|||||||
|
|
||||||
- **Health:** CPU/load, GPU act-freq, temps, **tmpfs overlay usage** (the
|
- **Health:** CPU/load, GPU act-freq, temps, **tmpfs overlay usage** (the
|
||||||
RAM-exhaustion → NVMe-keepalive risk — the single most important kiosk health
|
RAM-exhaustion → NVMe-keepalive risk — the single most important kiosk health
|
||||||
signal), NVMe-root link state, 802.1X/MACsec `Authorized`, SELinux enforcing,
|
signal), host **uptime** (server derives reboot-pending vs `reboot_at`),
|
||||||
chrony sync, `kiosk.service` active.
|
NVMe-root link state, 802.1X/MACsec `Authorized`, SELinux enforcing, chrony
|
||||||
|
clock health (sync state + NTP source/stratum/offset), `kiosk.service` active.
|
||||||
- **Version:** running image version (`/etc/fleet-image`), rootfs digest, kernel.
|
- **Version:** running image version (`/etc/fleet-image`), rootfs digest, kernel.
|
||||||
- **Usage:** kiosk URL, session uptime, **Chromium reload/crash count** (from the
|
- **Usage:** kiosk URL, session uptime, **Chromium reload/crash count** (from the
|
||||||
`kiosk-session` restart loop — the exact signal behind the GPU-bug diagnosis),
|
`kiosk-session` restart loop — the exact signal behind the GPU-bug diagnosis),
|
||||||
@@ -186,8 +187,9 @@ UKI).
|
|||||||
Host (by cert-serial / hostnqn)
|
Host (by cert-serial / hostnqn)
|
||||||
├─ identity: cert serial, hostnqn, MAC, starship-name (dnsmasq)
|
├─ identity: cert serial, hostnqn, MAC, starship-name (dnsmasq)
|
||||||
├─ version: image_version, rootfs digest, kernel, desired_version
|
├─ version: image_version, rootfs digest, kernel, desired_version
|
||||||
├─ health: cpu/load, gpu_freq, temps, tmpfs_used%, nvme_root, auth/macsec,
|
├─ health: cpu/load, gpu_freq, temps, tmpfs_used%, uptime, nvme_root,
|
||||||
│ selinux, chrony, kiosk_active, last_seen
|
│ auth/macsec, selinux, chrony (sync + source/stratum/offset),
|
||||||
|
│ kiosk_active, last_seen
|
||||||
├─ usage: kiosk_url, session_uptime, chromium_crashes, input_active, display
|
├─ usage: kiosk_url, session_uptime, chromium_crashes, input_active, display
|
||||||
└─ desired: target_version, kiosk_url, reboot_at, drain (server-authoritative)
|
└─ desired: target_version, kiosk_url, reboot_at, drain (server-authoritative)
|
||||||
```
|
```
|
||||||
|
|||||||
+18
-4
@@ -22,9 +22,12 @@ build VM** by CI (deps proxy-fetched, `go.sum`-pinned) and file-provisioned in.
|
|||||||
3. **Sends `Hello`** (image version, kernel, kiosk URL, root hash, starship name —
|
3. **Sends `Hello`** (image version, kernel, kiosk URL, root hash, starship name —
|
||||||
the fleet-generic facts, pulled from `fleet-hostapi` on `127.0.0.1:9099`).
|
the fleet-generic facts, pulled from `fleet-hostapi` on `127.0.0.1:9099`).
|
||||||
4. **Sends a `HealthSnapshot` every `FLEET_AGENT_INTERVAL`** — load/cpu/temp/mem/
|
4. **Sends a `HealthSnapshot` every `FLEET_AGENT_INTERVAL`** — load/cpu/temp/mem/
|
||||||
tmpfs, nvme-root, 802.1X-macsec, selinux, chrony, kiosk-active — probed
|
tmpfs, **uptime**, nvme-root, 802.1X-macsec, selinux, kiosk-active — probed
|
||||||
read-only from `/proc`+`/sys` (each probe degrades gracefully; a failure never
|
read-only from `/proc`+`/sys`, plus **chrony** clock health (sync state +
|
||||||
kills the stream).
|
source/stratum/offset relative to the NTP source; see note). Each probe
|
||||||
|
degrades gracefully; a failure never kills the stream. `uptime` lets the
|
||||||
|
manager derive **reboot-pending** (boot_time = last_seen − uptime vs the
|
||||||
|
requested `reboot_at`, all in the manager's clock frame).
|
||||||
5. **Receives `DesiredState`/`Command` and LOGS them.** This build does **not**
|
5. **Receives `DesiredState`/`Command` and LOGS them.** This build does **not**
|
||||||
act on them. The reconcile actuators (apply kiosk URL, reboot, drain) are a
|
act on them. The reconcile actuators (apply kiosk URL, reboot, drain) are a
|
||||||
deliberate follow-up (see `docs/TODO.md` #14) — the receive path is wired now so
|
deliberate follow-up (see `docs/TODO.md` #14) — the receive path is wired now so
|
||||||
@@ -33,6 +36,17 @@ build VM** by CI (deps proxy-fetched, `go.sum`-pinned) and file-provisioned in.
|
|||||||
A dropped stream is retried with jittered backoff; the manager treats stream loss
|
A dropped stream is retried with jittered backoff; the manager treats stream loss
|
||||||
as the host going **offline**.
|
as the host going **offline**.
|
||||||
|
|
||||||
|
> **Note — the chrony probe reads a file, not the command socket.** chronyd's unix
|
||||||
|
> command socket is owner-write-only and STIG sets `cmdport 0` (no UDP command
|
||||||
|
> port), so the sandboxed agent can't query it. Instead the image enables
|
||||||
|
> `log tracking`, and the agent reads the last line of
|
||||||
|
> `/var/log/chrony/tracking.log`. That read needs **two** access layers, both wired
|
||||||
|
> in: the **`chrony` supplementary group** (DAC — `/var/log/chrony` is `0750`) and
|
||||||
|
> a minimal **SELinux allow** for the `chronyd_var_log_t` label
|
||||||
|
> (`files/stig/fleet-agent-chronylog.te`). The lab NTP server must also be
|
||||||
|
> reachable (a `123/udp` host-firewall rule on the serving host) or the log only
|
||||||
|
> carries `unsynced` lines. See `docs/stig-deviations.md` #16.
|
||||||
|
|
||||||
## Prerequisite: the host must already be enrolled
|
## Prerequisite: the host must already be enrolled
|
||||||
|
|
||||||
The manager rejects the stream (`PermissionDenied`) until an inventory record
|
The manager rejects the stream (`PermissionDenied`) until an inventory record
|
||||||
@@ -56,7 +70,7 @@ identity, mirror `rd.dot1x.*`), `FLEET_AGENT_INTERVAL`, `FLEET_AGENT_HOSTAPI`.
|
|||||||
| `snapshot.go` | read-only `/proc`+`/sys` health probes + hostapi facts fetch |
|
| `snapshot.go` | read-only `/proc`+`/sys` health probes + hostapi facts fetch |
|
||||||
| `tpmkey/` | **copy** of `files/enrol/tpmkey` + read-only `LoadSigner` (see its header) |
|
| `tpmkey/` | **copy** of `files/enrol/tpmkey` + read-only `LoadSigner` (see its header) |
|
||||||
| `proto/` | **copy** of the generated `fleet.agent.v1` package (one `.proto` contract, two peers) |
|
| `proto/` | **copy** of the generated `fleet.agent.v1` package (one `.proto` contract, two peers) |
|
||||||
| `fleet-agent.service` | hardened unit (tss group for the TPM, INET egress) |
|
| `fleet-agent.service` | hardened unit (`tss` group for the TPM, `chrony` group to read chrony's tracking.log, INET egress) |
|
||||||
|
|
||||||
## Build / test
|
## Build / test
|
||||||
|
|
||||||
|
|||||||
@@ -69,7 +69,7 @@ Host (cert_serial)
|
|||||||
├─ binding : asset_tag, switch_port, location (recorded at approval)
|
├─ binding : asset_tag, switch_port, location (recorded at approval)
|
||||||
├─ enrolled : approver, at, fingerprint
|
├─ enrolled : approver, at, fingerprint
|
||||||
├─ version : image_version, rootfs_digest, kernel (agent-reported)
|
├─ version : image_version, rootfs_digest, kernel (agent-reported)
|
||||||
├─ health : load, cpu, temp, tmpfs%, nvme_root, auth, selinux, chrony, kiosk (latest snapshot; series in InfluxDB)
|
├─ health : load, cpu, temp, tmpfs%, uptime, nvme_root, auth, selinux, chrony (state+source/stratum/offset), kiosk (latest snapshot; series in InfluxDB)
|
||||||
├─ usage : kiosk_url, session_uptime, chromium_crashes, display
|
├─ usage : kiosk_url, session_uptime, chromium_crashes, display
|
||||||
└─ desired : target_version, kiosk_url, reboot_at, drain (SERVER-AUTHORITATIVE)
|
└─ desired : target_version, kiosk_url, reboot_at, drain (SERVER-AUTHORITATIVE)
|
||||||
```
|
```
|
||||||
@@ -141,7 +141,9 @@ inventory for offline delivery. Metrics writing is behind a `MetricsSink`
|
|||||||
Full bring-up (PKI, Let's Encrypt DNS-01, `register-admin`, compose) is in
|
Full bring-up (PKI, Let's Encrypt DNS-01, `register-admin`, compose) is in
|
||||||
[`DEPLOY.md`](DEPLOY.md). Config is layered: **`defaults.env`** (committed) +
|
[`DEPLOY.md`](DEPLOY.md). Config is layered: **`defaults.env`** (committed) +
|
||||||
**`.env`** (overrides/secrets), via `COMPOSE_ENV_FILES=defaults.env,.env`. In
|
**`.env`** (overrides/secrets), via `COMPOSE_ENV_FILES=defaults.env,.env`. In
|
||||||
short: set that + write your secrets to `.env` → `./server-pki.sh` +
|
short: set that + write your secrets to `.env` → `./server-pki.sh` (or, where
|
||||||
|
step-ca is co-located, the `cert-auto` **`agent-cert-renewer`** sidecar that
|
||||||
|
issues + auto-renews the agent-plane server cert and hot-reloads it — DEPLOY §2a) +
|
||||||
`../enrol-broker/admin-pki.sh … fleet-manager` → `docker compose up -d influxdb`
|
`../enrol-broker/admin-pki.sh … fleet-manager` → `docker compose up -d influxdb`
|
||||||
→ `docker compose run --rm fleet-manager register-admin <you>` → `docker compose
|
→ `docker compose run --rm fleet-manager register-admin <you>` → `docker compose
|
||||||
up -d fleet-manager traefik`. InfluxDB is private (published to loopback only);
|
up -d fleet-manager traefik`. InfluxDB is private (published to loopback only);
|
||||||
@@ -187,9 +189,11 @@ curl -s localhost:9500/api/hosts | jq '.hosts[] | {name:.identity.starship_name,
|
|||||||
- **Checkpoint 7 (done):** deployment — three-stage `Dockerfile` (npm→dist→
|
- **Checkpoint 7 (done):** deployment — three-stage `Dockerfile` (npm→dist→
|
||||||
`go build -tags embedspa`→distroless), `docker-compose.yml` (private InfluxDB +
|
`go build -tags embedspa`→distroless), `docker-compose.yml` (private InfluxDB +
|
||||||
host-networked manager), `defaults.env`, `server-pki.sh` (agent-plane fleet-CA
|
host-networked manager), `defaults.env`, `server-pki.sh` (agent-plane fleet-CA
|
||||||
server cert), and [`DEPLOY.md`](DEPLOY.md) (PKI, LE DNS-01, `register-admin`
|
server cert) **+ the `cert-auto` `agent-cert-renewer` sidecar** (auto-issues +
|
||||||
bootstrap). Design docs updated (manager marked built; resolved transport/auth/
|
auto-renews that cert from a co-located step-ca; the manager hot-reloads it via
|
||||||
metrics forks; per-host-versioning gap recorded).
|
`GetCertificate` in `certreload.go` — no restart), and [`DEPLOY.md`](DEPLOY.md)
|
||||||
|
(PKI, LE DNS-01, `register-admin` bootstrap). Design docs updated (manager marked
|
||||||
|
built; resolved transport/auth/metrics forks; per-host-versioning gap recorded).
|
||||||
- **Checkpoint 8 (done):** hardening + ingress — CI image build
|
- **Checkpoint 8 (done):** hardening + ingress — CI image build
|
||||||
(`fleet-manager-image.yaml`, pulled by tag), `backup.sh`; a **behind-proxy admin
|
(`fleet-manager-image.yaml`, pulled by tag), `backup.sh`; a **behind-proxy admin
|
||||||
mode** (passkey over plaintext behind a TLS-terminating proxy) fronted by a
|
mode** (passkey over plaintext behind a TLS-terminating proxy) fronted by a
|
||||||
|
|||||||
Reference in New Issue
Block a user