194 lines
10 KiB
Markdown
194 lines
10 KiB
Markdown
# Fleet management plane — design note
|
||
|
||
**Status:** DESIGN (not built). Captured 2026-06-28. Build phased — see "Phasing"
|
||
and `docs/TODO.md` #14.
|
||
|
||
Centralised monitoring + maintenance of the diskless kiosk fleet: **health,
|
||
version, usage, and maintenance actions**. This note is the authoritative design;
|
||
it deliberately reuses what already exists rather than bolting on a generic MDM,
|
||
because the base architecture (immutable UKI + NVMe-root, TPM identity, no
|
||
per-host state, segmented network) makes most off-the-shelf kiosk-management tools
|
||
a poor fit.
|
||
|
||
## Design philosophy (forced by the architecture invariants)
|
||
|
||
These are consequences of the CLAUDE.md invariants, not preferences:
|
||
|
||
1. **Server-authoritative, hosts are cattle.** The root is immutable erofs +
|
||
dm-verity on a tmpfs overlay — nothing survives reboot. So **all durable state
|
||
(history, inventory, desired-state) lives server-side**; the on-host agent is
|
||
stateless and buffers only in RAM (best-effort).
|
||
2. **Identity from the TPM, never baked.** No per-host state in any artifact
|
||
(invariant #1). The agent derives identity at runtime from the **TPM-held
|
||
cert** (key `0x81010001`, cert NV `0x01800001`) — the same identity used for
|
||
802.1X EAP-TLS and the NVMe hostnqn. No enrollment token in the image.
|
||
3. **Pull-based control, no inbound surface.** The boot LAN is locked down and
|
||
segmented; we do not open an always-on inbound control port on hosts. Hosts
|
||
**poll** a desired-state endpoint and reconcile. This also fits the immutable +
|
||
digest-pinned model: *convergence to a new version = reboot*.
|
||
4. **Reuse the existing buses.** Telemetry rides the already-built
|
||
`auditd → audit-collector relay → Wazuh` pipeline; version truth is the
|
||
existing digest-pinned deployment manifest; redeploy/drain mechanics are the
|
||
SPDK rolling-update flow (TODO #6). The management plane orchestrates these; it
|
||
does not reinvent them.
|
||
5. **Minimal, reproducible, sandboxed agent.** Same posture as `fleet-hostapi`
|
||
(loopback/least-priv, `DynamicUser`, fapolicyd-trusted, reproducible static
|
||
build).
|
||
|
||
## Components
|
||
|
||
### 1. Identity — mTLS with the TPM cert
|
||
|
||
The agent presents the TPM-held client cert over **mTLS** to the management
|
||
endpoint; the server authenticates the host by cert CN/SAN, which is the canonical
|
||
fleet identity (= NVMe hostnqn = 802.1X identity). Reuses the **step-ca fleet CA**
|
||
(`serving/dot1x-bench/`). No baked secret; the host also pins the server cert.
|
||
|
||
- **Secure path:** full mTLS, TPM cert. Strong, no secrets in the image.
|
||
- **Open/test path (`rd.dot1x=0`):** fall back to dnsmasq fleet-name + MAC as a
|
||
weak identity. **Lab only** — flagged as insufficient for production.
|
||
|
||
Inventory is keyed by **cert serial / hostnqn**, joined to the dnsmasq starship
|
||
name (`fleet-namer`), MAC, last-seen, running/desired version, and health.
|
||
|
||
### 2. On-host agent — extend `fleet-hostapi`
|
||
|
||
`fleet-hostapi` (`files/hostapi/`) is already the only fleet-authored component on
|
||
the hardware and already models os/hardware/net/kiosk facts. Add a **report/push
|
||
mode** (or a tiny sibling that reads its loopback `/v1/host` and ships it). Keep it
|
||
stateless; buffer in tmpfs only; identity from the TPM.
|
||
|
||
Reported facts:
|
||
|
||
- **Health:** CPU/load, GPU act-freq, temps, **tmpfs overlay usage** (the
|
||
RAM-exhaustion → NVMe-keepalive risk — the single most important kiosk health
|
||
signal), NVMe-root link state, 802.1X/MACsec `Authorized`, SELinux enforcing,
|
||
chrony sync, `kiosk.service` active.
|
||
- **Version:** running image version (`/etc/fleet-image`), rootfs digest, kernel.
|
||
- **Usage:** kiosk URL, session uptime, **Chromium reload/crash count** (from the
|
||
`kiosk-session` restart loop — the exact signal behind the GPU-bug diagnosis),
|
||
input idle/active (liveness only, never content — privacy-aware), display power
|
||
state.
|
||
|
||
Cadence: heartbeat (~30–60 s) + event-driven (crash, auth loss, version change).
|
||
|
||
### 3. Transport
|
||
|
||
**Phase 1 — ride the Wazuh pipeline.** Emit snapshots as structured JSON events
|
||
through the existing `audit-collector` relay → `wazuh1`. Add `fleet_health` /
|
||
`fleet_kiosk` decoders + rules alongside `fleet_audit` (`serving/siem/config/`).
|
||
Wazuh then provides search, dashboards, and alerting for free, and it's already
|
||
secured-path-aware.
|
||
|
||
**Phase 2 — metrics push** for high-resolution time-series (CPU/GPU/mem trends):
|
||
**push** via OTLP or Prometheus remote-write to a collector on the mgmt host →
|
||
VictoriaMetrics/Prometheus. **Push, never scrape** — scraping would require an
|
||
inbound port on every locked host, against the posture.
|
||
|
||
### 4. Control / maintenance — pull-based desired-state reconciliation
|
||
|
||
Each host periodically fetches **its** desired state (mTLS, keyed by TPM cert):
|
||
target version, kiosk URL, `reboot-at`, `drain`. The agent reconciles; because the
|
||
root is immutable, **convergence = reboot** (a reboot re-PXEs and picks up the new
|
||
UKI).
|
||
|
||
- The **deployment manifest is already the desired-state for versions** (one
|
||
commit per release, digest-pinned). Extend it to per-host / per-cohort desired
|
||
state. Rollback = revert the commit (unchanged model).
|
||
- **Maintenance actions** (reboot, drain, restart kiosk, change URL, request logs)
|
||
ride the same poll: the server sets the intent; the host obeys on next poll.
|
||
- This dovetails with the **SPDK rolling update** (TODO #6): "redeploy a cohort" =
|
||
bump the manifest → `spdk-loader` serves the new version online → the management
|
||
plane tells those hosts to drain/reboot → it watches `spdk-release list` CONNS
|
||
fall → the old version auto-reaps. The management plane *orchestrates* this loop.
|
||
- **No always-on inbound control socket.** Admin SSH (key-only ECDSA) stays purely
|
||
break-glass for exceptional manual ops.
|
||
|
||
### 5. Server side (serving host or a dedicated mgmt node, e.g. alongside `wazuh1`)
|
||
|
||
- **Mgmt service** (small Go, mirrors the `fleet-hostapi` style): mTLS endpoint
|
||
that accepts host reports and serves per-host desired-state; authenticates hosts
|
||
by TPM cert via the fleet CA.
|
||
- **Store:** time-series (VictoriaMetrics/Prometheus) for metrics; Wazuh /
|
||
OpenSearch for events/logs; git or a tiny DB for desired-state + inventory.
|
||
- **Dashboards:** Grafana (metric trends) + the Wazuh dashboard (events/alerts) +
|
||
a small fleet-overview board (offline hosts, version drift, crash rates, tmpfs
|
||
pressure). Could be a sibling of the `serving/kiosk-app` SPA.
|
||
- **Alerting:** host offline > N min; Chromium crash-loop; **running ≠ desired
|
||
version**; tmpfs near-full; MACsec/auth down; NVMe-root reconnect storms.
|
||
|
||
## Data model (sketch)
|
||
|
||
```
|
||
Host (by cert-serial / hostnqn)
|
||
├─ identity: cert serial, hostnqn, MAC, starship-name (dnsmasq)
|
||
├─ version: image_version, rootfs digest, kernel, desired_version
|
||
├─ health: cpu/load, gpu_freq, temps, tmpfs_used%, nvme_root, auth/macsec,
|
||
│ selinux, chrony, kiosk_active, last_seen
|
||
├─ usage: kiosk_url, session_uptime, chromium_crashes, input_active, display
|
||
└─ desired: target_version, kiosk_url, reboot_at, drain (server-authoritative)
|
||
```
|
||
|
||
## Mapping to existing building blocks
|
||
|
||
| Need | Reuse |
|
||
|---|---|
|
||
| Host identity | TPM cert / hostnqn / step-ca fleet CA (`serving/dot1x-bench/`) |
|
||
| On-host data | `fleet-hostapi` `/v1/host` (`files/hostapi/`) |
|
||
| Telemetry + alerting | audit-collector relay → Wazuh (`serving/siem/`) |
|
||
| Friendly names / inventory | dnsmasq `fleet-namer` (MAC → name → DNS) |
|
||
| Version truth + rollback | digest-pinned deployment manifest (git) |
|
||
| Redeploy / drain mechanics | SPDK `spdk-loader` + `spdk-release` (TODO #6) |
|
||
| Build / sign / publish | Gitea CI + OCI images |
|
||
|
||
## Phasing
|
||
|
||
1. **Phase 1 (mostly-built):** push `fleet-hostapi` snapshots as structured events
|
||
into Wazuh; add decoders/rules + a fleet dashboard → health, version-drift,
|
||
offline detection, usage, alerting. Reuses the existing pipeline end to end.
|
||
2. **Phase 2:** metrics push (OTLP / remote-write) → VictoriaMetrics + Grafana for
|
||
trend graphs.
|
||
3. **Phase 3:** mTLS desired-state pull (version pin, reboot/drain, kiosk URL),
|
||
wired to the deployment manifest + SPDK rolling-update flow.
|
||
|
||
## Build vs buy
|
||
|
||
Off-the-shelf fits poorly because the base is unusual:
|
||
|
||
- **FleetDM + osquery** — closest for the query/health side; worth evaluating. But
|
||
osquery enrollment carries per-host state, so it must be run carefully (identity
|
||
from TPM, enrollment in tmpfs only) or it breaks invariant #1.
|
||
- **Mender / balena / typical kiosk MDM** — assume mutable disk + their own OTA +
|
||
enrollment. We already have a *better* version mechanism (UKI + NVMe-root +
|
||
digest manifest), so their OTA is redundant and their disk model conflicts.
|
||
|
||
**Decision: build the thin mgmt service + reconcile loop; reuse Wazuh/Grafana for
|
||
the heavy lifting.** The hard parts (identity, immutable versioning, a telemetry
|
||
bus) already exist.
|
||
|
||
## Security posture
|
||
|
||
- mTLS + TPM cert = strong identity, zero baked secrets, reuses the 802.1X CA.
|
||
- Pull-based control = **no new inbound attack surface** on locked hosts.
|
||
- Telemetry stays on the mgmt/boot LAN, TLS'd (same gap as the Wazuh-relay-TLS
|
||
TODO) — no internet egress, air-gap intact.
|
||
- Agent stays minimal/sandboxed/fapolicyd-trusted/reproducible (`fleet-hostapi`
|
||
template); usage capture is liveness-only, never content (privacy).
|
||
- Fleet-wide inventory is recon-valuable → protect the management plane (auth +
|
||
segmentation); never expose its dashboards to the boot LAN.
|
||
|
||
## Open questions / decisions for build time
|
||
|
||
- **Telemetry encoding:** structured syslog/JSON into Wazuh (Phase 1) vs native
|
||
OTLP from the start. Leaning Wazuh-first for reuse.
|
||
- **Metrics store:** VictoriaMetrics vs Prometheus (VM is lighter for push +
|
||
long retention).
|
||
- **Desired-state store:** git (GitOps, aud/rollback for free, matches the
|
||
deployment-manifest model) vs a small DB (richer per-host scheduling).
|
||
- **Per-host vs per-cohort** desired state granularity (cohorts likely enough;
|
||
per-host for canaries).
|
||
- **Open-path identity:** MAC/fleet-name is weak — gate the management plane to the
|
||
secure path, or accept lab-only for the open path.
|
||
- **osquery/FleetDM** evaluation for the read side before building a bespoke
|
||
collector.
|