diskless-workstation/docs/fleet-management-design.md

# Fleet management plane — design note

**Status:** PARTLY BUILT. Captured 2026-06-28; **server-side control plane built
2026-07** (`serving/fleet-manager/`). The sections below are the original design;
the **as-built note** immediately following records what shipped and how the
open forks resolved. Build still phased — see "Phasing" and `docs/TODO.md` #14.

## As-built note (2026-07) — `serving/fleet-manager/`

The **server-side control plane + admin SPA is built** and supersedes several
"open questions" below. What shipped:

- **One service, two planes** (`serving/fleet-manager/`, Go): an **admin plane**
  (HTTPS + SPA + JSON API) and an **agent plane** (gRPC/mTLS). Runs as its own
  container on the mgmt host (`docker-compose.yml`: private InfluxDB +
  host-networked manager). Deployment runbook: `serving/fleet-manager/DEPLOY.md`.
- **Resolved forks (were "open questions / decisions"):**
  - *Steady-state transport (agent ↔ manager)* → **bespoke gRPC bidi stream**
    (`proto/fleet_agent.proto`: `FleetAgent.Session`), host-dialed mTLS, health/
    metrics up + desired-state down. Chosen over OTLP+Wazuh+control-stream: one
    stream, no polling, no inbound host port. Enrolment stays plain HTTP/JSON.
  - *Metrics store* → **InfluxDB 2.x** (was VictoriaMetrics/Prometheus). The
    manager is the **only** reader/writer (host holds no Influx credential);
    writes are line-protocol tagged with the mTLS-authenticated identity, reads
    are a Flux query behind a **field whitelist + hex-serial guard**. Graphs are
    rendered **in the SPA** (bespoke SVG), **not Grafana**.
  - *Dashboards* → the **LYSANDRA restricted-violet SPA** (Svelte 5 + Tailwind 4,
    embedded via `go:embed`): Monitor / Authorise / Lifecycle. Not Grafana/Wazuh
    dashboards for this plane (Wazuh stays the audit/SIEM path).
  - *Desired-state store* → a single **JSON file** (atomic writes) behind a
    `Store` interface — not git/DB yet; a DB is a drop-in. Server-authoritative.
  - *Admin/browser identity* → **WebAuthn/passkey (YubiKey)**, not browser mTLS:
    usernameless discoverable login, UV required, server-side session + CSRF.
    Bootstrap is a closed CLI (`register-admin`, one-time HMAC enrol link). The
    UI cert is **Let's Encrypt DNS-01** (mgmt-net host, hot-reloaded). The
    *manager→broker* hop stays mTLS (service-to-service, broker admin CA).
  - *Authorise pillar* → the **enrol-broker folded in** "alongside": the broker
    stays the state machine + step-ca signer; the manager drives its admin plane
    over mTLS (Accept-gated JSON approve) and **births the inventory record** on
    approve, recording the human passkey admin as approver.
- **Agent identity confirmed:** the agent reuses the **same TPM identity cert**
  as 802.1X EAP-TLS / NVMe hostnqn — "one identity, three stages". Peer cert
  serial is authoritative; an unenrolled serial is refused.
- **On-host agent — telemetry half BUILT (2026-07-05, `files/agent/`).** The
  gRPC client that feeds this plane. Reuses the host TPM identity for mTLS (read-only
  `tpmkey.LoadSigner` at `0x81000100` + cert from NV `0x01800100`; private key never
  leaves the TPM), opens `Session`, sends `Hello` + periodic `HealthSnapshot`
  (read-only `/proc`+`/sys` probes + `fleet-hostapi` facts). **Telemetry-only:** it
  receives `DesiredState`/`Command` but only **logs** them. Static out-of-VM CI
  binary, `2b-agent` provisioner (`enable_agent`), fapolicyd-trusted; mTLS proven
  against the go-tpm simulator (`files/agent/simtest`).
- **NOT yet built (recorded intent):** (1) the agent's **reconcile actuators** —
  apply kiosk_url, reboot_at/REBOOT, drain — deferred until the telemetry half is
  bench-verified; and (2) per-host **boot-version assignment**.
  `desired.target_version` is stored + pushed, but changing what a host *boots*
  needs dnsmasq cohort steering + multi-version UKI serving (`docs/TODO.md` #6). The
  version pin is labelled "recorded intent" in the Lifecycle UI.

Everything below is the original design context; treat the as-built note as
authoritative where they differ.

Centralised monitoring + maintenance of the diskless kiosk fleet: **health,
version, usage, and maintenance actions**. This note is the authoritative design;
it deliberately reuses what already exists rather than bolting on a generic MDM,
because the base architecture (immutable UKI + NVMe-root, TPM identity, no
per-host state, segmented network) makes most off-the-shelf kiosk-management tools
a poor fit.

## Design philosophy (forced by the architecture invariants)

These are consequences of the CLAUDE.md invariants, not preferences:

1. **Server-authoritative, hosts are cattle.** The root is immutable erofs +
   dm-verity on a tmpfs overlay — nothing survives reboot. So **all durable state
   (history, inventory, desired-state) lives server-side**; the on-host agent is
   stateless and buffers only in RAM (best-effort).
2. **Identity from the TPM, never baked.** No per-host state in any artifact
   (invariant #1). The agent derives identity at runtime from the **TPM-held
   cert** (key `0x81000100`, cert NV `0x01800100`) — the same identity used for
   802.1X EAP-TLS and the NVMe hostnqn. No enrollment token in the image.
3. **Pull-based control, no inbound surface.** The boot LAN is locked down and
   segmented; we do not open an always-on inbound control port on hosts. Hosts
   **poll** a desired-state endpoint and reconcile. This also fits the immutable +
   digest-pinned model: *convergence to a new version = reboot*.
4. **Reuse the existing buses.** Telemetry rides the already-built
   `auditd → audit-collector relay → Wazuh` pipeline; version truth is the
   existing digest-pinned deployment manifest; redeploy/drain mechanics are the
   SPDK rolling-update flow (TODO #6). The management plane orchestrates these; it
   does not reinvent them.
5. **Minimal, reproducible, sandboxed agent.** Same posture as `fleet-hostapi`
   (loopback/least-priv, `DynamicUser`, fapolicyd-trusted, reproducible static
   build).

## Components

### 1. Identity — mTLS with the TPM cert

The agent presents the TPM-held client cert over **mTLS** to the management
endpoint; the server authenticates the host by cert CN/SAN, which is the canonical
fleet identity (= NVMe hostnqn = 802.1X identity). Reuses the **step-ca fleet CA**
(`serving/dot1x-bench/`). No baked secret; the host also pins the server cert.

> **Bootstrap (where the cert comes from):** mTLS here assumes the host already
> holds a TPM cert — but an un-enrolled host does not. That chicken-and-egg is
> resolved by **enrolment**, which is the management plane's *one* unauthenticated
> endpoint and the moment the inventory record is born. See
> **[tpm-enrolment-design.md](tpm-enrolment-design.md)** — the enrolment broker is
> designed to fold into this mgmt service as its bootstrap routes.

- **Secure path:** full mTLS, TPM cert. Strong, no secrets in the image.
- **Open/test path (`rd.dot1x=0`):** fall back to dnsmasq fleet-name + MAC as a
  weak identity. **Lab only** — flagged as insufficient for production.

Inventory is keyed by **cert serial / hostnqn**, joined to the dnsmasq starship
name (`fleet-namer`), MAC, last-seen, running/desired version, and health.

### 2. On-host agent — extend `fleet-hostapi`

`fleet-hostapi` (`files/hostapi/`) is already the only fleet-authored component on
the hardware and already models os/hardware/net/kiosk facts. Add a **report/push
mode** (or a tiny sibling that reads its loopback `/v1/host` and ships it). Keep it
stateless; buffer in tmpfs only; identity from the TPM.

Reported facts:

- **Health:** CPU/load, GPU act-freq, temps, **tmpfs overlay usage** (the
  RAM-exhaustion → NVMe-keepalive risk — the single most important kiosk health
  signal), host **uptime** (server derives reboot-pending vs `reboot_at`),
  NVMe-root link state, 802.1X/MACsec `Authorized`, SELinux enforcing, chrony
  clock health (sync state + NTP source/stratum/offset), `kiosk.service` active.
- **Version:** running image version (`/etc/fleet-image`), rootfs digest, kernel.
- **Usage:** kiosk URL, session uptime, **Chromium reload/crash count** (from the
  `kiosk-session` restart loop — the exact signal behind the GPU-bug diagnosis),
  input idle/active (liveness only, never content — privacy-aware), display power
  state.

Cadence: heartbeat (~30–60 s) + event-driven (crash, auth loss, version change).

### 3. Transport

**Phase 1 — ride the Wazuh pipeline.** Emit snapshots as structured JSON events
through the existing `audit-collector` relay → `wazuh1`. Add `fleet_health` /
`fleet_kiosk` decoders + rules alongside `fleet_audit` (`serving/siem/config/`).
Wazuh then provides search, dashboards, and alerting for free, and it's already
secured-path-aware.

**Phase 2 — metrics push** for high-resolution time-series (CPU/GPU/mem trends):
**push** via OTLP or Prometheus remote-write to a collector on the mgmt host →
VictoriaMetrics/Prometheus. **Push, never scrape** — scraping would require an
inbound port on every locked host, against the posture.

### 4. Control / maintenance — pull-based desired-state reconciliation

Each host periodically fetches **its** desired state (mTLS, keyed by TPM cert):
target version, kiosk URL, `reboot-at`, `drain`. The agent reconciles; because the
root is immutable, **convergence = reboot** (a reboot re-PXEs and picks up the new
UKI).

- The **deployment manifest is already the desired-state for versions** (one
  commit per release, digest-pinned). Extend it to per-host / per-cohort desired
  state. Rollback = revert the commit (unchanged model).
- **Maintenance actions** (reboot, drain, restart kiosk, change URL, request logs)
  ride the same poll: the server sets the intent; the host obeys on next poll.
- This dovetails with the **SPDK rolling update** (TODO #6): "redeploy a cohort" =
  bump the manifest → `spdk-loader` serves the new version online → the management
  plane tells those hosts to drain/reboot → it watches `spdk-release list` CONNS
  fall → the old version auto-reaps. The management plane *orchestrates* this loop.
- **No always-on inbound control socket.** Admin SSH (key-only ECDSA) stays purely
  break-glass for exceptional manual ops.

### 5. Server side (serving host or a dedicated mgmt node, e.g. alongside `wazuh1`)

- **Mgmt service** (small Go, mirrors the `fleet-hostapi` style): mTLS endpoint
  that accepts host reports and serves per-host desired-state; authenticates hosts
  by TPM cert via the fleet CA.
- **Store:** time-series (VictoriaMetrics/Prometheus) for metrics; Wazuh /
  OpenSearch for events/logs; git or a tiny DB for desired-state + inventory.
- **Dashboards:** Grafana (metric trends) + the Wazuh dashboard (events/alerts) +
  a small fleet-overview board (offline hosts, version drift, crash rates, tmpfs
  pressure). Could be a sibling of the `serving/kiosk-app` SPA.
- **Alerting:** host offline > N min; Chromium crash-loop; **running ≠ desired
  version**; tmpfs near-full; MACsec/auth down; NVMe-root reconnect storms.

## Data model (sketch)

```
Host (by cert-serial / hostnqn)
  ├─ identity:   cert serial, hostnqn, MAC, starship-name (dnsmasq)
  ├─ version:    image_version, rootfs digest, kernel, desired_version
  ├─ health:     cpu/load, gpu_freq, temps, tmpfs_used%, uptime, nvme_root,
  │              auth/macsec, selinux, chrony (sync + source/stratum/offset),
  │              kiosk_active, last_seen
  ├─ usage:      kiosk_url, session_uptime, chromium_crashes, input_active, display
  └─ desired:    target_version, kiosk_url, reboot_at, drain   (server-authoritative)
```

## Mapping to existing building blocks

| Need | Reuse |
|---|---|
| Host identity | TPM cert / hostnqn / step-ca fleet CA (`serving/dot1x-bench/`) |
| On-host data | `fleet-hostapi` `/v1/host` (`files/hostapi/`) |
| Telemetry + alerting | audit-collector relay → Wazuh (`serving/siem/`) |
| Friendly names / inventory | dnsmasq `fleet-namer` (MAC → name → DNS) |
| Version truth + rollback | digest-pinned deployment manifest (git) |
| Redeploy / drain mechanics | SPDK `spdk-loader` + `spdk-release` (TODO #6) |
| Build / sign / publish | Gitea CI + OCI images |

## Phasing

1. **Phase 1 (mostly-built):** push `fleet-hostapi` snapshots as structured events
   into Wazuh; add decoders/rules + a fleet dashboard → health, version-drift,
   offline detection, usage, alerting. Reuses the existing pipeline end to end.
2. **Phase 2:** metrics push (OTLP / remote-write) → VictoriaMetrics + Grafana for
   trend graphs.
3. **Phase 3:** mTLS desired-state pull (version pin, reboot/drain, kiosk URL),
   wired to the deployment manifest + SPDK rolling-update flow.

## Build vs buy

Off-the-shelf fits poorly because the base is unusual:

- **FleetDM + osquery** — closest for the query/health side; worth evaluating. But
  osquery enrollment carries per-host state, so it must be run carefully (identity
  from TPM, enrollment in tmpfs only) or it breaks invariant #1.
- **Mender / balena / typical kiosk MDM** — assume mutable disk + their own OTA +
  enrollment. We already have a *better* version mechanism (UKI + NVMe-root +
  digest manifest), so their OTA is redundant and their disk model conflicts.

**Decision: build the thin mgmt service + reconcile loop; reuse Wazuh/Grafana for
the heavy lifting.** The hard parts (identity, immutable versioning, a telemetry
bus) already exist.

## Security posture

- mTLS + TPM cert = strong identity, zero baked secrets, reuses the 802.1X CA.
- Pull-based control = **no new inbound attack surface** on locked hosts.
- Telemetry stays on the mgmt/boot LAN, TLS'd (same gap as the Wazuh-relay-TLS
  TODO) — no internet egress, air-gap intact.
- Agent stays minimal/sandboxed/fapolicyd-trusted/reproducible (`fleet-hostapi`
  template); usage capture is liveness-only, never content (privacy).
- Fleet-wide inventory is recon-valuable → protect the management plane (auth +
  segmentation); never expose its dashboards to the boot LAN.

## Open questions / decisions for build time

- **Telemetry encoding:** structured syslog/JSON into Wazuh (Phase 1) vs native
  OTLP from the start. Leaning Wazuh-first for reuse.
- **Steady-state transport (agent ↔ manager):** the agent is **Go**, the path is
  **mTLS + host-dialed** (preserves "no inbound port"). Fork: bespoke **gRPC**
  stream (health up / desired-state down) vs reuse **OTLP-gRPC for metrics + Wazuh
  for events** + a thin gRPC control stream — the latter gets VictoriaMetrics/
  Grafana/Wazuh for free. `Connect` (connectrpc.com) if one protobuf IDL serving
  gRPC + JSON + browser is wanted. Note **enrolment** deliberately stays plain
  HTTP/JSON (stdlib, minimal, the one unauth edge) — see
  [tpm-enrolment-design.md](tpm-enrolment-design.md) "Implementation".
- **Metrics store:** VictoriaMetrics vs Prometheus (VM is lighter for push +
  long retention).
- **Desired-state store:** git (GitOps, aud/rollback for free, matches the
  deployment-manifest model) vs a small DB (richer per-host scheduling).
- **Per-host vs per-cohort** desired state granularity (cohorts likely enough;
  per-host for canaries).
- **Open-path identity:** MAC/fleet-name is weak — gate the management plane to the
  secure path, or accept lab-only for the open path.
- **osquery/FleetDM** evaluation for the read side before building a bespoke
  collector.