diskless-workstation/docs/fleet-management-design.md

Fleet management plane — design note

Status: PARTLY BUILT. Captured 2026-06-28; server-side control plane built 2026-07 (serving/fleet-manager/). The sections below are the original design; the as-built note immediately following records what shipped and how the open forks resolved. Build still phased — see "Phasing" and docs/TODO.md #14.

As-built note (2026-07) — serving/fleet-manager/

The server-side control plane + admin SPA is built and supersedes several "open questions" below. What shipped:

• One service, two planes (serving/fleet-manager/, Go): an admin plane (HTTPS + SPA + JSON API) and an agent plane (gRPC/mTLS). Runs as its own container on the mgmt host (docker-compose.yml: private InfluxDB + host-networked manager). Deployment runbook: serving/fleet-manager/DEPLOY.md.
• Resolved forks (were "open questions / decisions"):
  • Steady-state transport (agent ↔ manager) → bespoke gRPC bidi stream (proto/fleet_agent.proto: FleetAgent.Session), host-dialed mTLS, health/ metrics up + desired-state down. Chosen over OTLP+Wazuh+control-stream: one stream, no polling, no inbound host port. Enrolment stays plain HTTP/JSON.
  • Metrics store → InfluxDB 2.x (was VictoriaMetrics/Prometheus). The manager is the only reader/writer (host holds no Influx credential); writes are line-protocol tagged with the mTLS-authenticated identity, reads are a Flux query behind a field whitelist + hex-serial guard. Graphs are rendered in the SPA (bespoke SVG), not Grafana.
  • Dashboards → the LYSANDRA restricted-violet SPA (Svelte 5 + Tailwind 4, embedded via go:embed): Monitor / Authorise / Lifecycle. Not Grafana/Wazuh dashboards for this plane (Wazuh stays the audit/SIEM path).
  • Desired-state store → a single JSON file (atomic writes) behind a Store interface — not git/DB yet; a DB is a drop-in. Server-authoritative.
  • Admin/browser identity → WebAuthn/passkey (YubiKey), not browser mTLS: usernameless discoverable login, UV required, server-side session + CSRF. Bootstrap is a closed CLI (register-admin, one-time HMAC enrol link). The UI cert is Let's Encrypt DNS-01 (mgmt-net host, hot-reloaded). The manager→broker hop stays mTLS (service-to-service, broker admin CA).
  • Authorise pillar → the enrol-broker folded in "alongside": the broker stays the state machine + step-ca signer; the manager drives its admin plane over mTLS (Accept-gated JSON approve) and births the inventory record on approve, recording the human passkey admin as approver.
• Agent identity confirmed: the agent reuses the same TPM identity cert as 802.1X EAP-TLS / NVMe hostnqn — "one identity, three stages". Peer cert serial is authoritative; an unenrolled serial is refused.
• On-host agent — telemetry half BUILT (2026-07-05, files/agent/). The gRPC client that feeds this plane. Reuses the host TPM identity for mTLS (read-only tpmkey.LoadSigner at 0x81000100 + cert from NV 0x01800100; private key never leaves the TPM), opens Session, sends Hello + periodic HealthSnapshot (read-only /proc+/sys probes + fleet-hostapi facts). Telemetry-only: it receives DesiredState/Command but only logs them. Static out-of-VM CI binary, 2b-agent provisioner (enable_agent), fapolicyd-trusted; mTLS proven against the go-tpm simulator (files/agent/simtest).
• NOT yet built (recorded intent): (1) the agent's reconcile actuators — apply kiosk_url, reboot_at/REBOOT, drain — deferred until the telemetry half is bench-verified; and (2) per-host boot-version assignment. desired.target_version is stored + pushed, but changing what a host boots needs dnsmasq cohort steering + multi-version UKI serving (docs/TODO.md #6). The version pin is labelled "recorded intent" in the Lifecycle UI.

Everything below is the original design context; treat the as-built note as authoritative where they differ.

Centralised monitoring + maintenance of the diskless kiosk fleet: health, version, usage, and maintenance actions. This note is the authoritative design; it deliberately reuses what already exists rather than bolting on a generic MDM, because the base architecture (immutable UKI + NVMe-root, TPM identity, no per-host state, segmented network) makes most off-the-shelf kiosk-management tools a poor fit.

Design philosophy (forced by the architecture invariants)

These are consequences of the CLAUDE.md invariants, not preferences:

1. Server-authoritative, hosts are cattle. The root is immutable erofs + dm-verity on a tmpfs overlay — nothing survives reboot. So all durable state (history, inventory, desired-state) lives server-side; the on-host agent is stateless and buffers only in RAM (best-effort).
2. Identity from the TPM, never baked. No per-host state in any artifact (invariant #1). The agent derives identity at runtime from the TPM-held cert (key 0x81000100, cert NV 0x01800100) — the same identity used for 802.1X EAP-TLS and the NVMe hostnqn. No enrollment token in the image.
3. Pull-based control, no inbound surface. The boot LAN is locked down and segmented; we do not open an always-on inbound control port on hosts. Hosts poll a desired-state endpoint and reconcile. This also fits the immutable + digest-pinned model: convergence to a new version = reboot.
4. Reuse the existing buses. Telemetry rides the already-built auditd → audit-collector relay → Wazuh pipeline; version truth is the existing digest-pinned deployment manifest; redeploy/drain mechanics are the SPDK rolling-update flow (TODO #6). The management plane orchestrates these; it does not reinvent them.
5. Minimal, reproducible, sandboxed agent. Same posture as fleet-hostapi (loopback/least-priv, DynamicUser, fapolicyd-trusted, reproducible static build).

Components

1. Identity — mTLS with the TPM cert

The agent presents the TPM-held client cert over mTLS to the management endpoint; the server authenticates the host by cert CN/SAN, which is the canonical fleet identity (= NVMe hostnqn = 802.1X identity). Reuses the step-ca fleet CA (serving/dot1x-bench/). No baked secret; the host also pins the server cert.

Bootstrap (where the cert comes from): mTLS here assumes the host already holds a TPM cert — but an un-enrolled host does not. That chicken-and-egg is resolved by enrolment, which is the management plane's one unauthenticated endpoint and the moment the inventory record is born. See tpm-enrolment-design.md — the enrolment broker is designed to fold into this mgmt service as its bootstrap routes.

• Secure path: full mTLS, TPM cert. Strong, no secrets in the image.
• Open/test path (rd.dot1x=0): fall back to dnsmasq fleet-name + MAC as a weak identity. Lab only — flagged as insufficient for production.

Inventory is keyed by cert serial / hostnqn, joined to the dnsmasq starship name (fleet-namer), MAC, last-seen, running/desired version, and health.

2. On-host agent — extend fleet-hostapi

fleet-hostapi (files/hostapi/) is already the only fleet-authored component on the hardware and already models os/hardware/net/kiosk facts. Add a report/push mode (or a tiny sibling that reads its loopback /v1/host and ships it). Keep it stateless; buffer in tmpfs only; identity from the TPM.

Reported facts:

• Health: CPU/load, GPU act-freq, temps, tmpfs overlay usage (the RAM-exhaustion → NVMe-keepalive risk — the single most important kiosk health signal), host uptime (server derives reboot-pending vs reboot_at), NVMe-root link state, 802.1X/MACsec Authorized, SELinux enforcing, chrony clock health (sync state + NTP source/stratum/offset), kiosk.service active.
• Version: running image version (/etc/fleet-image), rootfs digest, kernel.
• Usage: kiosk URL, session uptime, Chromium reload/crash count (from the kiosk-session restart loop — the exact signal behind the GPU-bug diagnosis), input idle/active (liveness only, never content — privacy-aware), display power state.

Cadence: heartbeat (~30–60 s) + event-driven (crash, auth loss, version change).

3. Transport

Phase 1 — ride the Wazuh pipeline. Emit snapshots as structured JSON events through the existing audit-collector relay → wazuh1. Add fleet_health / fleet_kiosk decoders + rules alongside fleet_audit (serving/siem/config/). Wazuh then provides search, dashboards, and alerting for free, and it's already secured-path-aware.

Phase 2 — metrics push for high-resolution time-series (CPU/GPU/mem trends): push via OTLP or Prometheus remote-write to a collector on the mgmt host → VictoriaMetrics/Prometheus. Push, never scrape — scraping would require an inbound port on every locked host, against the posture.

4. Control / maintenance — pull-based desired-state reconciliation

Each host periodically fetches its desired state (mTLS, keyed by TPM cert): target version, kiosk URL, reboot-at, drain. The agent reconciles; because the root is immutable, convergence = reboot (a reboot re-PXEs and picks up the new UKI).

• The deployment manifest is already the desired-state for versions (one commit per release, digest-pinned). Extend it to per-host / per-cohort desired state. Rollback = revert the commit (unchanged model).
• Maintenance actions (reboot, drain, restart kiosk, change URL, request logs) ride the same poll: the server sets the intent; the host obeys on next poll.
• This dovetails with the SPDK rolling update (TODO #6): "redeploy a cohort" = bump the manifest → spdk-loader serves the new version online → the management plane tells those hosts to drain/reboot → it watches spdk-release list CONNS fall → the old version auto-reaps. The management plane orchestrates this loop.
• No always-on inbound control socket. Admin SSH (key-only ECDSA) stays purely break-glass for exceptional manual ops.

5. Server side (serving host or a dedicated mgmt node, e.g. alongside wazuh1)

• Mgmt service (small Go, mirrors the fleet-hostapi style): mTLS endpoint that accepts host reports and serves per-host desired-state; authenticates hosts by TPM cert via the fleet CA.
• Store: time-series (VictoriaMetrics/Prometheus) for metrics; Wazuh / OpenSearch for events/logs; git or a tiny DB for desired-state + inventory.
• Dashboards: Grafana (metric trends) + the Wazuh dashboard (events/alerts) + a small fleet-overview board (offline hosts, version drift, crash rates, tmpfs pressure). Could be a sibling of the serving/kiosk-app SPA.
• Alerting: host offline > N min; Chromium crash-loop; running ≠ desired version; tmpfs near-full; MACsec/auth down; NVMe-root reconnect storms.

Data model (sketch)

Host (by cert-serial / hostnqn)
  ├─ identity:   cert serial, hostnqn, MAC, starship-name (dnsmasq)
  ├─ version:    image_version, rootfs digest, kernel, desired_version
  ├─ health:     cpu/load, gpu_freq, temps, tmpfs_used%, uptime, nvme_root,
  │              auth/macsec, selinux, chrony (sync + source/stratum/offset),
  │              kiosk_active, last_seen
  ├─ usage:      kiosk_url, session_uptime, chromium_crashes, input_active, display
  └─ desired:    target_version, kiosk_url, reboot_at, drain   (server-authoritative)

Mapping to existing building blocks

Need	Reuse
Host identity	TPM cert / hostnqn / step-ca fleet CA (serving/dot1x-bench/)
On-host data	fleet-hostapi /v1/host (files/hostapi/)
Telemetry + alerting	audit-collector relay → Wazuh (serving/siem/)
Friendly names / inventory	dnsmasq fleet-namer (MAC → name → DNS)
Version truth + rollback	digest-pinned deployment manifest (git)
Redeploy / drain mechanics	SPDK spdk-loader + spdk-release (TODO #6)
Build / sign / publish	Gitea CI + OCI images

Phasing

1. Phase 1 (mostly-built): push fleet-hostapi snapshots as structured events into Wazuh; add decoders/rules + a fleet dashboard → health, version-drift, offline detection, usage, alerting. Reuses the existing pipeline end to end.
2. Phase 2: metrics push (OTLP / remote-write) → VictoriaMetrics + Grafana for trend graphs.
3. Phase 3: mTLS desired-state pull (version pin, reboot/drain, kiosk URL), wired to the deployment manifest + SPDK rolling-update flow.

Build vs buy

Off-the-shelf fits poorly because the base is unusual:

• FleetDM + osquery — closest for the query/health side; worth evaluating. But osquery enrollment carries per-host state, so it must be run carefully (identity from TPM, enrollment in tmpfs only) or it breaks invariant #1.
• Mender / balena / typical kiosk MDM — assume mutable disk + their own OTA + enrollment. We already have a better version mechanism (UKI + NVMe-root + digest manifest), so their OTA is redundant and their disk model conflicts.

Decision: build the thin mgmt service + reconcile loop; reuse Wazuh/Grafana for the heavy lifting. The hard parts (identity, immutable versioning, a telemetry bus) already exist.

Security posture

• mTLS + TPM cert = strong identity, zero baked secrets, reuses the 802.1X CA.
• Pull-based control = no new inbound attack surface on locked hosts.
• Telemetry stays on the mgmt/boot LAN, TLS'd (same gap as the Wazuh-relay-TLS TODO) — no internet egress, air-gap intact.
• Agent stays minimal/sandboxed/fapolicyd-trusted/reproducible (fleet-hostapi template); usage capture is liveness-only, never content (privacy).
• Fleet-wide inventory is recon-valuable → protect the management plane (auth + segmentation); never expose its dashboards to the boot LAN.

Open questions / decisions for build time

• Telemetry encoding: structured syslog/JSON into Wazuh (Phase 1) vs native OTLP from the start. Leaning Wazuh-first for reuse.
• Steady-state transport (agent ↔ manager): the agent is Go, the path is mTLS + host-dialed (preserves "no inbound port"). Fork: bespoke gRPC stream (health up / desired-state down) vs reuse OTLP-gRPC for metrics + Wazuh for events + a thin gRPC control stream — the latter gets VictoriaMetrics/ Grafana/Wazuh for free. Connect (connectrpc.com) if one protobuf IDL serving gRPC + JSON + browser is wanted. Note enrolment deliberately stays plain HTTP/JSON (stdlib, minimal, the one unauth edge) — see tpm-enrolment-design.md "Implementation".
• Metrics store: VictoriaMetrics vs Prometheus (VM is lighter for push + long retention).
• Desired-state store: git (GitOps, aud/rollback for free, matches the deployment-manifest model) vs a small DB (richer per-host scheduling).
• Per-host vs per-cohort desired state granularity (cohorts likely enough; per-host for canaries).
• Open-path identity: MAC/fleet-name is weak — gate the management plane to the secure path, or accept lab-only for the open path.
• osquery/FleetDM evaluation for the read side before building a bespoke collector.