fixing final SPDK deployment
This commit is contained in:
Audiobuzz
@@ -1126,3 +1126,20 @@ the wrapper, own the VA-API `--enable-features`:
|
||||
remains documented here if a minimal/auditable cmdline is ever wanted.
|
||||
**BENCH-VERIFY:** `chrome://policy` shows `ExtensionInstallBlocklist=["*"]` active;
|
||||
extensions can't install.
|
||||
|
||||
## [ ] 14. (DESIGN) Centralised fleet management plane (health/version/usage/maintenance)
|
||||
|
||||
**DESIGN CAPTURED 2026-06-28 — [docs/fleet-management-design.md](fleet-management-design.md).**
|
||||
Server-authoritative, pull-based, TPM-identified management plane reusing existing
|
||||
pieces: `fleet-hostapi` as the on-host data source, the audit-collector→Wazuh
|
||||
pipeline as telemetry transport, the digest-pinned deployment manifest as version
|
||||
truth, and the SPDK `spdk-loader`/`spdk-release` rolling-update flow (#6) as the
|
||||
redeploy/drain mechanics. Hosts stay stateless (identity from the TPM cert, no
|
||||
baked enrollment); control is pull/reconcile (no inbound surface); convergence to
|
||||
a new version = reboot. **Phased build:** (1) push hostapi snapshots as structured
|
||||
events into Wazuh + fleet dashboard (health/version-drift/offline/usage/alerting);
|
||||
(2) metrics push (OTLP/remote-write) → VictoriaMetrics + Grafana; (3) mTLS
|
||||
desired-state pull (version pin, reboot/drain, kiosk URL) wired to the deployment
|
||||
manifest + SPDK flow. Build-time decisions (telemetry encoding, metrics/desired-
|
||||
state store, per-host vs cohort, osquery/FleetDM eval) listed in the design note.
|
||||
Depends on the step-ca fleet CA (#8) for the mTLS identity.
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
# Fleet management plane — design note
|
||||
|
||||
**Status:** DESIGN (not built). Captured 2026-06-28. Build phased — see "Phasing"
|
||||
and `docs/TODO.md` #14.
|
||||
|
||||
Centralised monitoring + maintenance of the diskless kiosk fleet: **health,
|
||||
version, usage, and maintenance actions**. This note is the authoritative design;
|
||||
it deliberately reuses what already exists rather than bolting on a generic MDM,
|
||||
because the base architecture (immutable UKI + NVMe-root, TPM identity, no
|
||||
per-host state, segmented network) makes most off-the-shelf kiosk-management tools
|
||||
a poor fit.
|
||||
|
||||
## Design philosophy (forced by the architecture invariants)
|
||||
|
||||
These are consequences of the CLAUDE.md invariants, not preferences:
|
||||
|
||||
1. **Server-authoritative, hosts are cattle.** The root is immutable erofs +
|
||||
dm-verity on a tmpfs overlay — nothing survives reboot. So **all durable state
|
||||
(history, inventory, desired-state) lives server-side**; the on-host agent is
|
||||
stateless and buffers only in RAM (best-effort).
|
||||
2. **Identity from the TPM, never baked.** No per-host state in any artifact
|
||||
(invariant #1). The agent derives identity at runtime from the **TPM-held
|
||||
cert** (key `0x81010001`, cert NV `0x01800001`) — the same identity used for
|
||||
802.1X EAP-TLS and the NVMe hostnqn. No enrollment token in the image.
|
||||
3. **Pull-based control, no inbound surface.** The boot LAN is locked down and
|
||||
segmented; we do not open an always-on inbound control port on hosts. Hosts
|
||||
**poll** a desired-state endpoint and reconcile. This also fits the immutable +
|
||||
digest-pinned model: *convergence to a new version = reboot*.
|
||||
4. **Reuse the existing buses.** Telemetry rides the already-built
|
||||
`auditd → audit-collector relay → Wazuh` pipeline; version truth is the
|
||||
existing digest-pinned deployment manifest; redeploy/drain mechanics are the
|
||||
SPDK rolling-update flow (TODO #6). The management plane orchestrates these; it
|
||||
does not reinvent them.
|
||||
5. **Minimal, reproducible, sandboxed agent.** Same posture as `fleet-hostapi`
|
||||
(loopback/least-priv, `DynamicUser`, fapolicyd-trusted, reproducible static
|
||||
build).
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Identity — mTLS with the TPM cert
|
||||
|
||||
The agent presents the TPM-held client cert over **mTLS** to the management
|
||||
endpoint; the server authenticates the host by cert CN/SAN, which is the canonical
|
||||
fleet identity (= NVMe hostnqn = 802.1X identity). Reuses the **step-ca fleet CA**
|
||||
(`serving/dot1x-bench/`). No baked secret; the host also pins the server cert.
|
||||
|
||||
- **Secure path:** full mTLS, TPM cert. Strong, no secrets in the image.
|
||||
- **Open/test path (`rd.dot1x=0`):** fall back to dnsmasq fleet-name + MAC as a
|
||||
weak identity. **Lab only** — flagged as insufficient for production.
|
||||
|
||||
Inventory is keyed by **cert serial / hostnqn**, joined to the dnsmasq starship
|
||||
name (`fleet-namer`), MAC, last-seen, running/desired version, and health.
|
||||
|
||||
### 2. On-host agent — extend `fleet-hostapi`
|
||||
|
||||
`fleet-hostapi` (`files/hostapi/`) is already the only fleet-authored component on
|
||||
the hardware and already models os/hardware/net/kiosk facts. Add a **report/push
|
||||
mode** (or a tiny sibling that reads its loopback `/v1/host` and ships it). Keep it
|
||||
stateless; buffer in tmpfs only; identity from the TPM.
|
||||
|
||||
Reported facts:
|
||||
|
||||
- **Health:** CPU/load, GPU act-freq, temps, **tmpfs overlay usage** (the
|
||||
RAM-exhaustion → NVMe-keepalive risk — the single most important kiosk health
|
||||
signal), NVMe-root link state, 802.1X/MACsec `Authorized`, SELinux enforcing,
|
||||
chrony sync, `kiosk.service` active.
|
||||
- **Version:** running image version (`/etc/fleet-image`), rootfs digest, kernel.
|
||||
- **Usage:** kiosk URL, session uptime, **Chromium reload/crash count** (from the
|
||||
`kiosk-session` restart loop — the exact signal behind the GPU-bug diagnosis),
|
||||
input idle/active (liveness only, never content — privacy-aware), display power
|
||||
state.
|
||||
|
||||
Cadence: heartbeat (~30–60 s) + event-driven (crash, auth loss, version change).
|
||||
|
||||
### 3. Transport
|
||||
|
||||
**Phase 1 — ride the Wazuh pipeline.** Emit snapshots as structured JSON events
|
||||
through the existing `audit-collector` relay → `wazuh1`. Add `fleet_health` /
|
||||
`fleet_kiosk` decoders + rules alongside `fleet_audit` (`serving/siem/config/`).
|
||||
Wazuh then provides search, dashboards, and alerting for free, and it's already
|
||||
secured-path-aware.
|
||||
|
||||
**Phase 2 — metrics push** for high-resolution time-series (CPU/GPU/mem trends):
|
||||
**push** via OTLP or Prometheus remote-write to a collector on the mgmt host →
|
||||
VictoriaMetrics/Prometheus. **Push, never scrape** — scraping would require an
|
||||
inbound port on every locked host, against the posture.
|
||||
|
||||
### 4. Control / maintenance — pull-based desired-state reconciliation
|
||||
|
||||
Each host periodically fetches **its** desired state (mTLS, keyed by TPM cert):
|
||||
target version, kiosk URL, `reboot-at`, `drain`. The agent reconciles; because the
|
||||
root is immutable, **convergence = reboot** (a reboot re-PXEs and picks up the new
|
||||
UKI).
|
||||
|
||||
- The **deployment manifest is already the desired-state for versions** (one
|
||||
commit per release, digest-pinned). Extend it to per-host / per-cohort desired
|
||||
state. Rollback = revert the commit (unchanged model).
|
||||
- **Maintenance actions** (reboot, drain, restart kiosk, change URL, request logs)
|
||||
ride the same poll: the server sets the intent; the host obeys on next poll.
|
||||
- This dovetails with the **SPDK rolling update** (TODO #6): "redeploy a cohort" =
|
||||
bump the manifest → `spdk-loader` serves the new version online → the management
|
||||
plane tells those hosts to drain/reboot → it watches `spdk-release list` CONNS
|
||||
fall → the old version auto-reaps. The management plane *orchestrates* this loop.
|
||||
- **No always-on inbound control socket.** Admin SSH (key-only ECDSA) stays purely
|
||||
break-glass for exceptional manual ops.
|
||||
|
||||
### 5. Server side (serving host or a dedicated mgmt node, e.g. alongside `wazuh1`)
|
||||
|
||||
- **Mgmt service** (small Go, mirrors the `fleet-hostapi` style): mTLS endpoint
|
||||
that accepts host reports and serves per-host desired-state; authenticates hosts
|
||||
by TPM cert via the fleet CA.
|
||||
- **Store:** time-series (VictoriaMetrics/Prometheus) for metrics; Wazuh /
|
||||
OpenSearch for events/logs; git or a tiny DB for desired-state + inventory.
|
||||
- **Dashboards:** Grafana (metric trends) + the Wazuh dashboard (events/alerts) +
|
||||
a small fleet-overview board (offline hosts, version drift, crash rates, tmpfs
|
||||
pressure). Could be a sibling of the `serving/kiosk-app` SPA.
|
||||
- **Alerting:** host offline > N min; Chromium crash-loop; **running ≠ desired
|
||||
version**; tmpfs near-full; MACsec/auth down; NVMe-root reconnect storms.
|
||||
|
||||
## Data model (sketch)
|
||||
|
||||
```
|
||||
Host (by cert-serial / hostnqn)
|
||||
├─ identity: cert serial, hostnqn, MAC, starship-name (dnsmasq)
|
||||
├─ version: image_version, rootfs digest, kernel, desired_version
|
||||
├─ health: cpu/load, gpu_freq, temps, tmpfs_used%, nvme_root, auth/macsec,
|
||||
│ selinux, chrony, kiosk_active, last_seen
|
||||
├─ usage: kiosk_url, session_uptime, chromium_crashes, input_active, display
|
||||
└─ desired: target_version, kiosk_url, reboot_at, drain (server-authoritative)
|
||||
```
|
||||
|
||||
## Mapping to existing building blocks
|
||||
|
||||
| Need | Reuse |
|
||||
|---|---|
|
||||
| Host identity | TPM cert / hostnqn / step-ca fleet CA (`serving/dot1x-bench/`) |
|
||||
| On-host data | `fleet-hostapi` `/v1/host` (`files/hostapi/`) |
|
||||
| Telemetry + alerting | audit-collector relay → Wazuh (`serving/siem/`) |
|
||||
| Friendly names / inventory | dnsmasq `fleet-namer` (MAC → name → DNS) |
|
||||
| Version truth + rollback | digest-pinned deployment manifest (git) |
|
||||
| Redeploy / drain mechanics | SPDK `spdk-loader` + `spdk-release` (TODO #6) |
|
||||
| Build / sign / publish | Gitea CI + OCI images |
|
||||
|
||||
## Phasing
|
||||
|
||||
1. **Phase 1 (mostly-built):** push `fleet-hostapi` snapshots as structured events
|
||||
into Wazuh; add decoders/rules + a fleet dashboard → health, version-drift,
|
||||
offline detection, usage, alerting. Reuses the existing pipeline end to end.
|
||||
2. **Phase 2:** metrics push (OTLP / remote-write) → VictoriaMetrics + Grafana for
|
||||
trend graphs.
|
||||
3. **Phase 3:** mTLS desired-state pull (version pin, reboot/drain, kiosk URL),
|
||||
wired to the deployment manifest + SPDK rolling-update flow.
|
||||
|
||||
## Build vs buy
|
||||
|
||||
Off-the-shelf fits poorly because the base is unusual:
|
||||
|
||||
- **FleetDM + osquery** — closest for the query/health side; worth evaluating. But
|
||||
osquery enrollment carries per-host state, so it must be run carefully (identity
|
||||
from TPM, enrollment in tmpfs only) or it breaks invariant #1.
|
||||
- **Mender / balena / typical kiosk MDM** — assume mutable disk + their own OTA +
|
||||
enrollment. We already have a *better* version mechanism (UKI + NVMe-root +
|
||||
digest manifest), so their OTA is redundant and their disk model conflicts.
|
||||
|
||||
**Decision: build the thin mgmt service + reconcile loop; reuse Wazuh/Grafana for
|
||||
the heavy lifting.** The hard parts (identity, immutable versioning, a telemetry
|
||||
bus) already exist.
|
||||
|
||||
## Security posture
|
||||
|
||||
- mTLS + TPM cert = strong identity, zero baked secrets, reuses the 802.1X CA.
|
||||
- Pull-based control = **no new inbound attack surface** on locked hosts.
|
||||
- Telemetry stays on the mgmt/boot LAN, TLS'd (same gap as the Wazuh-relay-TLS
|
||||
TODO) — no internet egress, air-gap intact.
|
||||
- Agent stays minimal/sandboxed/fapolicyd-trusted/reproducible (`fleet-hostapi`
|
||||
template); usage capture is liveness-only, never content (privacy).
|
||||
- Fleet-wide inventory is recon-valuable → protect the management plane (auth +
|
||||
segmentation); never expose its dashboards to the boot LAN.
|
||||
|
||||
## Open questions / decisions for build time
|
||||
|
||||
- **Telemetry encoding:** structured syslog/JSON into Wazuh (Phase 1) vs native
|
||||
OTLP from the start. Leaning Wazuh-first for reuse.
|
||||
- **Metrics store:** VictoriaMetrics vs Prometheus (VM is lighter for push +
|
||||
long retention).
|
||||
- **Desired-state store:** git (GitOps, aud/rollback for free, matches the
|
||||
deployment-manifest model) vs a small DB (richer per-host scheduling).
|
||||
- **Per-host vs per-cohort** desired state granularity (cohorts likely enough;
|
||||
per-host for canaries).
|
||||
- **Open-path identity:** MAC/fleet-name is weak — gate the management plane to the
|
||||
secure path, or accept lab-only for the open path.
|
||||
- **osquery/FleetDM** evaluation for the read side before building a bespoke
|
||||
collector.
|
||||
Reference in New Issue
Block a user