From bc14b8986db582026551c406655d16b5b71e0dc7 Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Sun, 28 Jun 2026 08:44:49 +0000 Subject: [PATCH] fixing final SPDK deployment --- docs/TODO.md | 17 +++ docs/fleet-management-design.md | 193 ++++++++++++++++++++++++++++++++ serving/docker-compose.yml | 17 ++- serving/kiosk-app | 2 +- serving/spdk/spdk-release.sh | 11 +- serving/spdk/spdk-target.sh | 15 ++- 6 files changed, 243 insertions(+), 12 deletions(-) create mode 100644 docs/fleet-management-design.md diff --git a/docs/TODO.md b/docs/TODO.md index 4f10069..922c6bc 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1126,3 +1126,20 @@ the wrapper, own the VA-API `--enable-features`: remains documented here if a minimal/auditable cmdline is ever wanted. **BENCH-VERIFY:** `chrome://policy` shows `ExtensionInstallBlocklist=["*"]` active; extensions can't install. + +## [ ] 14. (DESIGN) Centralised fleet management plane (health/version/usage/maintenance) + +**DESIGN CAPTURED 2026-06-28 — [docs/fleet-management-design.md](fleet-management-design.md).** +Server-authoritative, pull-based, TPM-identified management plane reusing existing +pieces: `fleet-hostapi` as the on-host data source, the audit-collector→Wazuh +pipeline as telemetry transport, the digest-pinned deployment manifest as version +truth, and the SPDK `spdk-loader`/`spdk-release` rolling-update flow (#6) as the +redeploy/drain mechanics. Hosts stay stateless (identity from the TPM cert, no +baked enrollment); control is pull/reconcile (no inbound surface); convergence to +a new version = reboot. **Phased build:** (1) push hostapi snapshots as structured +events into Wazuh + fleet dashboard (health/version-drift/offline/usage/alerting); +(2) metrics push (OTLP/remote-write) → VictoriaMetrics + Grafana; (3) mTLS +desired-state pull (version pin, reboot/drain, kiosk URL) wired to the deployment +manifest + SPDK flow. Build-time decisions (telemetry encoding, metrics/desired- +state store, per-host vs cohort, osquery/FleetDM eval) listed in the design note. +Depends on the step-ca fleet CA (#8) for the mTLS identity. diff --git a/docs/fleet-management-design.md b/docs/fleet-management-design.md new file mode 100644 index 0000000..f115dcf --- /dev/null +++ b/docs/fleet-management-design.md @@ -0,0 +1,193 @@ +# Fleet management plane — design note + +**Status:** DESIGN (not built). Captured 2026-06-28. Build phased — see "Phasing" +and `docs/TODO.md` #14. + +Centralised monitoring + maintenance of the diskless kiosk fleet: **health, +version, usage, and maintenance actions**. This note is the authoritative design; +it deliberately reuses what already exists rather than bolting on a generic MDM, +because the base architecture (immutable UKI + NVMe-root, TPM identity, no +per-host state, segmented network) makes most off-the-shelf kiosk-management tools +a poor fit. + +## Design philosophy (forced by the architecture invariants) + +These are consequences of the CLAUDE.md invariants, not preferences: + +1. **Server-authoritative, hosts are cattle.** The root is immutable erofs + + dm-verity on a tmpfs overlay — nothing survives reboot. So **all durable state + (history, inventory, desired-state) lives server-side**; the on-host agent is + stateless and buffers only in RAM (best-effort). +2. **Identity from the TPM, never baked.** No per-host state in any artifact + (invariant #1). The agent derives identity at runtime from the **TPM-held + cert** (key `0x81010001`, cert NV `0x01800001`) — the same identity used for + 802.1X EAP-TLS and the NVMe hostnqn. No enrollment token in the image. +3. **Pull-based control, no inbound surface.** The boot LAN is locked down and + segmented; we do not open an always-on inbound control port on hosts. Hosts + **poll** a desired-state endpoint and reconcile. This also fits the immutable + + digest-pinned model: *convergence to a new version = reboot*. +4. **Reuse the existing buses.** Telemetry rides the already-built + `auditd → audit-collector relay → Wazuh` pipeline; version truth is the + existing digest-pinned deployment manifest; redeploy/drain mechanics are the + SPDK rolling-update flow (TODO #6). The management plane orchestrates these; it + does not reinvent them. +5. **Minimal, reproducible, sandboxed agent.** Same posture as `fleet-hostapi` + (loopback/least-priv, `DynamicUser`, fapolicyd-trusted, reproducible static + build). + +## Components + +### 1. Identity — mTLS with the TPM cert + +The agent presents the TPM-held client cert over **mTLS** to the management +endpoint; the server authenticates the host by cert CN/SAN, which is the canonical +fleet identity (= NVMe hostnqn = 802.1X identity). Reuses the **step-ca fleet CA** +(`serving/dot1x-bench/`). No baked secret; the host also pins the server cert. + +- **Secure path:** full mTLS, TPM cert. Strong, no secrets in the image. +- **Open/test path (`rd.dot1x=0`):** fall back to dnsmasq fleet-name + MAC as a + weak identity. **Lab only** — flagged as insufficient for production. + +Inventory is keyed by **cert serial / hostnqn**, joined to the dnsmasq starship +name (`fleet-namer`), MAC, last-seen, running/desired version, and health. + +### 2. On-host agent — extend `fleet-hostapi` + +`fleet-hostapi` (`files/hostapi/`) is already the only fleet-authored component on +the hardware and already models os/hardware/net/kiosk facts. Add a **report/push +mode** (or a tiny sibling that reads its loopback `/v1/host` and ships it). Keep it +stateless; buffer in tmpfs only; identity from the TPM. + +Reported facts: + +- **Health:** CPU/load, GPU act-freq, temps, **tmpfs overlay usage** (the + RAM-exhaustion → NVMe-keepalive risk — the single most important kiosk health + signal), NVMe-root link state, 802.1X/MACsec `Authorized`, SELinux enforcing, + chrony sync, `kiosk.service` active. +- **Version:** running image version (`/etc/fleet-image`), rootfs digest, kernel. +- **Usage:** kiosk URL, session uptime, **Chromium reload/crash count** (from the + `kiosk-session` restart loop — the exact signal behind the GPU-bug diagnosis), + input idle/active (liveness only, never content — privacy-aware), display power + state. + +Cadence: heartbeat (~30–60 s) + event-driven (crash, auth loss, version change). + +### 3. Transport + +**Phase 1 — ride the Wazuh pipeline.** Emit snapshots as structured JSON events +through the existing `audit-collector` relay → `wazuh1`. Add `fleet_health` / +`fleet_kiosk` decoders + rules alongside `fleet_audit` (`serving/siem/config/`). +Wazuh then provides search, dashboards, and alerting for free, and it's already +secured-path-aware. + +**Phase 2 — metrics push** for high-resolution time-series (CPU/GPU/mem trends): +**push** via OTLP or Prometheus remote-write to a collector on the mgmt host → +VictoriaMetrics/Prometheus. **Push, never scrape** — scraping would require an +inbound port on every locked host, against the posture. + +### 4. Control / maintenance — pull-based desired-state reconciliation + +Each host periodically fetches **its** desired state (mTLS, keyed by TPM cert): +target version, kiosk URL, `reboot-at`, `drain`. The agent reconciles; because the +root is immutable, **convergence = reboot** (a reboot re-PXEs and picks up the new +UKI). + +- The **deployment manifest is already the desired-state for versions** (one + commit per release, digest-pinned). Extend it to per-host / per-cohort desired + state. Rollback = revert the commit (unchanged model). +- **Maintenance actions** (reboot, drain, restart kiosk, change URL, request logs) + ride the same poll: the server sets the intent; the host obeys on next poll. +- This dovetails with the **SPDK rolling update** (TODO #6): "redeploy a cohort" = + bump the manifest → `spdk-loader` serves the new version online → the management + plane tells those hosts to drain/reboot → it watches `spdk-release list` CONNS + fall → the old version auto-reaps. The management plane *orchestrates* this loop. +- **No always-on inbound control socket.** Admin SSH (key-only ECDSA) stays purely + break-glass for exceptional manual ops. + +### 5. Server side (serving host or a dedicated mgmt node, e.g. alongside `wazuh1`) + +- **Mgmt service** (small Go, mirrors the `fleet-hostapi` style): mTLS endpoint + that accepts host reports and serves per-host desired-state; authenticates hosts + by TPM cert via the fleet CA. +- **Store:** time-series (VictoriaMetrics/Prometheus) for metrics; Wazuh / + OpenSearch for events/logs; git or a tiny DB for desired-state + inventory. +- **Dashboards:** Grafana (metric trends) + the Wazuh dashboard (events/alerts) + + a small fleet-overview board (offline hosts, version drift, crash rates, tmpfs + pressure). Could be a sibling of the `serving/kiosk-app` SPA. +- **Alerting:** host offline > N min; Chromium crash-loop; **running ≠ desired + version**; tmpfs near-full; MACsec/auth down; NVMe-root reconnect storms. + +## Data model (sketch) + +``` +Host (by cert-serial / hostnqn) + ├─ identity: cert serial, hostnqn, MAC, starship-name (dnsmasq) + ├─ version: image_version, rootfs digest, kernel, desired_version + ├─ health: cpu/load, gpu_freq, temps, tmpfs_used%, nvme_root, auth/macsec, + │ selinux, chrony, kiosk_active, last_seen + ├─ usage: kiosk_url, session_uptime, chromium_crashes, input_active, display + └─ desired: target_version, kiosk_url, reboot_at, drain (server-authoritative) +``` + +## Mapping to existing building blocks + +| Need | Reuse | +|---|---| +| Host identity | TPM cert / hostnqn / step-ca fleet CA (`serving/dot1x-bench/`) | +| On-host data | `fleet-hostapi` `/v1/host` (`files/hostapi/`) | +| Telemetry + alerting | audit-collector relay → Wazuh (`serving/siem/`) | +| Friendly names / inventory | dnsmasq `fleet-namer` (MAC → name → DNS) | +| Version truth + rollback | digest-pinned deployment manifest (git) | +| Redeploy / drain mechanics | SPDK `spdk-loader` + `spdk-release` (TODO #6) | +| Build / sign / publish | Gitea CI + OCI images | + +## Phasing + +1. **Phase 1 (mostly-built):** push `fleet-hostapi` snapshots as structured events + into Wazuh; add decoders/rules + a fleet dashboard → health, version-drift, + offline detection, usage, alerting. Reuses the existing pipeline end to end. +2. **Phase 2:** metrics push (OTLP / remote-write) → VictoriaMetrics + Grafana for + trend graphs. +3. **Phase 3:** mTLS desired-state pull (version pin, reboot/drain, kiosk URL), + wired to the deployment manifest + SPDK rolling-update flow. + +## Build vs buy + +Off-the-shelf fits poorly because the base is unusual: + +- **FleetDM + osquery** — closest for the query/health side; worth evaluating. But + osquery enrollment carries per-host state, so it must be run carefully (identity + from TPM, enrollment in tmpfs only) or it breaks invariant #1. +- **Mender / balena / typical kiosk MDM** — assume mutable disk + their own OTA + + enrollment. We already have a *better* version mechanism (UKI + NVMe-root + + digest manifest), so their OTA is redundant and their disk model conflicts. + +**Decision: build the thin mgmt service + reconcile loop; reuse Wazuh/Grafana for +the heavy lifting.** The hard parts (identity, immutable versioning, a telemetry +bus) already exist. + +## Security posture + +- mTLS + TPM cert = strong identity, zero baked secrets, reuses the 802.1X CA. +- Pull-based control = **no new inbound attack surface** on locked hosts. +- Telemetry stays on the mgmt/boot LAN, TLS'd (same gap as the Wazuh-relay-TLS + TODO) — no internet egress, air-gap intact. +- Agent stays minimal/sandboxed/fapolicyd-trusted/reproducible (`fleet-hostapi` + template); usage capture is liveness-only, never content (privacy). +- Fleet-wide inventory is recon-valuable → protect the management plane (auth + + segmentation); never expose its dashboards to the boot LAN. + +## Open questions / decisions for build time + +- **Telemetry encoding:** structured syslog/JSON into Wazuh (Phase 1) vs native + OTLP from the start. Leaning Wazuh-first for reuse. +- **Metrics store:** VictoriaMetrics vs Prometheus (VM is lighter for push + + long retention). +- **Desired-state store:** git (GitOps, aud/rollback for free, matches the + deployment-manifest model) vs a small DB (richer per-host scheduling). +- **Per-host vs per-cohort** desired state granularity (cohorts likely enough; + per-host for canaries). +- **Open-path identity:** MAC/fleet-name is weak — gate the management plane to the + secure path, or accept lab-only for the open path. +- **osquery/FleetDM** evaluation for the read side before building a bespoke + collector. diff --git a/serving/docker-compose.yml b/serving/docker-compose.yml index c48edec..dd31d46 100644 --- a/serving/docker-compose.yml +++ b/serving/docker-compose.yml @@ -285,13 +285,18 @@ services: TARGET_PORT: ${TARGET_PORT} SPDK_CPUMASK: ${SPDK_CPUMASK} NVMEOF_SVC: nvmeof - # Healthy once the RPC socket answers — gates the spdk-loader's startup. + # Healthy once spdk-target has written its readiness stamp (transport up + + # releases reconciled). A FILE check, not a python rpc.py probe: nvmf_tgt does + # a slow DPDK init (~50s observed) and busy-polls its reactor core, so a python + # healthcheck starves/times out and the container sticks at "unhealthy" even + # though the target is up. `test -f` is instant and contention-free. The stamp + # is removed on (re)start + exit, so it never falsely reports a dead target. healthcheck: - test: ["CMD-SHELL", "rpc.py -s /var/run/spdk/spdk.sock framework_get_config >/dev/null 2>&1 || exit 1"] - interval: 5s - timeout: 3s - retries: 24 - start_period: 10s + test: ["CMD-SHELL", "test -f /var/run/spdk/ready || exit 1"] + interval: 10s + timeout: 5s + retries: 6 + start_period: 120s # Persistent target: started ONCE, never restarted for a version change. # Rolling updates add/remove versions online (spdk-loader, or spdk-ctl.sh). entrypoint: ["/bin/bash", "/usr/local/bin/spdk-target"] diff --git a/serving/kiosk-app b/serving/kiosk-app index 538c7eb..4579d22 160000 --- a/serving/kiosk-app +++ b/serving/kiosk-app @@ -1 +1 @@ -Subproject commit 538c7eb87d67bc2839f55ed246de9f3740a2bc56 +Subproject commit 4579d227b096368469c7b5439d9a1b0f9858ba67 diff --git a/serving/spdk/spdk-release.sh b/serving/spdk/spdk-release.sh index fdb1192..d83e50a 100755 --- a/serving/spdk/spdk-release.sh +++ b/serving/spdk/spdk-release.sh @@ -98,8 +98,12 @@ except Exception: print(-1)' cmd_add() { local v=${1:?usage: add } local dir="$RELEASES_DIR/$v" - [ -s "$dir/rootfs.erofs" ] || die "$dir/rootfs.erofs missing/empty" - [ -s "$dir/rootfs.hash" ] || die "$dir/rootfs.hash missing/empty" + # Validation failures here RETURN (not die/exit): cmd_add is called per-dir by + # cmd_reconcile as `cmd_add … || warn`, and an `exit` inside a function aborts + # the WHOLE process (|| only catches a return), so one bad/stale dir would stop + # every OTHER release from being added. Return so reconcile skips just this one. + [ -s "$dir/rootfs.erofs" ] || { echo "ERROR: $dir/rootfs.erofs missing/empty" >&2; return 1; } + [ -s "$dir/rootfs.hash" ] || { echo "ERROR: $dir/rootfs.hash missing/empty" >&2; return 1; } # Guard: the dir name must match the version inside verity.json. A mismatch # means a stale/symlinked or mislabeled release dir (e.g. an old symlink into @@ -112,7 +116,8 @@ cmd_add() { local vv vv=$(vfield "$v" .image_version) if [ -n "$vv" ] && [ "${vv,,}" != "${v,,}" ]; then - die "release dir '$v' holds verity.json for version '$vv' (stale symlink / mislabeled dir). Refusing to add — remove '$dir' and re-stage." + echo "ERROR: release dir '$v' holds verity.json for version '$vv' (stale symlink / mislabeled dir). Refusing to add — remove '$dir' and re-stage." >&2 + return 1 fi local nqn du hu s diff --git a/serving/spdk/spdk-target.sh b/serving/spdk/spdk-target.sh index 22021a5..4280b9b 100755 --- a/serving/spdk/spdk-target.sh +++ b/serving/spdk/spdk-target.sh @@ -19,6 +19,12 @@ TADDR="${TARGET_ADDR:-0.0.0.0}" TPORT="${TARGET_PORT:-4420}" HUGEPAGES="${HUGEPAGES:-1024}" DISCOVERY_NQN="nqn.2014-08.org.nvmexpress.discovery" +# Readiness stamp for the container healthcheck. Written ONLY after the transport +# is up and releases are reconciled; removed on (re)start + exit. A file check is +# instant and python-free — the right probe for a box where SPDK busy-polls its +# (often single) reactor core and a python rpc.py healthcheck would starve/time +# out. Lives on the shared /run/spdk volume so it's cleared per container start. +READY_FILE="${READY_FILE:-/var/run/spdk/ready}" TGT=$(command -v nvmf_tgt || echo /usr/libexec/spdk/spdk/build/bin/nvmf_tgt) RPC=$(command -v rpc.py || echo /usr/libexec/spdk/scripts/rpc.py) @@ -29,6 +35,8 @@ RELEASE_CTL=$(command -v spdk-release || echo /usr/local/bin/spdk-release) RPC_SOCK="${RPC_SOCK:-}" rpc() { if [ -n "$RPC_SOCK" ]; then "$RPC" -s "$RPC_SOCK" "$@"; else "$RPC" "$@"; fi; } [ -n "$RPC_SOCK" ] && mkdir -p "$(dirname "$RPC_SOCK")" +mkdir -p "$(dirname "$READY_FILE")" +rm -f "$READY_FILE" # clear any stale readiness from a previous container start # Ensure hugepages exist — DPDK needs them. vm.nr_hugepages isn't namespaced, so # a privileged container writes the host value; re-applied after a host reboot @@ -51,8 +59,8 @@ TGT_ARGS=(-m "$CPUMASK") TGT_PID=$! # A TERM/INT here stops the target and DROPS EVERY connection — only on container # stop, never for a version change. Make the intent loud. -trap 'echo "==> stopping nvmf_tgt — this drops ALL host root connections" >&2; kill "$TGT_PID" 2>/dev/null || true' TERM INT -trap 'kill "$TGT_PID" 2>/dev/null || true' EXIT +trap 'echo "==> stopping nvmf_tgt — this drops ALL host root connections" >&2; rm -f "$READY_FILE"; kill "$TGT_PID" 2>/dev/null || true' TERM INT +trap 'rm -f "$READY_FILE"; kill "$TGT_PID" 2>/dev/null || true' EXIT # Wait for the RPC socket. for _ in $(seq 1 30); do @@ -91,6 +99,9 @@ echo "==> reconciling releases under $RELEASES_DIR" RELEASES_DIR="$RELEASES_DIR" TARGET_ADDR="$TADDR" TARGET_PORT="$TPORT" RPC_SOCK="$RPC_SOCK" \ bash "$RELEASE_CTL" reconcile || echo "WARNING: reconcile reported errors" >&2 +# Signal readiness for the healthcheck — transport is up and releases reconciled. +: > "$READY_FILE" + echo "==> nvmf_tgt up; releases served online. Add/remove versions with:" echo " docker compose exec ${NVMEOF_SVC:-nvmeof} spdk-release {add|remove|list} " echo " or host-side: serving/spdk/spdk-ctl.sh {load|unload|list} "