fixing final SPDK deployment

docs/TODO.md

@@ -1126,3 +1126,20 @@ the wrapper, own the VA-API `--enable-features`:
 remains documented here if a minimal/auditable cmdline is ever wanted.
 **BENCH-VERIFY:** `chrome://policy` shows `ExtensionInstallBlocklist=["*"]` active;
 extensions can't install.
+
+## [ ] 14. (DESIGN) Centralised fleet management plane (health/version/usage/maintenance)
+
+**DESIGN CAPTURED 2026-06-28 — [docs/fleet-management-design.md](fleet-management-design.md).**
+Server-authoritative, pull-based, TPM-identified management plane reusing existing
+pieces: `fleet-hostapi` as the on-host data source, the audit-collector→Wazuh
+pipeline as telemetry transport, the digest-pinned deployment manifest as version
+truth, and the SPDK `spdk-loader`/`spdk-release` rolling-update flow (#6) as the
+redeploy/drain mechanics. Hosts stay stateless (identity from the TPM cert, no
+baked enrollment); control is pull/reconcile (no inbound surface); convergence to
+a new version = reboot. **Phased build:** (1) push hostapi snapshots as structured
+events into Wazuh + fleet dashboard (health/version-drift/offline/usage/alerting);
+(2) metrics push (OTLP/remote-write) → VictoriaMetrics + Grafana; (3) mTLS
+desired-state pull (version pin, reboot/drain, kiosk URL) wired to the deployment
+manifest + SPDK flow. Build-time decisions (telemetry encoding, metrics/desired-
+state store, per-host vs cohort, osquery/FleetDM eval) listed in the design note.
+Depends on the step-ca fleet CA (#8) for the mTLS identity.

docs/fleet-management-design.md

@@ -0,0 +1,193 @@
+# Fleet management plane — design note
+
+**Status:** DESIGN (not built). Captured 2026-06-28. Build phased — see "Phasing"
+and `docs/TODO.md` #14.
+
+Centralised monitoring + maintenance of the diskless kiosk fleet: **health,
+version, usage, and maintenance actions**. This note is the authoritative design;
+it deliberately reuses what already exists rather than bolting on a generic MDM,
+because the base architecture (immutable UKI + NVMe-root, TPM identity, no
+per-host state, segmented network) makes most off-the-shelf kiosk-management tools
+a poor fit.
+
+## Design philosophy (forced by the architecture invariants)
+
+These are consequences of the CLAUDE.md invariants, not preferences:
+
+1. **Server-authoritative, hosts are cattle.** The root is immutable erofs +
+   dm-verity on a tmpfs overlay — nothing survives reboot. So **all durable state
+   (history, inventory, desired-state) lives server-side**; the on-host agent is
+   stateless and buffers only in RAM (best-effort).
+2. **Identity from the TPM, never baked.** No per-host state in any artifact
+   (invariant #1). The agent derives identity at runtime from the **TPM-held
+   cert** (key `0x81010001`, cert NV `0x01800001`) — the same identity used for
+   802.1X EAP-TLS and the NVMe hostnqn. No enrollment token in the image.
+3. **Pull-based control, no inbound surface.** The boot LAN is locked down and
+   segmented; we do not open an always-on inbound control port on hosts. Hosts
+   **poll** a desired-state endpoint and reconcile. This also fits the immutable +
+   digest-pinned model: *convergence to a new version = reboot*.
+4. **Reuse the existing buses.** Telemetry rides the already-built
+   `auditd → audit-collector relay → Wazuh` pipeline; version truth is the
+   existing digest-pinned deployment manifest; redeploy/drain mechanics are the
+   SPDK rolling-update flow (TODO #6). The management plane orchestrates these; it
+   does not reinvent them.
+5. **Minimal, reproducible, sandboxed agent.** Same posture as `fleet-hostapi`
+   (loopback/least-priv, `DynamicUser`, fapolicyd-trusted, reproducible static
+   build).
+
+## Components
+
+### 1. Identity — mTLS with the TPM cert
+
+The agent presents the TPM-held client cert over **mTLS** to the management
+endpoint; the server authenticates the host by cert CN/SAN, which is the canonical
+fleet identity (= NVMe hostnqn = 802.1X identity). Reuses the **step-ca fleet CA**
+(`serving/dot1x-bench/`). No baked secret; the host also pins the server cert.
+
+- **Secure path:** full mTLS, TPM cert. Strong, no secrets in the image.
+- **Open/test path (`rd.dot1x=0`):** fall back to dnsmasq fleet-name + MAC as a
+  weak identity. **Lab only** — flagged as insufficient for production.
+
+Inventory is keyed by **cert serial / hostnqn**, joined to the dnsmasq starship
+name (`fleet-namer`), MAC, last-seen, running/desired version, and health.
+
+### 2. On-host agent — extend `fleet-hostapi`
+
+`fleet-hostapi` (`files/hostapi/`) is already the only fleet-authored component on
+the hardware and already models os/hardware/net/kiosk facts. Add a **report/push
+mode** (or a tiny sibling that reads its loopback `/v1/host` and ships it). Keep it
+stateless; buffer in tmpfs only; identity from the TPM.
+
+Reported facts:
+
+- **Health:** CPU/load, GPU act-freq, temps, **tmpfs overlay usage** (the
+  RAM-exhaustion → NVMe-keepalive risk — the single most important kiosk health
+  signal), NVMe-root link state, 802.1X/MACsec `Authorized`, SELinux enforcing,
+  chrony sync, `kiosk.service` active.
+- **Version:** running image version (`/etc/fleet-image`), rootfs digest, kernel.
+- **Usage:** kiosk URL, session uptime, **Chromium reload/crash count** (from the
+  `kiosk-session` restart loop — the exact signal behind the GPU-bug diagnosis),
+  input idle/active (liveness only, never content — privacy-aware), display power
+  state.
+
+Cadence: heartbeat (~30–60 s) + event-driven (crash, auth loss, version change).
+
+### 3. Transport
+
+**Phase 1 — ride the Wazuh pipeline.** Emit snapshots as structured JSON events
+through the existing `audit-collector` relay → `wazuh1`. Add `fleet_health` /
+`fleet_kiosk` decoders + rules alongside `fleet_audit` (`serving/siem/config/`).
+Wazuh then provides search, dashboards, and alerting for free, and it's already
+secured-path-aware.
+
+**Phase 2 — metrics push** for high-resolution time-series (CPU/GPU/mem trends):
+**push** via OTLP or Prometheus remote-write to a collector on the mgmt host →
+VictoriaMetrics/Prometheus. **Push, never scrape** — scraping would require an
+inbound port on every locked host, against the posture.
+
+### 4. Control / maintenance — pull-based desired-state reconciliation
+
+Each host periodically fetches **its** desired state (mTLS, keyed by TPM cert):
+target version, kiosk URL, `reboot-at`, `drain`. The agent reconciles; because the
+root is immutable, **convergence = reboot** (a reboot re-PXEs and picks up the new
+UKI).
+
+- The **deployment manifest is already the desired-state for versions** (one
+  commit per release, digest-pinned). Extend it to per-host / per-cohort desired
+  state. Rollback = revert the commit (unchanged model).
+- **Maintenance actions** (reboot, drain, restart kiosk, change URL, request logs)
+  ride the same poll: the server sets the intent; the host obeys on next poll.
+- This dovetails with the **SPDK rolling update** (TODO #6): "redeploy a cohort" =
+  bump the manifest → `spdk-loader` serves the new version online → the management
+  plane tells those hosts to drain/reboot → it watches `spdk-release list` CONNS
+  fall → the old version auto-reaps. The management plane *orchestrates* this loop.
+- **No always-on inbound control socket.** Admin SSH (key-only ECDSA) stays purely
+  break-glass for exceptional manual ops.
+
+### 5. Server side (serving host or a dedicated mgmt node, e.g. alongside `wazuh1`)
+
+- **Mgmt service** (small Go, mirrors the `fleet-hostapi` style): mTLS endpoint
+  that accepts host reports and serves per-host desired-state; authenticates hosts
+  by TPM cert via the fleet CA.
+- **Store:** time-series (VictoriaMetrics/Prometheus) for metrics; Wazuh /
+  OpenSearch for events/logs; git or a tiny DB for desired-state + inventory.
+- **Dashboards:** Grafana (metric trends) + the Wazuh dashboard (events/alerts) +
+  a small fleet-overview board (offline hosts, version drift, crash rates, tmpfs
+  pressure). Could be a sibling of the `serving/kiosk-app` SPA.
+- **Alerting:** host offline > N min; Chromium crash-loop; **running ≠ desired
+  version**; tmpfs near-full; MACsec/auth down; NVMe-root reconnect storms.
+
+## Data model (sketch)
+
+```
+Host (by cert-serial / hostnqn)
+  ├─ identity:   cert serial, hostnqn, MAC, starship-name (dnsmasq)
+  ├─ version:    image_version, rootfs digest, kernel, desired_version
+  ├─ health:     cpu/load, gpu_freq, temps, tmpfs_used%, nvme_root, auth/macsec,
+  │              selinux, chrony, kiosk_active, last_seen
+  ├─ usage:      kiosk_url, session_uptime, chromium_crashes, input_active, display
+  └─ desired:    target_version, kiosk_url, reboot_at, drain   (server-authoritative)
+```
+
+## Mapping to existing building blocks
+
+| Need | Reuse |
+|---|---|
+| Host identity | TPM cert / hostnqn / step-ca fleet CA (`serving/dot1x-bench/`) |
+| On-host data | `fleet-hostapi` `/v1/host` (`files/hostapi/`) |
+| Telemetry + alerting | audit-collector relay → Wazuh (`serving/siem/`) |
+| Friendly names / inventory | dnsmasq `fleet-namer` (MAC → name → DNS) |
+| Version truth + rollback | digest-pinned deployment manifest (git) |
+| Redeploy / drain mechanics | SPDK `spdk-loader` + `spdk-release` (TODO #6) |
+| Build / sign / publish | Gitea CI + OCI images |
+
+## Phasing
+
+1. **Phase 1 (mostly-built):** push `fleet-hostapi` snapshots as structured events
+   into Wazuh; add decoders/rules + a fleet dashboard → health, version-drift,
+   offline detection, usage, alerting. Reuses the existing pipeline end to end.
+2. **Phase 2:** metrics push (OTLP / remote-write) → VictoriaMetrics + Grafana for
+   trend graphs.
+3. **Phase 3:** mTLS desired-state pull (version pin, reboot/drain, kiosk URL),
+   wired to the deployment manifest + SPDK rolling-update flow.
+
+## Build vs buy
+
+Off-the-shelf fits poorly because the base is unusual:
+
+- **FleetDM + osquery** — closest for the query/health side; worth evaluating. But
+  osquery enrollment carries per-host state, so it must be run carefully (identity
+  from TPM, enrollment in tmpfs only) or it breaks invariant #1.
+- **Mender / balena / typical kiosk MDM** — assume mutable disk + their own OTA +
+  enrollment. We already have a *better* version mechanism (UKI + NVMe-root +
+  digest manifest), so their OTA is redundant and their disk model conflicts.
+
+**Decision: build the thin mgmt service + reconcile loop; reuse Wazuh/Grafana for
+the heavy lifting.** The hard parts (identity, immutable versioning, a telemetry
+bus) already exist.
+
+## Security posture
+
+- mTLS + TPM cert = strong identity, zero baked secrets, reuses the 802.1X CA.
+- Pull-based control = **no new inbound attack surface** on locked hosts.
+- Telemetry stays on the mgmt/boot LAN, TLS'd (same gap as the Wazuh-relay-TLS
+  TODO) — no internet egress, air-gap intact.
+- Agent stays minimal/sandboxed/fapolicyd-trusted/reproducible (`fleet-hostapi`
+  template); usage capture is liveness-only, never content (privacy).
+- Fleet-wide inventory is recon-valuable → protect the management plane (auth +
+  segmentation); never expose its dashboards to the boot LAN.
+
+## Open questions / decisions for build time
+
+- **Telemetry encoding:** structured syslog/JSON into Wazuh (Phase 1) vs native
+  OTLP from the start. Leaning Wazuh-first for reuse.
+- **Metrics store:** VictoriaMetrics vs Prometheus (VM is lighter for push +
+  long retention).
+- **Desired-state store:** git (GitOps, aud/rollback for free, matches the
+  deployment-manifest model) vs a small DB (richer per-host scheduling).
+- **Per-host vs per-cohort** desired state granularity (cohorts likely enough;
+  per-host for canaries).
+- **Open-path identity:** MAC/fleet-name is weak — gate the management plane to the
+  secure path, or accept lab-only for the open path.
+- **osquery/FleetDM** evaluation for the read side before building a bespoke
+  collector.

serving/docker-compose.yml

@@ -285,13 +285,18 @@ services:
       TARGET_PORT: ${TARGET_PORT}
       SPDK_CPUMASK: ${SPDK_CPUMASK}
       NVMEOF_SVC: nvmeof
-    # Healthy once the RPC socket answers — gates the spdk-loader's startup.
+    # Healthy once spdk-target has written its readiness stamp (transport up +
+    # releases reconciled). A FILE check, not a python rpc.py probe: nvmf_tgt does
+    # a slow DPDK init (~50s observed) and busy-polls its reactor core, so a python
+    # healthcheck starves/times out and the container sticks at "unhealthy" even
+    # though the target is up. `test -f` is instant and contention-free. The stamp
+    # is removed on (re)start + exit, so it never falsely reports a dead target.
     healthcheck:
-      test: ["CMD-SHELL", "rpc.py -s /var/run/spdk/spdk.sock framework_get_config >/dev/null 2>&1 || exit 1"]
-      interval: 5s
-      timeout: 3s
-      retries: 24
-      start_period: 10s
+      test: ["CMD-SHELL", "test -f /var/run/spdk/ready || exit 1"]
+      interval: 10s
+      timeout: 5s
+      retries: 6
+      start_period: 120s
     # Persistent target: started ONCE, never restarted for a version change.
     # Rolling updates add/remove versions online (spdk-loader, or spdk-ctl.sh).
     entrypoint: ["/bin/bash", "/usr/local/bin/spdk-target"]

serving/spdk/spdk-release.sh

@@ -98,8 +98,12 @@ except Exception: print(-1)'
 cmd_add() {
     local v=${1:?usage: add <version>}
     local dir="$RELEASES_DIR/$v"
-    [ -s "$dir/rootfs.erofs" ] || die "$dir/rootfs.erofs missing/empty"
-    [ -s "$dir/rootfs.hash" ]  || die "$dir/rootfs.hash missing/empty"
+    # Validation failures here RETURN (not die/exit): cmd_add is called per-dir by
+    # cmd_reconcile as `cmd_add … || warn`, and an `exit` inside a function aborts
+    # the WHOLE process (|| only catches a return), so one bad/stale dir would stop
+    # every OTHER release from being added. Return so reconcile skips just this one.
+    [ -s "$dir/rootfs.erofs" ] || { echo "ERROR: $dir/rootfs.erofs missing/empty" >&2; return 1; }
+    [ -s "$dir/rootfs.hash" ]  || { echo "ERROR: $dir/rootfs.hash missing/empty" >&2; return 1; }
 
     # Guard: the dir name must match the version inside verity.json. A mismatch
     # means a stale/symlinked or mislabeled release dir (e.g. an old symlink into
@@ -112,7 +116,8 @@ cmd_add() {
     local vv
     vv=$(vfield "$v" .image_version)
     if [ -n "$vv" ] && [ "${vv,,}" != "${v,,}" ]; then
-        die "release dir '$v' holds verity.json for version '$vv' (stale symlink / mislabeled dir). Refusing to add — remove '$dir' and re-stage."
+        echo "ERROR: release dir '$v' holds verity.json for version '$vv' (stale symlink / mislabeled dir). Refusing to add — remove '$dir' and re-stage." >&2
+        return 1
     fi
 
     local nqn du hu s

serving/spdk/spdk-target.sh

@@ -19,6 +19,12 @@ TADDR="${TARGET_ADDR:-0.0.0.0}"
 TPORT="${TARGET_PORT:-4420}"
 HUGEPAGES="${HUGEPAGES:-1024}"
 DISCOVERY_NQN="nqn.2014-08.org.nvmexpress.discovery"
+# Readiness stamp for the container healthcheck. Written ONLY after the transport
+# is up and releases are reconciled; removed on (re)start + exit. A file check is
+# instant and python-free — the right probe for a box where SPDK busy-polls its
+# (often single) reactor core and a python rpc.py healthcheck would starve/time
+# out. Lives on the shared /run/spdk volume so it's cleared per container start.
+READY_FILE="${READY_FILE:-/var/run/spdk/ready}"
 
 TGT=$(command -v nvmf_tgt || echo /usr/libexec/spdk/spdk/build/bin/nvmf_tgt)
 RPC=$(command -v rpc.py   || echo /usr/libexec/spdk/scripts/rpc.py)
@@ -29,6 +35,8 @@ RELEASE_CTL=$(command -v spdk-release || echo /usr/local/bin/spdk-release)
 RPC_SOCK="${RPC_SOCK:-}"
 rpc() { if [ -n "$RPC_SOCK" ]; then "$RPC" -s "$RPC_SOCK" "$@"; else "$RPC" "$@"; fi; }
 [ -n "$RPC_SOCK" ] && mkdir -p "$(dirname "$RPC_SOCK")"
+mkdir -p "$(dirname "$READY_FILE")"
+rm -f "$READY_FILE"   # clear any stale readiness from a previous container start
 
 # Ensure hugepages exist — DPDK needs them. vm.nr_hugepages isn't namespaced, so
 # a privileged container writes the host value; re-applied after a host reboot
@@ -51,8 +59,8 @@ TGT_ARGS=(-m "$CPUMASK")
 TGT_PID=$!
 # A TERM/INT here stops the target and DROPS EVERY connection — only on container
 # stop, never for a version change. Make the intent loud.
-trap 'echo "==> stopping nvmf_tgt — this drops ALL host root connections" >&2; kill "$TGT_PID" 2>/dev/null || true' TERM INT
-trap 'kill "$TGT_PID" 2>/dev/null || true' EXIT
+trap 'echo "==> stopping nvmf_tgt — this drops ALL host root connections" >&2; rm -f "$READY_FILE"; kill "$TGT_PID" 2>/dev/null || true' TERM INT
+trap 'rm -f "$READY_FILE"; kill "$TGT_PID" 2>/dev/null || true' EXIT
 
 # Wait for the RPC socket.
 for _ in $(seq 1 30); do
@@ -91,6 +99,9 @@ echo "==> reconciling releases under $RELEASES_DIR"
 RELEASES_DIR="$RELEASES_DIR" TARGET_ADDR="$TADDR" TARGET_PORT="$TPORT" RPC_SOCK="$RPC_SOCK" \
     bash "$RELEASE_CTL" reconcile || echo "WARNING: reconcile reported errors" >&2
 
+# Signal readiness for the healthcheck — transport is up and releases reconciled.
+: > "$READY_FILE"
+
 echo "==> nvmf_tgt up; releases served online. Add/remove versions with:"
 echo "      docker compose exec ${NVMEOF_SVC:-nvmeof} spdk-release {add|remove|list} <version>"
 echo "    or host-side: serving/spdk/spdk-ctl.sh {load|unload|list} <version>"