Merge remote-tracking branch 'origin/main' into tpm-enroll

docs/TODO.md

@@ -645,15 +645,69 @@ must be built around **never restarting the process that holds live connections.
 - **Planned SPDK restarts only via drain/failover** (multipath fail-over, or a
   maintenance window with generous `ctrl_loss_tmo` so hosts recover not die). There
   is **no zero-impact single-node SPDK restart.**
-- **Implementation gap (current lab):** SPDK runs as a container with
-  `spdk-setup.sh` as entrypoint + the artifact image mounted → effectively
-  **single-version, configured at container start**, so changing versions =
-  container restart = drop all connections. The work here is a **long-lived target
-  + an RPC-driven control path** ("add release / remove release / drain status")
-  decoupled from container lifecycle, with per-version subsystems.
+- **[x] Long-lived target + online release control — IMPLEMENTED 2026-06-27.**
+  The old gap (SPDK ran `spdk-setup.sh` as entrypoint, single-version,
+  configured-at-start → changing versions = container restart = drop ALL
+  connections) is closed. Now in `serving/spdk/`:
+  - **`spdk-target.sh`** — persistent entrypoint: starts `nvmf_tgt` ONCE
+    (release-agnostic), creates the TCP transport + discovery listener, then
+    reconciles every staged release. Never restarted to change versions; on
+    (re)start re-adds everything under `./releases/` (a serving-host reboot
+    restores all served versions).
+  - **`spdk-release.sh`** — JSON-RPC control: `add`/`remove`/`list`/
+    `drain-status`/`reconcile`, **one subsystem per version** (distinct NQN, so
+    adding/removing one never disturbs others' connections), **drain-gated
+    removal** (refuses while a version has connected controllers unless
+    `--force`).
+  - **`spdk-ctl.sh`** — host ops: `load`/`unload`/`list`; stages a release's
+    artifacts from its OCI image into `./releases/<ver>/` and drives the live
+    target. Rolling update = `load` new → reboot hosts → `unload` old once
+    drained.
+  - **`spdk-loader.sh`** — DECLARATIVE rolling updates (the "init container that
+    stays up", compose service `spdk-loader`). On `compose up` after an
+    `IMAGE_VERSION` bump it stages + adds the new rootfs to the running target
+    ONLINE, then polls and **auto-reaps** any non-current version once it has
+    drained (0 conns) for a sustained grace window (`REAP_GRACE_SECONDS`, keep >
+    host `ctrl_loss_tmo`). So migration = bump `IMAGE_VERSION` in `.env` +
+    `docker compose up -d` (compose recreates the loader + the http/dnsmasq UKI
+    servers, but NOT the nvmeof target → no dropped connections). Single served
+    UKI ⇒ a 0-conn old version is genuinely orphaned (a returning host fetches
+    the current UKI), so auto-reap is safe; removal is still drain-gated +
+    grace-debounced. The loader + target share the RPC socket on a named volume
+    (`nvmf_tgt -r`); the loader reuses the SPDK image (has python3 + rpc.py).
+  - **No jq dependency:** `spdk-release.sh` parses RPC JSON with **python3** (the
+    SPDK image ships no jq); legacy artifacts (no per-version UUIDs in
+    verity.json) fall back to the fixed UUID pair with a warning.
+  - **Per-version namespace UUIDs** (`scripts/extract-artifacts.sh`): derived
+    deterministically (UUIDv5 of `image_version`), baked into the UKI cmdline +
+    `verity.json` (now also carries `subnqn`); the serving side reads them back
+    from each release's `verity.json`. Required so two versions don't collide on
+    the same `/dev/disk/by-id/nvme-uuid.<uuid>` host symlink.
+  - **Bootstrap unchanged:** `docker compose up` with one version still works —
+    the `/artifacts` image volume is seeded into `./releases/${IMAGE_VERSION}`.
+  - **PARTIALLY BENCH-VERIFIED 2026-06-28:** long-lived target boots a host end
+    to end — UKI served over HTTP → host boots → NVMe/TCP root connects to the
+    persistent `nvmf_tgt` (single version `spdk-permanence-02ffd7d`). Surfaced +
+    fixed several real bugs along the way: bootstrap seed must COPY not symlink
+    (a symlink into the version-pinned `/artifacts` re-pointed a stale release at
+    new data); `spdk-release` parses RPC JSON via **python3** (SPDK image has no
+    jq); legacy `verity.json` (no per-version UUIDs) falls back to the fixed pair;
+    mislabeled-dir guard (dir name vs verity.json `image_version`, case-insensitive);
+    dropped the redundant discovery-listener re-add (daemon logged "Listener
+    already exists"). **CI lowercases `image_version`** (gitea-main-ci vars step) —
+    a mixed-case branch (`SPDK-permanence`) had baked a capital
+    `fleet-SPDK-…​.efi` the lowercase boot URL couldn't fetch (iPXE "no such
+    file"); booted via a local retag workaround pending a lowercase rebuild.
+  - **STILL BENCH-VERIFY:** (a) `load` a 2nd version while a host is connected to
+    the 1st → 1st's connection survives (no AER/reset on the running host); (b)
+    `unload`/auto-reap drain-gate blocks while CONNS>0 and reaps after grace on a
+    LIVE host; (c) serving-host reboot reconciles all staged versions; (d)
+    per-version UUID build reproducible (two builds, same version → identical
+    `data_ns_uuid`/`hash_ns_uuid` + `root_hash`); (e) the full fix on a clean
+    LOWERCASE CI rebuild (no retag). See `serving/spdk/README.md`.
 - **Capacity:** each concurrently-served version costs target memory/hugepages +
   file-backed bdev overhead — plan for N live versions (current + previous +
-  canary) during long rollouts.
+  canary) during long rollouts (raise `HUGEPAGES`).
 - **Attach gating + discovery:** with several subsystems advertised, the dracut
   connect + per-subsystem hostNQN allow-list must ensure a vX host attaches **only**
   vX's namespace (the by-UUID verity ref protects *what mounts*, but gate the
@@ -824,6 +878,10 @@ smoke test.
      `docs/stig-deviations.md` #6 to reflect encrypted offload.
   (RELP+TLS via `omrelp`/`imrelp` is the alternative if guaranteed delivery is
   wanted alongside encryption.)
+- **fleet-hostapi: scope CORS to the kiosk origin.** The host-info agent ships
+  with `FLEET_HOSTAPI_ALLOW_ORIGIN=*` (lab). For prod set it to the exact kiosk
+  origin (default `http://boot.fleet.lab`) via the unit's commented
+  `Environment=` line (or a drop-in). See #12.
 
 ## [ ] 9. (OPTIONAL / nice-to-have) Convert provisioning to Ansible where it pays off
 
@@ -992,3 +1050,104 @@ non-destructive TPM capability check (device, key persist, NV roundtrip, openssl
 TPM2 provider, CSR signed by the TPM key) to de-risk the TPM half before any of
 this. Expected gap: the image installs `tpm2-tss` but no openssl TPM2 **provider**
 (`tpm2-openssl`) — add it to the build once the self-test confirms.
+
+## [~] 12. Workstation host-info API (`fleet-hostapi`) for the kiosk
+
+**IMPLEMENTED 2026-06-28.** A small **Go, read-only, loopback-only** HTTP agent on
+the workstation that reports host/OS/hardware facts to the kiosk SPA, per the
+[HOST-API.md](../HOST-API.md) contract (`GET http://localhost:9099/v1/host`).
+Source: `files/hostapi/` (stdlib-only, reproducible static build). Built + enabled
+by provisioner **2b-hostapi**; toolchain removed in 2d; binary fapolicyd-trusted
+in 2c-STIG-postfix (register #10). Chromium policy grants Private Network Access
+scoped to the kiosk origin (`files/kiosk/chromium-kiosk-policy.json`).
+
+- **Per-host safety:** host-specific fields (kernel, hostname, IP, MAC, board,
+  uptime) are read at RUNTIME; only image-generic facts (image version +
+  weston/chromium versions) are baked, to `/etc/fleet-hostinfo.json` (invariant
+  #1 preserved). Deliberately does NOT expose DMI serial (root-only/per-host) or
+  any TPM/cert/identity material.
+- **Hardening shipped:** loopback bind enforced in code; `DynamicUser`, no caps,
+  `ProtectSystem=strict`, `MemoryDenyWriteExecute`, `SystemCallFilter=@system-service`,
+  `IPAddressDeny=any` (loopback only). Runs unprivileged, reads world-readable
+  files only.
+
+**Lab posture LEFT AS-IS (decided 2026-06-28) — revisit for prod:**
+- **CORS `Allow-Origin=*`** → scope to the kiosk origin (`http://boot.fleet.lab`).
+  Consolidated into #8.
+- **SELinux:** runs in the default service domain (no custom policy module).
+  **BENCH-VERIFY** no AVC denials for `fleet-hostapi` on an enforcing booted host
+  (Wazuh/`fleet_audit` rule 100120 will surface any). If denials appear, decide:
+  write a confined SELinux policy module vs adjust the sandbox.
+- **No auth** (justified: loopback + read-only + single-user appliance). Option if
+  the threat model tightens: shared token in `kiosk.env` + header check.
+- **Info sensitivity:** exposes hostname/IP/MAC/board/versions to the kiosk. Drop
+  MAC/IP if that's later deemed too much.
+- **Reproducibility:** the `golang` toolchain version is a reproducibility-sensitive
+  input (a package bump changes the binary → verity hash); covered by the
+  build-twice gate. Pin the go version only if drift becomes a problem.
+
+**BENCH-VERIFY (functional):** (a) service starts under enforcing SELinux +
+fapolicyd (not blocked); (b) the kiosk SPA's System page fetches `/v1/host`
+successfully through the PNA allowance (preflight + GET); (c) facts render
+correctly on real hardware (board/CPU/GPU from DMI/DRM, not the virtio build VM).
+
+## [~] 13. Kiosk GPU compositing + Chromium flag ownership
+
+**GPU FIX APPLIED 2026-06-28.** The 820 (Intel HD620/i915) ran Chromium at ~150%
+CPU with periodic crashes: `--disable-gpu` in our `kiosk-session` forced software
+raster/compositing. Root cause confirmed from the live cmdline. Fixes in
+`files/kiosk/kiosk-session` + `diskless.pkr.hcl`:
+- **Runtime GPU probe** (fleet-generic image boots real HW *and* virtio VMs): a
+  real KMS driver (`i915`/`amdgpu`/…) + `/dev/dri/renderD128` → GPU path
+  (`--ignore-gpu-blocklist --enable-gpu-rasterization --enable-zero-copy`, weston
+  GL renderer); else software (`--disable-gpu` + `weston --use-pixman`) so the VMs
+  still render. No per-host state baked.
+- **kiosk user added to `render`** (renderD128 isn't logind-seat-ACL'd on EL9;
+  `video` only covers card0/KMS for weston).
+- **Merged the distro wrapper's `--disable-features` into ours** — Chromium takes
+  the LAST `--disable-features` (not the union), so ours was silently dropping the
+  wrapper's Lens/Gemini/Compose disables.
+
+**BENCH-VERIFY (on the 820):** GPU process alive (`chrome://gpu`), Compositor/
+ThreadPool threads near-idle + CPU drop, `gt_act_freq_mhz` rises under load, no
+SELinux AVC for `dri_device_t` from the kiosk domain (Wazuh `fleet_audit` 100120
+will surface any — if present, add a scoped SELinux allow or confirm the kiosk
+user's SELinux mapping is unconfined). Confirm VMs still render (software path).
+
+**DECIDED 2026-06-28 — Option 1 (policy lock), APPLIED.** The distro chromium
+wrapper (`/usr/bin/chromium-browser`) layers flags as `$CHROMIUM_FLAGS`
+(`/etc/chromium/chromium.conf` → VA-API enables + AI disables; overridable via
+`CHROMIUM_USER_FLAGS`) + **hardcoded** `$CHROMIUM_DISTRO_FLAGS`
+(`--enable-plugins --enable-extensions --enable-user-scripts --enable-printing
+--enable-sync --auto-ssl-client-auth`) + our `"$@"` (last). The hardcoded layer
+isn't removable via env/conf — only by bypassing the wrapper. Rather than bypass
+(Option 2, which would force us to replicate the wrapper's stdio sanitisation +
+own the VA-API feature list across updates), we **neutralise in managed policy**
+(policy beats cmdline flags, survives updates): `ExtensionInstallBlocklist=["*"]`
+added to `files/kiosk/chromium-kiosk-policy.json` (blocks all extensions → also
+kills the user-scripts surface). `--enable-sync` already locked by `SyncDisabled`;
+`--auto-ssl-client-auth` is moot (no browser-store client cert; the TPM 802.1X
+cert isn't one) — `AutoSelectCertificateForUrls` left unset. VA-API enables
+inherited from `chromium.conf` unchanged. Option 2 (full flag ownership, bypass
+the wrapper, own the VA-API `--enable-features`:
+`AllowQt,WaylandLinuxDrmSyncobj,WaylandPerSurfaceScale,WaylandUiScale,AcceleratedVideoDecodeLinuxGL,AcceleratedVideoDecodeLinuxZeroCopyGL,AcceleratedVideoEncoder`)
+remains documented here if a minimal/auditable cmdline is ever wanted.
+**BENCH-VERIFY:** `chrome://policy` shows `ExtensionInstallBlocklist=["*"]` active;
+extensions can't install.
+
+## [ ] 14. (DESIGN) Centralised fleet management plane (health/version/usage/maintenance)
+
+**DESIGN CAPTURED 2026-06-28 — [docs/fleet-management-design.md](fleet-management-design.md).**
+Server-authoritative, pull-based, TPM-identified management plane reusing existing
+pieces: `fleet-hostapi` as the on-host data source, the audit-collector→Wazuh
+pipeline as telemetry transport, the digest-pinned deployment manifest as version
+truth, and the SPDK `spdk-loader`/`spdk-release` rolling-update flow (#6) as the
+redeploy/drain mechanics. Hosts stay stateless (identity from the TPM cert, no
+baked enrollment); control is pull/reconcile (no inbound surface); convergence to
+a new version = reboot. **Phased build:** (1) push hostapi snapshots as structured
+events into Wazuh + fleet dashboard (health/version-drift/offline/usage/alerting);
+(2) metrics push (OTLP/remote-write) → VictoriaMetrics + Grafana; (3) mTLS
+desired-state pull (version pin, reboot/drain, kiosk URL) wired to the deployment
+manifest + SPDK flow. Build-time decisions (telemetry encoding, metrics/desired-
+state store, per-host vs cohort, osquery/FleetDM eval) listed in the design note.
+Depends on the step-ca fleet CA (#8) for the mTLS identity.