diskless-workstation/serving/fleet-manager/README.md

# fleet-manager — server-side fleet control plane

The management service for the diskless kiosk fleet, and the backend for the
admin SPA. It is the central place to **monitor**, **authorise**, and
**lifecycle-control** enrolled hosts. See
[`docs/fleet-management-design.md`](../../docs/fleet-management-design.md) and
[`docs/tpm-enrolment-design.md`](../../docs/tpm-enrolment-design.md) §5.

Unlike `fleet-hostapi` / `fleet-enrol` (baked into the reproducible verity image,
stdlib-only), this runs **server-side** as its own container on the management
network, so external deps are fine (`grpc-go`, `influxdb-client-go`).

## Two planes (distinct trust models)

| Plane | Transport | Browser/peer identity | Who | Purpose |
|---|---|---|---|---|
| **admin** | HTTPS via **Traefik** (:443) → manager | **WebAuthn/passkey** (YubiKey) → session | admin browsers | JSON API + SPA (monitor / authorise / lifecycle) |
| **agent** | gRPC + mTLS | **fleet CA** (host TPM cert) | enrolled hosts | bidi stream: health/metrics up, desired-state down |

The agent plane uses the host's **reused TPM identity cert** (the same one used
for 802.1X EAP-TLS and from which the NVMe `hostnqn` derives — "one identity,
three stages"). The admin plane authenticates the **human** with a hardware
**passkey** (phishing-resistant, origin-bound), not a client cert — so there's no
cert to import and nothing to phish. (The manager→broker hop is separately mTLS;
that's service-to-service — see [Authorise](#authorise-pillar).)

### Admin auth: passkeys/WebAuthn

- **Login** is usernameless (discoverable passkey) with `userVerification:
  required` — tap the YubiKey + PIN. A successful assertion mints an opaque,
  server-side session (`Secure`/`HttpOnly`/`SameSite=Strict` cookie); mutating
  requests must also echo the session CSRF token in `X-Fleet-CSRF`.
- **Bootstrap** is closed: `fleet-manager register-admin <name>` (run on the mgmt
  host) mints a one-time, HMAC-signed enrolment link; the admin opens it in the
  browser and registers their passkey. Single-use (jti ledger). Further admins can
  be enrolled from the UI once one exists.
- **RP is origin-bound**: `FLEET_MGR_RP_ID` / `FLEET_MGR_RP_ORIGIN` must be the
  UI's real hostname — for this deployment **`diskless-server.bitasylum.org`** /
  `https://diskless-server.bitasylum.org` (port-less, since Traefik serves :443).
- **TLS + cert** are owned by the **`traefik`** mgmt-plane ingress: it terminates
  a publicly-trusted **Let's Encrypt** cert (**Route 53 DNS-01** — the UI is on the
  mgmt network, not internet-reachable, so HTTP-01/TLS-ALPN-01 can't validate) on
  :443 and forwards plaintext to the manager, which runs
  `FLEET_MGR_ADMIN_BEHIND_PROXY=true` (passkey auth ON, plaintext on loopback —
  WebAuthn only needs the origin to match, not the app's own transport). Traefik
  is the k8s ingress-controller + cert-manager role in miniature. *(Self-TLS mode
  without Traefik is still available: `BEHIND_PROXY=false` + a cert at
  `FLEET_MGR_ADMIN_CERT/_KEY`, e.g. from `acme/renew-loop.sh`.)*

## Monitoring path (no host ever touches InfluxDB)

```
host agent ──gRPC bidi (mTLS, TPM cert)──▶ fleet-manager ──line protocol──▶ InfluxDB
           ◀──── desired-state (target ver / kiosk_url / reboot / drain) ──┘ (+ inventory health)
SPA ──GET /api/hosts/{serial}/metrics──▶ fleet-manager ──Flux──▶ InfluxDB ──JSON──▶ SPA renders charts
```

Hosts hold **no InfluxDB credential** and never reach it; the manager is the only
writer/reader, holding the Influx token server-side. Identity is taken from the
mTLS cert (anti-spoofing: a host can only write its own series). Push-only, no
inbound host port. Graphs are rendered **in the SPA**, not Grafana.

## Inventory record

Keyed by TPM **cert serial** (= hostnqn key). Born at enrolment approval, filled
by the agent stream and admin action:

```
Host (cert_serial)
  ├─ identity : cert_serial, hostnqn, mac, ek_hash, starship_name
  ├─ binding  : asset_tag, switch_port, location        (recorded at approval)
  ├─ enrolled : approver, at, fingerprint
  ├─ version  : image_version, rootfs_digest, kernel     (agent-reported)
  ├─ health   : load, cpu, temp, tmpfs%, nvme_root, auth, selinux, chrony, kiosk  (latest snapshot; series in InfluxDB)
  ├─ usage    : kiosk_url, session_uptime, chromium_crashes, display
  └─ desired  : target_version, kiosk_url, reboot_at, drain   (SERVER-AUTHORITATIVE)
```

Persisted as a single JSON file (atomic writes) on a mounted volume — stdlib-only,
inspectable, reproducible; a DB swap is a drop-in behind the `Store` interface.

## API (admin plane)

| Route | Auth | Status |
|---|---|---|
| `GET /healthz` | open | ✅ ckpt 1 |
| `POST /auth/register/begin\|finish` | enrol token | ✅ ckpt 5 — passkey registration ceremony |
| `POST /auth/login/begin\|finish` | passkey | ✅ ckpt 5 — usernameless discoverable login → session |
| `POST /auth/logout` | session | ✅ ckpt 5 |
| `GET /api/whoami` | session | ✅ ckpt 5 — resume session + recover CSRF token |
| `GET /api/hosts` | session | ✅ ckpt 1 — fleet overview |
| `GET /api/hosts/{serial}` | session | ✅ ckpt 1 — host detail |
| `PUT /api/hosts/{serial}/desired` | session + CSRF | ✅ ckpt 1 — set desired-state (audited to admin) |
| `GET /api/enrol/pending` | session | ✅ ckpt 3 — proxies the broker queue |
| `POST /api/enrol/{id}/approve` | session + CSRF | ✅ ckpt 3 — signs via broker + **births inventory** |
| `POST /api/enrol/{id}/deny` | session + CSRF | ✅ ckpt 3 |
| `GET /api/hosts/{serial}/metrics?field=&range=` | session | ✅ ckpt 4 — Flux time-series (field whitelist) |

`/api/*` is gated by the passkey session (mutations also need the CSRF header).

<a name="authorise-pillar"></a>The authorise pillar drives the enrol-broker's
admin plane over **mTLS** (the manager is a service client, gated by the dedicated
admin CA) asking for JSON, so an approve returns the decided enrolment; the manager
records the **human** approver (the signed-in passkey admin) and births the
inventory record. The broker stays the enrolment state machine + step-ca signer
(v1 "alongside" fold-in).

**Agent plane** — gRPC `FleetAgent.Session` bidi over mTLS (fleet CA): ✅ ckpt 2.
`Hello`/`HealthSnapshot`/`CommandResult` up; `DesiredState`/`Command` down.
Identity is the peer cert serial (never self-asserted); an unenrolled serial is
refused. Desired-state is pushed live on connect + on change, and is durable in
inventory for offline delivery. Metrics writing is behind a `MetricsSink`
(no-op now; InfluxDB in ckpt 4).

## Config (env)

| Var | Default | Notes |
|---|---|---|
| `FLEET_MGR_ADMIN_ADDR` | `127.0.0.1:9500` | admin listener (loopback — Traefik forwards here) |
| `FLEET_MGR_ADMIN_BEHIND_PROXY` | `false` | passkey ON, plaintext listener (Traefik/ingress terminates TLS). Compose defaults it `true` |
| `FLEET_MGR_ADMIN_CERT` / `_KEY` | — | UI server cert/key — **self-TLS mode only** (`BEHIND_PROXY=false`, no Traefik) |
| `FLEET_MGR_RP_ID` | — | WebAuthn RP ID = UI hostname (`diskless-server.bitasylum.org`) |
| `FLEET_MGR_RP_ORIGIN` | — | WebAuthn origin (`https://diskless-server.bitasylum.org`) |
| `FLEET_MGR_RP_NAME` | `Fleet Command` | RP display name |
| `FLEET_MGR_ENROLL_KEY_FILE` | — | shared HMAC key for enrolment tokens (CLI + service) |
| `FLEET_MGR_ADMIN_STORE` | `/var/lib/fleet-manager/admins.json` | registered-passkey store |
| `FLEET_MGR_SESSION_TTL_MIN` | `720` | session lifetime (minutes) |
| `FLEET_MGR_ADMIN_INSECURE` | `false` | plaintext, **no passkey auth** — dev only |
| `FLEET_MGR_AGENT_ADDR` | `127.0.0.1:9600` | gRPC agent listener |
| `FLEET_MGR_AGENT_CERT` / `_KEY` | — | agent-plane server cert/key |
| `FLEET_MGR_AGENT_CLIENT_CA` | — | **fleet CA** (verifies host TPM certs). Agent plane starts only when all three are set. |
| `FLEET_MGR_BROKER_URL` | — | enrol-broker admin base URL (e.g. `https://127.0.0.1:9443`). Authorise pillar enabled only when set. |
| `FLEET_MGR_BROKER_CLIENT_CERT` / `_KEY` | — | manager's client cert for the broker admin mTLS (admin CA) |
| `FLEET_MGR_BROKER_CA` | — | CA that signs the broker's admin server cert |
| `FLEET_MGR_INFLUX_URL` | — | InfluxDB URL (private mgmt net). Metrics enabled only when URL + token set. |
| `FLEET_MGR_INFLUX_TOKEN` | — | InfluxDB token — **server-side only**, never on hosts |
| `FLEET_MGR_INFLUX_ORG` | `fleet` | InfluxDB org |
| `FLEET_MGR_INFLUX_BUCKET` | `fleet_health` | InfluxDB bucket |
| `FLEET_MGR_INVENTORY` | `/var/lib/fleet-manager/inventory.json` | inventory file (mount a volume) |
| `FLEET_MGR_SEED` | `false` | inject demo hosts so the SPA has data — dev only |
| `HOSTNQN_BASE` | `nqn.2026-06.org.example.fleet:host-` | hostnqn prefix (match the broker) |

## Deploy

Full bring-up (PKI, Let's Encrypt DNS-01, `register-admin`, compose) is in
[`DEPLOY.md`](DEPLOY.md). In short: `cp .env.example .env` → `./server-pki.sh` +
`../enrol-broker/admin-pki.sh … fleet-manager` → `docker compose up -d influxdb`
→ `docker compose run --rm fleet-manager register-admin <you>` → `docker compose
up -d fleet-manager`. InfluxDB is private (published to loopback only); the
manager uses host networking to reach the loopback broker + bind a routable
agent listener.

## Dev run

```bash
FLEET_MGR_ADMIN_INSECURE=true FLEET_MGR_SEED=true \
  FLEET_MGR_ADMIN_ADDR=127.0.0.1:9500 FLEET_MGR_INVENTORY=/tmp/inv.json \
  go run .
curl -s localhost:9500/api/hosts | jq '.hosts[] | {name:.identity.starship_name, run:.version.image_version, want:.desired.target_version, online}'
```

## Build status

- **Checkpoint 1 (done):** Go skeleton, JSON-file inventory `Store`, admin JSON API
  (monitor + lifecycle), admin mTLS wiring, dev seed. Stdlib-only.
- **Checkpoint 2 (done):** gRPC agent plane — proto contract, mTLS (fleet CA)
  server, stream registry, identity-from-cert, snapshot→inventory, live
  desired-state push, `MetricsSink` seam. Full mTLS integration test.
- **Checkpoint 3 (done):** authorise pillar — the enrol-broker gained an
  `Accept`-gated JSON approve response (browsers/`verify-serving.sh` unchanged);
  the manager drives it over mTLS and births the inventory record on approve.
  httptest-mock-broker test.
- **Checkpoint 4 (done):** InfluxDB metrics gateway — snapshots written as points
  tagged with the authenticated identity (`MetricsSink`→`influxSink`), and a
  read API (`/api/hosts/{serial}/metrics`) that runs a Flux query with a **field
  whitelist** + hex-serial guard (no caller input reaches the query raw). Unit
  tests for point construction, query building, and unsafe-input rejection.
- **Checkpoint 5 (done):** admin **WebAuthn/passkey** auth replacing browser mTLS —
  credential store, `register-admin` bootstrap CLI (one-time HMAC enrol token),
  register/login ceremonies (usernameless + UV required), server-side sessions +
  CSRF middleware gating `/api/*`, admin TLS switched to server-auth with a
  hot-reloading cert (for LE DNS-01 renewals). Tests: enrol token, session
  middleware, cred-store persistence. Live passkey ceremony is a bench item.
- **Checkpoint 6 (done):** the **SPA** ([`ui/`](ui/)) — Svelte 5 + Tailwind 4,
  LYSANDRA restricted-violet theme, passkey login/enrol + Monitor/Authorise/
  Lifecycle + bespoke SVG charts. Embedded via `go:embed` behind the `embedspa`
  build tag; served same-origin over the admin plane. Smoke-tested end-to-end
  (index + assets + API through the SPA origin).
- **Checkpoint 7 (done):** deployment — three-stage `Dockerfile` (npm→dist→
  `go build -tags embedspa`→distroless), `docker-compose.yml` (private InfluxDB +
  host-networked manager), `.env.example`, `server-pki.sh` (agent-plane fleet-CA
  server cert), and [`DEPLOY.md`](DEPLOY.md) (PKI, LE DNS-01, `register-admin`
  bootstrap). Design docs updated (manager marked built; resolved transport/auth/
  metrics forks; per-host-versioning gap recorded).
- **Checkpoint 8 (done):** hardening + ingress — CI image build
  (`fleet-manager-image.yaml`, pulled by tag), `backup.sh`; a **behind-proxy admin
  mode** (passkey over plaintext behind a TLS-terminating proxy) fronted by a
  **Traefik** mgmt ingress (:443 + Route 53 DNS-01, retiring the acme side-car).
  This aligns with the k8s target (Traefik → ingress controller + cert-manager);
  the agent plane stays direct mTLS via OPNsense L3. See [`DEPLOY.md`](DEPLOY.md).