diskless-workstation/serving/fleet-manager/README.md

# fleet-manager — server-side fleet control plane

The management service for the diskless kiosk fleet, and the backend for the
admin SPA. It is the central place to **monitor**, **authorise**, and
**lifecycle-control** enrolled hosts. See
[`docs/fleet-management-design.md`](../../docs/fleet-management-design.md) and
[`docs/tpm-enrolment-design.md`](../../docs/tpm-enrolment-design.md) §5.

Unlike `fleet-hostapi` / `fleet-enrol` (baked into the reproducible verity image,
stdlib-only), this runs **server-side** as its own container on the management
network, so external deps are fine (`grpc-go`, `influxdb-client-go`).

## Two planes (distinct trust models)

| Plane | Transport | Browser/peer identity | Who | Purpose |
|---|---|---|---|---|
| **admin** | HTTPS via **Traefik** (:443) → manager | **WebAuthn/passkey** (YubiKey) → session | admin browsers | JSON API + SPA (monitor / authorise / lifecycle) |
| **agent** | gRPC + mTLS | **fleet CA** (host TPM cert) | enrolled hosts | bidi stream: health/metrics up, desired-state down |

The agent plane uses the host's **reused TPM identity cert** (the same one used
for 802.1X EAP-TLS and from which the NVMe `hostnqn` derives — "one identity,
three stages"). The admin plane authenticates the **human** with a hardware
**passkey** (phishing-resistant, origin-bound), not a client cert — so there's no
cert to import and nothing to phish. (The manager→broker hop is separately mTLS;
that's service-to-service — see [Authorise](#authorise-pillar).)

### Admin auth: passkeys/WebAuthn

- **Login** is usernameless (discoverable passkey) with `userVerification:
  required` — tap the YubiKey + PIN. A successful assertion mints an opaque,
  server-side session (`Secure`/`HttpOnly`/`SameSite=Strict` cookie); mutating
  requests must also echo the session CSRF token in `X-Fleet-CSRF`.
- **Bootstrap** is closed: `fleet-manager register-admin <name>` (run on the mgmt
  host) mints a one-time, HMAC-signed enrolment link; the admin opens it in the
  browser and registers their passkey. Single-use (jti ledger). Further admins can
  be enrolled from the UI once one exists.
- **RP is origin-bound**: `FLEET_MGR_RP_ID` / `FLEET_MGR_RP_ORIGIN` must be the
  UI's real hostname — for this deployment **`diskless-server.bitasylum.org`** /
  `https://diskless-server.bitasylum.org` (port-less, since Traefik serves :443).
- **TLS + cert** are owned by the **`traefik`** mgmt-plane ingress: it terminates
  a publicly-trusted **Let's Encrypt** cert (**Route 53 DNS-01** — the UI is on the
  mgmt network, not internet-reachable, so HTTP-01/TLS-ALPN-01 can't validate) on
  :443 and forwards plaintext to the manager, which runs
  passkey auth ON, plaintext on loopback (WebAuthn only needs the origin to match,
  not the app's own transport). The manager never terminates TLS itself. Traefik
  is the k8s ingress-controller + cert-manager role in miniature.

## Monitoring path (no host ever touches InfluxDB)

```
host agent ──gRPC bidi (mTLS, TPM cert)──▶ fleet-manager ──line protocol──▶ InfluxDB
           ◀──── desired-state (target ver / kiosk_url / reboot / drain) ──┘ (+ inventory health)
SPA ──GET /api/hosts/{serial}/metrics──▶ fleet-manager ──Flux──▶ InfluxDB ──JSON──▶ SPA renders charts
```

Hosts hold **no InfluxDB credential** and never reach it; the manager is the only
writer/reader, holding the Influx token server-side. Identity is taken from the
mTLS cert (anti-spoofing: a host can only write its own series). Push-only, no
inbound host port. Graphs are rendered **in the SPA**, not Grafana.

## Inventory record

Keyed by TPM **cert serial** (= hostnqn key). Born at enrolment approval, filled
by the agent stream and admin action:

```
Host (cert_serial)
  ├─ identity : cert_serial, hostnqn, mac, ek_hash, starship_name
  ├─ binding  : asset_tag, switch_port, location        (recorded at approval)
  ├─ enrolled : approver, at, fingerprint
  ├─ version  : image_version, rootfs_digest, kernel     (agent-reported)
  ├─ health   : load, cpu, temp, tmpfs%, uptime, nvme_root, auth, selinux, chrony (state+source/stratum/offset), kiosk  (latest snapshot; series in InfluxDB)
  ├─ usage    : kiosk_url, session_uptime, chromium_crashes, display
  └─ desired  : target_version, kiosk_url, reboot_at, drain   (SERVER-AUTHORITATIVE)
```

Persisted as a single JSON file (atomic writes) on a mounted volume — stdlib-only,
inspectable, reproducible; a DB swap is a drop-in behind the `Store` interface.

## API (admin plane)

| Route | Auth | Status |
|---|---|---|
| `GET /healthz` | open | ✅ ckpt 1 |
| `POST /auth/register/begin\|finish` | enrol token | ✅ ckpt 5 — passkey registration ceremony |
| `POST /auth/login/begin\|finish` | passkey | ✅ ckpt 5 — usernameless discoverable login → session |
| `POST /auth/logout` | session | ✅ ckpt 5 |
| `GET /api/whoami` | session | ✅ ckpt 5 — resume session + recover CSRF token |
| `GET /api/hosts` | session | ✅ ckpt 1 — fleet overview |
| `GET /api/hosts/{serial}` | session | ✅ ckpt 1 — host detail |
| `PUT /api/hosts/{serial}/desired` | session + CSRF | ✅ ckpt 1 — set desired-state (audited to admin) |
| `GET /api/enrol/pending` | session | ✅ ckpt 3 — proxies the broker queue |
| `POST /api/enrol/{id}/approve` | session + CSRF | ✅ ckpt 3 — signs via broker + **births inventory** |
| `POST /api/enrol/{id}/deny` | session + CSRF | ✅ ckpt 3 |
| `GET /api/hosts/{serial}/metrics?field=&range=` | session | ✅ ckpt 4 — Flux time-series (field whitelist) |

`/api/*` is gated by the passkey session (mutations also need the CSRF header).

<a name="authorise-pillar"></a>The authorise pillar drives the enrol-broker's
admin plane over **mTLS** (the manager is a service client, gated by the dedicated
admin CA) asking for JSON, so an approve returns the decided enrolment; the manager
records the **human** approver (the signed-in passkey admin) and births the
inventory record. The broker stays the enrolment state machine + step-ca signer
(v1 "alongside" fold-in).

**Agent plane** — gRPC `FleetAgent.Session` bidi over mTLS (fleet CA): ✅ ckpt 2.
`Hello`/`HealthSnapshot`/`CommandResult` up; `DesiredState`/`Command` down.
Identity is the peer cert serial (never self-asserted); an unenrolled serial is
refused. Desired-state is pushed live on connect + on change, and is durable in
inventory for offline delivery. Metrics writing is behind a `MetricsSink`
(no-op now; InfluxDB in ckpt 4).

## Config (env)

| Var | Default | Notes |
|---|---|---|
| `FLEET_MGR_ADMIN_ADDR` | `127.0.0.1:9500` | admin listener (loopback — Traefik forwards here; always plaintext) |
| `FLEET_MGR_RP_ID` | — | WebAuthn RP ID = UI hostname (`diskless-server.bitasylum.org`) |
| `FLEET_MGR_RP_ORIGIN` | — | WebAuthn origin (`https://diskless-server.bitasylum.org`) |
| `FLEET_MGR_RP_NAME` | `Fleet Command` | RP display name |
| `FLEET_MGR_ENROLL_KEY_FILE` | — | shared HMAC key for enrolment tokens (CLI + service) |
| `FLEET_MGR_ADMIN_STORE` | `/var/lib/fleet-manager/admins.json` | registered-passkey store |
| `FLEET_MGR_SESSION_TTL_MIN` | `720` | session lifetime (minutes) |
| `FLEET_MGR_ADMIN_INSECURE` | `false` | plaintext, **no passkey auth** — dev only |
| `FLEET_MGR_AGENT_ADDR` | `127.0.0.1:9600` | gRPC agent listener |
| `FLEET_MGR_AGENT_CERT` / `_KEY` | — | agent-plane server cert/key |
| `FLEET_MGR_AGENT_CLIENT_CA` | — | **fleet CA** (verifies host TPM certs). Agent plane starts only when all three are set. |
| `FLEET_MGR_BROKER_URL` | — | enrol-broker admin base URL (e.g. `https://127.0.0.1:9443`). Authorise pillar enabled only when set. |
| `FLEET_MGR_BROKER_CLIENT_CERT` / `_KEY` | — | manager's client cert for the broker admin mTLS (admin CA) |
| `FLEET_MGR_BROKER_CA` | — | CA that signs the broker's admin server cert |
| `FLEET_MGR_INFLUX_URL` | — | InfluxDB URL (private mgmt net). Metrics enabled only when URL + token set. |
| `FLEET_MGR_INFLUX_TOKEN` | — | InfluxDB token — **server-side only**, never on hosts |
| `FLEET_MGR_INFLUX_ORG` | `fleet` | InfluxDB org |
| `FLEET_MGR_INFLUX_BUCKET` | `fleet_health` | InfluxDB bucket |
| `FLEET_MGR_INVENTORY` | `/var/lib/fleet-manager/inventory.json` | inventory file (mount a volume) |
| `FLEET_MGR_SEED` | `false` | inject demo hosts so the SPA has data — dev only |
| `HOSTNQN_BASE` | `nqn.2026-06.org.example.fleet:host-` | hostnqn prefix (match the broker) |

## Deploy

Full bring-up (PKI, Let's Encrypt DNS-01, `register-admin`, compose) is in
[`DEPLOY.md`](DEPLOY.md). Config is layered: **`defaults.env`** (committed) +
**`.env`** (overrides/secrets), via `COMPOSE_ENV_FILES=defaults.env,.env`. In
short: set that + write your secrets to `.env` → `./server-pki.sh` (or, where
step-ca is co-located, the `cert-auto` **`agent-cert-renewer`** sidecar that
issues + auto-renews the agent-plane server cert and hot-reloads it — DEPLOY §2a) +
`../enrol-broker/admin-pki.sh … fleet-manager` → `docker compose up -d influxdb`
→ `docker compose run --rm fleet-manager register-admin <you>` → `docker compose
up -d fleet-manager traefik`. InfluxDB is private (published to loopback only);
the manager uses host networking to reach the loopback broker + bind a routable
agent listener.

## Dev run

```bash
FLEET_MGR_ADMIN_INSECURE=true FLEET_MGR_SEED=true \
  FLEET_MGR_ADMIN_ADDR=127.0.0.1:9500 FLEET_MGR_INVENTORY=/tmp/inv.json \
  go run .
curl -s localhost:9500/api/hosts | jq '.hosts[] | {name:.identity.starship_name, run:.version.image_version, want:.desired.target_version, online}'
```

## Build status

- **Checkpoint 1 (done):** Go skeleton, JSON-file inventory `Store`, admin JSON API
  (monitor + lifecycle), admin mTLS wiring, dev seed. Stdlib-only.
- **Checkpoint 2 (done):** gRPC agent plane — proto contract, mTLS (fleet CA)
  server, stream registry, identity-from-cert, snapshot→inventory, live
  desired-state push, `MetricsSink` seam. Full mTLS integration test.
- **Checkpoint 3 (done):** authorise pillar — the enrol-broker gained an
  `Accept`-gated JSON approve response (browsers/`verify-serving.sh` unchanged);
  the manager drives it over mTLS and births the inventory record on approve.
  httptest-mock-broker test.
- **Checkpoint 4 (done):** InfluxDB metrics gateway — snapshots written as points
  tagged with the authenticated identity (`MetricsSink`→`influxSink`), and a
  read API (`/api/hosts/{serial}/metrics`) that runs a Flux query with a **field
  whitelist** + hex-serial guard (no caller input reaches the query raw). Unit
  tests for point construction, query building, and unsafe-input rejection.
- **Checkpoint 5 (done):** admin **WebAuthn/passkey** auth replacing browser mTLS —
  credential store, `register-admin` bootstrap CLI (one-time HMAC enrol token),
  register/login ceremonies (usernameless + UV required), server-side sessions +
  CSRF middleware gating `/api/*`, admin TLS switched to server-auth with a
  hot-reloading cert (for LE DNS-01 renewals). Tests: enrol token, session
  middleware, cred-store persistence. Live passkey ceremony is a bench item.
- **Checkpoint 6 (done):** the **SPA** ([`ui/`](ui/)) — Svelte 5 + Tailwind 4,
  LYSANDRA restricted-violet theme, passkey login/enrol + Monitor/Authorise/
  Lifecycle + bespoke SVG charts. Embedded via `go:embed` behind the `embedspa`
  build tag; served same-origin over the admin plane. Smoke-tested end-to-end
  (index + assets + API through the SPA origin).
- **Checkpoint 7 (done):** deployment — three-stage `Dockerfile` (npm→dist→
  `go build -tags embedspa`→distroless), `docker-compose.yml` (private InfluxDB +
  host-networked manager), `defaults.env`, `server-pki.sh` (agent-plane fleet-CA
  server cert) **+ the `cert-auto` `agent-cert-renewer` sidecar** (auto-issues +
  auto-renews that cert from a co-located step-ca; the manager hot-reloads it via
  `GetCertificate` in `certreload.go` — no restart), and [`DEPLOY.md`](DEPLOY.md)
  (PKI, LE DNS-01, `register-admin` bootstrap). Design docs updated (manager marked
  built; resolved transport/auth/metrics forks; per-host-versioning gap recorded).
- **Checkpoint 8 (done):** hardening + ingress — CI image build
  (`fleet-manager-image.yaml`, pulled by tag), `backup.sh`; a **behind-proxy admin
  mode** (passkey over plaintext behind a TLS-terminating proxy) fronted by a
  **Traefik** mgmt ingress (:443 + Route 53 DNS-01, retiring the acme side-car).
  This aligns with the k8s target (Traefik → ingress controller + cert-manager);
  the agent plane stays direct mTLS via OPNsense L3. See [`DEPLOY.md`](DEPLOY.md).