diskless-workstation/serving/fleet-manager/README.md

fleet-manager — server-side fleet control plane

The management service for the diskless kiosk fleet, and the backend for the admin SPA. It is the central place to monitor, authorise, and lifecycle-control enrolled hosts. See docs/fleet-management-design.md and docs/tpm-enrolment-design.md §5.

Unlike fleet-hostapi / fleet-enrol (baked into the reproducible verity image, stdlib-only), this runs server-side as its own container on the management network, so external deps are fine (grpc-go, influxdb-client-go).

Two planes (distinct trust models)

Plane	Transport	Browser/peer identity	Who	Purpose
admin	HTTPS (server-auth)	WebAuthn/passkey (YubiKey) → session	admin browsers	JSON API + SPA (monitor / authorise / lifecycle)
agent	gRPC + mTLS	fleet CA (host TPM cert)	enrolled hosts	bidi stream: health/metrics up, desired-state down

The agent plane uses the host's reused TPM identity cert (the same one used for 802.1X EAP-TLS and from which the NVMe hostnqn derives — "one identity, three stages"). The admin plane authenticates the human with a hardware passkey (phishing-resistant, origin-bound), not a client cert — so there's no cert to import and nothing to phish. (The manager→broker hop is separately mTLS; that's service-to-service — see Authorise.)

Admin auth: passkeys/WebAuthn

• Login is usernameless (discoverable passkey) with userVerification: required — tap the YubiKey + PIN. A successful assertion mints an opaque, server-side session (Secure/HttpOnly/SameSite=Strict cookie); mutating requests must also echo the session CSRF token in X-Fleet-CSRF.
• Bootstrap is closed: fleet-manager register-admin <name> (run on the mgmt host) mints a one-time, HMAC-signed enrolment link; the admin opens it in the browser and registers their passkey. Single-use (jti ledger). Further admins can be enrolled from the UI once one exists.
• RP is origin-bound: FLEET_MGR_RP_ID / FLEET_MGR_RP_ORIGIN must be the UI's real hostname — for this deployment diskless-server.bitasylum.org / https://diskless-server.bitasylum.org.
• HTTPS cert for the UI is a publicly-trusted cert via Let's Encrypt DNS-01 (the UI is on the mgmt network, not internet-reachable, so HTTP-01 / TLS-ALPN-01 can't validate; DNS-01 proves control of bitasylum.org via a TXT record and supports wildcards). An external ACME client (lego/certbot + DNS provider) issues + renews to FLEET_MGR_ADMIN_CERT/_KEY; the manager hot-reloads the cert on renewal (no restart). A trusted cert also keeps the WebAuthn secure-context clean (no browser warnings).

Monitoring path (no host ever touches InfluxDB)

host agent ──gRPC bidi (mTLS, TPM cert)──▶ fleet-manager ──line protocol──▶ InfluxDB
           ◀──── desired-state (target ver / kiosk_url / reboot / drain) ──┘ (+ inventory health)
SPA ──GET /api/hosts/{serial}/metrics──▶ fleet-manager ──Flux──▶ InfluxDB ──JSON──▶ SPA renders charts

Hosts hold no InfluxDB credential and never reach it; the manager is the only writer/reader, holding the Influx token server-side. Identity is taken from the mTLS cert (anti-spoofing: a host can only write its own series). Push-only, no inbound host port. Graphs are rendered in the SPA, not Grafana.

Inventory record

Keyed by TPM cert serial (= hostnqn key). Born at enrolment approval, filled by the agent stream and admin action:

Host (cert_serial)
  ├─ identity : cert_serial, hostnqn, mac, ek_hash, starship_name
  ├─ binding  : asset_tag, switch_port, location        (recorded at approval)
  ├─ enrolled : approver, at, fingerprint
  ├─ version  : image_version, rootfs_digest, kernel     (agent-reported)
  ├─ health   : load, cpu, temp, tmpfs%, nvme_root, auth, selinux, chrony, kiosk  (latest snapshot; series in InfluxDB)
  ├─ usage    : kiosk_url, session_uptime, chromium_crashes, display
  └─ desired  : target_version, kiosk_url, reboot_at, drain   (SERVER-AUTHORITATIVE)

Persisted as a single JSON file (atomic writes) on a mounted volume — stdlib-only, inspectable, reproducible; a DB swap is a drop-in behind the Store interface.

API (admin plane)

Route	Auth	Status
GET /healthz	open	✅ ckpt 1
POST /auth/register/begin|finish	enrol token	✅ ckpt 5 — passkey registration ceremony
POST /auth/login/begin|finish	passkey	✅ ckpt 5 — usernameless discoverable login → session
POST /auth/logout	session	✅ ckpt 5
GET /api/whoami	session	✅ ckpt 5 — resume session + recover CSRF token
GET /api/hosts	session	✅ ckpt 1 — fleet overview
GET /api/hosts/{serial}	session	✅ ckpt 1 — host detail
PUT /api/hosts/{serial}/desired	session + CSRF	✅ ckpt 1 — set desired-state (audited to admin)
GET /api/enrol/pending	session	✅ ckpt 3 — proxies the broker queue
POST /api/enrol/{id}/approve	session + CSRF	✅ ckpt 3 — signs via broker + births inventory
POST /api/enrol/{id}/deny	session + CSRF	✅ ckpt 3
GET /api/hosts/{serial}/metrics?field=&range=	session	✅ ckpt 4 — Flux time-series (field whitelist)

/api/* is gated by the passkey session (mutations also need the CSRF header).

The authorise pillar drives the enrol-broker's admin plane over mTLS (the manager is a service client, gated by the dedicated admin CA) asking for JSON, so an approve returns the decided enrolment; the manager records the human approver (the signed-in passkey admin) and births the inventory record. The broker stays the enrolment state machine + step-ca signer (v1 "alongside" fold-in).

Agent plane — gRPC FleetAgent.Session bidi over mTLS (fleet CA): ✅ ckpt 2. Hello/HealthSnapshot/CommandResult up; DesiredState/Command down. Identity is the peer cert serial (never self-asserted); an unenrolled serial is refused. Desired-state is pushed live on connect + on change, and is durable in inventory for offline delivery. Metrics writing is behind a MetricsSink (no-op now; InfluxDB in ckpt 4).

Config (env)

Var	Default	Notes
FLEET_MGR_ADMIN_ADDR	127.0.0.1:9500	admin listener
FLEET_MGR_ADMIN_CERT / _KEY	—	UI server cert/key (server-auth; LE DNS-01, hot-reloaded)
FLEET_MGR_RP_ID	—	WebAuthn RP ID = UI hostname (diskless-server.bitasylum.org)
FLEET_MGR_RP_ORIGIN	—	WebAuthn origin (https://diskless-server.bitasylum.org)
FLEET_MGR_RP_NAME	Fleet Command	RP display name
FLEET_MGR_ENROLL_KEY_FILE	—	shared HMAC key for enrolment tokens (CLI + service)
FLEET_MGR_ADMIN_STORE	/var/lib/fleet-manager/admins.json	registered-passkey store
FLEET_MGR_SESSION_TTL_MIN	720	session lifetime (minutes)
FLEET_MGR_ADMIN_INSECURE	false	plaintext, no passkey auth — dev only
FLEET_MGR_AGENT_ADDR	127.0.0.1:9600	gRPC agent listener
FLEET_MGR_AGENT_CERT / _KEY	—	agent-plane server cert/key
FLEET_MGR_AGENT_CLIENT_CA	—	fleet CA (verifies host TPM certs). Agent plane starts only when all three are set.
FLEET_MGR_BROKER_URL	—	enrol-broker admin base URL (e.g. https://127.0.0.1:9443). Authorise pillar enabled only when set.
FLEET_MGR_BROKER_CLIENT_CERT / _KEY	—	manager's client cert for the broker admin mTLS (admin CA)
FLEET_MGR_BROKER_CA	—	CA that signs the broker's admin server cert
FLEET_MGR_INFLUX_URL	—	InfluxDB URL (private mgmt net). Metrics enabled only when URL + token set.
FLEET_MGR_INFLUX_TOKEN	—	InfluxDB token — server-side only, never on hosts
FLEET_MGR_INFLUX_ORG	fleet	InfluxDB org
FLEET_MGR_INFLUX_BUCKET	fleet_health	InfluxDB bucket
FLEET_MGR_INVENTORY	/var/lib/fleet-manager/inventory.json	inventory file (mount a volume)
FLEET_MGR_SEED	false	inject demo hosts so the SPA has data — dev only
HOSTNQN_BASE	nqn.2026-06.org.example.fleet:host-	hostnqn prefix (match the broker)

InfluxDB env lands in checkpoint 4.

Dev run

FLEET_MGR_ADMIN_INSECURE=true FLEET_MGR_SEED=true \
  FLEET_MGR_ADMIN_ADDR=127.0.0.1:9500 FLEET_MGR_INVENTORY=/tmp/inv.json \
  go run .
curl -s localhost:9500/api/hosts | jq '.hosts[] | {name:.identity.starship_name, run:.version.image_version, want:.desired.target_version, online}'

Build status

• Checkpoint 1 (done): Go skeleton, JSON-file inventory Store, admin JSON API (monitor + lifecycle), admin mTLS wiring, dev seed. Stdlib-only.
• Checkpoint 2 (done): gRPC agent plane — proto contract, mTLS (fleet CA) server, stream registry, identity-from-cert, snapshot→inventory, live desired-state push, MetricsSink seam. Full mTLS integration test.
• Checkpoint 3 (done): authorise pillar — the enrol-broker gained an Accept-gated JSON approve response (browsers/verify-serving.sh unchanged); the manager drives it over mTLS and births the inventory record on approve. httptest-mock-broker test.
• Checkpoint 4 (done): InfluxDB metrics gateway — snapshots written as points tagged with the authenticated identity (MetricsSink→influxSink), and a read API (/api/hosts/{serial}/metrics) that runs a Flux query with a field whitelist + hex-serial guard (no caller input reaches the query raw). Unit tests for point construction, query building, and unsafe-input rejection.
• Checkpoint 5 (done): admin WebAuthn/passkey auth replacing browser mTLS — credential store, register-admin bootstrap CLI (one-time HMAC enrol token), register/login ceremonies (usernameless + UV required), server-side sessions + CSRF middleware gating /api/*, admin TLS switched to server-auth with a hot-reloading cert (for LE DNS-01 renewals). Tests: enrol token, session middleware, cred-store persistence. Live passkey ceremony is a bench item.
• Checkpoint 6 (done): the SPA (ui/) — Svelte 5 + Tailwind 4, LYSANDRA restricted-violet theme, passkey login/enrol + Monitor/Authorise/ Lifecycle + bespoke SVG charts. Embedded via go:embed behind the embedspa build tag; served same-origin over the admin plane. Smoke-tested end-to-end (index + assets + API through the SPA origin).
• Next: Docker/compose + PKI/RP/ACME + docs (ckpt 7).