diskless-workstation/serving/fleet-manager

fleet-manager — server-side fleet control plane

The management service for the diskless kiosk fleet, and the backend for the admin SPA. It is the central place to monitor, authorise, and lifecycle-control enrolled hosts. See docs/fleet-management-design.md and docs/tpm-enrolment-design.md §5.

Unlike fleet-hostapi / fleet-enrol (baked into the reproducible verity image, stdlib-only), this runs server-side as its own container on the management network, so external deps are fine (grpc-go, influxdb-client-go).

Two planes (distinct trust models)

Plane	Transport	Browser/peer identity	Who	Purpose
admin	HTTPS via Traefik (:443) → manager	WebAuthn/passkey (YubiKey) → session	admin browsers	JSON API + SPA (monitor / authorise / lifecycle)
agent	gRPC + mTLS	fleet CA (host TPM cert)	enrolled hosts	bidi stream: health/metrics up, desired-state down

The agent plane uses the host's reused TPM identity cert (the same one used for 802.1X EAP-TLS and from which the NVMe hostnqn derives — "one identity, three stages"). The admin plane authenticates the human with a hardware passkey (phishing-resistant, origin-bound), not a client cert — so there's no cert to import and nothing to phish. (The manager→broker hop is separately mTLS; that's service-to-service — see Authorise.)

Admin auth: passkeys/WebAuthn

• Login is usernameless (discoverable passkey) with userVerification: required — tap the YubiKey + PIN. A successful assertion mints an opaque, server-side session (Secure/HttpOnly/SameSite=Strict cookie); mutating requests must also echo the session CSRF token in X-Fleet-CSRF.
• Bootstrap is closed: fleet-manager register-admin <name> (run on the mgmt host) mints a one-time, HMAC-signed enrolment link; the admin opens it in the browser and registers their passkey. Single-use (jti ledger). Further admins can be enrolled from the UI once one exists.
• RP is origin-bound: FLEET_MGR_RP_ID / FLEET_MGR_RP_ORIGIN must be the UI's real hostname — for this deployment diskless-server.bitasylum.org / https://diskless-server.bitasylum.org (port-less, since Traefik serves :443).
• TLS + cert are owned by the traefik mgmt-plane ingress: it terminates a publicly-trusted Let's Encrypt cert (Route 53 DNS-01 — the UI is on the mgmt network, not internet-reachable, so HTTP-01/TLS-ALPN-01 can't validate) on :443 and forwards plaintext to the manager, which runs passkey auth ON, plaintext on loopback (WebAuthn only needs the origin to match, not the app's own transport). The manager never terminates TLS itself. Traefik is the k8s ingress-controller + cert-manager role in miniature.

Monitoring path (no host ever touches InfluxDB)

host agent ──gRPC bidi (mTLS, TPM cert)──▶ fleet-manager ──line protocol──▶ InfluxDB
           ◀──── desired-state (target ver / kiosk_url / reboot / drain) ──┘ (+ inventory health)
SPA ──GET /api/hosts/{serial}/metrics──▶ fleet-manager ──Flux──▶ InfluxDB ──JSON──▶ SPA renders charts

Hosts hold no InfluxDB credential and never reach it; the manager is the only writer/reader, holding the Influx token server-side. Identity is taken from the mTLS cert (anti-spoofing: a host can only write its own series). Push-only, no inbound host port. Graphs are rendered in the SPA, not Grafana.

Inventory record

Keyed by TPM cert serial (= hostnqn key). Born at enrolment approval, filled by the agent stream and admin action:

Host (cert_serial)
  ├─ identity : cert_serial, hostnqn, mac, ek_hash, starship_name
  ├─ binding  : asset_tag, switch_port, location        (recorded at approval)
  ├─ enrolled : approver, at, fingerprint
  ├─ version  : image_version, rootfs_digest, kernel     (agent-reported)
  ├─ health   : load, cpu, temp, tmpfs%, nvme_root, auth, selinux, chrony, kiosk  (latest snapshot; series in InfluxDB)
  ├─ usage    : kiosk_url, session_uptime, chromium_crashes, display
  └─ desired  : target_version, kiosk_url, reboot_at, drain   (SERVER-AUTHORITATIVE)

Persisted as a single JSON file (atomic writes) on a mounted volume — stdlib-only, inspectable, reproducible; a DB swap is a drop-in behind the Store interface.

API (admin plane)

Route	Auth	Status
GET /healthz	open	✅ ckpt 1
POST /auth/register/begin|finish	enrol token	✅ ckpt 5 — passkey registration ceremony
POST /auth/login/begin|finish	passkey	✅ ckpt 5 — usernameless discoverable login → session
POST /auth/logout	session	✅ ckpt 5
GET /api/whoami	session	✅ ckpt 5 — resume session + recover CSRF token
GET /api/hosts	session	✅ ckpt 1 — fleet overview
GET /api/hosts/{serial}	session	✅ ckpt 1 — host detail
PUT /api/hosts/{serial}/desired	session + CSRF	✅ ckpt 1 — set desired-state (audited to admin)
GET /api/enrol/pending	session	✅ ckpt 3 — proxies the broker queue
POST /api/enrol/{id}/approve	session + CSRF	✅ ckpt 3 — signs via broker + births inventory
POST /api/enrol/{id}/deny	session + CSRF	✅ ckpt 3
GET /api/hosts/{serial}/metrics?field=&range=	session	✅ ckpt 4 — Flux time-series (field whitelist)

/api/* is gated by the passkey session (mutations also need the CSRF header).

The authorise pillar drives the enrol-broker's admin plane over mTLS (the manager is a service client, gated by the dedicated admin CA) asking for JSON, so an approve returns the decided enrolment; the manager records the human approver (the signed-in passkey admin) and births the inventory record. The broker stays the enrolment state machine + step-ca signer (v1 "alongside" fold-in).

Agent plane — gRPC FleetAgent.Session bidi over mTLS (fleet CA): ✅ ckpt 2. Hello/HealthSnapshot/CommandResult up; DesiredState/Command down. Identity is the peer cert serial (never self-asserted); an unenrolled serial is refused. Desired-state is pushed live on connect + on change, and is durable in inventory for offline delivery. Metrics writing is behind a MetricsSink (no-op now; InfluxDB in ckpt 4).

Config (env)

Var	Default	Notes
FLEET_MGR_ADMIN_ADDR	127.0.0.1:9500	admin listener (loopback — Traefik forwards here; always plaintext)
FLEET_MGR_RP_ID	—	WebAuthn RP ID = UI hostname (diskless-server.bitasylum.org)
FLEET_MGR_RP_ORIGIN	—	WebAuthn origin (https://diskless-server.bitasylum.org)
FLEET_MGR_RP_NAME	Fleet Command	RP display name
FLEET_MGR_ENROLL_KEY_FILE	—	shared HMAC key for enrolment tokens (CLI + service)
FLEET_MGR_ADMIN_STORE	/var/lib/fleet-manager/admins.json	registered-passkey store
FLEET_MGR_SESSION_TTL_MIN	720	session lifetime (minutes)
FLEET_MGR_ADMIN_INSECURE	false	plaintext, no passkey auth — dev only
FLEET_MGR_AGENT_ADDR	127.0.0.1:9600	gRPC agent listener
FLEET_MGR_AGENT_CERT / _KEY	—	agent-plane server cert/key
FLEET_MGR_AGENT_CLIENT_CA	—	fleet CA (verifies host TPM certs). Agent plane starts only when all three are set.
FLEET_MGR_BROKER_URL	—	enrol-broker admin base URL (e.g. https://127.0.0.1:9443). Authorise pillar enabled only when set.
FLEET_MGR_BROKER_CLIENT_CERT / _KEY	—	manager's client cert for the broker admin mTLS (admin CA)
FLEET_MGR_BROKER_CA	—	CA that signs the broker's admin server cert
FLEET_MGR_INFLUX_URL	—	InfluxDB URL (private mgmt net). Metrics enabled only when URL + token set.
FLEET_MGR_INFLUX_TOKEN	—	InfluxDB token — server-side only, never on hosts
FLEET_MGR_INFLUX_ORG	fleet	InfluxDB org
FLEET_MGR_INFLUX_BUCKET	fleet_health	InfluxDB bucket
FLEET_MGR_INVENTORY	/var/lib/fleet-manager/inventory.json	inventory file (mount a volume)
FLEET_MGR_SEED	false	inject demo hosts so the SPA has data — dev only
HOSTNQN_BASE	nqn.2026-06.org.example.fleet:host-	hostnqn prefix (match the broker)

Deploy

Full bring-up (PKI, Let's Encrypt DNS-01, register-admin, compose) is in DEPLOY.md. Config is layered: defaults.env (committed) + .env (overrides/secrets), via COMPOSE_ENV_FILES=defaults.env,.env. In short: set that + write your secrets to .env → ./server-pki.sh + ../enrol-broker/admin-pki.sh … fleet-manager → docker compose up -d influxdb → docker compose run --rm fleet-manager register-admin <you> → docker compose up -d fleet-manager traefik. InfluxDB is private (published to loopback only); the manager uses host networking to reach the loopback broker + bind a routable agent listener.

Dev run

FLEET_MGR_ADMIN_INSECURE=true FLEET_MGR_SEED=true \
  FLEET_MGR_ADMIN_ADDR=127.0.0.1:9500 FLEET_MGR_INVENTORY=/tmp/inv.json \
  go run .
curl -s localhost:9500/api/hosts | jq '.hosts[] | {name:.identity.starship_name, run:.version.image_version, want:.desired.target_version, online}'

Build status

• Checkpoint 1 (done): Go skeleton, JSON-file inventory Store, admin JSON API (monitor + lifecycle), admin mTLS wiring, dev seed. Stdlib-only.
• Checkpoint 2 (done): gRPC agent plane — proto contract, mTLS (fleet CA) server, stream registry, identity-from-cert, snapshot→inventory, live desired-state push, MetricsSink seam. Full mTLS integration test.
• Checkpoint 3 (done): authorise pillar — the enrol-broker gained an Accept-gated JSON approve response (browsers/verify-serving.sh unchanged); the manager drives it over mTLS and births the inventory record on approve. httptest-mock-broker test.
• Checkpoint 4 (done): InfluxDB metrics gateway — snapshots written as points tagged with the authenticated identity (MetricsSink→influxSink), and a read API (/api/hosts/{serial}/metrics) that runs a Flux query with a field whitelist + hex-serial guard (no caller input reaches the query raw). Unit tests for point construction, query building, and unsafe-input rejection.
• Checkpoint 5 (done): admin WebAuthn/passkey auth replacing browser mTLS — credential store, register-admin bootstrap CLI (one-time HMAC enrol token), register/login ceremonies (usernameless + UV required), server-side sessions + CSRF middleware gating /api/*, admin TLS switched to server-auth with a hot-reloading cert (for LE DNS-01 renewals). Tests: enrol token, session middleware, cred-store persistence. Live passkey ceremony is a bench item.
• Checkpoint 6 (done): the SPA (ui/) — Svelte 5 + Tailwind 4, LYSANDRA restricted-violet theme, passkey login/enrol + Monitor/Authorise/ Lifecycle + bespoke SVG charts. Embedded via go:embed behind the embedspa build tag; served same-origin over the admin plane. Smoke-tested end-to-end (index + assets + API through the SPA origin).
• Checkpoint 7 (done): deployment — three-stage Dockerfile (npm→dist→ go build -tags embedspa→distroless), docker-compose.yml (private InfluxDB + host-networked manager), defaults.env, server-pki.sh (agent-plane fleet-CA server cert), and DEPLOY.md (PKI, LE DNS-01, register-admin bootstrap). Design docs updated (manager marked built; resolved transport/auth/ metrics forks; per-host-versioning gap recorded).
• Checkpoint 8 (done): hardening + ingress — CI image build (fleet-manager-image.yaml, pulled by tag), backup.sh; a behind-proxy admin mode (passkey over plaintext behind a TLS-terminating proxy) fronted by a Traefik mgmt ingress (:443 + Route 53 DNS-01, retiring the acme side-car). This aligns with the k8s target (Traefik → ingress controller + cert-manager); the agent plane stays direct mTLS via OPNsense L3. See DEPLOY.md.