diskless-workstation/serving/fleet-manager

fleet-manager — server-side fleet control plane

The management service for the diskless kiosk fleet, and the backend for the admin SPA. It is the central place to monitor, authorise, and lifecycle-control enrolled hosts. See docs/fleet-management-design.md and docs/tpm-enrolment-design.md §5.

Unlike fleet-hostapi / fleet-enrol (baked into the reproducible verity image, stdlib-only), this runs server-side as its own container on the management network, so external deps are fine (grpc-go, influxdb-client-go).

Two mTLS planes (distinct trust anchors)

Plane	Transport	Verifies	Who	Purpose
admin	HTTPS + mTLS	dedicated admin CA (admin-pki.sh)	admin browsers	JSON API + SPA (monitor / authorise / lifecycle)
agent	gRPC + mTLS	fleet CA (host TPM cert)	enrolled hosts	bidi stream: health/metrics up, desired-state down

The split is load-bearing: the agent plane uses the host's reused TPM identity cert (the same one used for 802.1X EAP-TLS and from which the NVMe hostnqn derives — "one identity, three stages"), while the admin plane uses a separate CA so an enrolled host can never reach approve/deny/lifecycle. See the identity discussion in the enrolment design §5.

Monitoring path (no host ever touches InfluxDB)

host agent ──gRPC bidi (mTLS, TPM cert)──▶ fleet-manager ──line protocol──▶ InfluxDB
           ◀──── desired-state (target ver / kiosk_url / reboot / drain) ──┘ (+ inventory health)
SPA ──GET /api/hosts/{serial}/metrics──▶ fleet-manager ──Flux──▶ InfluxDB ──JSON──▶ SPA renders charts

Hosts hold no InfluxDB credential and never reach it; the manager is the only writer/reader, holding the Influx token server-side. Identity is taken from the mTLS cert (anti-spoofing: a host can only write its own series). Push-only, no inbound host port. Graphs are rendered in the SPA, not Grafana.

Inventory record

Keyed by TPM cert serial (= hostnqn key). Born at enrolment approval, filled by the agent stream and admin action:

Host (cert_serial)
  ├─ identity : cert_serial, hostnqn, mac, ek_hash, starship_name
  ├─ binding  : asset_tag, switch_port, location        (recorded at approval)
  ├─ enrolled : approver, at, fingerprint
  ├─ version  : image_version, rootfs_digest, kernel     (agent-reported)
  ├─ health   : load, cpu, temp, tmpfs%, nvme_root, auth, selinux, chrony, kiosk  (latest snapshot; series in InfluxDB)
  ├─ usage    : kiosk_url, session_uptime, chromium_crashes, display
  └─ desired  : target_version, kiosk_url, reboot_at, drain   (SERVER-AUTHORITATIVE)

Persisted as a single JSON file (atomic writes) on a mounted volume — stdlib-only, inspectable, reproducible; a DB swap is a drop-in behind the Store interface.

API (admin plane)

Route	Status
GET /healthz	✅ ckpt 1
GET /api/hosts	✅ ckpt 1 — fleet overview
GET /api/hosts/{serial}	✅ ckpt 1 — host detail
PUT /api/hosts/{serial}/desired	✅ ckpt 1 — set desired-state (audited to admin CN)
GET /api/enrol/pending	✅ ckpt 3 — proxies the broker queue
POST /api/enrol/{id}/approve	✅ ckpt 3 — signs via broker + births inventory (identity/binding/enrolled)
POST /api/enrol/{id}/deny	✅ ckpt 3
GET /api/hosts/{serial}/metrics?field=&range=	✅ ckpt 4 — Flux time-series (field whitelist)

The authorise pillar drives the enrol-broker's admin plane over mTLS (the manager is a service client, gated by the dedicated admin CA) asking for JSON, so an approve returns the decided enrolment; the manager records the human approver (its own admin mTLS CN) and births the inventory record. The broker stays the enrolment state machine + step-ca signer (v1 "alongside" fold-in).

Agent plane — gRPC FleetAgent.Session bidi over mTLS (fleet CA): ✅ ckpt 2. Hello/HealthSnapshot/CommandResult up; DesiredState/Command down. Identity is the peer cert serial (never self-asserted); an unenrolled serial is refused. Desired-state is pushed live on connect + on change, and is durable in inventory for offline delivery. Metrics writing is behind a MetricsSink (no-op now; InfluxDB in ckpt 4).

Config (env)

Var	Default	Notes
FLEET_MGR_ADMIN_ADDR	127.0.0.1:9500	admin listener
FLEET_MGR_ADMIN_CERT / _KEY	—	admin server cert/key (mTLS)
FLEET_MGR_ADMIN_CLIENT_CA	—	dedicated admin CA (NOT the fleet CA)
FLEET_MGR_ADMIN_INSECURE	false	plaintext, no client-cert — dev only
FLEET_MGR_AGENT_ADDR	127.0.0.1:9600	gRPC agent listener
FLEET_MGR_AGENT_CERT / _KEY	—	agent-plane server cert/key
FLEET_MGR_AGENT_CLIENT_CA	—	fleet CA (verifies host TPM certs). Agent plane starts only when all three are set.
FLEET_MGR_BROKER_URL	—	enrol-broker admin base URL (e.g. https://127.0.0.1:9443). Authorise pillar enabled only when set.
FLEET_MGR_BROKER_CLIENT_CERT / _KEY	—	manager's client cert for the broker admin mTLS (admin CA)
FLEET_MGR_BROKER_CA	—	CA that signs the broker's admin server cert
FLEET_MGR_INFLUX_URL	—	InfluxDB URL (private mgmt net). Metrics enabled only when URL + token set.
FLEET_MGR_INFLUX_TOKEN	—	InfluxDB token — server-side only, never on hosts
FLEET_MGR_INFLUX_ORG	fleet	InfluxDB org
FLEET_MGR_INFLUX_BUCKET	fleet_health	InfluxDB bucket
FLEET_MGR_INVENTORY	/var/lib/fleet-manager/inventory.json	inventory file (mount a volume)
FLEET_MGR_SEED	false	inject demo hosts so the SPA has data — dev only
HOSTNQN_BASE	nqn.2026-06.org.example.fleet:host-	hostnqn prefix (match the broker)

InfluxDB env lands in checkpoint 4.

Dev run

FLEET_MGR_ADMIN_INSECURE=true FLEET_MGR_SEED=true \
  FLEET_MGR_ADMIN_ADDR=127.0.0.1:9500 FLEET_MGR_INVENTORY=/tmp/inv.json \
  go run .
curl -s localhost:9500/api/hosts | jq '.hosts[] | {name:.identity.starship_name, run:.version.image_version, want:.desired.target_version, online}'

Build status

• Checkpoint 1 (done): Go skeleton, JSON-file inventory Store, admin JSON API (monitor + lifecycle), admin mTLS wiring, dev seed. Stdlib-only.
• Checkpoint 2 (done): gRPC agent plane — proto contract, mTLS (fleet CA) server, stream registry, identity-from-cert, snapshot→inventory, live desired-state push, MetricsSink seam. Full mTLS integration test.
• Checkpoint 3 (done): authorise pillar — the enrol-broker gained an Accept-gated JSON approve response (browsers/verify-serving.sh unchanged); the manager drives it over mTLS and births the inventory record on approve. httptest-mock-broker test.
• Checkpoint 4 (done): InfluxDB metrics gateway — snapshots written as points tagged with the authenticated identity (MetricsSink→influxSink), and a read API (/api/hosts/{serial}/metrics) that runs a Flux query with a field whitelist + hex-serial guard (no caller input reaches the query raw). Unit tests for point construction, query building, and unsafe-input rejection.
• Next: themed SPA (ckpt 5) → Docker/compose + PKI + docs (ckpt 6).