Point the compose service at the registry image the new CI workflow pushes
(git.bitasylum.org/audiobuzz/fleet-manager:${FLEET_MANAGER_VERSION:-latest}) so
mgmt hosts `docker compose pull` instead of building at deploy time — matching
dnsmasq/spdk/kiosk. `build: .` stays for local dev. Add FLEET_MANAGER_VERSION to
.env.example.
Follow-up to comma-separated RP_ORIGIN support: the enrolment URL builder used
the raw origin string, which would embed the whole comma list. Take the first
origin for the link.
Passkeys bind to the exact scheme://host:port the browser used, so a single
configured origin fails "Error validating origin" when reached via a different
form (e.g. :9500 direct vs a 443 proxy). Split FLEET_MGR_RP_ORIGIN on commas so
several exact origins can be allowed. Single-value config is unchanged.
Flipping FLEET_MGR_ADMIN_INSECURE=false before bootstrapping an admin crash-looped
on a cryptic "enrol key: open /data/enroll.key: no such file or directory". The
key is created by `register-admin`, so on ErrNotExist say exactly that (and note
the insecure escape hatch) instead of surfacing the raw errno.
Split-horizon DNS made lego's propagation check query the container's internal
resolver, which doesn't serve the public _acme-challenge TXT record, so the
challenge timed out. Add ACME_RESOLVERS (comma-separated host:port, default
8.8.8.8:53) → one --dns.resolvers flag each. Documented in .env + DEPLOY.md.
The goacme/lego image ships the binary at /lego, not on PATH, so the bare `lego`
call failed ("not found") once the entrypoint was overridden with the shell.
Detect it (command -v, then /lego fallback) and invoke via $LEGO.
lego auto-discovers the Route 53 zone via ListHostedZonesByName, so the zone ID
is optional — but pinning it disambiguates overlapping public/private zones and
lets the IAM policy drop the list permission. Thread it through compose + .env
+ DEPLOY.md (no script change; lego reads it from the environment).
Auto-issue + auto-renew the admin-UI TLS cert with a long-running compose
side-car instead of a host cron. The `acme` service (goacme/lego image, running
acme/renew-loop.sh) obtains the cert over DNS-01 via the AWS Route 53 API,
writes fullchain/privkey into ./pki/admin/ — the SAME dir the manager reads
(./pki:/pki:ro) — and chowns them to the nonroot runtime uid so the read-only
container can load the key. The manager's certReloader hot-swaps on the mtime
bump, no restart. Loop renews within 30 days of expiry; ACME_STAGING=1 gives a
dry-run vs LE staging.
- docker-compose.yml: `acme` service + `acme-state` volume (lego account/cert
store). ACME_DOMAIN reuses FLEET_MGR_RP_ID so cert host == passkey origin.
- .env.example: ACME_EMAIL + AWS creds (or instance role) + ACME_STAGING.
- DEPLOY.md §2c: rewritten to the side-car flow — scoped Route 53 IAM policy,
bring-acme-up-first ordering, flip off insecure, and an origin-match note.
(The docker-compose.yml commit also carries the local harbor.bitasylum.org
influxdb image mirror edit.)
A just-approved inventory record has a zero-value last_seen (Go 0001-01-01),
which parsed to a pre-epoch instant and slipped past the !t guard, rendering as
"739800d ago". Treat any at-or-before-epoch timestamp as "never" — the honest
state for a born-but-never-connected host (its health/version stay "—" until an
agent dials the gRPC plane).
The distroless nonroot runtime (uid 65532) couldn't write /data/inventory.json
because a named volume is created root-owned — approve failed with permission
denied on the inventory birth. Pre-create /data owned by nonroot in the image so
a fresh volume inherits it via Docker copy-up; an already-created root-owned
volume still needs a one-time chown (documented).
DEPLOY.md:
- 2b now covers BOTH broker deployments — the main serving stack
(enrol-pki/broker, ENROL_ADMIN_INSECURE=true by default → must flip to false)
and the bench stack — with the mint-once-into-broker-dir, copy-CA-to-manager
flow and an mTLS verify curl.
- Troubleshooting section: the data-volume permission-denied fix, the
"authorise plane not configured" (recreate + single-instance) fix, and the
"wrong version number" (broker plaintext) fix.
Every plane logged its state EXCEPT the broker success path, which was silent —
so "authorise pillar working" was indistinguishable from "not logged". Add an
ENABLED line matching the InfluxDB/agent/admin convention.
The compose wires the authorise (broker) and agent planes to fixed cert paths,
so a first bring-up that hasn't run the PKI steps yet crash-looped on the first
missing file (e.g. /pki/broker/clients/fleet-manager.crt) instead of booting.
Treat a configured-but-absent cert as "plane not yet provisioned": log a loud
WARNING and disable that plane, so the manager boots and each plane lights up
as its PKI appears. A malformed/unreadable cert still hard-fails (real
misconfig). The admin plane stays a hard requirement (FLEET_MGR_ADMIN_INSECURE
is its bench escape). Adds a missingFiles() helper; hub push path already
nil-guards a disabled agent plane.
Complete the deployment story for the server-side control plane:
- docker-compose.yml: private InfluxDB (published to loopback only) + a
host-networked fleet-manager (reaches the loopback enrol-broker on
127.0.0.1:9443 and binds a routable gRPC agent listener, like the broker).
- .env.example: RP origin (diskless-server.bitasylum.org), Influx token/DB
password, cert/PKI paths; the Influx token wires influxdb + manager with
one value.
- server-pki.sh: issue the agent-plane server cert from the fleet CA (step)
and stage the fleet root as the client-verify CA.
- DEPLOY.md: end-to-end runbook — three-cert PKI (fleet-CA agent cert, broker
admin-client cert, Let's Encrypt DNS-01 UI cert), register-admin passkey
bootstrap, compose up, hardware-free smoke test, backups.
- .dockerignore: keep host node_modules/dist/pki/secrets out of the build
context so the embedded SPA is the one built in-image.
- .gitignore: never commit .env / issued PKI / data.
Design docs marked as-built: fleet-management-design.md as-built note
(resolved forks — gRPC bidi transport, InfluxDB metrics, SPA graphs, passkey
admin auth, broker fold-in "alongside"); tpm-enrolment-design.md (broker
fold-in + agent transport fork resolved); TODO.md #14 → partly built, with
the on-host agent + per-host boot-version assignment (#6) called out as the
remaining work.
Two-stage build: node builds the Svelte SPA to dist, then golang compiles the
binary with the SPA embedded (-tags embedspa). Committed .pb.go stubs mean no
protoc. Ships as a small static distroless/nonroot image; same binary serves the
register-admin subcommand. (Not build-verified here — no docker in the
devcontainer — but the go build -tags embedspa + npm build are both proven.)
Per the option-A decision (deploy current design without per-host versioning):
add an honest note on the Lifecycle panel that kiosk-url/reboot/drain reconcile
via the agent but per-host boot-version assignment (dnsmasq cohort steering) is
not yet wired, so a version pin is stored + pushed but does not change what the
host boots. Avoids implying the pin takes effect.
Replace the free-text target-version field with a select populated from versions
known across the fleet (each host's running + targeted version, newest first),
plus a 'custom...' escape to a text input for a version not yet seen. Avoids
typos on the pin. (Authoritative serving-side source is a follow-up.)
- Contrast: lighten the muted/faint text tokens (AA on the hull panels) and make
Monitor host-card data values bright ink against dim faint labels, so version/
tmpfs/seen details read clearly on the blue-grey panels (was grey-on-grey).
- Lifecycle: reboot selector now offers none | now | at... — 'now' sets
reboot_at to the current time (agent reboots on reconcile); still confirm-gated
as a high-consequence action.
The themed single-page admin UI (serving/fleet-manager/ui/), embedded into the
binary and served same-origin over the admin plane.
- Svelte 5 + Tailwind 4 + Vite; a sibling of serving/kiosk-app. Theme = LYSANDRA
'restricted-violet' (high-security evolution of the kiosk's flight-deck DNA):
amber deck, VIOLET reserved for privileged/destructive + auth surfaces, teal
telemetry, red alert, clearance chrome (corner-reticle panels, scanlines, mono
clearance labels, secure-link status bar).
- Passkey gate: Login (usernameless discoverable WebAuthn) + Enrol (from the
register-admin one-time link) via navigator.credentials, with base64url<->buffer
helpers and the go-webauthn response shapes.
- Three pillars over the JSON API: Monitor (fleet roster + version-drift + health
+ per-host detail with bespoke SVG telemetry charts), Authorise (pending queue:
fingerprint cross-check, approve+bind in violet, deny), Lifecycle (desired-state
form; drain/reboot confirm-gated). Session-aware fetch client (cookie + CSRF).
- go:embed behind the build tag (spa_embed.go / spa_stub.go) so the
default go build/test need no dist/; mounted at / (api/auth/healthz win).
Verified: npm build clean (71KB JS / 33KB CSS + self-hosted fonts); default +
-tags embedspa go builds; smoke test serves index + assets + API through the SPA
origin. Against an insecure+seeded backend the deck renders passkey-free for
theme work. dist/ + node_modules gitignored.
The human->UI hop now authenticates with a hardware passkey (YubiKey), not an
mTLS client cert: phishing-resistant, origin-bound, better UX. (manager->broker
stays mTLS — service-to-service.)
- webauthn.go: credential store (registered admins + passkeys + a one-time-token
jti replay ledger; JSON file, atomic writes) + the webauthn.User impl + RP setup.
- cli.go: 'fleet-manager register-admin <name>' mints a short-lived HMAC-signed
enrolment link over a shared enrol key; the browser completes the WebAuthn
create ceremony. No anyone-can-register endpoint; single-use via the ledger.
- authapi.go: register/login ceremonies (usernameless discoverable login,
userVerification required) + logout + whoami; short-lived in-memory ceremony
store between begin/finish.
- session.go: opaque server-side sessions (Secure/HttpOnly/SameSite=Strict cookie)
+ requireSession middleware gating /api/*, with a CSRF header check on mutations.
adminName now comes from the session (audit attribution).
- main.go: register-admin subcommand dispatch; admin TLS switched to SERVER-AUTH
(no client cert) with a HOT-RELOADING cert (certReloader) so a Let's Encrypt
DNS-01 renewal is picked up without a restart. RP = diskless-server.bitasylum.org.
/api/* mounted behind requireSession; /auth/* + /healthz public.
Cert strategy (per design decision): the UI HTTPS cert is a publicly-trusted LE
cert via DNS-01 (mgmt-network UI isn't internet-reachable, so HTTP-01/TLS-ALPN-01
can't validate; DNS-01 proves control of bitasylum.org via TXT + supports
wildcards). External ACME client issues+renews to the cert path; manager reloads.
Deps: github.com/go-webauthn/webauthn (server-side). Tests: enrol-token
mint/verify/tamper/expiry, session middleware (401/403/CSRF/expiry), cred-store
persistence + jti ledger. build/test/vet/gofmt clean. Live passkey ceremony
(virtual authenticator / real YubiKey) is a bench item.
The manager is the ONLY thing that talks to InfluxDB, both directions; hosts hold
no token and never reach it.
- influx.go: influxSink implements MetricsSink — each agent HealthSnapshot is
written as a fleet_health point, identity (serial/hostnqn/starship) as TAGS
from the AUTHENTICATED stream (a host can only write its own series), telemetry
as fields. QueryField answers the SPA's read: a Flux query built with a strict
FIELD WHITELIST + hex-serial guard + parsed-duration range, so no caller input
is interpolated raw (injection-safe). Server-side influxdb-client-go/v2.
- admin.go: GET /api/hosts/{serial}/metrics?field=&range= -> time-series JSON;
503 when InfluxDB isn't wired, 400 on a bad range, 404 for unknown host.
- main.go: sink created when FLEET_MGR_INFLUX_URL+_TOKEN set, fed to the agent
hub; nil -> no-op sink so the manager runs without metrics.
Tests: snapshotPoint tags/fields/time, fluxQuery contents + %q quoting, and
QueryField rejects a non-hex serial + non-whitelisted field. build/test/vet/gofmt
clean. Live InfluxDB path validated in the compose bench (ckpt 6).
The AUTHORISE pillar: the manager owns the enrolment queue UX, the enrol-broker
stays the state machine + step-ca signer (v1 'alongside' fold-in).
enrol-broker (minimal, backward-compatible change):
- admin approve/deny now return the decided Enrolment as JSON when the caller
sends Accept: application/json. Browsers still redirect; other API callers
(verify-serving.sh) still get 204. The cert PEM stays unexported (only the
requesting host gets it via /status) — the JSON carries cert_serial, hostnqn,
context, fingerprint, cn.
fleet-manager:
- broker.go: mTLS client for the broker admin plane (Pending/Approve/Deny),
nil when unconfigured so the manager runs standalone.
- admin.go: GET /api/enrol/pending (proxy), POST /api/enrol/{id}/approve
(drives broker, then BIRTHS the inventory record — identity from the returned
serial/hostnqn/context, binding from the operator's optional body, enrolled
with the HUMAN approver = the manager's admin mTLS CN), POST .../deny.
503 when the broker isn't wired.
Test: httptest mock broker asserts the Accept:application/json approve, pending
proxy, inventory birth (identity+binding+enrolled), and the disabled-503 path.
Broker + manager both build/vet/test/gofmt clean.
The steady-state host<->manager transport, resolving the design's OPEN FORK to
bespoke gRPC (not OTLP+Wazuh). One long-lived, host-dialed, mTLS bidi stream:
health/metrics up, desired-state/commands down — no polling, real-time control,
still no inbound host port.
- proto/fleet_agent.proto: FleetAgent.Session(stream AgentUp) -> stream ManagerDown.
AgentUp = Hello | HealthSnapshot | CommandResult; ManagerDown = DesiredState |
Command(reboot/drain/resync). int64 unix times (no well-known-type dep).
Generated stubs committed (CI needs no protoc).
- agent.go: gRPC server with mTLS verified by the FLEET CA (the host's reused TPM
identity cert — distinct from the admin plane's admin CA). Identity is the peer
cert SERIAL, taken from the verified cert = inventory key, never self-asserted;
an unenrolled serial is refused (enrolment gates the stream). Per-conn send
channel decouples admin pushes from stream.Send. Stream registry (serial->conn)
with reconnect eviction. On connect + on change, desired-state is pushed live;
inventory stays authoritative so offline hosts reconcile on reconnect. Metrics
go through a MetricsSink seam (no-op now; InfluxDB in ckpt 4).
- main.go: agent plane starts only when its cert/key/CA are set; admin can run
standalone. Lifecycle PUT now live-pushes and reports pushed_live.
Full mTLS integration test (in-process fleet CA + host client cert): initial
desired-state on connect, Hello+snapshot -> inventory + online presence, live
desired-state push, and unenrolled-serial rejection. go build/test/vet/gofmt clean.
Deps (server-side only): google.golang.org/grpc + protobuf.
The checkpoint-1 commit included the 9.1M compiled 'fleet-manager' binary. Remove
it from tracking and gitignore it (/fleet-manager + *.test), matching the
enrol-broker convention. Build artifacts don't belong in git.
First slice of the server-side fleet control plane (serving/fleet-manager/), the
backend for the admin SPA (monitor / authorise / lifecycle). See
docs/fleet-management-design.md + tpm-enrolment-design.md §5.
This checkpoint (stdlib-only):
- Inventory record (identity/binding/enrolled/version/health/usage/desired),
keyed by TPM cert serial = hostnqn key, per the design's 'inventory record born
at enrolment'. JSON-file Store (atomic writes, mutex) behind an interface so a
DB swap is a drop-in.
- Admin plane: mTLS server verified by the DEDICATED admin CA (never the fleet CA,
so a host cert can't reach admin) + a plaintext dev mode. JSON API:
GET /api/hosts, GET /api/hosts/{serial}, PUT /api/hosts/{serial}/desired
(desired-state audited to the admin client-cert CN).
- Dev seed (FLEET_MGR_SEED) so the SPA has data before real enrolment/agent flow.
Smoke-tested: list/get/desired-PUT (incl. version-drift + drain), 404s, atomic
persistence. go build + go vet + gofmt clean.
Agent gRPC plane, broker-fold-in authorise, InfluxDB metrics, and the themed SPA
follow in checkpoints 2-6.
Audiobuzz