diskless-workstation/serving/fleet-manager/DEPLOY.md

fleet-manager — deployment runbook

Bring-up of the fleet-manager (control plane + admin SPA) and its private InfluxDB on the management host. This is a lab/on-prem deployment; adapt the PKI/ACME steps to your environment. See README.md for the design and docs/fleet-management-design.md for the architecture.

Where it runs. The management host — the trusted side, alongside the enrol-broker and step-ca (serving/dot1x-bench/). Never on the boot LAN. The admin plane (passkey UI) faces admins; the agent plane (gRPC/mTLS) faces enrolled hosts; both are segmented from the boot network.

Prerequisites

• Docker + compose, and the step CLI (smallstep) on the host.
• The fleet step-ca already bootstrapped (serving/dot1x-bench/ — docker compose up -d step-ca && ./step-ca/bootstrap.sh), so ../dot1x-bench/pki/root_ca.crt and the enrol provisioner exist.
• The enrol-broker running (serving/dot1x-bench/ compose) with its admin plane on 127.0.0.1:9443.
• A DNS zone you control for the UI hostname (bitasylum.org) and an ACME client that can do DNS-01 (e.g. lego or certbot + your DNS provider).
• A YubiKey (or other WebAuthn authenticator) for the bootstrap admin.

1. Configure

cd serving/fleet-manager
cp .env.example .env
# Edit .env:
#   FLEET_MGR_RP_ID / _ORIGIN   -> your UI hostname (diskless-server.bitasylum.org)
#   FLEET_MGR_INFLUX_TOKEN      -> a long random secret (openssl rand -hex 32)
#   INFLUX_ADMIN_PASSWORD       -> a strong password

The Influx token is handed to both InfluxDB (its init admin token) and the manager, so one value wires them with no copy step. .env is gitignored.

2. PKI

Three certs, three trust roots — do not conflate them (see README "two planes"):

2a. Agent-plane server cert (fleet CA)

Enrolled hosts dial the gRPC agent plane and verify the manager against the fleet root they hold, so this cert is issued by the fleet CA:

# SANs = every name/IP a host might dial the agent plane on.
AGENT_SANS=diskless-server.bitasylum.org,10.0.0.1 \
  ./server-pki.sh
# -> ./pki/agent/{server.crt,server.key,fleet-ca.crt}

2b. Broker admin-client cert (broker admin CA)

The manager is an mTLS client of the enrol-broker's admin plane. Mint its client cert from the broker's dedicated admin CA (NOT the fleet CA — reusing the fleet CA would let any enrolled host authenticate to the admin plane). Mint the whole set once into the broker's own PKI dir (so the broker gets its server.crt/server.key too), then copy the CA + client cert to the manager so both sides share one admin CA.

The broker's PKI dir + its "serve mTLS" switch differ by which stack runs it:

• Main serving stack (serving/docker-compose.yml) — broker mounts ./enrol-pki:/pki, and the admin plane is plaintext by default (ENROL_ADMIN_INSECURE=true in serving/.env). Set it false, then:

  cd ..                                     # serving/
  ./enrol-broker/admin-pki.sh ./enrol-pki/broker fleet-manager
  #   -> enrol-pki/broker/{admin-ca.crt, server.crt, server.key, clients/fleet-manager.*}
  docker compose up -d enrol-broker         # restart so it serves mTLS (no cert error in logs)
  mkdir -p fleet-manager/pki/broker
  cp    enrol-pki/broker/admin-ca.crt  fleet-manager/pki/broker/
  cp -r enrol-pki/broker/clients       fleet-manager/pki/broker/
  

• Bench stack (serving/dot1x-bench/) — broker mounts ./pki:/pki and is mTLS-by-default (cert paths hardcoded in that compose):

  ../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager
  cp -r ../dot1x-bench/pki/broker ./pki/broker
  

Either way the manager ends up with ./pki/broker/{admin-ca.crt, clients/fleet-manager.{crt,key}}. If you (re)generate a fresh admin CA, re-copy it to the manager — a stale copy won't verify. Verify the mTLS end to end:

curl -s --cert ./pki/broker/clients/fleet-manager.crt \
        --key  ./pki/broker/clients/fleet-manager.key \
        --cacert ./pki/broker/admin-ca.crt \
        https://127.0.0.1:9443/pending ; echo    # expect [] or null (empty queue)

2c. Admin UI ingress — Traefik on :443 (Let's Encrypt DNS-01 via Route 53)

The admin UI is fronted by the traefik compose service: it terminates the LE cert on :443 and forwards plaintext to the manager (behind-proxy mode, loopback :9500). The UI hostname resolves on the mgmt net and is not internet-reachable, so only DNS-01 can validate — Traefik automates the _acme-challenge TXT record via the AWS Route 53 API and auto-renews. Only the admin plane rides Traefik; the agent gRPC/mTLS plane (:9600) is direct (see Network paths).

Set the UI hostname in traefik/dynamic/fleet-manager.yml — the Host() rule (file config isn't env-interpolated), matching FLEET_MGR_RP_ID.

IAM policy — AWS creds scoped to the bitasylum.org hosted zone (put AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY in .env, or attach an instance role and omit them):

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["route53:GetChange"], "Resource": "arn:aws:route53:::change/*" },
    { "Effect": "Allow", "Action": ["route53:ListHostedZonesByName"], "Resource": "*" },
    { "Effect": "Allow",
      "Action": ["route53:ListResourceRecordSets", "route53:ChangeResourceRecordSets"],
      "Resource": "arn:aws:route53:::hostedzone/<YOUR_ZONE_ID>" }
  ]
}

AWS_HOSTED_ZONE_ID is optional: the ACME client auto-discovers the zone via ListHostedZonesByName (hence that statement). Set it (Traefik reads it from the env) to pin the zone — required if a public + private zone share bitasylum.org, and it lets you drop the ListHostedZonesByName statement for a tighter policy.

Run it:

# .env: ACME_EMAIL + the AWS creds above. For a first DRY-RUN vs LE staging
# (untrusted, no rate limit), add the caserver flag to the traefik `command:` in
# docker-compose.yml, prove issuance, then remove it for the real cert.
docker compose up -d traefik
docker compose logs -f traefik    # watch the DNS-01 challenge succeed + cert stored

Traefik runs the propagation check against public resolvers (8.8.8.8:53,1.1.1.1:53 in the command) — needed because split-horizon DNS won't serve the public _acme-challenge record. The cert persists in the traefik-acme volume (acme.json) and Traefik renews it automatically. A publicly-trusted cert also keeps the WebAuthn secure-context clean.

Origin match: passkeys are origin-bound, so FLEET_MGR_RP_ORIGIN must equal the browser's origin. With Traefik on :443 that's the port-less https://diskless-server.bitasylum.org; FLEET_MGR_RP_ID is the host. The manager binds loopback (127.0.0.1:9500) — only Traefik reaches it; never expose that port directly.

Self-TLS alternative (no Traefik): set FLEET_MGR_ADMIN_BEHIND_PROXY=false and give the manager a cert at /pki/admin/{fullchain,privkey}.pem (e.g. via the acme/renew-loop.sh helper under a harbor.bitasylum.org/dockerhub/goacme/lego container). The manager then terminates TLS itself and the origin includes the port it listens on.

Network paths (three planes, three trust stages)

Plane	When	From	Path	Auth
Enrolment (/enrol/)	pre-auth (no cert yet)	boot LAN	boot nginx passthrough → broker :9090	none (the one unauth edge)
Agent (gRPC :9600)	post-auth (802.1X'd, holds TPM cert)	authenticated host VLAN	direct → manager :9600, via OPNsense L3 (allow only host-VLAN→:9600)	mTLS, TPM cert (fleet CA)
Admin (UI :443)	anytime	mgmt/admin network	Traefik → manager :9500	WebAuthn passkey

The agent plane does not go through the boot nginx or Traefik: it connects only after EAP-TLS/MACsec, and Traefik can't terminate the fleet-CA client-cert mTLS. Segment it with an OPNsense firewall rule allowing only the authenticated host VLAN → manager:9600. In the k8s target this becomes a dedicated TCP/TLS- passthrough ingress (the pod still sees the client cert), separate from the admin Ingress.

3. Start InfluxDB

docker compose up -d influxdb
docker compose ps    # wait for healthy

First start runs setup with the org/bucket/token/retention from .env.

4. Bootstrap the first admin (passkey)

Admin auth is closed — there is no password. Mint a one-time enrolment link and register your passkey in the browser:

docker compose run --rm fleet-manager register-admin "your-name"
# prints: https://diskless-server.bitasylum.org/#/enrol?token=...

The register-admin subcommand generates the shared enrol-HMAC key on first run (into the fleet-data volume at /data/enroll.key) and mints an HMAC-signed, single-use token (jti ledger prevents replay). Open the link on a machine with your YubiKey, tap + PIN, and the passkey is registered. Further admins can be enrolled from the UI once one exists.

5. Start the manager + ingress

docker compose up -d fleet-manager traefik
docker compose logs -f fleet-manager

Healthy startup logs the four planes: admin (passkey, behind TLS proxy — plaintext on loopback), agent gRPC/mTLS, authorise (broker), and the InfluxDB gateway. Any plane whose config is absent logs DISABLED and the rest still run. Traefik terminates :443 and forwards to the manager's loopback listener.

Then browse to https://diskless-server.bitasylum.org/, log in with the passkey, and you should see Monitor / Authorise / Lifecycle.

Smoke test without hardware

To render the SPA against demo data with no passkey/PKI:

docker compose run --rm -e FLEET_MGR_ADMIN_INSECURE=true -e FLEET_MGR_SEED=true \
  -e FLEET_MGR_ADMIN_ADDR=0.0.0.0:9500 -p 9500:9500 fleet-manager
# http://localhost:9500/  (whoami returns "dev-insecure"; two seeded hosts)

FLEET_MGR_ADMIN_INSECURE=true is dev/lab only — plaintext, no passkey.

Backups

Back up the fleet-data volume (inventory.json, admins.json, enroll.key) and the influx-data volume. Losing admins.json locks every admin out — you'd re-bootstrap with register-admin. Losing enroll.key invalidates any outstanding (unused) enrol links.

./backup.sh [dest-dir] tars both volumes (read-only, safe to run live) into timestamped .tar.gz files (default ./backups, keeps the newest BACKUP_KEEP=14). Schedule it with a systemd timer / cron and copy the output offsite:

./backup.sh /srv/backups/fleet-manager
# restore (manager stopped) — see the header of backup.sh

If you run compose under a non-default project name, pass COMPOSE_PROJECT_NAME=<name> so it finds the <name>_fleet-data volume.

Troubleshooting

• permission denied writing the inventory on approve — the runtime image is distroless nonroot (uid 65532), but a named volume is created root-owned, so /data isn't writable. A rebuilt image pre-creates /data owned by nonroot (a fresh volume inherits it), but an already-created volume must be fixed once:

  docker volume ls | grep fleet          # find the name, e.g. fleet-manager_fleet-data
  docker run --rm -v <vol>:/data harbor.bitasylum.org/dockerhub/library/alpine:3.20 chown -R 65532:65532 /data
  

• authorise plane not configured (no enrol-broker wired) in the SPA — the manager has the pillar disabled (s.broker == nil). Check docker compose logs fleet-manager | grep -i pillar: a DISABLED/WARNING line names the missing URL or cert file. After staging the broker PKI, the container must be recreated (docker compose up -d --force-recreate fleet-manager) — a plain restart won't re-enable a plane that soft-disabled at boot. Also make sure only one manager instance is bound to :9500 (docker ps | grep manager); a stale parallel instance serves the old state.
• TLS connect error: wrong version number to the broker — the broker admin plane is answering in plaintext: it's running with ENROL_ADMIN_INSECURE=true (the main serving-stack default). Set it false, ensure the broker's server.crt/server.key/admin-ca.crt exist in its PKI mount, and restart it (see step 2b).

Hardening status

Done: distroless-nonroot runtime; passkey admin auth on :443 via the Traefik mgmt ingress with an auto-renewing LE cert (Route 53 DNS-01); mTLS on both the agent (fleet CA) and manager→broker (admin CA) planes; secrets kept server-side (.env/PKI gitignored); optional planes soft-disable instead of crash-looping; CI builds + pushes the image (fleet-manager-image.yaml); backup.sh for the durable volumes.

Still open (operator to close per environment):

• Network segmentation. The admin (:443 Traefik, forwards to loopback :9500) and agent (:9600) listeners must be reachable only from their zones — admin from the mgmt network, agent from the authenticated host VLAN via OPNsense L3 (allow only host-VLAN → :9600). Never the boot LAN. Enforce at the firewall; this repo can't. See Network paths.
• Agent-plane server cert. server-pki.sh isn't run yet (nothing connects to the agent plane until the on-host agent exists — it soft-disables meanwhile). Issue it when the agent lands.
• Digest pinning. For a locked deployment, pin fleet-manager, influxdb, and traefik by @sha256:… (CLAUDE.md invariant #12) rather than the tags used here for convenience.

What is NOT yet wired (recorded intent)

Per-host boot-version assignment (desired.target_version) is stored and pushed to the agent, but does not yet change what a host boots: that needs dnsmasq cohort steering + multi-version UKI serving (docs/TODO.md #6). Kiosk URL / reboot / drain do reconcile via the agent today. The Lifecycle page labels the version pin as recorded intent.