diskless-workstation/serving/fleet-manager/DEPLOY.md

fleet-manager — deployment runbook

Bring-up of the fleet-manager (control plane + admin SPA) and its private InfluxDB on the management host. This is a lab/on-prem deployment; adapt the PKI/ACME steps to your environment. See README.md for the design and docs/fleet-management-design.md for the architecture.

Where it runs. The management host — the trusted side, alongside the enrol-broker and step-ca (serving/dot1x-bench/). Never on the boot LAN. The admin plane (passkey UI) faces admins; the agent plane (gRPC/mTLS) faces enrolled hosts; both are segmented from the boot network.

Prerequisites

• Docker + compose, and the step CLI (smallstep) on the host.
• The fleet step-ca already bootstrapped (serving/dot1x-bench/ — docker compose up -d step-ca && ./step-ca/bootstrap.sh), so ../dot1x-bench/pki/root_ca.crt and the enrol provisioner exist.
• The enrol-broker running (serving/dot1x-bench/ compose) with its admin plane on 127.0.0.1:9443.
• A DNS zone you control for the UI hostname (bitasylum.org) and an ACME client that can do DNS-01 (e.g. lego or certbot + your DNS provider).
• A YubiKey (or other WebAuthn authenticator) for the bootstrap admin.

1. Configure

cd serving/fleet-manager
cp .env.example .env
# Edit .env:
#   FLEET_MGR_RP_ID / _ORIGIN   -> your UI hostname (diskless-server.bitasylum.org)
#   FLEET_MGR_INFLUX_TOKEN      -> a long random secret (openssl rand -hex 32)
#   INFLUX_ADMIN_PASSWORD       -> a strong password

The Influx token is handed to both InfluxDB (its init admin token) and the manager, so one value wires them with no copy step. .env is gitignored.

2. PKI

Three certs, three trust roots — do not conflate them (see README "two planes"):

2a. Agent-plane server cert (fleet CA)

Enrolled hosts dial the gRPC agent plane and verify the manager against the fleet root they hold, so this cert is issued by the fleet CA:

# SANs = every name/IP a host might dial the agent plane on.
AGENT_SANS=diskless-server.bitasylum.org,10.0.0.1 \
  ./server-pki.sh
# -> ./pki/agent/{server.crt,server.key,fleet-ca.crt}

2b. Broker admin-client cert (broker admin CA)

The manager is an mTLS client of the enrol-broker's admin plane. Mint its client cert from the broker's dedicated admin CA (NOT the fleet CA — reusing the fleet CA would let any enrolled host authenticate to the admin plane). Mint the whole set once into the broker's own PKI dir (so the broker gets its server.crt/server.key too), then copy the CA + client cert to the manager so both sides share one admin CA.

The broker's PKI dir + its "serve mTLS" switch differ by which stack runs it:

• Main serving stack (serving/docker-compose.yml) — broker mounts ./enrol-pki:/pki, and the admin plane is plaintext by default (ENROL_ADMIN_INSECURE=true in serving/.env). Set it false, then:

  cd ..                                     # serving/
  ./enrol-broker/admin-pki.sh ./enrol-pki/broker fleet-manager
  #   -> enrol-pki/broker/{admin-ca.crt, server.crt, server.key, clients/fleet-manager.*}
  docker compose up -d enrol-broker         # restart so it serves mTLS (no cert error in logs)
  mkdir -p fleet-manager/pki/broker
  cp    enrol-pki/broker/admin-ca.crt  fleet-manager/pki/broker/
  cp -r enrol-pki/broker/clients       fleet-manager/pki/broker/
  

• Bench stack (serving/dot1x-bench/) — broker mounts ./pki:/pki and is mTLS-by-default (cert paths hardcoded in that compose):

  ../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager
  cp -r ../dot1x-bench/pki/broker ./pki/broker
  

Either way the manager ends up with ./pki/broker/{admin-ca.crt, clients/fleet-manager.{crt,key}}. If you (re)generate a fresh admin CA, re-copy it to the manager — a stale copy won't verify. Verify the mTLS end to end:

curl -s --cert ./pki/broker/clients/fleet-manager.crt \
        --key  ./pki/broker/clients/fleet-manager.key \
        --cacert ./pki/broker/admin-ca.crt \
        https://127.0.0.1:9443/pending ; echo    # expect [] or null (empty queue)

2c. Admin UI cert (Let's Encrypt DNS-01)

The UI hostname resolves on the mgmt net and is not internet-reachable, so HTTP-01/TLS-ALPN-01 can't validate. Use DNS-01, which proves control of the zone via a TXT record (and supports wildcards). Example with lego:

# Credentials for your DNS provider are provider-specific (env vars).
lego --dns <provider> --domains diskless-server.bitasylum.org \
     --email you@example.org --accept-tos run
# Point the manager at the issued files (it hot-reloads on renewal):
mkdir -p ./pki/admin
cp .lego/certificates/diskless-server.bitasylum.org.crt  ./pki/admin/fullchain.pem
cp .lego/certificates/diskless-server.bitasylum.org.key  ./pki/admin/privkey.pem

Automate renewal (cron/systemd-timer) to rewrite those two files; the manager's certReloader notices the mtime bump and swaps the cert live — no restart. A publicly-trusted cert also keeps the WebAuthn secure-context clean (no browser warnings). Alternative: front the loopback admin listener with your existing mgmt reverse proxy and terminate the cert there; then set FLEET_MGR_ADMIN_INSECURE=false still but let the proxy hold the cert.

3. Start InfluxDB

docker compose up -d influxdb
docker compose ps    # wait for healthy

First start runs setup with the org/bucket/token/retention from .env.

4. Bootstrap the first admin (passkey)

Admin auth is closed — there is no password. Mint a one-time enrolment link and register your passkey in the browser:

docker compose run --rm fleet-manager register-admin "your-name"
# prints: https://diskless-server.bitasylum.org/#/enrol?token=...

The register-admin subcommand generates the shared enrol-HMAC key on first run (into the fleet-data volume at /data/enroll.key) and mints an HMAC-signed, single-use token (jti ledger prevents replay). Open the link on a machine with your YubiKey, tap + PIN, and the passkey is registered. Further admins can be enrolled from the UI once one exists.

5. Start the manager

docker compose up -d fleet-manager
docker compose logs -f fleet-manager

Healthy startup logs the four planes: admin HTTPS+passkey, agent gRPC/mTLS, authorise (broker), and the InfluxDB gateway. Any plane whose config is absent logs DISABLED and the rest still run.

Then browse to https://diskless-server.bitasylum.org/, log in with the passkey, and you should see Monitor / Authorise / Lifecycle.

Smoke test without hardware

To render the SPA against demo data with no passkey/PKI:

docker compose run --rm -e FLEET_MGR_ADMIN_INSECURE=true -e FLEET_MGR_SEED=true \
  -e FLEET_MGR_ADMIN_ADDR=0.0.0.0:9500 -p 9500:9500 fleet-manager
# http://localhost:9500/  (whoami returns "dev-insecure"; two seeded hosts)

FLEET_MGR_ADMIN_INSECURE=true is dev/lab only — plaintext, no passkey.

Backups

Back up the fleet-data volume (inventory.json, admins.json, enroll.key) and the influx-data volume. Losing admins.json locks every admin out — you'd re-bootstrap with register-admin. Losing enroll.key invalidates any outstanding (unused) enrol links.

Troubleshooting

• permission denied writing the inventory on approve — the runtime image is distroless nonroot (uid 65532), but a named volume is created root-owned, so /data isn't writable. A rebuilt image pre-creates /data owned by nonroot (a fresh volume inherits it), but an already-created volume must be fixed once:

  docker volume ls | grep fleet          # find the name, e.g. fleet-manager_fleet-data
  docker run --rm -v <vol>:/data alpine chown -R 65532:65532 /data
  

• authorise plane not configured (no enrol-broker wired) in the SPA — the manager has the pillar disabled (s.broker == nil). Check docker compose logs fleet-manager | grep -i pillar: a DISABLED/WARNING line names the missing URL or cert file. After staging the broker PKI, the container must be recreated (docker compose up -d --force-recreate fleet-manager) — a plain restart won't re-enable a plane that soft-disabled at boot. Also make sure only one manager instance is bound to :9500 (docker ps | grep manager); a stale parallel instance serves the old state.
• TLS connect error: wrong version number to the broker — the broker admin plane is answering in plaintext: it's running with ENROL_ADMIN_INSECURE=true (the main serving-stack default). Set it false, ensure the broker's server.crt/server.key/admin-ca.crt exist in its PKI mount, and restart it (see step 2b).

What is NOT yet wired (recorded intent)

Per-host boot-version assignment (desired.target_version) is stored and pushed to the agent, but does not yet change what a host boots: that needs dnsmasq cohort steering + multi-version UKI serving (docs/TODO.md #6). Kiosk URL / reboot / drain do reconcile via the agent today. The Lifecycle page labels the version pin as recorded intent.