diskless-workstation/serving/fleet-manager/DEPLOY.md

fleet-manager — deployment runbook

Bring-up of the fleet-manager (control plane + admin SPA) and its private InfluxDB on the management host. This is a lab/on-prem deployment; adapt the PKI/ACME steps to your environment. See README.md for the design and docs/fleet-management-design.md for the architecture.

Where it runs. The management host — the trusted side, alongside the enrol-broker and step-ca (serving/dot1x-bench/). Never on the boot LAN. The admin plane (passkey UI) faces admins; the agent plane (gRPC/mTLS) faces enrolled hosts; both are segmented from the boot network.

Prerequisites

• Docker + compose, and the step CLI (smallstep) on the host.
• The fleet step-ca already bootstrapped (serving/dot1x-bench/ — docker compose up -d step-ca && ./step-ca/bootstrap.sh), so ../dot1x-bench/pki/root_ca.crt and the enrol provisioner exist.
• The enrol-broker running (serving/dot1x-bench/ compose) with its admin plane on 127.0.0.1:9443.
• A DNS zone you control for the UI hostname (bitasylum.org) and an ACME client that can do DNS-01 (e.g. lego or certbot + your DNS provider).
• A YubiKey (or other WebAuthn authenticator) for the bootstrap admin.

1. Configure

cd serving/fleet-manager
cp .env.example .env
# Edit .env:
#   FLEET_MGR_RP_ID / _ORIGIN   -> your UI hostname (diskless-server.bitasylum.org)
#   FLEET_MGR_INFLUX_TOKEN      -> a long random secret (openssl rand -hex 32)
#   INFLUX_ADMIN_PASSWORD       -> a strong password

The Influx token is handed to both InfluxDB (its init admin token) and the manager, so one value wires them with no copy step. .env is gitignored.

2. PKI

Three certs, three trust roots — do not conflate them (see README "two planes"):

2a. Agent-plane server cert (fleet CA)

Enrolled hosts dial the gRPC agent plane and verify the manager against the fleet root they hold, so this cert is issued by the fleet CA:

# SANs = every name/IP a host might dial the agent plane on.
AGENT_SANS=diskless-server.bitasylum.org,10.0.0.1 \
  ./server-pki.sh
# -> ./pki/agent/{server.crt,server.key,fleet-ca.crt}

2b. Broker admin-client cert (broker admin CA)

The manager is an mTLS client of the enrol-broker's admin plane. Mint its client cert from the broker's dedicated admin CA (NOT the fleet CA — reusing the fleet CA would let any enrolled host authenticate to the admin plane). Mint the whole set once into the broker's own PKI dir (so the broker gets its server.crt/server.key too), then copy the CA + client cert to the manager so both sides share one admin CA.

The broker's PKI dir + its "serve mTLS" switch differ by which stack runs it:

• Main serving stack (serving/docker-compose.yml) — broker mounts ./enrol-pki:/pki, and the admin plane is plaintext by default (ENROL_ADMIN_INSECURE=true in serving/.env). Set it false, then:

  cd ..                                     # serving/
  ./enrol-broker/admin-pki.sh ./enrol-pki/broker fleet-manager
  #   -> enrol-pki/broker/{admin-ca.crt, server.crt, server.key, clients/fleet-manager.*}
  docker compose up -d enrol-broker         # restart so it serves mTLS (no cert error in logs)
  mkdir -p fleet-manager/pki/broker
  cp    enrol-pki/broker/admin-ca.crt  fleet-manager/pki/broker/
  cp -r enrol-pki/broker/clients       fleet-manager/pki/broker/
  

• Bench stack (serving/dot1x-bench/) — broker mounts ./pki:/pki and is mTLS-by-default (cert paths hardcoded in that compose):

  ../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager
  cp -r ../dot1x-bench/pki/broker ./pki/broker
  

Either way the manager ends up with ./pki/broker/{admin-ca.crt, clients/fleet-manager.{crt,key}}. If you (re)generate a fresh admin CA, re-copy it to the manager — a stale copy won't verify. Verify the mTLS end to end:

curl -s --cert ./pki/broker/clients/fleet-manager.crt \
        --key  ./pki/broker/clients/fleet-manager.key \
        --cacert ./pki/broker/admin-ca.crt \
        https://127.0.0.1:9443/pending ; echo    # expect [] or null (empty queue)

2c. Admin UI cert — Let's Encrypt DNS-01 via Route 53 (the acme side-car)

The UI hostname resolves on the mgmt net and is not internet-reachable, so HTTP-01/TLS-ALPN-01 can't validate. Use DNS-01, which proves control of the zone via a TXT record. The acme compose service does this automatically: a long-running side-car (acme/renew-loop.sh, on the goacme/lego image) issues the cert on first start and renews within 30 days of expiry, writing it into ./pki/admin/ — the same dir the manager reads — so the manager's certReloader hot-swaps it live, no restart.

IAM policy — give the side-car AWS creds scoped to the bitasylum.org hosted zone (put AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY in .env, or attach an instance role and omit them):

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["route53:GetChange"], "Resource": "arn:aws:route53:::change/*" },
    { "Effect": "Allow", "Action": ["route53:ListHostedZonesByName"], "Resource": "*" },
    { "Effect": "Allow",
      "Action": ["route53:ListResourceRecordSets", "route53:ChangeResourceRecordSets"],
      "Resource": "arn:aws:route53:::hostedzone/<YOUR_ZONE_ID>" }
  ]
}

AWS_HOSTED_ZONE_ID is optional: lego auto-discovers the zone via ListHostedZonesByName (hence that statement). Set it in .env to pin the zone — required if a public + private zone share bitasylum.org (auto-discovery may pick the wrong one), and it lets you drop the ListHostedZonesByName statement for a tighter policy.

Run it (bring the side-car up first so the cert exists before the manager):

# In .env: set ACME_EMAIL + the AWS creds above. ACME_DOMAIN is taken from
# FLEET_MGR_RP_ID. Optionally ACME_STAGING=1 for a first dry-run (untrusted, no
# rate limit) to prove Route 53 access, then re-run without it for a real cert.
docker compose up -d acme
docker compose logs -f acme      # wait for "staged /out/{fullchain,privkey}.pem"
ls -l pki/admin/                 # fullchain.pem + privkey.pem, owned by uid 65532

Then turn off insecure mode and (re)start the manager — the defaults FLEET_MGR_ADMIN_CERT=/pki/admin/fullchain.pem / _KEY=/pki/admin/privkey.pem already point at what the side-car wrote:

# ensure FLEET_MGR_ADMIN_INSECURE=false in .env, then:
docker compose up -d --force-recreate fleet-manager
docker compose logs fleet-manager | grep -i listening   # "admin (HTTPS + passkey) listening"

A publicly-trusted cert also keeps the WebAuthn secure-context clean (no browser warnings). The side-car keeps renewing on its 12-hour loop; the manager reloads each new cert automatically.

Split-horizon DNS: the side-car defaults ACME_RESOLVERS=8.8.8.8:53 so lego checks TXT propagation against public DNS. Without it, lego uses the container's internal resolver, which doesn't serve the public _acme-challenge record, and the challenge times out. Override if 8.8.8.8 is blocked (e.g. 1.1.1.1:53, comma-separated for several).

Origin match: the cert CN/SAN (FLEET_MGR_RP_ID) must equal the origin the browser uses and FLEET_MGR_RP_ORIGIN — passkeys are origin-bound. The admin listener binds loopback (127.0.0.1:9500) by default; expose it as https://diskless-server.bitasylum.org either by binding a routable FLEET_MGR_ADMIN_ADDR or fronting it with a mgmt proxy that passes TLS through (don't re-terminate, or the passkey origin/cert won't match).

3. Start InfluxDB

docker compose up -d influxdb
docker compose ps    # wait for healthy

First start runs setup with the org/bucket/token/retention from .env.

4. Bootstrap the first admin (passkey)

Admin auth is closed — there is no password. Mint a one-time enrolment link and register your passkey in the browser:

docker compose run --rm fleet-manager register-admin "your-name"
# prints: https://diskless-server.bitasylum.org/#/enrol?token=...

The register-admin subcommand generates the shared enrol-HMAC key on first run (into the fleet-data volume at /data/enroll.key) and mints an HMAC-signed, single-use token (jti ledger prevents replay). Open the link on a machine with your YubiKey, tap + PIN, and the passkey is registered. Further admins can be enrolled from the UI once one exists.

5. Start the manager

docker compose up -d fleet-manager
docker compose logs -f fleet-manager

Healthy startup logs the four planes: admin HTTPS+passkey, agent gRPC/mTLS, authorise (broker), and the InfluxDB gateway. Any plane whose config is absent logs DISABLED and the rest still run.

Then browse to https://diskless-server.bitasylum.org/, log in with the passkey, and you should see Monitor / Authorise / Lifecycle.

Smoke test without hardware

To render the SPA against demo data with no passkey/PKI:

docker compose run --rm -e FLEET_MGR_ADMIN_INSECURE=true -e FLEET_MGR_SEED=true \
  -e FLEET_MGR_ADMIN_ADDR=0.0.0.0:9500 -p 9500:9500 fleet-manager
# http://localhost:9500/  (whoami returns "dev-insecure"; two seeded hosts)

FLEET_MGR_ADMIN_INSECURE=true is dev/lab only — plaintext, no passkey.

Backups

Back up the fleet-data volume (inventory.json, admins.json, enroll.key) and the influx-data volume. Losing admins.json locks every admin out — you'd re-bootstrap with register-admin. Losing enroll.key invalidates any outstanding (unused) enrol links.

./backup.sh [dest-dir] tars both volumes (read-only, safe to run live) into timestamped .tar.gz files (default ./backups, keeps the newest BACKUP_KEEP=14). Schedule it with a systemd timer / cron and copy the output offsite:

./backup.sh /srv/backups/fleet-manager
# restore (manager stopped) — see the header of backup.sh

If you run compose under a non-default project name, pass COMPOSE_PROJECT_NAME=<name> so it finds the <name>_fleet-data volume.

Troubleshooting

• permission denied writing the inventory on approve — the runtime image is distroless nonroot (uid 65532), but a named volume is created root-owned, so /data isn't writable. A rebuilt image pre-creates /data owned by nonroot (a fresh volume inherits it), but an already-created volume must be fixed once:

  docker volume ls | grep fleet          # find the name, e.g. fleet-manager_fleet-data
  docker run --rm -v <vol>:/data alpine chown -R 65532:65532 /data
  

• authorise plane not configured (no enrol-broker wired) in the SPA — the manager has the pillar disabled (s.broker == nil). Check docker compose logs fleet-manager | grep -i pillar: a DISABLED/WARNING line names the missing URL or cert file. After staging the broker PKI, the container must be recreated (docker compose up -d --force-recreate fleet-manager) — a plain restart won't re-enable a plane that soft-disabled at boot. Also make sure only one manager instance is bound to :9500 (docker ps | grep manager); a stale parallel instance serves the old state.
• TLS connect error: wrong version number to the broker — the broker admin plane is answering in plaintext: it's running with ENROL_ADMIN_INSECURE=true (the main serving-stack default). Set it false, ensure the broker's server.crt/server.key/admin-ca.crt exist in its PKI mount, and restart it (see step 2b).

What is NOT yet wired (recorded intent)

Per-host boot-version assignment (desired.target_version) is stored and pushed to the agent, but does not yet change what a host boots: that needs dnsmasq cohort steering + multi-version UKI serving (docs/TODO.md #6). Kiosk URL / reboot / drain do reconcile via the agent today. The Lifecycle page labels the version pin as recorded intent.