diskless-workstation/serving/fleet-manager/DEPLOY.md

# fleet-manager — deployment runbook

Bring-up of the fleet-manager (control plane + admin SPA) and its private
InfluxDB on the **management host**. This is a lab/on-prem deployment; adapt the
PKI/ACME steps to your environment. See [`README.md`](README.md) for the design
and [`docs/fleet-management-design.md`](../../docs/fleet-management-design.md) for
the architecture.

> **Where it runs.** The management host — the trusted side, alongside the
> enrol-broker and step-ca (`serving/dot1x-bench/`). **Never on the boot LAN.**
> The admin plane (passkey UI) faces admins; the agent plane (gRPC/mTLS) faces
> enrolled hosts; both are segmented from the boot network.

## Prerequisites

- Docker + compose, and the `step` CLI (smallstep) on the host.
- The **fleet step-ca** already bootstrapped (`serving/dot1x-bench/` —
  `docker compose up -d step-ca && ./step-ca/bootstrap.sh`), so
  `../dot1x-bench/pki/root_ca.crt` and the `enrol` provisioner exist.
- The **enrol-broker** running (`serving/dot1x-bench/` compose) with its admin
  plane on `127.0.0.1:9443`.
- A DNS zone you control for the UI hostname (`bitasylum.org`) and an ACME
  client that can do **DNS-01** (e.g. `lego` or `certbot` + your DNS provider).
- A **YubiKey** (or other WebAuthn authenticator) for the bootstrap admin.

## 1. Configure

Config is **layered**: `defaults.env` (committed defaults) + `.env` (your
overrides + secrets, gitignored). Compose reads both — later wins — via
`COMPOSE_ENV_FILES`, so a changed default in `defaults.env` reaches you without
re-copying anything; `.env` stays small.

```bash
cd serving/fleet-manager
export COMPOSE_ENV_FILES=defaults.env,.env     # once per shell (or a Makefile/.envrc)
# .env holds ONLY overrides + secrets — at minimum:
cat > .env <<EOF
FLEET_MGR_INFLUX_TOKEN=$(openssl rand -hex 32)
INFLUX_ADMIN_PASSWORD=$(openssl rand -hex 24)
ACME_EMAIL=you@bitasylum.org
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
# override any other default from defaults.env here, e.g. FLEET_MGR_RP_ID=...
EOF
```

`COMPOSE_ENV_FILES` requires every listed file to exist, so create `.env` before
running compose. Per-command alternative: `docker compose --env-file defaults.env
--env-file .env …`. The Influx token is handed to **both** InfluxDB (its init
admin token) and the manager, so one value wires them.

Or skip the export entirely and use the **Makefile**, which sets
`COMPOSE_ENV_FILES` for you: `make up` / `make logs SERVICE=fleet-manager` /
`make register-admin NAME=you` / `make config` (`make help` lists targets).

## 2. PKI

Three certs, three trust roots — do not conflate them (see README "two planes"):

### 2a. Agent-plane server cert (fleet CA)

Enrolled hosts dial the gRPC agent plane and verify the manager against the
**fleet root** they hold, so this cert is issued by the fleet CA. Its SANs MUST
include every name a host dials — the fleet-DNS name `manager.fleet.lab` (what the
on-host agent uses) **and** the host FQDN. A missing SAN fails the agent TLS
handshake with `certificate is valid for … not manager.fleet.lab`.

**Recommended — auto-renewer (step-ca co-located on this host).** The
`agent-cert-renewer` sidecar issues the cert once, then runs `step ca renew
--daemon` to rotate it before the CA's ~90-day cap. Renewal **preserves the
subject + SANs**, so the drift above cannot recur, and the manager hot-reloads the
rotated cert with no restart (`GetCertificate`, `certreload.go`). No `step` CLI on
the host, no manual re-mint. Tune SANs / paths via `defaults.env` (see the
`cert-auto` block); it mounts the CA root/intermediate + `enrol` provisioner
password read-only from `../dot1x-bench/pki`.

```bash
docker compose --profile cert-auto up -d agent-cert-renewer
# -> ./pki/agent/{server.crt,server.key,fleet-ca.crt}, auto-renewed thereafter.
```

**Manual alternative — `server-pki.sh`** (needs the `step` CLI on the host, or run
it inside the step-ca container; re-run every ~90 days). Use where step-ca is
**not** co-located:

```bash
# SANs = every name/IP a host might dial the agent plane on.
AGENT_SANS=manager.fleet.lab,diskless-server.bitasylum.org \
  ./server-pki.sh
# -> ./pki/agent/{server.crt,server.key,fleet-ca.crt}
```

### 2b. Broker admin-client cert (broker admin CA)

The manager is an mTLS **client** of the enrol-broker's admin plane. Mint its
client cert from the broker's **dedicated admin CA** (NOT the fleet CA — reusing
the fleet CA would let any enrolled host authenticate to the admin plane). Mint
the whole set **once** into the broker's own PKI dir (so the broker gets its
`server.crt`/`server.key` too), then copy the CA + client cert to the manager so
**both sides share one admin CA**.

The broker's PKI dir + its "serve mTLS" switch differ by which stack runs it:

- **Main serving stack** (`serving/docker-compose.yml`) — broker mounts
  `./enrol-pki:/pki`, and the admin plane is **plaintext by default**
  (`ENROL_ADMIN_INSECURE=true` in `serving/.env`). Set it `false`, then:
  ```bash
  cd ..                                     # serving/
  ./enrol-broker/admin-pki.sh ./enrol-pki/broker fleet-manager
  #   -> enrol-pki/broker/{admin-ca.crt, server.crt, server.key, clients/fleet-manager.*}
  docker compose up -d enrol-broker         # restart so it serves mTLS (no cert error in logs)
  mkdir -p fleet-manager/pki/broker
  cp    enrol-pki/broker/admin-ca.crt  fleet-manager/pki/broker/
  cp -r enrol-pki/broker/clients       fleet-manager/pki/broker/
  ```
- **Bench stack** (`serving/dot1x-bench/`) — broker mounts `./pki:/pki` and is
  mTLS-by-default (cert paths hardcoded in that compose):
  ```bash
  ../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager
  cp -r ../dot1x-bench/pki/broker ./pki/broker
  ```

Either way the manager ends up with `./pki/broker/{admin-ca.crt,
clients/fleet-manager.{crt,key}}`. If you (re)generate a fresh admin CA, **re-copy**
it to the manager — a stale copy won't verify. Verify the mTLS end to end:
```bash
curl -s --cert ./pki/broker/clients/fleet-manager.crt \
        --key  ./pki/broker/clients/fleet-manager.key \
        --cacert ./pki/broker/admin-ca.crt \
        https://127.0.0.1:9443/pending ; echo    # expect [] or null (empty queue)
```

### 2c. Admin UI ingress — Traefik on :443 (Let's Encrypt DNS-01 via Route 53)

The admin UI is fronted by the **`traefik`** compose service: it terminates the LE
cert on :443 and forwards plaintext to the manager (behind-proxy mode, loopback
:9500). The UI hostname resolves on the mgmt net and is **not internet-reachable**,
so only **DNS-01** can validate — Traefik automates the `_acme-challenge` TXT
record via the AWS Route 53 API and auto-renews. **Only the admin plane rides
Traefik**; the agent gRPC/mTLS plane (:9600) is direct (see [Network paths](#network-paths)).

**Set the UI hostname** in `traefik/dynamic/fleet-manager.yml` — the `Host()` rule
(file config isn't env-interpolated), matching `FLEET_MGR_RP_ID`.

**IAM policy** — AWS creds scoped to the `bitasylum.org` hosted zone (put
`AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` in `.env`, or attach an instance role
and omit them):

```json
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["route53:GetChange"], "Resource": "arn:aws:route53:::change/*" },
    { "Effect": "Allow", "Action": ["route53:ListHostedZonesByName"], "Resource": "*" },
    { "Effect": "Allow",
      "Action": ["route53:ListResourceRecordSets", "route53:ChangeResourceRecordSets"],
      "Resource": "arn:aws:route53:::hostedzone/<YOUR_ZONE_ID>" }
  ]
}
```

`AWS_HOSTED_ZONE_ID` is **optional**: the ACME client auto-discovers the zone via
`ListHostedZonesByName` (hence that statement). Set it (Traefik reads it from the
env) to pin the zone — required if a **public + private** zone share `bitasylum.org`,
and it lets you **drop the `ListHostedZonesByName` statement** for a tighter policy.

**Run it:**

```bash
# .env: ACME_EMAIL + the AWS creds above. For a first DRY-RUN vs LE staging
# (untrusted, no rate limit), add the caserver flag to the traefik `command:` in
# docker-compose.yml, prove issuance, then remove it for the real cert.
docker compose up -d traefik
docker compose logs -f traefik    # watch the DNS-01 challenge succeed + cert stored
```

Traefik runs the propagation check against **public** resolvers
(`8.8.8.8:53,1.1.1.1:53` in the command) — needed because split-horizon DNS won't
serve the public `_acme-challenge` record. The cert persists in the `traefik-acme`
volume (`acme.json`) and Traefik renews it automatically. A publicly-trusted cert
also keeps the WebAuthn secure-context clean.

> **Origin match:** passkeys are origin-bound, so `FLEET_MGR_RP_ORIGIN` must equal
> the browser's origin. With Traefik on :443 that's the **port-less**
> `https://diskless-server.bitasylum.org`; `FLEET_MGR_RP_ID` is the host. The
> manager binds loopback (`127.0.0.1:9500`) — only Traefik reaches it; never expose
> that port directly.

<a name="network-paths"></a>
### Network paths (three planes, three trust stages)

| Plane | When | From | Path | Auth |
|---|---|---|---|---|
| Enrolment (`/enrol/`) | pre-auth (no cert yet) | boot LAN | boot nginx passthrough → broker :9090 | none (the one unauth edge) |
| Agent (gRPC :9600) | post-auth (802.1X'd, holds TPM cert) | authenticated host VLAN | **direct** → manager :9600, via **OPNsense L3** (allow only host-VLAN→:9600) | **mTLS, TPM cert (fleet CA)** |
| Admin (UI :443) | anytime | mgmt/admin network | **Traefik** → manager :9500 | WebAuthn passkey |

The agent plane does **not** go through the boot nginx or Traefik: it connects
only after EAP-TLS/MACsec, and Traefik can't terminate the fleet-CA client-cert
mTLS. Segment it with an **OPNsense firewall rule** allowing only the authenticated
host VLAN → `manager:9600`. In the k8s target this becomes a dedicated TCP/TLS-
**passthrough** ingress (the pod still sees the client cert), separate from the
admin Ingress.

## 3. Start InfluxDB

```bash
docker compose up -d influxdb
docker compose ps    # wait for healthy
```

First start runs setup with the org/bucket/token/retention from `.env`.

## 4. Bootstrap the first admin (passkey)

Admin auth is closed — there is no password. Mint a **one-time enrolment link**
and register your passkey in the browser:

```bash
docker compose run --rm fleet-manager register-admin "your-name"
# prints: https://diskless-server.bitasylum.org/#/enrol?token=...
```

The `register-admin` subcommand generates the shared enrol-HMAC key on first run
(into the `fleet-data` volume at `/data/enroll.key`) and mints an HMAC-signed,
single-use token (jti ledger prevents replay). Open the link on a machine with
your YubiKey, tap + PIN, and the passkey is registered. Further admins can be
enrolled from the UI once one exists.

## 5. Start the manager + ingress

```bash
docker compose up -d fleet-manager traefik
docker compose logs -f fleet-manager
```

Healthy startup logs the four planes: admin (`passkey, behind TLS proxy` — plaintext
on loopback), agent gRPC/mTLS, authorise (broker), and the InfluxDB gateway. Any
plane whose config is absent logs `DISABLED` and the rest still run. Traefik
terminates :443 and forwards to the manager's loopback listener.

Then browse to `https://diskless-server.bitasylum.org/`, log in with the passkey,
and you should see **Monitor / Authorise / Lifecycle**.

## Smoke test without hardware

To render the SPA against demo data with no passkey/PKI:

```bash
docker compose run --rm -e FLEET_MGR_ADMIN_INSECURE=true -e FLEET_MGR_SEED=true \
  -e FLEET_MGR_ADMIN_ADDR=0.0.0.0:9500 -p 9500:9500 fleet-manager
# http://localhost:9500/  (whoami returns "dev-insecure"; two seeded hosts)
```

`FLEET_MGR_ADMIN_INSECURE=true` is **dev/lab only** — plaintext, no passkey.

## Backups

Back up the `fleet-data` volume (`inventory.json`, `admins.json`, `enroll.key`)
and the `influx-data` volume. Losing `admins.json` locks every admin out — you'd
re-bootstrap with `register-admin`. Losing `enroll.key` invalidates any
outstanding (unused) enrol links.

`./backup.sh [dest-dir]` tars both volumes (read-only, safe to run live) into
timestamped `.tar.gz` files (default `./backups`, keeps the newest
`BACKUP_KEEP=14`). Schedule it with a systemd timer / cron and copy the output
offsite:

```bash
./backup.sh /srv/backups/fleet-manager
# restore (manager stopped) — see the header of backup.sh
```

If you run compose under a non-default project name, pass
`COMPOSE_PROJECT_NAME=<name>` so it finds the `<name>_fleet-data` volume.

## Troubleshooting

- **`permission denied` writing the inventory on approve** — the runtime image is
  distroless `nonroot` (uid **65532**), but a named volume is created root-owned,
  so `/data` isn't writable. A rebuilt image pre-creates `/data` owned by nonroot
  (a *fresh* volume inherits it), but an **already-created** volume must be fixed
  once:
  ```bash
  docker volume ls | grep fleet          # find the name, e.g. fleet-manager_fleet-data
  docker run --rm -v <vol>:/data harbor.bitasylum.org/dockerhub/library/alpine:3.20 chown -R 65532:65532 /data
  ```
- **`authorise plane not configured (no enrol-broker wired)` in the SPA** — the
  manager has the pillar disabled (`s.broker == nil`). Check
  `docker compose logs fleet-manager | grep -i pillar`: a `DISABLED`/`WARNING`
  line names the missing URL or cert file. After staging the broker PKI, the
  container must be **recreated** (`docker compose up -d --force-recreate
  fleet-manager`) — a plain restart won't re-enable a plane that soft-disabled at
  boot. Also make sure only **one** manager instance is bound to `:9500`
  (`docker ps | grep manager`); a stale parallel instance serves the old state.
- **`TLS connect error: wrong version number` to the broker** — the broker admin
  plane is answering in **plaintext**: it's running with `ENROL_ADMIN_INSECURE=true`
  (the main serving-stack default). Set it `false`, ensure the broker's
  `server.crt`/`server.key`/`admin-ca.crt` exist in its PKI mount, and restart it
  (see step 2b).

## Hardening status

Done: distroless-nonroot runtime; passkey admin auth on :443 via the **Traefik**
mgmt ingress with an auto-renewing LE cert (Route 53 DNS-01); mTLS on both the
agent (fleet CA) and manager→broker (admin CA) planes; secrets kept server-side
(`.env`/PKI gitignored); optional planes soft-disable instead of crash-looping; CI
builds + pushes the image (`fleet-manager-image.yaml`); `backup.sh` for the durable
volumes.

Still open (operator to close per environment):
- **Network segmentation.** The admin (`:443` Traefik, forwards to loopback `:9500`)
  and agent (`:9600`) listeners must be reachable only from their zones — admin from
  the mgmt network, agent from the authenticated host VLAN via **OPNsense L3** (allow
  only host-VLAN → `:9600`). Never the boot LAN. Enforce at the firewall; this repo
  can't. See [Network paths](#network-paths).
- **Agent-plane server cert.** Issue it (§2a) once the on-host agent is dialing in
  — until then the agent plane soft-disables. On a bench where step-ca is
  co-located, prefer the `cert-auto` auto-renewer so it never silently expires;
  its SANs MUST include `manager.fleet.lab` (the fleet-DNS name hosts dial).
- **Digest pinning.** For a locked deployment, pin `fleet-manager`, `influxdb`,
  and `traefik` by `@sha256:…` (CLAUDE.md invariant #12) rather than the tags used
  here for convenience.

## What is NOT yet wired (recorded intent)

Per-host **boot-version assignment** (`desired.target_version`) is stored and
pushed to the agent, but does not yet change what a host boots: that needs
dnsmasq cohort steering + multi-version UKI serving (`docs/TODO.md` #6). Kiosk
URL / reboot / drain **do** reconcile via the agent today. The Lifecycle page
labels the version pin as recorded intent.