fleet-manager: auto-renew the agent-plane server cert (step-ca sidecar + hot-reload)

The agent-plane server cert was issued by hand (server-pki.sh / docker exec
step ca certificate), which needs the step CLI on the host, re-specifies SANs
each time, and silently expires at the CA's 90-day cap — the exact failure that
just broke agent TLS (cert minted with a stale SAN, missing manager.fleet.lab).

Automate it:
- agent-cert-renewer sidecar (smallstep step-cli image, compose profile
  cert-auto): issues the cert once from the co-located fleet step-ca, then
  'step ca renew --daemon' rotates it before the cap. Renewal preserves the
  subject + SANs, so SAN drift cannot recur. Chowns to the manager's nonroot
  uid so the distroless manager reads the key. Opt-in + only valid where the CA
  is co-located; server-pki.sh stays as the manual/remote-CA path.
- manager hot-reloads via GetCertificate (certreload.go): a rotated cert on
  disk is picked up on the next handshake with no restart and no dropped agent
  streams; a half-written pair keeps the last good cert serving. Unit-tested.
- defaults.env documents the cert-auto knobs; DEPLOY.md §2a recommends the
  renewer and flags the mandatory manager.fleet.lab SAN.
This commit is contained in:
Audiobuzz
2026-07-10 07:42:18 +00:00
parent 1c68c37824
commit 1db3977238
8 changed files with 360 additions and 12 deletions
+27 -5
View File
@@ -60,11 +60,32 @@ Three certs, three trust roots — do not conflate them (see README "two planes"
### 2a. Agent-plane server cert (fleet CA)
Enrolled hosts dial the gRPC agent plane and verify the manager against the
**fleet root** they hold, so this cert is issued by the fleet CA:
**fleet root** they hold, so this cert is issued by the fleet CA. Its SANs MUST
include every name a host dials — the fleet-DNS name `manager.fleet.lab` (what the
on-host agent uses) **and** the host FQDN. A missing SAN fails the agent TLS
handshake with `certificate is valid for … not manager.fleet.lab`.
**Recommended — auto-renewer (step-ca co-located on this host).** The
`agent-cert-renewer` sidecar issues the cert once, then runs `step ca renew
--daemon` to rotate it before the CA's ~90-day cap. Renewal **preserves the
subject + SANs**, so the drift above cannot recur, and the manager hot-reloads the
rotated cert with no restart (`GetCertificate`, `certreload.go`). No `step` CLI on
the host, no manual re-mint. Tune SANs / paths via `defaults.env` (see the
`cert-auto` block); it mounts the CA root/intermediate + `enrol` provisioner
password read-only from `../dot1x-bench/pki`.
```bash
docker compose --profile cert-auto up -d agent-cert-renewer
# -> ./pki/agent/{server.crt,server.key,fleet-ca.crt}, auto-renewed thereafter.
```
**Manual alternative — `server-pki.sh`** (needs the `step` CLI on the host, or run
it inside the step-ca container; re-run every ~90 days). Use where step-ca is
**not** co-located:
```bash
# SANs = every name/IP a host might dial the agent plane on.
AGENT_SANS=diskless-server.bitasylum.org,10.0.0.1 \
AGENT_SANS=manager.fleet.lab,diskless-server.bitasylum.org \
./server-pki.sh
# -> ./pki/agent/{server.crt,server.key,fleet-ca.crt}
```
@@ -293,9 +314,10 @@ Still open (operator to close per environment):
the mgmt network, agent from the authenticated host VLAN via **OPNsense L3** (allow
only host-VLAN → `:9600`). Never the boot LAN. Enforce at the firewall; this repo
can't. See [Network paths](#network-paths).
- **Agent-plane server cert.** `server-pki.sh` isn't run yet (nothing connects to
the agent plane until the on-host agent exists — it soft-disables meanwhile).
Issue it when the agent lands.
- **Agent-plane server cert.** Issue it (§2a) once the on-host agent is dialing in
— until then the agent plane soft-disables. On a bench where step-ca is
co-located, prefer the `cert-auto` auto-renewer so it never silently expires;
its SANs MUST include `manager.fleet.lab` (the fleet-DNS name hosts dial).
- **Digest pinning.** For a locked deployment, pin `fleet-manager`, `influxdb`,
and `traefik` by `@sha256:…` (CLAUDE.md invariant #12) rather than the tags used
here for convenience.