fleet-manager: auto-renew the agent-plane server cert (step-ca sidecar + hot-reload)

The agent-plane server cert was issued by hand (server-pki.sh / docker exec
step ca certificate), which needs the step CLI on the host, re-specifies SANs
each time, and silently expires at the CA's 90-day cap — the exact failure that
just broke agent TLS (cert minted with a stale SAN, missing manager.fleet.lab).

Automate it:
- agent-cert-renewer sidecar (smallstep step-cli image, compose profile
  cert-auto): issues the cert once from the co-located fleet step-ca, then
  'step ca renew --daemon' rotates it before the cap. Renewal preserves the
  subject + SANs, so SAN drift cannot recur. Chowns to the manager's nonroot
  uid so the distroless manager reads the key. Opt-in + only valid where the CA
  is co-located; server-pki.sh stays as the manual/remote-CA path.
- manager hot-reloads via GetCertificate (certreload.go): a rotated cert on
  disk is picked up on the next handshake with no restart and no dropped agent
  streams; a half-written pair keeps the last good cert serving. Unit-tested.
- defaults.env documents the cert-auto knobs; DEPLOY.md §2a recommends the
  renewer and flags the mandatory manager.fleet.lab SAN.

serving/fleet-manager/DEPLOY.md

@@ -60,11 +60,32 @@ Three certs, three trust roots — do not conflate them (see README "two planes"
 ### 2a. Agent-plane server cert (fleet CA)
 
 Enrolled hosts dial the gRPC agent plane and verify the manager against the
-**fleet root** they hold, so this cert is issued by the fleet CA:
+**fleet root** they hold, so this cert is issued by the fleet CA. Its SANs MUST
+include every name a host dials — the fleet-DNS name `manager.fleet.lab` (what the
+on-host agent uses) **and** the host FQDN. A missing SAN fails the agent TLS
+handshake with `certificate is valid for … not manager.fleet.lab`.
+
+**Recommended — auto-renewer (step-ca co-located on this host).** The
+`agent-cert-renewer` sidecar issues the cert once, then runs `step ca renew
+--daemon` to rotate it before the CA's ~90-day cap. Renewal **preserves the
+subject + SANs**, so the drift above cannot recur, and the manager hot-reloads the
+rotated cert with no restart (`GetCertificate`, `certreload.go`). No `step` CLI on
+the host, no manual re-mint. Tune SANs / paths via `defaults.env` (see the
+`cert-auto` block); it mounts the CA root/intermediate + `enrol` provisioner
+password read-only from `../dot1x-bench/pki`.
+
+```bash
+docker compose --profile cert-auto up -d agent-cert-renewer
+# -> ./pki/agent/{server.crt,server.key,fleet-ca.crt}, auto-renewed thereafter.
+```
+
+**Manual alternative — `server-pki.sh`** (needs the `step` CLI on the host, or run
+it inside the step-ca container; re-run every ~90 days). Use where step-ca is
+**not** co-located:
 
 ```bash
 # SANs = every name/IP a host might dial the agent plane on.
-AGENT_SANS=diskless-server.bitasylum.org,10.0.0.1 \
+AGENT_SANS=manager.fleet.lab,diskless-server.bitasylum.org \
   ./server-pki.sh
 # -> ./pki/agent/{server.crt,server.key,fleet-ca.crt}
 ```
@@ -293,9 +314,10 @@ Still open (operator to close per environment):
   the mgmt network, agent from the authenticated host VLAN via **OPNsense L3** (allow
   only host-VLAN → `:9600`). Never the boot LAN. Enforce at the firewall; this repo
   can't. See [Network paths](#network-paths).
-- **Agent-plane server cert.** `server-pki.sh` isn't run yet (nothing connects to
+- **Agent-plane server cert.** Issue it (§2a) once the on-host agent is dialing in
-  the agent plane until the on-host agent exists — it soft-disables meanwhile).
+  — until then the agent plane soft-disables. On a bench where step-ca is
-  Issue it when the agent lands.
+  co-located, prefer the `cert-auto` auto-renewer so it never silently expires;
+  its SANs MUST include `manager.fleet.lab` (the fleet-DNS name hosts dial).
 - **Digest pinning.** For a locked deployment, pin `fleet-manager`, `influxdb`,
   and `traefik` by `@sha256:…` (CLAUDE.md invariant #12) rather than the tags used
   here for convenience.

serving/fleet-manager/agent-cert-renewer.sh

@@ -0,0 +1,72 @@
+#!/bin/sh
+# agent-cert-renewer.sh — issue-once + auto-renew the fleet-manager AGENT-PLANE
+# server cert from the co-located fleet step-ca. Runs in the smallstep step-cli
+# image as the `agent-cert-renewer` compose sidecar (profile cert-auto).
+#
+# This replaces the manual `server-pki.sh` + `docker exec … step ca certificate`
+# dance and the 90-day re-mint chore:
+#   - FIRST run (no cert yet): issues the cert via the 'enrol' provisioner, with
+#     ALL SANs hosts might dial (manager.fleet.lab + the host FQDN).
+#   - THEREAFTER: `step ca renew --daemon` rotates it well before the CA's cap.
+#     Renewal RE-ISSUES THE SAME IDENTITY (subject + SANs preserved), so the SAN
+#     drift that once broke agent TLS cannot recur.
+# The manager hot-reloads the rotated pair via GetCertificate (certreload.go) —
+# no restart, no dropped agent streams. Files are chowned to the manager's
+# nonroot uid so the distroless manager can read the key.
+#
+# Renewal authenticates to the CA with the CERT ITSELF (mTLS) — the provisioner
+# password is only needed for the initial issuance.
+set -eu
+
+CRT=${AGENT_CERT:-/pki/agent/server.crt}
+KEY=${AGENT_KEY:-/pki/agent/server.key}
+BUNDLE=${AGENT_CLIENT_CA:-/pki/agent/fleet-ca.crt}
+CA_URL=${CA_URL:-https://localhost:8443}
+CA_ROOT=${CA_ROOT:-/ca/root_ca.crt}
+CA_INT=${CA_INTERMEDIATE:-/ca/intermediate_ca.crt}
+PROVISIONER=${PROVISIONER:-enrol}
+PW_FILE=${PROVISIONER_PASSWORD_FILE:-/ca/password}
+SANS=${AGENT_SANS:-manager.fleet.lab}
+EXPIRES_IN=${RENEW_EXPIRES_IN:-720h}   # renew when < this remains (of the CA's cap)
+NOT_AFTER=${CERT_NOT_AFTER:-2160h}     # first-issue lifetime (must be <= CA max)
+OWNER=${OWNER:-65532:65532}            # the distroless manager's nonroot uid:gid
+
+primary=${SANS%%,*}
+san_args=""
+oldifs=$IFS; IFS=,
+for s in $SANS; do san_args="$san_args --san $s"; done
+IFS=$oldifs
+
+fixperms() {
+	chown "$OWNER" "$CRT" "$KEY" 2>/dev/null || true
+	chmod 0644 "$CRT" 2>/dev/null || true
+	chmod 0600 "$KEY" 2>/dev/null || true
+}
+
+# Client-verify bundle (intermediate + root). Build it if absent so a fresh
+# bring-up needs only this sidecar, not a separate server-pki.sh run. The manager
+# loads it once at startup — the root/intermediate don't rotate on this cadence.
+if [ ! -s "$BUNDLE" ]; then
+	{ [ -s "$CA_INT" ] && cat "$CA_INT"; cat "$CA_ROOT"; } > "$BUNDLE"
+	chown "$OWNER" "$BUNDLE" 2>/dev/null || true
+	chmod 0644 "$BUNDLE" 2>/dev/null || true
+	echo "agent-cert-renewer: wrote client-verify bundle $BUNDLE"
+fi
+
+# Issue once if there's no cert yet (provisioner + password). Renewal afterwards
+# needs neither.
+if [ ! -s "$CRT" ] || [ ! -s "$KEY" ]; then
+	echo "agent-cert-renewer: issuing CN=$primary (SANs=$SANS) from $CA_URL"
+	# shellcheck disable=SC2086  # san_args is intentionally word-split
+	step ca certificate "$primary" "$CRT" "$KEY" \
+		--ca-url "$CA_URL" --root "$CA_ROOT" \
+		--provisioner "$PROVISIONER" --provisioner-password-file "$PW_FILE" \
+		--not-after "$NOT_AFTER" --force $san_args
+	fixperms
+fi
+
+echo "agent-cert-renewer: watching $CRT (renew when < $EXPIRES_IN remains)"
+exec step ca renew --daemon --expires-in "$EXPIRES_IN" \
+	--ca-url "$CA_URL" --root "$CA_ROOT" \
+	--exec "sh -c 'chown $OWNER $CRT $KEY; chmod 0644 $CRT; chmod 0600 $KEY'" \
+	"$CRT" "$KEY"

serving/fleet-manager/agent.go

@@ -278,7 +278,10 @@ func serialHex(cert *x509.Certificate) string {
 // keepalive) with the hub registered. Split from the listen/serve so tests can
 // drive it on an ephemeral listener.
 func agentServer(cfg config, hub *agentHub) (*grpc.Server, error) {
-	cert, err := tls.LoadX509KeyPair(cfg.agentCert, cfg.agentKey)
+	// GetCertificate (not a fixed Certificates slice) so the agent-cert-renewer
+	// sidecar can rotate server.crt/key on disk and the manager picks it up on the
+	// next handshake without a restart. See certreload.go.
+	reloader, err := newCertReloader(cfg.agentCert, cfg.agentKey)
 	if err != nil {
 		return nil, fmt.Errorf("agent server cert: %w", err)
 	}
@@ -291,10 +294,10 @@ func agentServer(cfg config, hub *agentHub) (*grpc.Server, error) {
 		return nil, fmt.Errorf("no certificates parsed from FLEET_MGR_AGENT_CLIENT_CA")
 	}
 	creds := credentials.NewTLS(&tls.Config{
-		MinVersion:   tls.VersionTLS12,
+		MinVersion:     tls.VersionTLS12,
-		Certificates: []tls.Certificate{cert},
+		GetCertificate: reloader.GetCertificate,
-		ClientAuth:   tls.RequireAndVerifyClientCert,
+		ClientAuth:     tls.RequireAndVerifyClientCert,
-		ClientCAs:    pool,
+		ClientCAs:      pool,
 	})
 	gs := grpc.NewServer(
 		grpc.Creds(creds),

serving/fleet-manager/certreload.go

@@ -0,0 +1,77 @@
+// certreload.go — hot-reloading server cert for the agent plane.
+//
+// The agent-plane server cert (issued by the fleet CA) is rotated OUT of process
+// by the `agent-cert-renewer` sidecar running `step ca renew` (docker-compose.yml,
+// profile cert-auto). For that rotation to take effect WITHOUT restarting the
+// manager — which would drop every live agent stream — the TLS config serves the
+// cert through GetCertificate instead of a fixed Certificates slice, and this
+// reloader re-reads server.crt/server.key from disk when they change.
+//
+// Reload is lazy + modtime-gated: each handshake checks server.crt's modtime and
+// re-reads the pair only when it advances. A failed reload (a half-written pair
+// mid-renewal, transient bad perms) is logged and the LAST GOOD cert keeps
+// serving — a renewer hiccup must never take the agent plane down — and because a
+// failed reload does not advance the cached modtime, the next handshake retries
+// until the write settles.
+package main
+
+import (
+	"crypto/tls"
+	"log"
+	"os"
+	"sync"
+	"time"
+)
+
+type certReloader struct {
+	certPath, keyPath string
+
+	mu      sync.RWMutex
+	cert    *tls.Certificate
+	modTime time.Time
+}
+
+// newCertReloader loads the pair once (fail-fast if missing/bad at startup) and
+// returns a reloader whose GetCertificate hook is wired into the TLS config.
+func newCertReloader(certPath, keyPath string) (*certReloader, error) {
+	r := &certReloader{certPath: certPath, keyPath: keyPath}
+	if err := r.reload(); err != nil {
+		return nil, err
+	}
+	return r, nil
+}
+
+func (r *certReloader) reload() error {
+	c, err := tls.LoadX509KeyPair(r.certPath, r.keyPath)
+	if err != nil {
+		return err
+	}
+	var mt time.Time
+	if fi, err := os.Stat(r.certPath); err == nil {
+		mt = fi.ModTime()
+	}
+	r.mu.Lock()
+	r.cert, r.modTime = &c, mt
+	r.mu.Unlock()
+	return nil
+}
+
+// GetCertificate is the tls.Config hook. It reloads when server.crt's modtime has
+// advanced past the cached copy, otherwise returns the cached cert unchanged.
+func (r *certReloader) GetCertificate(*tls.ClientHelloInfo) (*tls.Certificate, error) {
+	r.mu.RLock()
+	cur, mt := r.cert, r.modTime
+	r.mu.RUnlock()
+
+	if fi, err := os.Stat(r.certPath); err == nil && fi.ModTime().After(mt) {
+		if err := r.reload(); err != nil {
+			log.Printf("agent plane: cert reload failed, keeping previous: %v", err)
+		} else {
+			r.mu.RLock()
+			cur = r.cert
+			r.mu.RUnlock()
+			log.Printf("agent plane: reloaded rotated server cert %s", r.certPath)
+		}
+	}
+	return cur, nil
+}

serving/fleet-manager/certreload_test.go

@@ -0,0 +1,109 @@
+package main
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/pem"
+	"math/big"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+// writePair mints a throwaway self-signed leaf with the given CN and writes it as
+// a cert/key PEM pair, so the reloader has a real, loadable x509 key pair.
+func writePair(t *testing.T, certPath, keyPath, cn string) {
+	t.Helper()
+	key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	if err != nil {
+		t.Fatal(err)
+	}
+	tmpl := &x509.Certificate{
+		SerialNumber: big.NewInt(1),
+		Subject:      pkix.Name{CommonName: cn},
+		NotBefore:    time.Now().Add(-time.Hour),
+		NotAfter:     time.Now().Add(time.Hour),
+	}
+	der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key)
+	if err != nil {
+		t.Fatal(err)
+	}
+	kder, err := x509.MarshalPKCS8PrivateKey(key)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(certPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(keyPath, pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: kder}), 0o600); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func leafCN(t *testing.T, r *certReloader) string {
+	t.Helper()
+	c, err := r.GetCertificate(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	leaf, err := x509.ParseCertificate(c.Certificate[0])
+	if err != nil {
+		t.Fatal(err)
+	}
+	return leaf.Subject.CommonName
+}
+
+func TestCertReloaderHotSwap(t *testing.T) {
+	dir := t.TempDir()
+	crt := filepath.Join(dir, "server.crt")
+	key := filepath.Join(dir, "server.key")
+
+	writePair(t, crt, key, "old")
+	r, err := newCertReloader(crt, key)
+	if err != nil {
+		t.Fatalf("newCertReloader: %v", err)
+	}
+	if cn := leafCN(t, r); cn != "old" {
+		t.Fatalf("initial CN = %q, want old", cn)
+	}
+
+	// Rotate the pair on disk with a strictly-later modtime — GetCertificate must
+	// pick it up without any explicit reload call (the renewer-rotates-in-place case).
+	writePair(t, crt, key, "new")
+	future := time.Now().Add(2 * time.Second)
+	if err := os.Chtimes(crt, future, future); err != nil {
+		t.Fatal(err)
+	}
+	if cn := leafCN(t, r); cn != "new" {
+		t.Fatalf("post-rotation CN = %q, want new", cn)
+	}
+}
+
+func TestCertReloaderKeepsPreviousOnBadReload(t *testing.T) {
+	dir := t.TempDir()
+	crt := filepath.Join(dir, "server.crt")
+	key := filepath.Join(dir, "server.key")
+
+	writePair(t, crt, key, "good")
+	r, err := newCertReloader(crt, key)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// Simulate a half-written rotation: cert file replaced with garbage + a newer
+	// modtime. Reload fails; the last good cert must keep serving.
+	if err := os.WriteFile(crt, []byte("not a pem"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	future := time.Now().Add(2 * time.Second)
+	if err := os.Chtimes(crt, future, future); err != nil {
+		t.Fatal(err)
+	}
+	if cn := leafCN(t, r); cn != "good" {
+		t.Fatalf("after bad reload CN = %q, want good (kept previous)", cn)
+	}
+}

serving/fleet-manager/defaults.env

@@ -42,10 +42,31 @@ AWS_HOSTED_ZONE_ID=
 
 # --- agent plane (gRPC + mTLS, fleet CA) ---------------------------------------
 # Routable mgmt address enrolled hosts dial (":9600" = all interfaces). The
-# server cert is issued by the fleet CA (server-pki.sh); the client CA is the
+# server cert is issued by the fleet CA (server-pki.sh, or the auto-renewer
-# fleet root so host TPM certs verify. Paths are fixed in docker-compose.yml.
+# below); the client CA is the fleet root+intermediate so host TPM certs verify.
+# Paths are fixed in docker-compose.yml.
 FLEET_MGR_AGENT_ADDR=:9600
 
+# --- agent-plane cert auto-renewer (compose profile: cert-auto) ----------------
+# Only used when you run the `agent-cert-renewer` sidecar (step-ca co-located on
+# this host). It issues the server cert once, then auto-renews before the CA cap,
+# preserving the SANs. Defaults suit the bench; override host paths in .env if
+# your step-ca lays out its PKI differently.
+#   AGENT_CERT_SANS         all names hosts dial (first = CN). manager.fleet.lab
+#                           is the fleet-DNS name; keep the host FQDN too.
+#   AGENT_CA_URL            the co-located fleet step-ca (host-loopback).
+#   AGENT_CERT_NOT_AFTER    first-issue lifetime (<= CA max, currently 2160h).
+#   AGENT_CERT_RENEW_EXPIRES_IN  renew when less than this remains.
+#   FLEET_CA_ROOT / _INTERMEDIATE / _PROVISIONER_PW  host paths to the CA material.
+AGENT_CERT_SANS=manager.fleet.lab,diskless-server.bitasylum.org
+AGENT_CA_URL=https://localhost:8443
+AGENT_CERT_NOT_AFTER=2160h
+AGENT_CERT_RENEW_EXPIRES_IN=720h
+FLEET_CA_PROVISIONER=enrol
+FLEET_CA_ROOT=../dot1x-bench/pki/root_ca.crt
+FLEET_CA_INTERMEDIATE=../dot1x-bench/pki/intermediate_ca.crt
+FLEET_CA_PROVISIONER_PW=../dot1x-bench/pki/stepca/secrets/password
+
 # --- authorise pillar (enrol-broker admin plane over mTLS) ---------------------
 # The broker runs network_mode:host on 127.0.0.1:9443. The manager's client cert
 # is minted by ../enrol-broker/admin-pki.sh (admin CA), into ./pki/broker/ .

serving/fleet-manager/docker-compose.yml

@@ -118,6 +118,45 @@ services:
       # cert. (UI TLS is Traefik's — no admin cert here.) See DEPLOY.md/server-pki.sh.
       - ./pki:/pki:ro
 
+  # --- agent-plane server-cert auto-renewer (optional; step-ca co-located) -----
+  # Issues the manager's agent-plane server cert ONCE from the fleet step-ca, then
+  # runs `step ca renew --daemon` to rotate it well before the CA's ~90-day cap —
+  # replacing the manual server-pki.sh + `docker exec … step ca certificate` dance
+  # and the periodic re-mint. Renewal PRESERVES the subject + SANs of the first
+  # issuance, so agent-TLS SAN drift can't recur. The manager hot-reloads the
+  # rotated cert via GetCertificate (certreload.go) — no restart, no dropped hub.
+  #
+  # OPT-IN via `--profile cert-auto`, and ONLY valid where the fleet step-ca
+  # (serving/dot1x-bench) runs on THIS host: it mounts the CA root + intermediate +
+  # 'enrol' provisioner password read-only and reaches the CA over host-loopback.
+  # Where the CA is remote, leave this off and issue via server-pki.sh (or ACME).
+  #
+  #   docker compose --profile cert-auto up -d agent-cert-renewer fleet-manager
+  agent-cert-renewer:
+    image: harbor.bitasylum.org/dockerhub/smallstep/step-cli:latest
+    profiles: ["cert-auto"]
+    restart: unless-stopped
+    network_mode: host                       # reach the co-located CA on ${AGENT_CA_URL}
+    user: "0:0"                              # writes into /pki/agent, then chowns to the manager uid
+    environment:
+      STEPPATH: /tmp/step
+      HOME: /tmp
+      AGENT_SANS: "${AGENT_CERT_SANS:-manager.fleet.lab,diskless-server.bitasylum.org}"
+      CA_URL: "${AGENT_CA_URL:-https://localhost:8443}"
+      PROVISIONER: "${FLEET_CA_PROVISIONER:-enrol}"
+      RENEW_EXPIRES_IN: "${AGENT_CERT_RENEW_EXPIRES_IN:-720h}"
+      CERT_NOT_AFTER: "${AGENT_CERT_NOT_AFTER:-2160h}"
+      OWNER: "65532:65532"
+    entrypoint: ["/bin/sh", "/renew/agent-cert-renewer.sh"]
+    volumes:
+      - ./pki/agent:/pki/agent                                    # shared with the manager (rw here)
+      - ./agent-cert-renewer.sh:/renew/agent-cert-renewer.sh:ro
+      # fleet CA material (read-only), from the co-located step-ca stack. Override
+      # the host paths in .env if your bench lays them out differently.
+      - ${FLEET_CA_ROOT:-../dot1x-bench/pki/root_ca.crt}:/ca/root_ca.crt:ro
+      - ${FLEET_CA_INTERMEDIATE:-../dot1x-bench/pki/intermediate_ca.crt}:/ca/intermediate_ca.crt:ro
+      - ${FLEET_CA_PROVISIONER_PW:-../dot1x-bench/pki/stepca/secrets/password}:/ca/password:ro
+
   # --- management-plane ingress (Traefik) — admin UI on :443 ------------------
   # Terminates the Let's Encrypt cert (Route 53 DNS-01) on :443 and forwards
   # plaintext to the manager (behind-proxy mode, loopback :9500). Host networking

serving/fleet-manager/server-pki.sh

@@ -8,6 +8,11 @@
 # fleet CA — this script issues it via `step ca certificate` against the running
 # fleet step-ca (serving/dot1x-bench/), the same CA that signs enrolment certs.
 #
+# NOTE: where the fleet step-ca is CO-LOCATED on this host, prefer the
+# `agent-cert-renewer` compose sidecar (docker-compose.yml, profile cert-auto) —
+# it does this issuance AND auto-renews before the CA's ~90-day cap, preserving
+# SANs, with no `step` CLI on the host. This script is the manual / remote-CA path.
+#
 # This does NOT issue:
 #   - the ADMIN-plane UI cert   -> owned by the Traefik ingress (Let's Encrypt
 #                                  DNS-01); the manager never terminates TLS. See DEPLOY.md §2c.