From 1db397723864c0b9fdae9ab9547eb4c975facd6f Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 10 Jul 2026 07:42:18 +0000 Subject: [PATCH] fleet-manager: auto-renew the agent-plane server cert (step-ca sidecar + hot-reload) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The agent-plane server cert was issued by hand (server-pki.sh / docker exec step ca certificate), which needs the step CLI on the host, re-specifies SANs each time, and silently expires at the CA's 90-day cap — the exact failure that just broke agent TLS (cert minted with a stale SAN, missing manager.fleet.lab). Automate it: - agent-cert-renewer sidecar (smallstep step-cli image, compose profile cert-auto): issues the cert once from the co-located fleet step-ca, then 'step ca renew --daemon' rotates it before the cap. Renewal preserves the subject + SANs, so SAN drift cannot recur. Chowns to the manager's nonroot uid so the distroless manager reads the key. Opt-in + only valid where the CA is co-located; server-pki.sh stays as the manual/remote-CA path. - manager hot-reloads via GetCertificate (certreload.go): a rotated cert on disk is picked up on the next handshake with no restart and no dropped agent streams; a half-written pair keeps the last good cert serving. Unit-tested. - defaults.env documents the cert-auto knobs; DEPLOY.md §2a recommends the renewer and flags the mandatory manager.fleet.lab SAN. --- serving/fleet-manager/DEPLOY.md | 32 +++++- serving/fleet-manager/agent-cert-renewer.sh | 72 +++++++++++++ serving/fleet-manager/agent.go | 13 ++- serving/fleet-manager/certreload.go | 77 ++++++++++++++ serving/fleet-manager/certreload_test.go | 109 ++++++++++++++++++++ serving/fleet-manager/defaults.env | 25 ++++- serving/fleet-manager/docker-compose.yml | 39 +++++++ serving/fleet-manager/server-pki.sh | 5 + 8 files changed, 360 insertions(+), 12 deletions(-) create mode 100755 serving/fleet-manager/agent-cert-renewer.sh create mode 100644 serving/fleet-manager/certreload.go create mode 100644 serving/fleet-manager/certreload_test.go diff --git a/serving/fleet-manager/DEPLOY.md b/serving/fleet-manager/DEPLOY.md index 0904863..7fd07a8 100644 --- a/serving/fleet-manager/DEPLOY.md +++ b/serving/fleet-manager/DEPLOY.md @@ -60,11 +60,32 @@ Three certs, three trust roots — do not conflate them (see README "two planes" ### 2a. Agent-plane server cert (fleet CA) Enrolled hosts dial the gRPC agent plane and verify the manager against the -**fleet root** they hold, so this cert is issued by the fleet CA: +**fleet root** they hold, so this cert is issued by the fleet CA. Its SANs MUST +include every name a host dials — the fleet-DNS name `manager.fleet.lab` (what the +on-host agent uses) **and** the host FQDN. A missing SAN fails the agent TLS +handshake with `certificate is valid for … not manager.fleet.lab`. + +**Recommended — auto-renewer (step-ca co-located on this host).** The +`agent-cert-renewer` sidecar issues the cert once, then runs `step ca renew +--daemon` to rotate it before the CA's ~90-day cap. Renewal **preserves the +subject + SANs**, so the drift above cannot recur, and the manager hot-reloads the +rotated cert with no restart (`GetCertificate`, `certreload.go`). No `step` CLI on +the host, no manual re-mint. Tune SANs / paths via `defaults.env` (see the +`cert-auto` block); it mounts the CA root/intermediate + `enrol` provisioner +password read-only from `../dot1x-bench/pki`. + +```bash +docker compose --profile cert-auto up -d agent-cert-renewer +# -> ./pki/agent/{server.crt,server.key,fleet-ca.crt}, auto-renewed thereafter. +``` + +**Manual alternative — `server-pki.sh`** (needs the `step` CLI on the host, or run +it inside the step-ca container; re-run every ~90 days). Use where step-ca is +**not** co-located: ```bash # SANs = every name/IP a host might dial the agent plane on. -AGENT_SANS=diskless-server.bitasylum.org,10.0.0.1 \ +AGENT_SANS=manager.fleet.lab,diskless-server.bitasylum.org \ ./server-pki.sh # -> ./pki/agent/{server.crt,server.key,fleet-ca.crt} ``` @@ -293,9 +314,10 @@ Still open (operator to close per environment): the mgmt network, agent from the authenticated host VLAN via **OPNsense L3** (allow only host-VLAN → `:9600`). Never the boot LAN. Enforce at the firewall; this repo can't. See [Network paths](#network-paths). -- **Agent-plane server cert.** `server-pki.sh` isn't run yet (nothing connects to - the agent plane until the on-host agent exists — it soft-disables meanwhile). - Issue it when the agent lands. +- **Agent-plane server cert.** Issue it (§2a) once the on-host agent is dialing in + — until then the agent plane soft-disables. On a bench where step-ca is + co-located, prefer the `cert-auto` auto-renewer so it never silently expires; + its SANs MUST include `manager.fleet.lab` (the fleet-DNS name hosts dial). - **Digest pinning.** For a locked deployment, pin `fleet-manager`, `influxdb`, and `traefik` by `@sha256:…` (CLAUDE.md invariant #12) rather than the tags used here for convenience. diff --git a/serving/fleet-manager/agent-cert-renewer.sh b/serving/fleet-manager/agent-cert-renewer.sh new file mode 100755 index 0000000..699b9eb --- /dev/null +++ b/serving/fleet-manager/agent-cert-renewer.sh @@ -0,0 +1,72 @@ +#!/bin/sh +# agent-cert-renewer.sh — issue-once + auto-renew the fleet-manager AGENT-PLANE +# server cert from the co-located fleet step-ca. Runs in the smallstep step-cli +# image as the `agent-cert-renewer` compose sidecar (profile cert-auto). +# +# This replaces the manual `server-pki.sh` + `docker exec … step ca certificate` +# dance and the 90-day re-mint chore: +# - FIRST run (no cert yet): issues the cert via the 'enrol' provisioner, with +# ALL SANs hosts might dial (manager.fleet.lab + the host FQDN). +# - THEREAFTER: `step ca renew --daemon` rotates it well before the CA's cap. +# Renewal RE-ISSUES THE SAME IDENTITY (subject + SANs preserved), so the SAN +# drift that once broke agent TLS cannot recur. +# The manager hot-reloads the rotated pair via GetCertificate (certreload.go) — +# no restart, no dropped agent streams. Files are chowned to the manager's +# nonroot uid so the distroless manager can read the key. +# +# Renewal authenticates to the CA with the CERT ITSELF (mTLS) — the provisioner +# password is only needed for the initial issuance. +set -eu + +CRT=${AGENT_CERT:-/pki/agent/server.crt} +KEY=${AGENT_KEY:-/pki/agent/server.key} +BUNDLE=${AGENT_CLIENT_CA:-/pki/agent/fleet-ca.crt} +CA_URL=${CA_URL:-https://localhost:8443} +CA_ROOT=${CA_ROOT:-/ca/root_ca.crt} +CA_INT=${CA_INTERMEDIATE:-/ca/intermediate_ca.crt} +PROVISIONER=${PROVISIONER:-enrol} +PW_FILE=${PROVISIONER_PASSWORD_FILE:-/ca/password} +SANS=${AGENT_SANS:-manager.fleet.lab} +EXPIRES_IN=${RENEW_EXPIRES_IN:-720h} # renew when < this remains (of the CA's cap) +NOT_AFTER=${CERT_NOT_AFTER:-2160h} # first-issue lifetime (must be <= CA max) +OWNER=${OWNER:-65532:65532} # the distroless manager's nonroot uid:gid + +primary=${SANS%%,*} +san_args="" +oldifs=$IFS; IFS=, +for s in $SANS; do san_args="$san_args --san $s"; done +IFS=$oldifs + +fixperms() { + chown "$OWNER" "$CRT" "$KEY" 2>/dev/null || true + chmod 0644 "$CRT" 2>/dev/null || true + chmod 0600 "$KEY" 2>/dev/null || true +} + +# Client-verify bundle (intermediate + root). Build it if absent so a fresh +# bring-up needs only this sidecar, not a separate server-pki.sh run. The manager +# loads it once at startup — the root/intermediate don't rotate on this cadence. +if [ ! -s "$BUNDLE" ]; then + { [ -s "$CA_INT" ] && cat "$CA_INT"; cat "$CA_ROOT"; } > "$BUNDLE" + chown "$OWNER" "$BUNDLE" 2>/dev/null || true + chmod 0644 "$BUNDLE" 2>/dev/null || true + echo "agent-cert-renewer: wrote client-verify bundle $BUNDLE" +fi + +# Issue once if there's no cert yet (provisioner + password). Renewal afterwards +# needs neither. +if [ ! -s "$CRT" ] || [ ! -s "$KEY" ]; then + echo "agent-cert-renewer: issuing CN=$primary (SANs=$SANS) from $CA_URL" + # shellcheck disable=SC2086 # san_args is intentionally word-split + step ca certificate "$primary" "$CRT" "$KEY" \ + --ca-url "$CA_URL" --root "$CA_ROOT" \ + --provisioner "$PROVISIONER" --provisioner-password-file "$PW_FILE" \ + --not-after "$NOT_AFTER" --force $san_args + fixperms +fi + +echo "agent-cert-renewer: watching $CRT (renew when < $EXPIRES_IN remains)" +exec step ca renew --daemon --expires-in "$EXPIRES_IN" \ + --ca-url "$CA_URL" --root "$CA_ROOT" \ + --exec "sh -c 'chown $OWNER $CRT $KEY; chmod 0644 $CRT; chmod 0600 $KEY'" \ + "$CRT" "$KEY" diff --git a/serving/fleet-manager/agent.go b/serving/fleet-manager/agent.go index 10e9b79..439d2ac 100644 --- a/serving/fleet-manager/agent.go +++ b/serving/fleet-manager/agent.go @@ -278,7 +278,10 @@ func serialHex(cert *x509.Certificate) string { // keepalive) with the hub registered. Split from the listen/serve so tests can // drive it on an ephemeral listener. func agentServer(cfg config, hub *agentHub) (*grpc.Server, error) { - cert, err := tls.LoadX509KeyPair(cfg.agentCert, cfg.agentKey) + // GetCertificate (not a fixed Certificates slice) so the agent-cert-renewer + // sidecar can rotate server.crt/key on disk and the manager picks it up on the + // next handshake without a restart. See certreload.go. + reloader, err := newCertReloader(cfg.agentCert, cfg.agentKey) if err != nil { return nil, fmt.Errorf("agent server cert: %w", err) } @@ -291,10 +294,10 @@ func agentServer(cfg config, hub *agentHub) (*grpc.Server, error) { return nil, fmt.Errorf("no certificates parsed from FLEET_MGR_AGENT_CLIENT_CA") } creds := credentials.NewTLS(&tls.Config{ - MinVersion: tls.VersionTLS12, - Certificates: []tls.Certificate{cert}, - ClientAuth: tls.RequireAndVerifyClientCert, - ClientCAs: pool, + MinVersion: tls.VersionTLS12, + GetCertificate: reloader.GetCertificate, + ClientAuth: tls.RequireAndVerifyClientCert, + ClientCAs: pool, }) gs := grpc.NewServer( grpc.Creds(creds), diff --git a/serving/fleet-manager/certreload.go b/serving/fleet-manager/certreload.go new file mode 100644 index 0000000..cb221a7 --- /dev/null +++ b/serving/fleet-manager/certreload.go @@ -0,0 +1,77 @@ +// certreload.go — hot-reloading server cert for the agent plane. +// +// The agent-plane server cert (issued by the fleet CA) is rotated OUT of process +// by the `agent-cert-renewer` sidecar running `step ca renew` (docker-compose.yml, +// profile cert-auto). For that rotation to take effect WITHOUT restarting the +// manager — which would drop every live agent stream — the TLS config serves the +// cert through GetCertificate instead of a fixed Certificates slice, and this +// reloader re-reads server.crt/server.key from disk when they change. +// +// Reload is lazy + modtime-gated: each handshake checks server.crt's modtime and +// re-reads the pair only when it advances. A failed reload (a half-written pair +// mid-renewal, transient bad perms) is logged and the LAST GOOD cert keeps +// serving — a renewer hiccup must never take the agent plane down — and because a +// failed reload does not advance the cached modtime, the next handshake retries +// until the write settles. +package main + +import ( + "crypto/tls" + "log" + "os" + "sync" + "time" +) + +type certReloader struct { + certPath, keyPath string + + mu sync.RWMutex + cert *tls.Certificate + modTime time.Time +} + +// newCertReloader loads the pair once (fail-fast if missing/bad at startup) and +// returns a reloader whose GetCertificate hook is wired into the TLS config. +func newCertReloader(certPath, keyPath string) (*certReloader, error) { + r := &certReloader{certPath: certPath, keyPath: keyPath} + if err := r.reload(); err != nil { + return nil, err + } + return r, nil +} + +func (r *certReloader) reload() error { + c, err := tls.LoadX509KeyPair(r.certPath, r.keyPath) + if err != nil { + return err + } + var mt time.Time + if fi, err := os.Stat(r.certPath); err == nil { + mt = fi.ModTime() + } + r.mu.Lock() + r.cert, r.modTime = &c, mt + r.mu.Unlock() + return nil +} + +// GetCertificate is the tls.Config hook. It reloads when server.crt's modtime has +// advanced past the cached copy, otherwise returns the cached cert unchanged. +func (r *certReloader) GetCertificate(*tls.ClientHelloInfo) (*tls.Certificate, error) { + r.mu.RLock() + cur, mt := r.cert, r.modTime + r.mu.RUnlock() + + if fi, err := os.Stat(r.certPath); err == nil && fi.ModTime().After(mt) { + if err := r.reload(); err != nil { + log.Printf("agent plane: cert reload failed, keeping previous: %v", err) + } else { + r.mu.RLock() + cur = r.cert + r.mu.RUnlock() + log.Printf("agent plane: reloaded rotated server cert %s", r.certPath) + } + } + return cur, nil +} diff --git a/serving/fleet-manager/certreload_test.go b/serving/fleet-manager/certreload_test.go new file mode 100644 index 0000000..7bc0519 --- /dev/null +++ b/serving/fleet-manager/certreload_test.go @@ -0,0 +1,109 @@ +package main + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "math/big" + "os" + "path/filepath" + "testing" + "time" +) + +// writePair mints a throwaway self-signed leaf with the given CN and writes it as +// a cert/key PEM pair, so the reloader has a real, loadable x509 key pair. +func writePair(t *testing.T, certPath, keyPath, cn string) { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(1), + Subject: pkix.Name{CommonName: cn}, + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + } + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) + if err != nil { + t.Fatal(err) + } + kder, err := x509.MarshalPKCS8PrivateKey(key) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(certPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(keyPath, pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: kder}), 0o600); err != nil { + t.Fatal(err) + } +} + +func leafCN(t *testing.T, r *certReloader) string { + t.Helper() + c, err := r.GetCertificate(nil) + if err != nil { + t.Fatal(err) + } + leaf, err := x509.ParseCertificate(c.Certificate[0]) + if err != nil { + t.Fatal(err) + } + return leaf.Subject.CommonName +} + +func TestCertReloaderHotSwap(t *testing.T) { + dir := t.TempDir() + crt := filepath.Join(dir, "server.crt") + key := filepath.Join(dir, "server.key") + + writePair(t, crt, key, "old") + r, err := newCertReloader(crt, key) + if err != nil { + t.Fatalf("newCertReloader: %v", err) + } + if cn := leafCN(t, r); cn != "old" { + t.Fatalf("initial CN = %q, want old", cn) + } + + // Rotate the pair on disk with a strictly-later modtime — GetCertificate must + // pick it up without any explicit reload call (the renewer-rotates-in-place case). + writePair(t, crt, key, "new") + future := time.Now().Add(2 * time.Second) + if err := os.Chtimes(crt, future, future); err != nil { + t.Fatal(err) + } + if cn := leafCN(t, r); cn != "new" { + t.Fatalf("post-rotation CN = %q, want new", cn) + } +} + +func TestCertReloaderKeepsPreviousOnBadReload(t *testing.T) { + dir := t.TempDir() + crt := filepath.Join(dir, "server.crt") + key := filepath.Join(dir, "server.key") + + writePair(t, crt, key, "good") + r, err := newCertReloader(crt, key) + if err != nil { + t.Fatal(err) + } + + // Simulate a half-written rotation: cert file replaced with garbage + a newer + // modtime. Reload fails; the last good cert must keep serving. + if err := os.WriteFile(crt, []byte("not a pem"), 0o644); err != nil { + t.Fatal(err) + } + future := time.Now().Add(2 * time.Second) + if err := os.Chtimes(crt, future, future); err != nil { + t.Fatal(err) + } + if cn := leafCN(t, r); cn != "good" { + t.Fatalf("after bad reload CN = %q, want good (kept previous)", cn) + } +} diff --git a/serving/fleet-manager/defaults.env b/serving/fleet-manager/defaults.env index 08699ca..e9889dd 100644 --- a/serving/fleet-manager/defaults.env +++ b/serving/fleet-manager/defaults.env @@ -42,10 +42,31 @@ AWS_HOSTED_ZONE_ID= # --- agent plane (gRPC + mTLS, fleet CA) --------------------------------------- # Routable mgmt address enrolled hosts dial (":9600" = all interfaces). The -# server cert is issued by the fleet CA (server-pki.sh); the client CA is the -# fleet root so host TPM certs verify. Paths are fixed in docker-compose.yml. +# server cert is issued by the fleet CA (server-pki.sh, or the auto-renewer +# below); the client CA is the fleet root+intermediate so host TPM certs verify. +# Paths are fixed in docker-compose.yml. FLEET_MGR_AGENT_ADDR=:9600 +# --- agent-plane cert auto-renewer (compose profile: cert-auto) ---------------- +# Only used when you run the `agent-cert-renewer` sidecar (step-ca co-located on +# this host). It issues the server cert once, then auto-renews before the CA cap, +# preserving the SANs. Defaults suit the bench; override host paths in .env if +# your step-ca lays out its PKI differently. +# AGENT_CERT_SANS all names hosts dial (first = CN). manager.fleet.lab +# is the fleet-DNS name; keep the host FQDN too. +# AGENT_CA_URL the co-located fleet step-ca (host-loopback). +# AGENT_CERT_NOT_AFTER first-issue lifetime (<= CA max, currently 2160h). +# AGENT_CERT_RENEW_EXPIRES_IN renew when less than this remains. +# FLEET_CA_ROOT / _INTERMEDIATE / _PROVISIONER_PW host paths to the CA material. +AGENT_CERT_SANS=manager.fleet.lab,diskless-server.bitasylum.org +AGENT_CA_URL=https://localhost:8443 +AGENT_CERT_NOT_AFTER=2160h +AGENT_CERT_RENEW_EXPIRES_IN=720h +FLEET_CA_PROVISIONER=enrol +FLEET_CA_ROOT=../dot1x-bench/pki/root_ca.crt +FLEET_CA_INTERMEDIATE=../dot1x-bench/pki/intermediate_ca.crt +FLEET_CA_PROVISIONER_PW=../dot1x-bench/pki/stepca/secrets/password + # --- authorise pillar (enrol-broker admin plane over mTLS) --------------------- # The broker runs network_mode:host on 127.0.0.1:9443. The manager's client cert # is minted by ../enrol-broker/admin-pki.sh (admin CA), into ./pki/broker/ . diff --git a/serving/fleet-manager/docker-compose.yml b/serving/fleet-manager/docker-compose.yml index c607436..f29b8f6 100644 --- a/serving/fleet-manager/docker-compose.yml +++ b/serving/fleet-manager/docker-compose.yml @@ -118,6 +118,45 @@ services: # cert. (UI TLS is Traefik's — no admin cert here.) See DEPLOY.md/server-pki.sh. - ./pki:/pki:ro + # --- agent-plane server-cert auto-renewer (optional; step-ca co-located) ----- + # Issues the manager's agent-plane server cert ONCE from the fleet step-ca, then + # runs `step ca renew --daemon` to rotate it well before the CA's ~90-day cap — + # replacing the manual server-pki.sh + `docker exec … step ca certificate` dance + # and the periodic re-mint. Renewal PRESERVES the subject + SANs of the first + # issuance, so agent-TLS SAN drift can't recur. The manager hot-reloads the + # rotated cert via GetCertificate (certreload.go) — no restart, no dropped hub. + # + # OPT-IN via `--profile cert-auto`, and ONLY valid where the fleet step-ca + # (serving/dot1x-bench) runs on THIS host: it mounts the CA root + intermediate + + # 'enrol' provisioner password read-only and reaches the CA over host-loopback. + # Where the CA is remote, leave this off and issue via server-pki.sh (or ACME). + # + # docker compose --profile cert-auto up -d agent-cert-renewer fleet-manager + agent-cert-renewer: + image: harbor.bitasylum.org/dockerhub/smallstep/step-cli:latest + profiles: ["cert-auto"] + restart: unless-stopped + network_mode: host # reach the co-located CA on ${AGENT_CA_URL} + user: "0:0" # writes into /pki/agent, then chowns to the manager uid + environment: + STEPPATH: /tmp/step + HOME: /tmp + AGENT_SANS: "${AGENT_CERT_SANS:-manager.fleet.lab,diskless-server.bitasylum.org}" + CA_URL: "${AGENT_CA_URL:-https://localhost:8443}" + PROVISIONER: "${FLEET_CA_PROVISIONER:-enrol}" + RENEW_EXPIRES_IN: "${AGENT_CERT_RENEW_EXPIRES_IN:-720h}" + CERT_NOT_AFTER: "${AGENT_CERT_NOT_AFTER:-2160h}" + OWNER: "65532:65532" + entrypoint: ["/bin/sh", "/renew/agent-cert-renewer.sh"] + volumes: + - ./pki/agent:/pki/agent # shared with the manager (rw here) + - ./agent-cert-renewer.sh:/renew/agent-cert-renewer.sh:ro + # fleet CA material (read-only), from the co-located step-ca stack. Override + # the host paths in .env if your bench lays them out differently. + - ${FLEET_CA_ROOT:-../dot1x-bench/pki/root_ca.crt}:/ca/root_ca.crt:ro + - ${FLEET_CA_INTERMEDIATE:-../dot1x-bench/pki/intermediate_ca.crt}:/ca/intermediate_ca.crt:ro + - ${FLEET_CA_PROVISIONER_PW:-../dot1x-bench/pki/stepca/secrets/password}:/ca/password:ro + # --- management-plane ingress (Traefik) — admin UI on :443 ------------------ # Terminates the Let's Encrypt cert (Route 53 DNS-01) on :443 and forwards # plaintext to the manager (behind-proxy mode, loopback :9500). Host networking diff --git a/serving/fleet-manager/server-pki.sh b/serving/fleet-manager/server-pki.sh index b8672c7..ca8f9cf 100755 --- a/serving/fleet-manager/server-pki.sh +++ b/serving/fleet-manager/server-pki.sh @@ -8,6 +8,11 @@ # fleet CA — this script issues it via `step ca certificate` against the running # fleet step-ca (serving/dot1x-bench/), the same CA that signs enrolment certs. # +# NOTE: where the fleet step-ca is CO-LOCATED on this host, prefer the +# `agent-cert-renewer` compose sidecar (docker-compose.yml, profile cert-auto) — +# it does this issuance AND auto-renews before the CA's ~90-day cap, preserving +# SANs, with no `step` CLI on the host. This script is the manual / remote-CA path. +# # This does NOT issue: # - the ADMIN-plane UI cert -> owned by the Traefik ingress (Let's Encrypt # DNS-01); the manager never terminates TLS. See DEPLOY.md §2c.