Audiobuzz
1db3977238
The agent-plane server cert was issued by hand (server-pki.sh / docker exec step ca certificate), which needs the step CLI on the host, re-specifies SANs each time, and silently expires at the CA's 90-day cap — the exact failure that just broke agent TLS (cert minted with a stale SAN, missing manager.fleet.lab). Automate it: - agent-cert-renewer sidecar (smallstep step-cli image, compose profile cert-auto): issues the cert once from the co-located fleet step-ca, then 'step ca renew --daemon' rotates it before the cap. Renewal preserves the subject + SANs, so SAN drift cannot recur. Chowns to the manager's nonroot uid so the distroless manager reads the key. Opt-in + only valid where the CA is co-located; server-pki.sh stays as the manual/remote-CA path. - manager hot-reloads via GetCertificate (certreload.go): a rotated cert on disk is picked up on the next handshake with no restart and no dropped agent streams; a half-written pair keeps the last good cert serving. Unit-tested. - defaults.env documents the cert-auto knobs; DEPLOY.md §2a recommends the renewer and flags the mandatory manager.fleet.lab SAN.
78 lines
2.4 KiB
Go
78 lines
2.4 KiB
Go
// certreload.go — hot-reloading server cert for the agent plane.
|
|
//
|
|
// The agent-plane server cert (issued by the fleet CA) is rotated OUT of process
|
|
// by the `agent-cert-renewer` sidecar running `step ca renew` (docker-compose.yml,
|
|
// profile cert-auto). For that rotation to take effect WITHOUT restarting the
|
|
// manager — which would drop every live agent stream — the TLS config serves the
|
|
// cert through GetCertificate instead of a fixed Certificates slice, and this
|
|
// reloader re-reads server.crt/server.key from disk when they change.
|
|
//
|
|
// Reload is lazy + modtime-gated: each handshake checks server.crt's modtime and
|
|
// re-reads the pair only when it advances. A failed reload (a half-written pair
|
|
// mid-renewal, transient bad perms) is logged and the LAST GOOD cert keeps
|
|
// serving — a renewer hiccup must never take the agent plane down — and because a
|
|
// failed reload does not advance the cached modtime, the next handshake retries
|
|
// until the write settles.
|
|
package main
|
|
|
|
import (
|
|
"crypto/tls"
|
|
"log"
|
|
"os"
|
|
"sync"
|
|
"time"
|
|
)
|
|
|
|
type certReloader struct {
|
|
certPath, keyPath string
|
|
|
|
mu sync.RWMutex
|
|
cert *tls.Certificate
|
|
modTime time.Time
|
|
}
|
|
|
|
// newCertReloader loads the pair once (fail-fast if missing/bad at startup) and
|
|
// returns a reloader whose GetCertificate hook is wired into the TLS config.
|
|
func newCertReloader(certPath, keyPath string) (*certReloader, error) {
|
|
r := &certReloader{certPath: certPath, keyPath: keyPath}
|
|
if err := r.reload(); err != nil {
|
|
return nil, err
|
|
}
|
|
return r, nil
|
|
}
|
|
|
|
func (r *certReloader) reload() error {
|
|
c, err := tls.LoadX509KeyPair(r.certPath, r.keyPath)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
var mt time.Time
|
|
if fi, err := os.Stat(r.certPath); err == nil {
|
|
mt = fi.ModTime()
|
|
}
|
|
r.mu.Lock()
|
|
r.cert, r.modTime = &c, mt
|
|
r.mu.Unlock()
|
|
return nil
|
|
}
|
|
|
|
// GetCertificate is the tls.Config hook. It reloads when server.crt's modtime has
|
|
// advanced past the cached copy, otherwise returns the cached cert unchanged.
|
|
func (r *certReloader) GetCertificate(*tls.ClientHelloInfo) (*tls.Certificate, error) {
|
|
r.mu.RLock()
|
|
cur, mt := r.cert, r.modTime
|
|
r.mu.RUnlock()
|
|
|
|
if fi, err := os.Stat(r.certPath); err == nil && fi.ModTime().After(mt) {
|
|
if err := r.reload(); err != nil {
|
|
log.Printf("agent plane: cert reload failed, keeping previous: %v", err)
|
|
} else {
|
|
r.mu.RLock()
|
|
cur = r.cert
|
|
r.mu.RUnlock()
|
|
log.Printf("agent plane: reloaded rotated server cert %s", r.certPath)
|
|
}
|
|
}
|
|
return cur, nil
|
|
}
|