The agent-plane server cert was issued by hand (server-pki.sh / docker exec
step ca certificate), which needs the step CLI on the host, re-specifies SANs
each time, and silently expires at the CA's 90-day cap — the exact failure that
just broke agent TLS (cert minted with a stale SAN, missing manager.fleet.lab).
Automate it:
- agent-cert-renewer sidecar (smallstep step-cli image, compose profile
cert-auto): issues the cert once from the co-located fleet step-ca, then
'step ca renew --daemon' rotates it before the cap. Renewal preserves the
subject + SANs, so SAN drift cannot recur. Chowns to the manager's nonroot
uid so the distroless manager reads the key. Opt-in + only valid where the CA
is co-located; server-pki.sh stays as the manual/remote-CA path.
- manager hot-reloads via GetCertificate (certreload.go): a rotated cert on
disk is picked up on the next handshake with no restart and no dropped agent
streams; a half-written pair keeps the last good cert serving. Unit-tested.
- defaults.env documents the cert-auto knobs; DEPLOY.md §2a recommends the
renewer and flags the mandatory manager.fleet.lab SAN.
Rename .env.example -> defaults.env and make it the committed defaults layer.
Compose reads defaults.env then .env (later wins) via COMPOSE_ENV_FILES, feeding
BOTH container env and ${...} interpolation — so a changed default here reaches
every run without re-copying, and .env shrinks to just overrides + secrets.
No compose service restructure: the existing interpolated environment: blocks
already build the container env; the layering just supplies their values from two
files. (env_file: directive alone wouldn't work — it doesn't feed the image-tag /
cross-service Influx / ACME_EMAIL interpolation this compose relies on.)
Docs (DEPLOY §1, compose header, README) updated: export
COMPOSE_ENV_FILES=defaults.env,.env, keep only secrets/overrides in .env.
Follow the code removal through the rest of the tree now that Traefik owns UI TLS:
- compose: drop FLEET_MGR_ADMIN_BEHIND_PROXY + ADMIN_CERT/_KEY from the manager
env; correct the ./pki mount comment (agent + broker certs only).
- .env.example: drop the BEHIND_PROXY + cert vars + self-TLS note.
- DEPLOY.md / README: remove the self-TLS "alternative" and the BEHIND_PROXY /
ADMIN_CERT config rows.
- server-pki.sh: point the admin UI cert at Traefik, not ./pki/admin.
- delete serving/fleet-manager/acme/ (the retired ACME side-car).
Management-plane ingress: a host-networked Traefik terminates the LE cert on :443
(Route 53 DNS-01, public-resolver propagation check) and forwards plaintext to
the manager running FLEET_MGR_ADMIN_BEHIND_PROXY=true on loopback :9500. Only the
admin plane rides Traefik — the agent gRPC/mTLS plane (:9600) stays direct (Traefik
can't terminate fleet-CA mTLS; hosts reach it via OPNsense L3 routing). This is the
k8s ingress-controller + cert-manager role in miniature.
- traefik/dynamic/fleet-manager.yml: file-provider router (Host rule -> manager),
passHostHeader so the WebAuthn RP origin matches.
- compose: traefik service (static config as CLI flags so ${ACME_EMAIL} interpolates)
+ traefik-acme volume; manager switched to behind-proxy defaults, RP_ORIGIN now
the port-less https origin.
- Retire the acme side-car (Traefik owns ACME now): remove the service + acme-state
volume; renew-loop.sh kept + relabelled as the self-TLS-mode cert helper.
- :443 only (admin UI is https-only) so it can't collide with a boot-LAN nginx :80.
Point the compose service at the registry image the new CI workflow pushes
(git.bitasylum.org/audiobuzz/fleet-manager:${FLEET_MANAGER_VERSION:-latest}) so
mgmt hosts `docker compose pull` instead of building at deploy time — matching
dnsmasq/spdk/kiosk. `build: .` stays for local dev. Add FLEET_MANAGER_VERSION to
.env.example.
Split-horizon DNS made lego's propagation check query the container's internal
resolver, which doesn't serve the public _acme-challenge TXT record, so the
challenge timed out. Add ACME_RESOLVERS (comma-separated host:port, default
8.8.8.8:53) → one --dns.resolvers flag each. Documented in .env + DEPLOY.md.
lego auto-discovers the Route 53 zone via ListHostedZonesByName, so the zone ID
is optional — but pinning it disambiguates overlapping public/private zones and
lets the IAM policy drop the list permission. Thread it through compose + .env
+ DEPLOY.md (no script change; lego reads it from the environment).
Auto-issue + auto-renew the admin-UI TLS cert with a long-running compose
side-car instead of a host cron. The `acme` service (goacme/lego image, running
acme/renew-loop.sh) obtains the cert over DNS-01 via the AWS Route 53 API,
writes fullchain/privkey into ./pki/admin/ — the SAME dir the manager reads
(./pki:/pki:ro) — and chowns them to the nonroot runtime uid so the read-only
container can load the key. The manager's certReloader hot-swaps on the mtime
bump, no restart. Loop renews within 30 days of expiry; ACME_STAGING=1 gives a
dry-run vs LE staging.
- docker-compose.yml: `acme` service + `acme-state` volume (lego account/cert
store). ACME_DOMAIN reuses FLEET_MGR_RP_ID so cert host == passkey origin.
- .env.example: ACME_EMAIL + AWS creds (or instance role) + ACME_STAGING.
- DEPLOY.md §2c: rewritten to the side-car flow — scoped Route 53 IAM policy,
bring-acme-up-first ordering, flip off insecure, and an origin-match note.
(The docker-compose.yml commit also carries the local harbor.bitasylum.org
influxdb image mirror edit.)
Complete the deployment story for the server-side control plane:
- docker-compose.yml: private InfluxDB (published to loopback only) + a
host-networked fleet-manager (reaches the loopback enrol-broker on
127.0.0.1:9443 and binds a routable gRPC agent listener, like the broker).
- .env.example: RP origin (diskless-server.bitasylum.org), Influx token/DB
password, cert/PKI paths; the Influx token wires influxdb + manager with
one value.
- server-pki.sh: issue the agent-plane server cert from the fleet CA (step)
and stage the fleet root as the client-verify CA.
- DEPLOY.md: end-to-end runbook — three-cert PKI (fleet-CA agent cert, broker
admin-client cert, Let's Encrypt DNS-01 UI cert), register-admin passkey
bootstrap, compose up, hardware-free smoke test, backups.
- .dockerignore: keep host node_modules/dist/pki/secrets out of the build
context so the embedded SPA is the one built in-image.
- .gitignore: never commit .env / issued PKI / data.
Design docs marked as-built: fleet-management-design.md as-built note
(resolved forks — gRPC bidi transport, InfluxDB metrics, SPA graphs, passkey
admin auth, broker fold-in "alongside"); tpm-enrolment-design.md (broker
fold-in + agent transport fork resolved); TODO.md #14 → partly built, with
the on-host agent + per-host boot-version assignment (#6) called out as the
remaining work.
Audiobuzz