The agent-plane server cert was issued by hand (server-pki.sh / docker exec
step ca certificate), which needs the step CLI on the host, re-specifies SANs
each time, and silently expires at the CA's 90-day cap — the exact failure that
just broke agent TLS (cert minted with a stale SAN, missing manager.fleet.lab).
Automate it:
- agent-cert-renewer sidecar (smallstep step-cli image, compose profile
cert-auto): issues the cert once from the co-located fleet step-ca, then
'step ca renew --daemon' rotates it before the cap. Renewal preserves the
subject + SANs, so SAN drift cannot recur. Chowns to the manager's nonroot
uid so the distroless manager reads the key. Opt-in + only valid where the CA
is co-located; server-pki.sh stays as the manual/remote-CA path.
- manager hot-reloads via GetCertificate (certreload.go): a rotated cert on
disk is picked up on the next handshake with no restart and no dropped agent
streams; a half-written pair keeps the last good cert serving. Unit-tested.
- defaults.env documents the cert-auto knobs; DEPLOY.md §2a recommends the
renewer and flags the mandatory manager.fleet.lab SAN.
`make up` / `logs` / `restart` / `pull` / `build` / `config` / `register-admin`
/ `backup` — each exports COMPOSE_ENV_FILES=defaults.env,.env so plain `make ...`
layers the committed defaults under .env with no manual export or --env-file.
Guards on .env existing (COMPOSE_ENV_FILES requires it) with a clear message.
`make help` lists targets. DEPLOY §1 points at it.
Rename .env.example -> defaults.env and make it the committed defaults layer.
Compose reads defaults.env then .env (later wins) via COMPOSE_ENV_FILES, feeding
BOTH container env and ${...} interpolation — so a changed default here reaches
every run without re-copying, and .env shrinks to just overrides + secrets.
No compose service restructure: the existing interpolated environment: blocks
already build the container env; the layering just supplies their values from two
files. (env_file: directive alone wouldn't work — it doesn't feed the image-tag /
cross-service Influx / ACME_EMAIL interpolation this compose relies on.)
Docs (DEPLOY §1, compose header, README) updated: export
COMPOSE_ENV_FILES=defaults.env,.env, keep only secrets/overrides in .env.
Follow the code removal through the rest of the tree now that Traefik owns UI TLS:
- compose: drop FLEET_MGR_ADMIN_BEHIND_PROXY + ADMIN_CERT/_KEY from the manager
env; correct the ./pki mount comment (agent + broker certs only).
- .env.example: drop the BEHIND_PROXY + cert vars + self-TLS note.
- DEPLOY.md / README: remove the self-TLS "alternative" and the BEHIND_PROXY /
ADMIN_CERT config rows.
- server-pki.sh: point the admin UI cert at Traefik, not ./pki/admin.
- delete serving/fleet-manager/acme/ (the retired ACME side-car).
Route the ad-hoc images I introduced through harbor.bitasylum.org (the standing
rule): backup.sh uses harbor.bitasylum.org/dockerhub/library/alpine:3.20 (via an
overridable ALPINE_IMAGE); the DEPLOY.md volume-chown fix + the goacme/lego prose
use the Harbor paths too. Compose (influxdb/traefik) + Dockerfile bases are already
harbor-mirrored; git.bitasylum.org/audiobuzz/* stays (CI-built).
- .env.example: RP_ORIGIN port-less, FLEET_MGR_ADMIN_BEHIND_PROXY=true, cert vars
labelled self-TLS-only; ACME vars are now Traefik's (dropped sidecar-only ones).
- DEPLOY.md §2c rewritten for Traefik (443 + Route 53 DNS-01), plus a "Network
paths" table answering how each plane is reached (enrolment via boot nginx,
agent direct mTLS via OPNsense L3, admin via Traefik). Start sequence + hardening
status updated (admin-exposure item resolved).
- README: admin transport = Traefik; behind-proxy explained; config table gains
BEHIND_PROXY; checkpoint 8 (hardening + Traefik ingress).
DEPLOY.md gains a "Hardening status" section (done vs operator-to-close), and
TODO #14 records the deployment as CI-built + acme-cert + backup, with the open
items called out: network-segment the :9500/:9600 listeners to the mgmt net, run
server-pki.sh when the agent lands, choose :9500 vs a 443 proxy, and digest-pin
the images.
Snapshot fleet-data (inventory + admins.json + enroll.key — the admin-lockout
risk) and influx-data to timestamped tarballs via a throwaway alpine container
(read-only, live-safe), with BACKUP_KEEP retention. Documented in DEPLOY.md
Backups with a restore pointer.
Split-horizon DNS made lego's propagation check query the container's internal
resolver, which doesn't serve the public _acme-challenge TXT record, so the
challenge timed out. Add ACME_RESOLVERS (comma-separated host:port, default
8.8.8.8:53) → one --dns.resolvers flag each. Documented in .env + DEPLOY.md.
lego auto-discovers the Route 53 zone via ListHostedZonesByName, so the zone ID
is optional — but pinning it disambiguates overlapping public/private zones and
lets the IAM policy drop the list permission. Thread it through compose + .env
+ DEPLOY.md (no script change; lego reads it from the environment).
Auto-issue + auto-renew the admin-UI TLS cert with a long-running compose
side-car instead of a host cron. The `acme` service (goacme/lego image, running
acme/renew-loop.sh) obtains the cert over DNS-01 via the AWS Route 53 API,
writes fullchain/privkey into ./pki/admin/ — the SAME dir the manager reads
(./pki:/pki:ro) — and chowns them to the nonroot runtime uid so the read-only
container can load the key. The manager's certReloader hot-swaps on the mtime
bump, no restart. Loop renews within 30 days of expiry; ACME_STAGING=1 gives a
dry-run vs LE staging.
- docker-compose.yml: `acme` service + `acme-state` volume (lego account/cert
store). ACME_DOMAIN reuses FLEET_MGR_RP_ID so cert host == passkey origin.
- .env.example: ACME_EMAIL + AWS creds (or instance role) + ACME_STAGING.
- DEPLOY.md §2c: rewritten to the side-car flow — scoped Route 53 IAM policy,
bring-acme-up-first ordering, flip off insecure, and an origin-match note.
(The docker-compose.yml commit also carries the local harbor.bitasylum.org
influxdb image mirror edit.)
The distroless nonroot runtime (uid 65532) couldn't write /data/inventory.json
because a named volume is created root-owned — approve failed with permission
denied on the inventory birth. Pre-create /data owned by nonroot in the image so
a fresh volume inherits it via Docker copy-up; an already-created root-owned
volume still needs a one-time chown (documented).
DEPLOY.md:
- 2b now covers BOTH broker deployments — the main serving stack
(enrol-pki/broker, ENROL_ADMIN_INSECURE=true by default → must flip to false)
and the bench stack — with the mint-once-into-broker-dir, copy-CA-to-manager
flow and an mTLS verify curl.
- Troubleshooting section: the data-volume permission-denied fix, the
"authorise plane not configured" (recreate + single-instance) fix, and the
"wrong version number" (broker plaintext) fix.
Complete the deployment story for the server-side control plane:
- docker-compose.yml: private InfluxDB (published to loopback only) + a
host-networked fleet-manager (reaches the loopback enrol-broker on
127.0.0.1:9443 and binds a routable gRPC agent listener, like the broker).
- .env.example: RP origin (diskless-server.bitasylum.org), Influx token/DB
password, cert/PKI paths; the Influx token wires influxdb + manager with
one value.
- server-pki.sh: issue the agent-plane server cert from the fleet CA (step)
and stage the fleet root as the client-verify CA.
- DEPLOY.md: end-to-end runbook — three-cert PKI (fleet-CA agent cert, broker
admin-client cert, Let's Encrypt DNS-01 UI cert), register-admin passkey
bootstrap, compose up, hardware-free smoke test, backups.
- .dockerignore: keep host node_modules/dist/pki/secrets out of the build
context so the embedded SPA is the one built in-image.
- .gitignore: never commit .env / issued PKI / data.
Design docs marked as-built: fleet-management-design.md as-built note
(resolved forks — gRPC bidi transport, InfluxDB metrics, SPA graphs, passkey
admin auth, broker fold-in "alongside"); tpm-enrolment-design.md (broker
fold-in + agent transport fork resolved); TODO.md #14 → partly built, with
the on-host agent + per-host boot-version assignment (#6) called out as the
remaining work.
Audiobuzz