From 9479b70b04897834bea8b7ac39f8d7af97857300 Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Sat, 4 Jul 2026 10:13:51 +0000 Subject: [PATCH] fleet-manager: add acme side-car for the admin-UI cert (LE DNS-01 via Route 53) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Auto-issue + auto-renew the admin-UI TLS cert with a long-running compose side-car instead of a host cron. The `acme` service (goacme/lego image, running acme/renew-loop.sh) obtains the cert over DNS-01 via the AWS Route 53 API, writes fullchain/privkey into ./pki/admin/ — the SAME dir the manager reads (./pki:/pki:ro) — and chowns them to the nonroot runtime uid so the read-only container can load the key. The manager's certReloader hot-swaps on the mtime bump, no restart. Loop renews within 30 days of expiry; ACME_STAGING=1 gives a dry-run vs LE staging. - docker-compose.yml: `acme` service + `acme-state` volume (lego account/cert store). ACME_DOMAIN reuses FLEET_MGR_RP_ID so cert host == passkey origin. - .env.example: ACME_EMAIL + AWS creds (or instance role) + ACME_STAGING. - DEPLOY.md §2c: rewritten to the side-car flow — scoped Route 53 IAM policy, bring-acme-up-first ordering, flip off insecure, and an origin-match note. (The docker-compose.yml commit also carries the local harbor.bitasylum.org influxdb image mirror edit.) --- serving/fleet-manager/.env.example | 11 ++++ serving/fleet-manager/DEPLOY.md | 68 ++++++++++++++++++------ serving/fleet-manager/acme/renew-loop.sh | 51 ++++++++++++++++++ serving/fleet-manager/docker-compose.yml | 30 ++++++++++- 4 files changed, 143 insertions(+), 17 deletions(-) create mode 100755 serving/fleet-manager/acme/renew-loop.sh diff --git a/serving/fleet-manager/.env.example b/serving/fleet-manager/.env.example index 1b1761c..365b8ed 100644 --- a/serving/fleet-manager/.env.example +++ b/serving/fleet-manager/.env.example @@ -19,6 +19,17 @@ FLEET_MGR_ADMIN_KEY=/pki/admin/privkey.pem # Dev-only escape hatch: plaintext, NO passkey auth. NEVER set true in prod. FLEET_MGR_ADMIN_INSECURE=false +# --- admin UI cert: Let's Encrypt DNS-01 via Route 53 (the `acme` side-car) ----- +# The side-car issues + auto-renews the UI cert into ./pki/admin/ ; the manager +# hot-reloads it. ACME_DOMAIN is taken from FLEET_MGR_RP_ID above. Give it AWS +# creds scoped to the bitasylum.org hosted zone (IAM policy in DEPLOY.md §2c), or +# omit them and attach an instance role. +ACME_EMAIL=you@bitasylum.org +AWS_ACCESS_KEY_ID= +AWS_SECRET_ACCESS_KEY= +AWS_REGION=us-east-1 +# ACME_STAGING=1 # uncomment for a dry-run vs LE staging (UNTRUSTED certs, no rate limit) + # --- agent plane (gRPC + mTLS, fleet CA) --------------------------------------- # Routable mgmt address enrolled hosts dial (":9600" = all interfaces). The # server cert is issued by the fleet CA (server-pki.sh); the client CA is the diff --git a/serving/fleet-manager/DEPLOY.md b/serving/fleet-manager/DEPLOY.md index 58fb598..b8c7365 100644 --- a/serving/fleet-manager/DEPLOY.md +++ b/serving/fleet-manager/DEPLOY.md @@ -93,28 +93,64 @@ curl -s --cert ./pki/broker/clients/fleet-manager.crt \ https://127.0.0.1:9443/pending ; echo # expect [] or null (empty queue) ``` -### 2c. Admin UI cert (Let's Encrypt DNS-01) +### 2c. Admin UI cert — Let's Encrypt DNS-01 via Route 53 (the `acme` side-car) The UI hostname resolves on the mgmt net and is **not internet-reachable**, so HTTP-01/TLS-ALPN-01 can't validate. Use **DNS-01**, which proves control of the -zone via a TXT record (and supports wildcards). Example with `lego`: +zone via a TXT record. The `acme` compose service does this automatically: a +long-running side-car (`acme/renew-loop.sh`, on the `goacme/lego` image) issues +the cert on first start and renews within 30 days of expiry, writing it into +`./pki/admin/` — the same dir the manager reads — so the manager's `certReloader` +hot-swaps it live, no restart. -```bash -# Credentials for your DNS provider are provider-specific (env vars). -lego --dns --domains diskless-server.bitasylum.org \ - --email you@example.org --accept-tos run -# Point the manager at the issued files (it hot-reloads on renewal): -mkdir -p ./pki/admin -cp .lego/certificates/diskless-server.bitasylum.org.crt ./pki/admin/fullchain.pem -cp .lego/certificates/diskless-server.bitasylum.org.key ./pki/admin/privkey.pem +**IAM policy** — give the side-car AWS creds scoped to the `bitasylum.org` hosted +zone (put `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` in `.env`, or attach an +instance role and omit them): + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { "Effect": "Allow", "Action": ["route53:GetChange"], "Resource": "arn:aws:route53:::change/*" }, + { "Effect": "Allow", "Action": ["route53:ListHostedZonesByName"], "Resource": "*" }, + { "Effect": "Allow", + "Action": ["route53:ListResourceRecordSets", "route53:ChangeResourceRecordSets"], + "Resource": "arn:aws:route53:::hostedzone/" } + ] +} ``` -Automate renewal (cron/systemd-timer) to rewrite those two files; the manager's -`certReloader` notices the mtime bump and swaps the cert live — no restart. A -publicly-trusted cert also keeps the WebAuthn secure-context clean (no browser -warnings). *Alternative:* front the loopback admin listener with your existing -mgmt reverse proxy and terminate the cert there; then set -`FLEET_MGR_ADMIN_INSECURE=false` still but let the proxy hold the cert. +**Run it** (bring the side-car up first so the cert exists before the manager): + +```bash +# In .env: set ACME_EMAIL + the AWS creds above. ACME_DOMAIN is taken from +# FLEET_MGR_RP_ID. Optionally ACME_STAGING=1 for a first dry-run (untrusted, no +# rate limit) to prove Route 53 access, then re-run without it for a real cert. +docker compose up -d acme +docker compose logs -f acme # wait for "staged /out/{fullchain,privkey}.pem" +ls -l pki/admin/ # fullchain.pem + privkey.pem, owned by uid 65532 +``` + +Then turn off insecure mode and (re)start the manager — the defaults +`FLEET_MGR_ADMIN_CERT=/pki/admin/fullchain.pem` / `_KEY=/pki/admin/privkey.pem` +already point at what the side-car wrote: + +```bash +# ensure FLEET_MGR_ADMIN_INSECURE=false in .env, then: +docker compose up -d --force-recreate fleet-manager +docker compose logs fleet-manager | grep -i listening # "admin (HTTPS + passkey) listening" +``` + +A publicly-trusted cert also keeps the WebAuthn secure-context clean (no browser +warnings). The side-car keeps renewing on its 12-hour loop; the manager reloads +each new cert automatically. + +> **Origin match:** the cert CN/SAN (`FLEET_MGR_RP_ID`) must equal the origin the +> browser uses *and* `FLEET_MGR_RP_ORIGIN` — passkeys are origin-bound. The admin +> listener binds loopback (`127.0.0.1:9500`) by default; expose it as +> `https://diskless-server.bitasylum.org` either by binding a routable +> `FLEET_MGR_ADMIN_ADDR` or fronting it with a mgmt proxy that passes TLS through +> (don't re-terminate, or the passkey origin/cert won't match). ## 3. Start InfluxDB diff --git a/serving/fleet-manager/acme/renew-loop.sh b/serving/fleet-manager/acme/renew-loop.sh new file mode 100755 index 0000000..ccf5134 --- /dev/null +++ b/serving/fleet-manager/acme/renew-loop.sh @@ -0,0 +1,51 @@ +#!/bin/sh +# renew-loop.sh — admin-UI cert issue/renew loop for the `acme` side-car. +# +# Runs inside the goacme/lego image (alpine-based). Obtains the fleet-manager +# admin-UI cert from Let's Encrypt via DNS-01 over AWS Route 53, and writes it +# into /out — bind-mounted to the SAME ./pki/admin dir the manager reads, so the +# manager's certReloader hot-swaps it on the next mtime bump (NO restart). +# +# The UI is on the mgmt network (not internet-reachable), so only DNS-01 can +# validate; Route 53 automates the _acme-challenge TXT record via the AWS API. +# lego renews only within --days of expiry, so this can loop harmlessly. +# +# Env (from compose/.env): ACME_DOMAIN, ACME_EMAIL, AWS_* creds (or an instance +# role), optional ACME_STAGING=1 for a dry-run against LE staging. +set -eu + +DOMAIN=${ACME_DOMAIN:?set ACME_DOMAIN} +EMAIL=${ACME_EMAIL:?set ACME_EMAIL} +UID_OUT=${RUNTIME_UID:-65532} # distroless "nonroot" — must be able to read the key +INTERVAL_OK=${INTERVAL_OK:-43200} # 12h between checks after success +INTERVAL_ERR=${INTERVAL_ERR:-3600} # 1h retry after a failure + +STAGING="" +[ -n "${ACME_STAGING:-}" ] && { + STAGING="--server https://acme-staging-v02.api.letsencrypt.org/directory" + echo "acme: using LE STAGING — certs are UNTRUSTED (dry run)" +} + +while true; do + # First pass issues; later passes renew when near expiry. + if [ -f "/le/certificates/$DOMAIN.crt" ]; then + set -- renew --days 30 + else + set -- run + fi + echo "acme: lego $* for $DOMAIN (Route 53 DNS-01)" + + # shellcheck disable=SC2086 # $STAGING is intentionally two words when set + if lego --accept-tos --email "$EMAIL" --dns route53 --domains "$DOMAIN" \ + --path /le $STAGING "$@"; then + install -m 644 "/le/certificates/$DOMAIN.crt" /out/fullchain.pem + install -m 600 "/le/certificates/$DOMAIN.key" /out/privkey.pem + chown "$UID_OUT:$UID_OUT" /out/fullchain.pem /out/privkey.pem 2>/dev/null \ + || echo "acme: WARN could not chown to $UID_OUT (manager may not read the key)" + echo "acme: staged /out/{fullchain,privkey}.pem; next check in ${INTERVAL_OK}s" + sleep "$INTERVAL_OK" + else + echo "acme: lego FAILED (check AWS creds / zone perms); retry in ${INTERVAL_ERR}s" + sleep "$INTERVAL_ERR" + fi +done diff --git a/serving/fleet-manager/docker-compose.yml b/serving/fleet-manager/docker-compose.yml index 2c3d467..527edd4 100644 --- a/serving/fleet-manager/docker-compose.yml +++ b/serving/fleet-manager/docker-compose.yml @@ -27,7 +27,7 @@ services: # --- metrics store (private; only the manager reads/writes it) -------------- influxdb: - image: influxdb:2.7 + image: harbor.bitasylum.org/dockerhub/library/influxdb:2.7 restart: unless-stopped # Publish to loopback only — the host-networked manager reaches it at # 127.0.0.1:8086; it is never on the boot/mgmt LAN. @@ -107,12 +107,40 @@ services: - fleet-data:/data # PKI, read-only: admin LE cert, agent-plane fleet-CA server cert, and the # broker admin-client cert. See DEPLOY.md / server-pki.sh for what lands here. + # The admin LE cert under ./pki/admin/ is produced by the `acme` side-car. - ./pki:/pki:ro + # --- admin-UI cert renewal side-car (Let's Encrypt DNS-01 via Route 53) ----- + # Long-running "compose cron": issues the UI cert on first start, then renews + # within 30 days of expiry. Writes into ./pki/admin/ — the SAME dir the manager + # reads (./pki:/pki:ro) — so the manager hot-reloads it with no restart. Needs + # outbound reach to Let's Encrypt + the AWS Route 53 API (bridge NAT). + # Bring this up BEFORE the manager (or the manager crash-loops until the cert + # exists, then self-heals). See DEPLOY.md §2c. + acme: + image: goacme/lego:v4.17.4 # mirror via harbor if no dockerhub egress + restart: unless-stopped + entrypoint: ["/bin/sh", "/renew-loop.sh"] + environment: + ACME_DOMAIN: "${FLEET_MGR_RP_ID:?set FLEET_MGR_RP_ID in .env}" # = the UI hostname + ACME_EMAIL: "${ACME_EMAIL:?set ACME_EMAIL in .env}" + ACME_STAGING: "${ACME_STAGING:-}" # set 1 for a dry-run + # AWS creds for the Route 53 DNS-01 challenge (or omit + use an instance role). + AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID:-}" + AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY:-}" + AWS_REGION: "${AWS_REGION:-us-east-1}" + volumes: + - ./acme/renew-loop.sh:/renew-loop.sh:ro + - acme-state:/le # lego account + cert store — PERSIST + - ./pki/admin:/out # shared with the manager's ./pki/admin + networks: + - fleet-mgmt + volumes: influx-data: influx-config: fleet-data: + acme-state: networks: fleet-mgmt: