fleet-manager: add acme side-car for the admin-UI cert (LE DNS-01 via Route 53)

Auto-issue + auto-renew the admin-UI TLS cert with a long-running compose
side-car instead of a host cron. The `acme` service (goacme/lego image, running
acme/renew-loop.sh) obtains the cert over DNS-01 via the AWS Route 53 API,
writes fullchain/privkey into ./pki/admin/ — the SAME dir the manager reads
(./pki:/pki:ro) — and chowns them to the nonroot runtime uid so the read-only
container can load the key. The manager's certReloader hot-swaps on the mtime
bump, no restart. Loop renews within 30 days of expiry; ACME_STAGING=1 gives a
dry-run vs LE staging.

- docker-compose.yml: `acme` service + `acme-state` volume (lego account/cert
  store). ACME_DOMAIN reuses FLEET_MGR_RP_ID so cert host == passkey origin.
- .env.example: ACME_EMAIL + AWS creds (or instance role) + ACME_STAGING.
- DEPLOY.md §2c: rewritten to the side-car flow — scoped Route 53 IAM policy,
  bring-acme-up-first ordering, flip off insecure, and an origin-match note.

(The docker-compose.yml commit also carries the local harbor.bitasylum.org
influxdb image mirror edit.)

serving/fleet-manager/.env.example

@@ -19,6 +19,17 @@ FLEET_MGR_ADMIN_KEY=/pki/admin/privkey.pem
 # Dev-only escape hatch: plaintext, NO passkey auth. NEVER set true in prod.
 FLEET_MGR_ADMIN_INSECURE=false
 
+# --- admin UI cert: Let's Encrypt DNS-01 via Route 53 (the `acme` side-car) -----
+# The side-car issues + auto-renews the UI cert into ./pki/admin/ ; the manager
+# hot-reloads it. ACME_DOMAIN is taken from FLEET_MGR_RP_ID above. Give it AWS
+# creds scoped to the bitasylum.org hosted zone (IAM policy in DEPLOY.md §2c), or
+# omit them and attach an instance role.
+ACME_EMAIL=you@bitasylum.org
+AWS_ACCESS_KEY_ID=
+AWS_SECRET_ACCESS_KEY=
+AWS_REGION=us-east-1
+# ACME_STAGING=1   # uncomment for a dry-run vs LE staging (UNTRUSTED certs, no rate limit)
+
 # --- agent plane (gRPC + mTLS, fleet CA) ---------------------------------------
 # Routable mgmt address enrolled hosts dial (":9600" = all interfaces). The
 # server cert is issued by the fleet CA (server-pki.sh); the client CA is the

serving/fleet-manager/DEPLOY.md

@@ -93,28 +93,64 @@ curl -s --cert ./pki/broker/clients/fleet-manager.crt \
         https://127.0.0.1:9443/pending ; echo    # expect [] or null (empty queue)
 ```
 
-### 2c. Admin UI cert (Let's Encrypt DNS-01)
+### 2c. Admin UI cert — Let's Encrypt DNS-01 via Route 53 (the `acme` side-car)
 
 The UI hostname resolves on the mgmt net and is **not internet-reachable**, so
 HTTP-01/TLS-ALPN-01 can't validate. Use **DNS-01**, which proves control of the
-zone via a TXT record (and supports wildcards). Example with `lego`:
+zone via a TXT record. The `acme` compose service does this automatically: a
+long-running side-car (`acme/renew-loop.sh`, on the `goacme/lego` image) issues
+the cert on first start and renews within 30 days of expiry, writing it into
+`./pki/admin/` — the same dir the manager reads — so the manager's `certReloader`
+hot-swaps it live, no restart.
 
-```bash
+**IAM policy** — give the side-car AWS creds scoped to the `bitasylum.org` hosted
-# Credentials for your DNS provider are provider-specific (env vars).
+zone (put `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` in `.env`, or attach an
-lego --dns <provider> --domains diskless-server.bitasylum.org \
+instance role and omit them):
-     --email you@example.org --accept-tos run
+
-# Point the manager at the issued files (it hot-reloads on renewal):
+```json
-mkdir -p ./pki/admin
+{
-cp .lego/certificates/diskless-server.bitasylum.org.crt  ./pki/admin/fullchain.pem
+  "Version": "2012-10-17",
-cp .lego/certificates/diskless-server.bitasylum.org.key  ./pki/admin/privkey.pem
+  "Statement": [
+    { "Effect": "Allow", "Action": ["route53:GetChange"], "Resource": "arn:aws:route53:::change/*" },
+    { "Effect": "Allow", "Action": ["route53:ListHostedZonesByName"], "Resource": "*" },
+    { "Effect": "Allow",
+      "Action": ["route53:ListResourceRecordSets", "route53:ChangeResourceRecordSets"],
+      "Resource": "arn:aws:route53:::hostedzone/<YOUR_ZONE_ID>" }
+  ]
+}
 ```
 
-Automate renewal (cron/systemd-timer) to rewrite those two files; the manager's
+**Run it** (bring the side-car up first so the cert exists before the manager):
-`certReloader` notices the mtime bump and swaps the cert live — no restart. A
+
-publicly-trusted cert also keeps the WebAuthn secure-context clean (no browser
+```bash
-warnings). *Alternative:* front the loopback admin listener with your existing
+# In .env: set ACME_EMAIL + the AWS creds above. ACME_DOMAIN is taken from
-mgmt reverse proxy and terminate the cert there; then set
+# FLEET_MGR_RP_ID. Optionally ACME_STAGING=1 for a first dry-run (untrusted, no
-`FLEET_MGR_ADMIN_INSECURE=false` still but let the proxy hold the cert.
+# rate limit) to prove Route 53 access, then re-run without it for a real cert.
+docker compose up -d acme
+docker compose logs -f acme      # wait for "staged /out/{fullchain,privkey}.pem"
+ls -l pki/admin/                 # fullchain.pem + privkey.pem, owned by uid 65532
+```
+
+Then turn off insecure mode and (re)start the manager — the defaults
+`FLEET_MGR_ADMIN_CERT=/pki/admin/fullchain.pem` / `_KEY=/pki/admin/privkey.pem`
+already point at what the side-car wrote:
+
+```bash
+# ensure FLEET_MGR_ADMIN_INSECURE=false in .env, then:
+docker compose up -d --force-recreate fleet-manager
+docker compose logs fleet-manager | grep -i listening   # "admin (HTTPS + passkey) listening"
+```
+
+A publicly-trusted cert also keeps the WebAuthn secure-context clean (no browser
+warnings). The side-car keeps renewing on its 12-hour loop; the manager reloads
+each new cert automatically.
+
+> **Origin match:** the cert CN/SAN (`FLEET_MGR_RP_ID`) must equal the origin the
+> browser uses *and* `FLEET_MGR_RP_ORIGIN` — passkeys are origin-bound. The admin
+> listener binds loopback (`127.0.0.1:9500`) by default; expose it as
+> `https://diskless-server.bitasylum.org` either by binding a routable
+> `FLEET_MGR_ADMIN_ADDR` or fronting it with a mgmt proxy that passes TLS through
+> (don't re-terminate, or the passkey origin/cert won't match).
 
 ## 3. Start InfluxDB
 

serving/fleet-manager/acme/renew-loop.sh

@@ -0,0 +1,51 @@
+#!/bin/sh
+# renew-loop.sh — admin-UI cert issue/renew loop for the `acme` side-car.
+#
+# Runs inside the goacme/lego image (alpine-based). Obtains the fleet-manager
+# admin-UI cert from Let's Encrypt via DNS-01 over AWS Route 53, and writes it
+# into /out — bind-mounted to the SAME ./pki/admin dir the manager reads, so the
+# manager's certReloader hot-swaps it on the next mtime bump (NO restart).
+#
+# The UI is on the mgmt network (not internet-reachable), so only DNS-01 can
+# validate; Route 53 automates the _acme-challenge TXT record via the AWS API.
+# lego renews only within --days of expiry, so this can loop harmlessly.
+#
+# Env (from compose/.env): ACME_DOMAIN, ACME_EMAIL, AWS_* creds (or an instance
+# role), optional ACME_STAGING=1 for a dry-run against LE staging.
+set -eu
+
+DOMAIN=${ACME_DOMAIN:?set ACME_DOMAIN}
+EMAIL=${ACME_EMAIL:?set ACME_EMAIL}
+UID_OUT=${RUNTIME_UID:-65532}          # distroless "nonroot" — must be able to read the key
+INTERVAL_OK=${INTERVAL_OK:-43200}      # 12h between checks after success
+INTERVAL_ERR=${INTERVAL_ERR:-3600}     # 1h retry after a failure
+
+STAGING=""
+[ -n "${ACME_STAGING:-}" ] && {
+  STAGING="--server https://acme-staging-v02.api.letsencrypt.org/directory"
+  echo "acme: using LE STAGING — certs are UNTRUSTED (dry run)"
+}
+
+while true; do
+  # First pass issues; later passes renew when near expiry.
+  if [ -f "/le/certificates/$DOMAIN.crt" ]; then
+    set -- renew --days 30
+  else
+    set -- run
+  fi
+  echo "acme: lego $* for $DOMAIN (Route 53 DNS-01)"
+
+  # shellcheck disable=SC2086  # $STAGING is intentionally two words when set
+  if lego --accept-tos --email "$EMAIL" --dns route53 --domains "$DOMAIN" \
+          --path /le $STAGING "$@"; then
+    install -m 644 "/le/certificates/$DOMAIN.crt" /out/fullchain.pem
+    install -m 600 "/le/certificates/$DOMAIN.key" /out/privkey.pem
+    chown "$UID_OUT:$UID_OUT" /out/fullchain.pem /out/privkey.pem 2>/dev/null \
+      || echo "acme: WARN could not chown to $UID_OUT (manager may not read the key)"
+    echo "acme: staged /out/{fullchain,privkey}.pem; next check in ${INTERVAL_OK}s"
+    sleep "$INTERVAL_OK"
+  else
+    echo "acme: lego FAILED (check AWS creds / zone perms); retry in ${INTERVAL_ERR}s"
+    sleep "$INTERVAL_ERR"
+  fi
+done

serving/fleet-manager/docker-compose.yml

@@ -27,7 +27,7 @@
 services:
   # --- metrics store (private; only the manager reads/writes it) --------------
   influxdb:
-    image: influxdb:2.7
+    image: harbor.bitasylum.org/dockerhub/library/influxdb:2.7
     restart: unless-stopped
     # Publish to loopback only — the host-networked manager reaches it at
     # 127.0.0.1:8086; it is never on the boot/mgmt LAN.
@@ -107,12 +107,40 @@ services:
       - fleet-data:/data
       # PKI, read-only: admin LE cert, agent-plane fleet-CA server cert, and the
       # broker admin-client cert. See DEPLOY.md / server-pki.sh for what lands here.
+      # The admin LE cert under ./pki/admin/ is produced by the `acme` side-car.
       - ./pki:/pki:ro
 
+  # --- admin-UI cert renewal side-car (Let's Encrypt DNS-01 via Route 53) -----
+  # Long-running "compose cron": issues the UI cert on first start, then renews
+  # within 30 days of expiry. Writes into ./pki/admin/ — the SAME dir the manager
+  # reads (./pki:/pki:ro) — so the manager hot-reloads it with no restart. Needs
+  # outbound reach to Let's Encrypt + the AWS Route 53 API (bridge NAT).
+  # Bring this up BEFORE the manager (or the manager crash-loops until the cert
+  # exists, then self-heals). See DEPLOY.md §2c.
+  acme:
+    image: goacme/lego:v4.17.4       # mirror via harbor if no dockerhub egress
+    restart: unless-stopped
+    entrypoint: ["/bin/sh", "/renew-loop.sh"]
+    environment:
+      ACME_DOMAIN: "${FLEET_MGR_RP_ID:?set FLEET_MGR_RP_ID in .env}"  # = the UI hostname
+      ACME_EMAIL: "${ACME_EMAIL:?set ACME_EMAIL in .env}"
+      ACME_STAGING: "${ACME_STAGING:-}"                               # set 1 for a dry-run
+      # AWS creds for the Route 53 DNS-01 challenge (or omit + use an instance role).
+      AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID:-}"
+      AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY:-}"
+      AWS_REGION: "${AWS_REGION:-us-east-1}"
+    volumes:
+      - ./acme/renew-loop.sh:/renew-loop.sh:ro
+      - acme-state:/le               # lego account + cert store — PERSIST
+      - ./pki/admin:/out             # shared with the manager's ./pki/admin
+    networks:
+      - fleet-mgmt
+
 volumes:
   influx-data:
   influx-config:
   fleet-data:
+  acme-state:
 
 networks:
   fleet-mgmt: