fleet-manager: add acme side-car for the admin-UI cert (LE DNS-01 via Route 53)
Auto-issue + auto-renew the admin-UI TLS cert with a long-running compose side-car instead of a host cron. The `acme` service (goacme/lego image, running acme/renew-loop.sh) obtains the cert over DNS-01 via the AWS Route 53 API, writes fullchain/privkey into ./pki/admin/ — the SAME dir the manager reads (./pki:/pki:ro) — and chowns them to the nonroot runtime uid so the read-only container can load the key. The manager's certReloader hot-swaps on the mtime bump, no restart. Loop renews within 30 days of expiry; ACME_STAGING=1 gives a dry-run vs LE staging. - docker-compose.yml: `acme` service + `acme-state` volume (lego account/cert store). ACME_DOMAIN reuses FLEET_MGR_RP_ID so cert host == passkey origin. - .env.example: ACME_EMAIL + AWS creds (or instance role) + ACME_STAGING. - DEPLOY.md §2c: rewritten to the side-car flow — scoped Route 53 IAM policy, bring-acme-up-first ordering, flip off insecure, and an origin-match note. (The docker-compose.yml commit also carries the local harbor.bitasylum.org influxdb image mirror edit.)
This commit is contained in:
Audiobuzz
@@ -19,6 +19,17 @@ FLEET_MGR_ADMIN_KEY=/pki/admin/privkey.pem
|
||||
# Dev-only escape hatch: plaintext, NO passkey auth. NEVER set true in prod.
|
||||
FLEET_MGR_ADMIN_INSECURE=false
|
||||
|
||||
# --- admin UI cert: Let's Encrypt DNS-01 via Route 53 (the `acme` side-car) -----
|
||||
# The side-car issues + auto-renews the UI cert into ./pki/admin/ ; the manager
|
||||
# hot-reloads it. ACME_DOMAIN is taken from FLEET_MGR_RP_ID above. Give it AWS
|
||||
# creds scoped to the bitasylum.org hosted zone (IAM policy in DEPLOY.md §2c), or
|
||||
# omit them and attach an instance role.
|
||||
ACME_EMAIL=you@bitasylum.org
|
||||
AWS_ACCESS_KEY_ID=
|
||||
AWS_SECRET_ACCESS_KEY=
|
||||
AWS_REGION=us-east-1
|
||||
# ACME_STAGING=1 # uncomment for a dry-run vs LE staging (UNTRUSTED certs, no rate limit)
|
||||
|
||||
# --- agent plane (gRPC + mTLS, fleet CA) ---------------------------------------
|
||||
# Routable mgmt address enrolled hosts dial (":9600" = all interfaces). The
|
||||
# server cert is issued by the fleet CA (server-pki.sh); the client CA is the
|
||||
|
||||
@@ -93,28 +93,64 @@ curl -s --cert ./pki/broker/clients/fleet-manager.crt \
|
||||
https://127.0.0.1:9443/pending ; echo # expect [] or null (empty queue)
|
||||
```
|
||||
|
||||
### 2c. Admin UI cert (Let's Encrypt DNS-01)
|
||||
### 2c. Admin UI cert — Let's Encrypt DNS-01 via Route 53 (the `acme` side-car)
|
||||
|
||||
The UI hostname resolves on the mgmt net and is **not internet-reachable**, so
|
||||
HTTP-01/TLS-ALPN-01 can't validate. Use **DNS-01**, which proves control of the
|
||||
zone via a TXT record (and supports wildcards). Example with `lego`:
|
||||
zone via a TXT record. The `acme` compose service does this automatically: a
|
||||
long-running side-car (`acme/renew-loop.sh`, on the `goacme/lego` image) issues
|
||||
the cert on first start and renews within 30 days of expiry, writing it into
|
||||
`./pki/admin/` — the same dir the manager reads — so the manager's `certReloader`
|
||||
hot-swaps it live, no restart.
|
||||
|
||||
```bash
|
||||
# Credentials for your DNS provider are provider-specific (env vars).
|
||||
lego --dns <provider> --domains diskless-server.bitasylum.org \
|
||||
--email you@example.org --accept-tos run
|
||||
# Point the manager at the issued files (it hot-reloads on renewal):
|
||||
mkdir -p ./pki/admin
|
||||
cp .lego/certificates/diskless-server.bitasylum.org.crt ./pki/admin/fullchain.pem
|
||||
cp .lego/certificates/diskless-server.bitasylum.org.key ./pki/admin/privkey.pem
|
||||
**IAM policy** — give the side-car AWS creds scoped to the `bitasylum.org` hosted
|
||||
zone (put `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` in `.env`, or attach an
|
||||
instance role and omit them):
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{ "Effect": "Allow", "Action": ["route53:GetChange"], "Resource": "arn:aws:route53:::change/*" },
|
||||
{ "Effect": "Allow", "Action": ["route53:ListHostedZonesByName"], "Resource": "*" },
|
||||
{ "Effect": "Allow",
|
||||
"Action": ["route53:ListResourceRecordSets", "route53:ChangeResourceRecordSets"],
|
||||
"Resource": "arn:aws:route53:::hostedzone/<YOUR_ZONE_ID>" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Automate renewal (cron/systemd-timer) to rewrite those two files; the manager's
|
||||
`certReloader` notices the mtime bump and swaps the cert live — no restart. A
|
||||
publicly-trusted cert also keeps the WebAuthn secure-context clean (no browser
|
||||
warnings). *Alternative:* front the loopback admin listener with your existing
|
||||
mgmt reverse proxy and terminate the cert there; then set
|
||||
`FLEET_MGR_ADMIN_INSECURE=false` still but let the proxy hold the cert.
|
||||
**Run it** (bring the side-car up first so the cert exists before the manager):
|
||||
|
||||
```bash
|
||||
# In .env: set ACME_EMAIL + the AWS creds above. ACME_DOMAIN is taken from
|
||||
# FLEET_MGR_RP_ID. Optionally ACME_STAGING=1 for a first dry-run (untrusted, no
|
||||
# rate limit) to prove Route 53 access, then re-run without it for a real cert.
|
||||
docker compose up -d acme
|
||||
docker compose logs -f acme # wait for "staged /out/{fullchain,privkey}.pem"
|
||||
ls -l pki/admin/ # fullchain.pem + privkey.pem, owned by uid 65532
|
||||
```
|
||||
|
||||
Then turn off insecure mode and (re)start the manager — the defaults
|
||||
`FLEET_MGR_ADMIN_CERT=/pki/admin/fullchain.pem` / `_KEY=/pki/admin/privkey.pem`
|
||||
already point at what the side-car wrote:
|
||||
|
||||
```bash
|
||||
# ensure FLEET_MGR_ADMIN_INSECURE=false in .env, then:
|
||||
docker compose up -d --force-recreate fleet-manager
|
||||
docker compose logs fleet-manager | grep -i listening # "admin (HTTPS + passkey) listening"
|
||||
```
|
||||
|
||||
A publicly-trusted cert also keeps the WebAuthn secure-context clean (no browser
|
||||
warnings). The side-car keeps renewing on its 12-hour loop; the manager reloads
|
||||
each new cert automatically.
|
||||
|
||||
> **Origin match:** the cert CN/SAN (`FLEET_MGR_RP_ID`) must equal the origin the
|
||||
> browser uses *and* `FLEET_MGR_RP_ORIGIN` — passkeys are origin-bound. The admin
|
||||
> listener binds loopback (`127.0.0.1:9500`) by default; expose it as
|
||||
> `https://diskless-server.bitasylum.org` either by binding a routable
|
||||
> `FLEET_MGR_ADMIN_ADDR` or fronting it with a mgmt proxy that passes TLS through
|
||||
> (don't re-terminate, or the passkey origin/cert won't match).
|
||||
|
||||
## 3. Start InfluxDB
|
||||
|
||||
|
||||
Executable
+51
@@ -0,0 +1,51 @@
|
||||
#!/bin/sh
|
||||
# renew-loop.sh — admin-UI cert issue/renew loop for the `acme` side-car.
|
||||
#
|
||||
# Runs inside the goacme/lego image (alpine-based). Obtains the fleet-manager
|
||||
# admin-UI cert from Let's Encrypt via DNS-01 over AWS Route 53, and writes it
|
||||
# into /out — bind-mounted to the SAME ./pki/admin dir the manager reads, so the
|
||||
# manager's certReloader hot-swaps it on the next mtime bump (NO restart).
|
||||
#
|
||||
# The UI is on the mgmt network (not internet-reachable), so only DNS-01 can
|
||||
# validate; Route 53 automates the _acme-challenge TXT record via the AWS API.
|
||||
# lego renews only within --days of expiry, so this can loop harmlessly.
|
||||
#
|
||||
# Env (from compose/.env): ACME_DOMAIN, ACME_EMAIL, AWS_* creds (or an instance
|
||||
# role), optional ACME_STAGING=1 for a dry-run against LE staging.
|
||||
set -eu
|
||||
|
||||
DOMAIN=${ACME_DOMAIN:?set ACME_DOMAIN}
|
||||
EMAIL=${ACME_EMAIL:?set ACME_EMAIL}
|
||||
UID_OUT=${RUNTIME_UID:-65532} # distroless "nonroot" — must be able to read the key
|
||||
INTERVAL_OK=${INTERVAL_OK:-43200} # 12h between checks after success
|
||||
INTERVAL_ERR=${INTERVAL_ERR:-3600} # 1h retry after a failure
|
||||
|
||||
STAGING=""
|
||||
[ -n "${ACME_STAGING:-}" ] && {
|
||||
STAGING="--server https://acme-staging-v02.api.letsencrypt.org/directory"
|
||||
echo "acme: using LE STAGING — certs are UNTRUSTED (dry run)"
|
||||
}
|
||||
|
||||
while true; do
|
||||
# First pass issues; later passes renew when near expiry.
|
||||
if [ -f "/le/certificates/$DOMAIN.crt" ]; then
|
||||
set -- renew --days 30
|
||||
else
|
||||
set -- run
|
||||
fi
|
||||
echo "acme: lego $* for $DOMAIN (Route 53 DNS-01)"
|
||||
|
||||
# shellcheck disable=SC2086 # $STAGING is intentionally two words when set
|
||||
if lego --accept-tos --email "$EMAIL" --dns route53 --domains "$DOMAIN" \
|
||||
--path /le $STAGING "$@"; then
|
||||
install -m 644 "/le/certificates/$DOMAIN.crt" /out/fullchain.pem
|
||||
install -m 600 "/le/certificates/$DOMAIN.key" /out/privkey.pem
|
||||
chown "$UID_OUT:$UID_OUT" /out/fullchain.pem /out/privkey.pem 2>/dev/null \
|
||||
|| echo "acme: WARN could not chown to $UID_OUT (manager may not read the key)"
|
||||
echo "acme: staged /out/{fullchain,privkey}.pem; next check in ${INTERVAL_OK}s"
|
||||
sleep "$INTERVAL_OK"
|
||||
else
|
||||
echo "acme: lego FAILED (check AWS creds / zone perms); retry in ${INTERVAL_ERR}s"
|
||||
sleep "$INTERVAL_ERR"
|
||||
fi
|
||||
done
|
||||
@@ -27,7 +27,7 @@
|
||||
services:
|
||||
# --- metrics store (private; only the manager reads/writes it) --------------
|
||||
influxdb:
|
||||
image: influxdb:2.7
|
||||
image: harbor.bitasylum.org/dockerhub/library/influxdb:2.7
|
||||
restart: unless-stopped
|
||||
# Publish to loopback only — the host-networked manager reaches it at
|
||||
# 127.0.0.1:8086; it is never on the boot/mgmt LAN.
|
||||
@@ -107,12 +107,40 @@ services:
|
||||
- fleet-data:/data
|
||||
# PKI, read-only: admin LE cert, agent-plane fleet-CA server cert, and the
|
||||
# broker admin-client cert. See DEPLOY.md / server-pki.sh for what lands here.
|
||||
# The admin LE cert under ./pki/admin/ is produced by the `acme` side-car.
|
||||
- ./pki:/pki:ro
|
||||
|
||||
# --- admin-UI cert renewal side-car (Let's Encrypt DNS-01 via Route 53) -----
|
||||
# Long-running "compose cron": issues the UI cert on first start, then renews
|
||||
# within 30 days of expiry. Writes into ./pki/admin/ — the SAME dir the manager
|
||||
# reads (./pki:/pki:ro) — so the manager hot-reloads it with no restart. Needs
|
||||
# outbound reach to Let's Encrypt + the AWS Route 53 API (bridge NAT).
|
||||
# Bring this up BEFORE the manager (or the manager crash-loops until the cert
|
||||
# exists, then self-heals). See DEPLOY.md §2c.
|
||||
acme:
|
||||
image: goacme/lego:v4.17.4 # mirror via harbor if no dockerhub egress
|
||||
restart: unless-stopped
|
||||
entrypoint: ["/bin/sh", "/renew-loop.sh"]
|
||||
environment:
|
||||
ACME_DOMAIN: "${FLEET_MGR_RP_ID:?set FLEET_MGR_RP_ID in .env}" # = the UI hostname
|
||||
ACME_EMAIL: "${ACME_EMAIL:?set ACME_EMAIL in .env}"
|
||||
ACME_STAGING: "${ACME_STAGING:-}" # set 1 for a dry-run
|
||||
# AWS creds for the Route 53 DNS-01 challenge (or omit + use an instance role).
|
||||
AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID:-}"
|
||||
AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY:-}"
|
||||
AWS_REGION: "${AWS_REGION:-us-east-1}"
|
||||
volumes:
|
||||
- ./acme/renew-loop.sh:/renew-loop.sh:ro
|
||||
- acme-state:/le # lego account + cert store — PERSIST
|
||||
- ./pki/admin:/out # shared with the manager's ./pki/admin
|
||||
networks:
|
||||
- fleet-mgmt
|
||||
|
||||
volumes:
|
||||
influx-data:
|
||||
influx-config:
|
||||
fleet-data:
|
||||
acme-state:
|
||||
|
||||
networks:
|
||||
fleet-mgmt:
|
||||
|
||||
Reference in New Issue
Block a user