diff --git a/serving/fleet-manager/.env.example b/serving/fleet-manager/.env.example index f38c08a..1e259c2 100644 --- a/serving/fleet-manager/.env.example +++ b/serving/fleet-manager/.env.example @@ -2,40 +2,36 @@ # .env is gitignored — it holds the Influx token and DB password. # --- WebAuthn RP (admin passkeys are ORIGIN-BOUND — must be the UI's real URL) -- -# For this deployment the UI is https://diskless-server.bitasylum.org . +# The UI is fronted by Traefik on :443, so the origin is PORT-LESS. FLEET_MGR_RP_ID=diskless-server.bitasylum.org FLEET_MGR_RP_ORIGIN=https://diskless-server.bitasylum.org FLEET_MGR_RP_NAME=Fleet Command -# --- admin plane HTTPS --------------------------------------------------------- -# The UI cert is a publicly-trusted Let's Encrypt cert via DNS-01 (the UI is on -# the mgmt net, not internet-reachable, so HTTP-01/TLS-ALPN-01 can't validate; -# DNS-01 proves control of bitasylum.org via a TXT record). An external ACME -# client (lego/certbot + your DNS provider) writes fullchain/privkey into -# ./pki/admin/ ; the manager hot-reloads on renewal. See DEPLOY.md. +# --- admin plane (behind the Traefik ingress) ---------------------------------- +# Traefik terminates the LE cert on :443 and forwards plaintext here; the manager +# keeps passkey auth ON but serves plaintext on loopback (only Traefik reaches it). FLEET_MGR_ADMIN_ADDR=127.0.0.1:9500 -FLEET_MGR_ADMIN_CERT=/pki/admin/fullchain.pem -FLEET_MGR_ADMIN_KEY=/pki/admin/privkey.pem +FLEET_MGR_ADMIN_BEHIND_PROXY=true # Dev-only escape hatch: plaintext, NO passkey auth. NEVER set true in prod. FLEET_MGR_ADMIN_INSECURE=false +# Only for self-TLS mode (BEHIND_PROXY=false + a cert, no Traefik — see DEPLOY.md): +FLEET_MGR_ADMIN_CERT=/pki/admin/fullchain.pem +FLEET_MGR_ADMIN_KEY=/pki/admin/privkey.pem -# --- admin UI cert: Let's Encrypt DNS-01 via Route 53 (the `acme` side-car) ----- -# The side-car issues + auto-renews the UI cert into ./pki/admin/ ; the manager -# hot-reloads it. ACME_DOMAIN is taken from FLEET_MGR_RP_ID above. Give it AWS -# creds scoped to the bitasylum.org hosted zone (IAM policy in DEPLOY.md §2c), or -# omit them and attach an instance role. +# --- Traefik ACME (Let's Encrypt DNS-01 via Route 53) -------------------------- +# Traefik issues + auto-renews the :443 cert. Set the UI hostname in +# traefik/dynamic/fleet-manager.yml (== FLEET_MGR_RP_ID). Give it AWS creds scoped +# to the bitasylum.org hosted zone (IAM policy in DEPLOY.md §2c), or attach an +# instance role and leave them blank. ACME_EMAIL=you@bitasylum.org AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= AWS_REGION=us-east-1 -# Optional but recommended: pin the Route 53 zone. Skips ListHostedZonesByName -# (so you can drop it from the IAM policy) and disambiguates public/private zones. +# Optional but recommended: pin the Route 53 zone (skip ListHostedZonesByName; +# disambiguate public/private zones). Traefik reads it from the env. AWS_HOSTED_ZONE_ID= -# Public resolver(s) for the DNS-01 propagation check (comma-separated host:port). -# Needed because internal/split-horizon DNS won't resolve the public challenge -# record. Defaults to Google DNS; override if 8.8.8.8 is blocked (e.g. 1.1.1.1:53). -ACME_RESOLVERS=8.8.8.8:53 -# ACME_STAGING=1 # uncomment for a dry-run vs LE staging (UNTRUSTED certs, no rate limit) +# First dry-run vs LE staging: add the caserver flag to the traefik command (see +# docker-compose.yml) rather than an env var. # --- agent plane (gRPC + mTLS, fleet CA) --------------------------------------- # Routable mgmt address enrolled hosts dial (":9600" = all interfaces). The diff --git a/serving/fleet-manager/DEPLOY.md b/serving/fleet-manager/DEPLOY.md index 66e7177..2ae465f 100644 --- a/serving/fleet-manager/DEPLOY.md +++ b/serving/fleet-manager/DEPLOY.md @@ -93,19 +93,21 @@ curl -s --cert ./pki/broker/clients/fleet-manager.crt \ https://127.0.0.1:9443/pending ; echo # expect [] or null (empty queue) ``` -### 2c. Admin UI cert — Let's Encrypt DNS-01 via Route 53 (the `acme` side-car) +### 2c. Admin UI ingress — Traefik on :443 (Let's Encrypt DNS-01 via Route 53) -The UI hostname resolves on the mgmt net and is **not internet-reachable**, so -HTTP-01/TLS-ALPN-01 can't validate. Use **DNS-01**, which proves control of the -zone via a TXT record. The `acme` compose service does this automatically: a -long-running side-car (`acme/renew-loop.sh`, on the `goacme/lego` image) issues -the cert on first start and renews within 30 days of expiry, writing it into -`./pki/admin/` — the same dir the manager reads — so the manager's `certReloader` -hot-swaps it live, no restart. +The admin UI is fronted by the **`traefik`** compose service: it terminates the LE +cert on :443 and forwards plaintext to the manager (behind-proxy mode, loopback +:9500). The UI hostname resolves on the mgmt net and is **not internet-reachable**, +so only **DNS-01** can validate — Traefik automates the `_acme-challenge` TXT +record via the AWS Route 53 API and auto-renews. **Only the admin plane rides +Traefik**; the agent gRPC/mTLS plane (:9600) is direct (see [Network paths](#network-paths)). -**IAM policy** — give the side-car AWS creds scoped to the `bitasylum.org` hosted -zone (put `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` in `.env`, or attach an -instance role and omit them): +**Set the UI hostname** in `traefik/dynamic/fleet-manager.yml` — the `Host()` rule +(file config isn't env-interpolated), matching `FLEET_MGR_RP_ID`. + +**IAM policy** — AWS creds scoped to the `bitasylum.org` hosted zone (put +`AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` in `.env`, or attach an instance role +and omit them): ```json { @@ -120,49 +122,53 @@ instance role and omit them): } ``` -`AWS_HOSTED_ZONE_ID` is **optional**: lego auto-discovers the zone via -`ListHostedZonesByName` (hence that statement). Set it in `.env` to pin the zone -— required if a **public + private** zone share `bitasylum.org` (auto-discovery -may pick the wrong one), and it lets you **drop the `ListHostedZonesByName` -statement** for a tighter policy. +`AWS_HOSTED_ZONE_ID` is **optional**: the ACME client auto-discovers the zone via +`ListHostedZonesByName` (hence that statement). Set it (Traefik reads it from the +env) to pin the zone — required if a **public + private** zone share `bitasylum.org`, +and it lets you **drop the `ListHostedZonesByName` statement** for a tighter policy. -**Run it** (bring the side-car up first so the cert exists before the manager): +**Run it:** ```bash -# In .env: set ACME_EMAIL + the AWS creds above. ACME_DOMAIN is taken from -# FLEET_MGR_RP_ID. Optionally ACME_STAGING=1 for a first dry-run (untrusted, no -# rate limit) to prove Route 53 access, then re-run without it for a real cert. -docker compose up -d acme -docker compose logs -f acme # wait for "staged /out/{fullchain,privkey}.pem" -ls -l pki/admin/ # fullchain.pem + privkey.pem, owned by uid 65532 +# .env: ACME_EMAIL + the AWS creds above. For a first DRY-RUN vs LE staging +# (untrusted, no rate limit), add the caserver flag to the traefik `command:` in +# docker-compose.yml, prove issuance, then remove it for the real cert. +docker compose up -d traefik +docker compose logs -f traefik # watch the DNS-01 challenge succeed + cert stored ``` -Then turn off insecure mode and (re)start the manager — the defaults -`FLEET_MGR_ADMIN_CERT=/pki/admin/fullchain.pem` / `_KEY=/pki/admin/privkey.pem` -already point at what the side-car wrote: +Traefik runs the propagation check against **public** resolvers +(`8.8.8.8:53,1.1.1.1:53` in the command) — needed because split-horizon DNS won't +serve the public `_acme-challenge` record. The cert persists in the `traefik-acme` +volume (`acme.json`) and Traefik renews it automatically. A publicly-trusted cert +also keeps the WebAuthn secure-context clean. -```bash -# ensure FLEET_MGR_ADMIN_INSECURE=false in .env, then: -docker compose up -d --force-recreate fleet-manager -docker compose logs fleet-manager | grep -i listening # "admin (HTTPS + passkey) listening" -``` +> **Origin match:** passkeys are origin-bound, so `FLEET_MGR_RP_ORIGIN` must equal +> the browser's origin. With Traefik on :443 that's the **port-less** +> `https://diskless-server.bitasylum.org`; `FLEET_MGR_RP_ID` is the host. The +> manager binds loopback (`127.0.0.1:9500`) — only Traefik reaches it; never expose +> that port directly. -A publicly-trusted cert also keeps the WebAuthn secure-context clean (no browser -warnings). The side-car keeps renewing on its 12-hour loop; the manager reloads -each new cert automatically. +> **Self-TLS alternative (no Traefik):** set `FLEET_MGR_ADMIN_BEHIND_PROXY=false` +> and give the manager a cert at `/pki/admin/{fullchain,privkey}.pem` (e.g. via the +> `acme/renew-loop.sh` helper under a `goacme/lego` container). The manager then +> terminates TLS itself and the origin includes the port it listens on. -> **Split-horizon DNS:** the side-car defaults `ACME_RESOLVERS=8.8.8.8:53` so -> lego checks TXT propagation against **public** DNS. Without it, lego uses the -> container's internal resolver, which doesn't serve the public -> `_acme-challenge` record, and the challenge times out. Override if 8.8.8.8 is -> blocked (e.g. `1.1.1.1:53`, comma-separated for several). + +### Network paths (three planes, three trust stages) -> **Origin match:** the cert CN/SAN (`FLEET_MGR_RP_ID`) must equal the origin the -> browser uses *and* `FLEET_MGR_RP_ORIGIN` — passkeys are origin-bound. The admin -> listener binds loopback (`127.0.0.1:9500`) by default; expose it as -> `https://diskless-server.bitasylum.org` either by binding a routable -> `FLEET_MGR_ADMIN_ADDR` or fronting it with a mgmt proxy that passes TLS through -> (don't re-terminate, or the passkey origin/cert won't match). +| Plane | When | From | Path | Auth | +|---|---|---|---|---| +| Enrolment (`/enrol/`) | pre-auth (no cert yet) | boot LAN | boot nginx passthrough → broker :9090 | none (the one unauth edge) | +| Agent (gRPC :9600) | post-auth (802.1X'd, holds TPM cert) | authenticated host VLAN | **direct** → manager :9600, via **OPNsense L3** (allow only host-VLAN→:9600) | **mTLS, TPM cert (fleet CA)** | +| Admin (UI :443) | anytime | mgmt/admin network | **Traefik** → manager :9500 | WebAuthn passkey | + +The agent plane does **not** go through the boot nginx or Traefik: it connects +only after EAP-TLS/MACsec, and Traefik can't terminate the fleet-CA client-cert +mTLS. Segment it with an **OPNsense firewall rule** allowing only the authenticated +host VLAN → `manager:9600`. In the k8s target this becomes a dedicated TCP/TLS- +**passthrough** ingress (the pod still sees the client cert), separate from the +admin Ingress. ## 3. Start InfluxDB @@ -189,16 +195,17 @@ single-use token (jti ledger prevents replay). Open the link on a machine with your YubiKey, tap + PIN, and the passkey is registered. Further admins can be enrolled from the UI once one exists. -## 5. Start the manager +## 5. Start the manager + ingress ```bash -docker compose up -d fleet-manager +docker compose up -d fleet-manager traefik docker compose logs -f fleet-manager ``` -Healthy startup logs the four planes: admin HTTPS+passkey, agent gRPC/mTLS, -authorise (broker), and the InfluxDB gateway. Any plane whose config is absent -logs `DISABLED` and the rest still run. +Healthy startup logs the four planes: admin (`passkey, behind TLS proxy` — plaintext +on loopback), agent gRPC/mTLS, authorise (broker), and the InfluxDB gateway. Any +plane whose config is absent logs `DISABLED` and the rest still run. Traefik +terminates :443 and forwards to the manager's loopback listener. Then browse to `https://diskless-server.bitasylum.org/`, log in with the passkey, and you should see **Monitor / Authorise / Lifecycle**. @@ -262,24 +269,25 @@ If you run compose under a non-default project name, pass ## Hardening status -Done: distroless-nonroot runtime; passkey admin auth over HTTPS with an -auto-renewing LE cert (`acme` side-car); mTLS on both the agent (fleet CA) and -manager→broker (admin CA) planes; secrets kept server-side (`.env`/PKI gitignored); -optional planes soft-disable instead of crash-looping; CI builds + pushes the -image (`fleet-manager-image.yaml`); `backup.sh` for the durable volumes. +Done: distroless-nonroot runtime; passkey admin auth on :443 via the **Traefik** +mgmt ingress with an auto-renewing LE cert (Route 53 DNS-01); mTLS on both the +agent (fleet CA) and manager→broker (admin CA) planes; secrets kept server-side +(`.env`/PKI gitignored); optional planes soft-disable instead of crash-looping; CI +builds + pushes the image (`fleet-manager-image.yaml`); `backup.sh` for the durable +volumes. Still open (operator to close per environment): -- **Network segmentation.** The admin (`:9500`) and agent (`:9600`) listeners must - be reachable only from the **mgmt network** — never the boot LAN. Enforce with a - host firewall / security group; this repo can't. +- **Network segmentation.** The admin (`:443` Traefik, forwards to loopback `:9500`) + and agent (`:9600`) listeners must be reachable only from their zones — admin from + the mgmt network, agent from the authenticated host VLAN via **OPNsense L3** (allow + only host-VLAN → `:9600`). Never the boot LAN. Enforce at the firewall; this repo + can't. See [Network paths](#network-paths). - **Agent-plane server cert.** `server-pki.sh` isn't run yet (nothing connects to the agent plane until the on-host agent exists — it soft-disables meanwhile). Issue it when the agent lands. -- **Admin exposure.** Decide `:9500` direct vs a 443 mgmt proxy (TLS passthrough) - and align `FLEET_MGR_RP_ORIGIN` — see §2c. - **Digest pinning.** For a locked deployment, pin `fleet-manager`, `influxdb`, - and the `goacme/lego` images by `@sha256:…` (CLAUDE.md invariant #12) rather than - the tags used here for convenience. + and `traefik` by `@sha256:…` (CLAUDE.md invariant #12) rather than the tags used + here for convenience. ## What is NOT yet wired (recorded intent) diff --git a/serving/fleet-manager/README.md b/serving/fleet-manager/README.md index 94f5f26..ea5b3d4 100644 --- a/serving/fleet-manager/README.md +++ b/serving/fleet-manager/README.md @@ -14,7 +14,7 @@ network, so external deps are fine (`grpc-go`, `influxdb-client-go`). | Plane | Transport | Browser/peer identity | Who | Purpose | |---|---|---|---|---| -| **admin** | HTTPS (server-auth) | **WebAuthn/passkey** (YubiKey) → session | admin browsers | JSON API + SPA (monitor / authorise / lifecycle) | +| **admin** | HTTPS via **Traefik** (:443) → manager | **WebAuthn/passkey** (YubiKey) → session | admin browsers | JSON API + SPA (monitor / authorise / lifecycle) | | **agent** | gRPC + mTLS | **fleet CA** (host TPM cert) | enrolled hosts | bidi stream: health/metrics up, desired-state down | The agent plane uses the host's **reused TPM identity cert** (the same one used @@ -36,14 +36,16 @@ that's service-to-service — see [Authorise](#authorise-pillar).) be enrolled from the UI once one exists. - **RP is origin-bound**: `FLEET_MGR_RP_ID` / `FLEET_MGR_RP_ORIGIN` must be the UI's real hostname — for this deployment **`diskless-server.bitasylum.org`** / - `https://diskless-server.bitasylum.org`. -- **HTTPS cert** for the UI is a publicly-trusted cert via **Let's Encrypt - DNS-01** (the UI is on the mgmt network, not internet-reachable, so HTTP-01 / - TLS-ALPN-01 can't validate; DNS-01 proves control of `bitasylum.org` via a TXT - record and supports wildcards). An external ACME client (lego/certbot + DNS - provider) issues + renews to `FLEET_MGR_ADMIN_CERT/_KEY`; the manager - **hot-reloads** the cert on renewal (no restart). A trusted cert also keeps the - WebAuthn secure-context clean (no browser warnings). + `https://diskless-server.bitasylum.org` (port-less, since Traefik serves :443). +- **TLS + cert** are owned by the **`traefik`** mgmt-plane ingress: it terminates + a publicly-trusted **Let's Encrypt** cert (**Route 53 DNS-01** — the UI is on the + mgmt network, not internet-reachable, so HTTP-01/TLS-ALPN-01 can't validate) on + :443 and forwards plaintext to the manager, which runs + `FLEET_MGR_ADMIN_BEHIND_PROXY=true` (passkey auth ON, plaintext on loopback — + WebAuthn only needs the origin to match, not the app's own transport). Traefik + is the k8s ingress-controller + cert-manager role in miniature. *(Self-TLS mode + without Traefik is still available: `BEHIND_PROXY=false` + a cert at + `FLEET_MGR_ADMIN_CERT/_KEY`, e.g. from `acme/renew-loop.sh`.)* ## Monitoring path (no host ever touches InfluxDB) @@ -114,8 +116,9 @@ inventory for offline delivery. Metrics writing is behind a `MetricsSink` | Var | Default | Notes | |---|---|---| -| `FLEET_MGR_ADMIN_ADDR` | `127.0.0.1:9500` | admin listener | -| `FLEET_MGR_ADMIN_CERT` / `_KEY` | — | UI server cert/key (server-auth; **LE DNS-01**, hot-reloaded) | +| `FLEET_MGR_ADMIN_ADDR` | `127.0.0.1:9500` | admin listener (loopback — Traefik forwards here) | +| `FLEET_MGR_ADMIN_BEHIND_PROXY` | `false` | passkey ON, plaintext listener (Traefik/ingress terminates TLS). Compose defaults it `true` | +| `FLEET_MGR_ADMIN_CERT` / `_KEY` | — | UI server cert/key — **self-TLS mode only** (`BEHIND_PROXY=false`, no Traefik) | | `FLEET_MGR_RP_ID` | — | WebAuthn RP ID = UI hostname (`diskless-server.bitasylum.org`) | | `FLEET_MGR_RP_ORIGIN` | — | WebAuthn origin (`https://diskless-server.bitasylum.org`) | | `FLEET_MGR_RP_NAME` | `Fleet Command` | RP display name | @@ -189,3 +192,9 @@ curl -s localhost:9500/api/hosts | jq '.hosts[] | {name:.identity.starship_name, server cert), and [`DEPLOY.md`](DEPLOY.md) (PKI, LE DNS-01, `register-admin` bootstrap). Design docs updated (manager marked built; resolved transport/auth/ metrics forks; per-host-versioning gap recorded). +- **Checkpoint 8 (done):** hardening + ingress — CI image build + (`fleet-manager-image.yaml`, pulled by tag), `backup.sh`; a **behind-proxy admin + mode** (passkey over plaintext behind a TLS-terminating proxy) fronted by a + **Traefik** mgmt ingress (:443 + Route 53 DNS-01, retiring the acme side-car). + This aligns with the k8s target (Traefik → ingress controller + cert-manager); + the agent plane stays direct mTLS via OPNsense L3. See [`DEPLOY.md`](DEPLOY.md).