From 0360c1c6d1f201726fa582d1436b366a9aa5fd5a Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Sat, 4 Jul 2026 09:56:09 +0000 Subject: [PATCH] fleet-manager: fix nonroot data-volume ownership + document broker wiring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The distroless nonroot runtime (uid 65532) couldn't write /data/inventory.json because a named volume is created root-owned — approve failed with permission denied on the inventory birth. Pre-create /data owned by nonroot in the image so a fresh volume inherits it via Docker copy-up; an already-created root-owned volume still needs a one-time chown (documented). DEPLOY.md: - 2b now covers BOTH broker deployments — the main serving stack (enrol-pki/broker, ENROL_ADMIN_INSECURE=true by default → must flip to false) and the bench stack — with the mint-once-into-broker-dir, copy-CA-to-manager flow and an mTLS verify curl. - Troubleshooting section: the data-volume permission-denied fix, the "authorise plane not configured" (recreate + single-instance) fix, and the "wrong version number" (broker plaintext) fix. --- serving/fleet-manager/DEPLOY.md | 61 +++++++++++++++++++++++++++++--- serving/fleet-manager/Dockerfile | 14 ++++++-- 2 files changed, 68 insertions(+), 7 deletions(-) diff --git a/serving/fleet-manager/DEPLOY.md b/serving/fleet-manager/DEPLOY.md index 7f96e31..58fb598 100644 --- a/serving/fleet-manager/DEPLOY.md +++ b/serving/fleet-manager/DEPLOY.md @@ -57,12 +57,40 @@ AGENT_SANS=diskless-server.bitasylum.org,10.0.0.1 \ The manager is an mTLS **client** of the enrol-broker's admin plane. Mint its client cert from the broker's **dedicated admin CA** (NOT the fleet CA — reusing -the fleet CA would let any enrolled host authenticate to the admin plane): +the fleet CA would let any enrolled host authenticate to the admin plane). Mint +the whole set **once** into the broker's own PKI dir (so the broker gets its +`server.crt`/`server.key` too), then copy the CA + client cert to the manager so +**both sides share one admin CA**. +The broker's PKI dir + its "serve mTLS" switch differ by which stack runs it: + +- **Main serving stack** (`serving/docker-compose.yml`) — broker mounts + `./enrol-pki:/pki`, and the admin plane is **plaintext by default** + (`ENROL_ADMIN_INSECURE=true` in `serving/.env`). Set it `false`, then: + ```bash + cd .. # serving/ + ./enrol-broker/admin-pki.sh ./enrol-pki/broker fleet-manager + # -> enrol-pki/broker/{admin-ca.crt, server.crt, server.key, clients/fleet-manager.*} + docker compose up -d enrol-broker # restart so it serves mTLS (no cert error in logs) + mkdir -p fleet-manager/pki/broker + cp enrol-pki/broker/admin-ca.crt fleet-manager/pki/broker/ + cp -r enrol-pki/broker/clients fleet-manager/pki/broker/ + ``` +- **Bench stack** (`serving/dot1x-bench/`) — broker mounts `./pki:/pki` and is + mTLS-by-default (cert paths hardcoded in that compose): + ```bash + ../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager + cp -r ../dot1x-bench/pki/broker ./pki/broker + ``` + +Either way the manager ends up with `./pki/broker/{admin-ca.crt, +clients/fleet-manager.{crt,key}}`. If you (re)generate a fresh admin CA, **re-copy** +it to the manager — a stale copy won't verify. Verify the mTLS end to end: ```bash -../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager -cp -r ../dot1x-bench/pki/broker ./pki/broker -# -> ./pki/broker/{admin-ca.crt, clients/fleet-manager.{crt,key}} +curl -s --cert ./pki/broker/clients/fleet-manager.crt \ + --key ./pki/broker/clients/fleet-manager.key \ + --cacert ./pki/broker/admin-ca.crt \ + https://127.0.0.1:9443/pending ; echo # expect [] or null (empty queue) ``` ### 2c. Admin UI cert (Let's Encrypt DNS-01) @@ -146,6 +174,31 @@ and the `influx-data` volume. Losing `admins.json` locks every admin out — you re-bootstrap with `register-admin`. Losing `enroll.key` invalidates any outstanding (unused) enrol links. +## Troubleshooting + +- **`permission denied` writing the inventory on approve** — the runtime image is + distroless `nonroot` (uid **65532**), but a named volume is created root-owned, + so `/data` isn't writable. A rebuilt image pre-creates `/data` owned by nonroot + (a *fresh* volume inherits it), but an **already-created** volume must be fixed + once: + ```bash + docker volume ls | grep fleet # find the name, e.g. fleet-manager_fleet-data + docker run --rm -v :/data alpine chown -R 65532:65532 /data + ``` +- **`authorise plane not configured (no enrol-broker wired)` in the SPA** — the + manager has the pillar disabled (`s.broker == nil`). Check + `docker compose logs fleet-manager | grep -i pillar`: a `DISABLED`/`WARNING` + line names the missing URL or cert file. After staging the broker PKI, the + container must be **recreated** (`docker compose up -d --force-recreate + fleet-manager`) — a plain restart won't re-enable a plane that soft-disabled at + boot. Also make sure only **one** manager instance is bound to `:9500` + (`docker ps | grep manager`); a stale parallel instance serves the old state. +- **`TLS connect error: wrong version number` to the broker** — the broker admin + plane is answering in **plaintext**: it's running with `ENROL_ADMIN_INSECURE=true` + (the main serving-stack default). Set it `false`, ensure the broker's + `server.crt`/`server.key`/`admin-ca.crt` exist in its PKI mount, and restart it + (see step 2b). + ## What is NOT yet wired (recorded intent) Per-host **boot-version assignment** (`desired.target_version`) is stored and diff --git a/serving/fleet-manager/Dockerfile b/serving/fleet-manager/Dockerfile index ec267d2..8f6cecb 100644 --- a/serving/fleet-manager/Dockerfile +++ b/serving/fleet-manager/Dockerfile @@ -8,7 +8,7 @@ # docker build -t fleet-manager serving/fleet-manager # ---- 1. build the SPA ------------------------------------------------------ -FROM node:24-alpine AS ui +FROM harbor.bitasylum.org/dockerhub/library/node:24-alpine AS ui WORKDIR /ui COPY ui/package.json ui/package-lock.json ./ RUN npm ci --no-audit --no-fund @@ -16,7 +16,7 @@ COPY ui/ ./ RUN npm run build # -> /ui/dist # ---- 2. build the Go binary (SPA embedded) -------------------------------- -FROM golang:1.26-alpine AS build +FROM harbor.bitasylum.org/dockerhub/library/golang:1.26-alpine AS build WORKDIR /src COPY go.mod go.sum ./ RUN go mod download @@ -25,11 +25,19 @@ COPY --from=ui /ui/dist ./ui/dist ENV CGO_ENABLED=0 GOTOOLCHAIN=local RUN go build -tags embedspa -trimpath -buildvcs=false \ -ldflags '-s -w -buildid=' -o /fleet-manager . +# Seed dir whose ownership a FRESH named volume inherits via Docker's copy-up, so +# the nonroot runtime (uid 65532) can write inventory/admins/enroll.key under +# /data without a manual chown. (An ALREADY-created root-owned volume must still +# be chowned or recreated once — see DEPLOY.md.) +RUN mkdir -p /data-seed && touch /data-seed/.keep # ---- 3. minimal runtime ---------------------------------------------------- # distroless/static: static binary + ca-certificates + tzdata, non-root by -# default. The data dir (inventory + admin store) is a mounted, writable volume. +# default (uid 65532). The data dir (inventory + admin store + enrol key) is a +# mounted, writable volume — pre-created here owned by nonroot so a fresh volume +# inherits that ownership. FROM gcr.io/distroless/static-debian12:nonroot COPY --from=build /fleet-manager /usr/local/bin/fleet-manager +COPY --from=build --chown=65532:65532 /data-seed /data USER nonroot:nonroot ENTRYPOINT ["/usr/local/bin/fleet-manager"]