fleet-manager: fix nonroot data-volume ownership + document broker wiring
The distroless nonroot runtime (uid 65532) couldn't write /data/inventory.json because a named volume is created root-owned — approve failed with permission denied on the inventory birth. Pre-create /data owned by nonroot in the image so a fresh volume inherits it via Docker copy-up; an already-created root-owned volume still needs a one-time chown (documented). DEPLOY.md: - 2b now covers BOTH broker deployments — the main serving stack (enrol-pki/broker, ENROL_ADMIN_INSECURE=true by default → must flip to false) and the bench stack — with the mint-once-into-broker-dir, copy-CA-to-manager flow and an mTLS verify curl. - Troubleshooting section: the data-volume permission-denied fix, the "authorise plane not configured" (recreate + single-instance) fix, and the "wrong version number" (broker plaintext) fix.
This commit is contained in:
Audiobuzz
@@ -57,12 +57,40 @@ AGENT_SANS=diskless-server.bitasylum.org,10.0.0.1 \
|
||||
|
||||
The manager is an mTLS **client** of the enrol-broker's admin plane. Mint its
|
||||
client cert from the broker's **dedicated admin CA** (NOT the fleet CA — reusing
|
||||
the fleet CA would let any enrolled host authenticate to the admin plane):
|
||||
the fleet CA would let any enrolled host authenticate to the admin plane). Mint
|
||||
the whole set **once** into the broker's own PKI dir (so the broker gets its
|
||||
`server.crt`/`server.key` too), then copy the CA + client cert to the manager so
|
||||
**both sides share one admin CA**.
|
||||
|
||||
The broker's PKI dir + its "serve mTLS" switch differ by which stack runs it:
|
||||
|
||||
- **Main serving stack** (`serving/docker-compose.yml`) — broker mounts
|
||||
`./enrol-pki:/pki`, and the admin plane is **plaintext by default**
|
||||
(`ENROL_ADMIN_INSECURE=true` in `serving/.env`). Set it `false`, then:
|
||||
```bash
|
||||
cd .. # serving/
|
||||
./enrol-broker/admin-pki.sh ./enrol-pki/broker fleet-manager
|
||||
# -> enrol-pki/broker/{admin-ca.crt, server.crt, server.key, clients/fleet-manager.*}
|
||||
docker compose up -d enrol-broker # restart so it serves mTLS (no cert error in logs)
|
||||
mkdir -p fleet-manager/pki/broker
|
||||
cp enrol-pki/broker/admin-ca.crt fleet-manager/pki/broker/
|
||||
cp -r enrol-pki/broker/clients fleet-manager/pki/broker/
|
||||
```
|
||||
- **Bench stack** (`serving/dot1x-bench/`) — broker mounts `./pki:/pki` and is
|
||||
mTLS-by-default (cert paths hardcoded in that compose):
|
||||
```bash
|
||||
../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager
|
||||
cp -r ../dot1x-bench/pki/broker ./pki/broker
|
||||
```
|
||||
|
||||
Either way the manager ends up with `./pki/broker/{admin-ca.crt,
|
||||
clients/fleet-manager.{crt,key}}`. If you (re)generate a fresh admin CA, **re-copy**
|
||||
it to the manager — a stale copy won't verify. Verify the mTLS end to end:
|
||||
```bash
|
||||
../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager
|
||||
cp -r ../dot1x-bench/pki/broker ./pki/broker
|
||||
# -> ./pki/broker/{admin-ca.crt, clients/fleet-manager.{crt,key}}
|
||||
curl -s --cert ./pki/broker/clients/fleet-manager.crt \
|
||||
--key ./pki/broker/clients/fleet-manager.key \
|
||||
--cacert ./pki/broker/admin-ca.crt \
|
||||
https://127.0.0.1:9443/pending ; echo # expect [] or null (empty queue)
|
||||
```
|
||||
|
||||
### 2c. Admin UI cert (Let's Encrypt DNS-01)
|
||||
@@ -146,6 +174,31 @@ and the `influx-data` volume. Losing `admins.json` locks every admin out — you
|
||||
re-bootstrap with `register-admin`. Losing `enroll.key` invalidates any
|
||||
outstanding (unused) enrol links.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **`permission denied` writing the inventory on approve** — the runtime image is
|
||||
distroless `nonroot` (uid **65532**), but a named volume is created root-owned,
|
||||
so `/data` isn't writable. A rebuilt image pre-creates `/data` owned by nonroot
|
||||
(a *fresh* volume inherits it), but an **already-created** volume must be fixed
|
||||
once:
|
||||
```bash
|
||||
docker volume ls | grep fleet # find the name, e.g. fleet-manager_fleet-data
|
||||
docker run --rm -v <vol>:/data alpine chown -R 65532:65532 /data
|
||||
```
|
||||
- **`authorise plane not configured (no enrol-broker wired)` in the SPA** — the
|
||||
manager has the pillar disabled (`s.broker == nil`). Check
|
||||
`docker compose logs fleet-manager | grep -i pillar`: a `DISABLED`/`WARNING`
|
||||
line names the missing URL or cert file. After staging the broker PKI, the
|
||||
container must be **recreated** (`docker compose up -d --force-recreate
|
||||
fleet-manager`) — a plain restart won't re-enable a plane that soft-disabled at
|
||||
boot. Also make sure only **one** manager instance is bound to `:9500`
|
||||
(`docker ps | grep manager`); a stale parallel instance serves the old state.
|
||||
- **`TLS connect error: wrong version number` to the broker** — the broker admin
|
||||
plane is answering in **plaintext**: it's running with `ENROL_ADMIN_INSECURE=true`
|
||||
(the main serving-stack default). Set it `false`, ensure the broker's
|
||||
`server.crt`/`server.key`/`admin-ca.crt` exist in its PKI mount, and restart it
|
||||
(see step 2b).
|
||||
|
||||
## What is NOT yet wired (recorded intent)
|
||||
|
||||
Per-host **boot-version assignment** (`desired.target_version`) is stored and
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
# docker build -t fleet-manager serving/fleet-manager
|
||||
|
||||
# ---- 1. build the SPA ------------------------------------------------------
|
||||
FROM node:24-alpine AS ui
|
||||
FROM harbor.bitasylum.org/dockerhub/library/node:24-alpine AS ui
|
||||
WORKDIR /ui
|
||||
COPY ui/package.json ui/package-lock.json ./
|
||||
RUN npm ci --no-audit --no-fund
|
||||
@@ -16,7 +16,7 @@ COPY ui/ ./
|
||||
RUN npm run build # -> /ui/dist
|
||||
|
||||
# ---- 2. build the Go binary (SPA embedded) --------------------------------
|
||||
FROM golang:1.26-alpine AS build
|
||||
FROM harbor.bitasylum.org/dockerhub/library/golang:1.26-alpine AS build
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
@@ -25,11 +25,19 @@ COPY --from=ui /ui/dist ./ui/dist
|
||||
ENV CGO_ENABLED=0 GOTOOLCHAIN=local
|
||||
RUN go build -tags embedspa -trimpath -buildvcs=false \
|
||||
-ldflags '-s -w -buildid=' -o /fleet-manager .
|
||||
# Seed dir whose ownership a FRESH named volume inherits via Docker's copy-up, so
|
||||
# the nonroot runtime (uid 65532) can write inventory/admins/enroll.key under
|
||||
# /data without a manual chown. (An ALREADY-created root-owned volume must still
|
||||
# be chowned or recreated once — see DEPLOY.md.)
|
||||
RUN mkdir -p /data-seed && touch /data-seed/.keep
|
||||
|
||||
# ---- 3. minimal runtime ----------------------------------------------------
|
||||
# distroless/static: static binary + ca-certificates + tzdata, non-root by
|
||||
# default. The data dir (inventory + admin store) is a mounted, writable volume.
|
||||
# default (uid 65532). The data dir (inventory + admin store + enrol key) is a
|
||||
# mounted, writable volume — pre-created here owned by nonroot so a fresh volume
|
||||
# inherits that ownership.
|
||||
FROM gcr.io/distroless/static-debian12:nonroot
|
||||
COPY --from=build /fleet-manager /usr/local/bin/fleet-manager
|
||||
COPY --from=build --chown=65532:65532 /data-seed /data
|
||||
USER nonroot:nonroot
|
||||
ENTRYPOINT ["/usr/local/bin/fleet-manager"]
|
||||
|
||||
Reference in New Issue
Block a user