fleet-manager: fix nonroot data-volume ownership + document broker wiring
The distroless nonroot runtime (uid 65532) couldn't write /data/inventory.json because a named volume is created root-owned — approve failed with permission denied on the inventory birth. Pre-create /data owned by nonroot in the image so a fresh volume inherits it via Docker copy-up; an already-created root-owned volume still needs a one-time chown (documented). DEPLOY.md: - 2b now covers BOTH broker deployments — the main serving stack (enrol-pki/broker, ENROL_ADMIN_INSECURE=true by default → must flip to false) and the bench stack — with the mint-once-into-broker-dir, copy-CA-to-manager flow and an mTLS verify curl. - Troubleshooting section: the data-volume permission-denied fix, the "authorise plane not configured" (recreate + single-instance) fix, and the "wrong version number" (broker plaintext) fix.
This commit is contained in:
@@ -57,12 +57,40 @@ AGENT_SANS=diskless-server.bitasylum.org,10.0.0.1 \
|
|||||||
|
|
||||||
The manager is an mTLS **client** of the enrol-broker's admin plane. Mint its
|
The manager is an mTLS **client** of the enrol-broker's admin plane. Mint its
|
||||||
client cert from the broker's **dedicated admin CA** (NOT the fleet CA — reusing
|
client cert from the broker's **dedicated admin CA** (NOT the fleet CA — reusing
|
||||||
the fleet CA would let any enrolled host authenticate to the admin plane):
|
the fleet CA would let any enrolled host authenticate to the admin plane). Mint
|
||||||
|
the whole set **once** into the broker's own PKI dir (so the broker gets its
|
||||||
|
`server.crt`/`server.key` too), then copy the CA + client cert to the manager so
|
||||||
|
**both sides share one admin CA**.
|
||||||
|
|
||||||
|
The broker's PKI dir + its "serve mTLS" switch differ by which stack runs it:
|
||||||
|
|
||||||
|
- **Main serving stack** (`serving/docker-compose.yml`) — broker mounts
|
||||||
|
`./enrol-pki:/pki`, and the admin plane is **plaintext by default**
|
||||||
|
(`ENROL_ADMIN_INSECURE=true` in `serving/.env`). Set it `false`, then:
|
||||||
|
```bash
|
||||||
|
cd .. # serving/
|
||||||
|
./enrol-broker/admin-pki.sh ./enrol-pki/broker fleet-manager
|
||||||
|
# -> enrol-pki/broker/{admin-ca.crt, server.crt, server.key, clients/fleet-manager.*}
|
||||||
|
docker compose up -d enrol-broker # restart so it serves mTLS (no cert error in logs)
|
||||||
|
mkdir -p fleet-manager/pki/broker
|
||||||
|
cp enrol-pki/broker/admin-ca.crt fleet-manager/pki/broker/
|
||||||
|
cp -r enrol-pki/broker/clients fleet-manager/pki/broker/
|
||||||
|
```
|
||||||
|
- **Bench stack** (`serving/dot1x-bench/`) — broker mounts `./pki:/pki` and is
|
||||||
|
mTLS-by-default (cert paths hardcoded in that compose):
|
||||||
```bash
|
```bash
|
||||||
../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager
|
../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager
|
||||||
cp -r ../dot1x-bench/pki/broker ./pki/broker
|
cp -r ../dot1x-bench/pki/broker ./pki/broker
|
||||||
# -> ./pki/broker/{admin-ca.crt, clients/fleet-manager.{crt,key}}
|
```
|
||||||
|
|
||||||
|
Either way the manager ends up with `./pki/broker/{admin-ca.crt,
|
||||||
|
clients/fleet-manager.{crt,key}}`. If you (re)generate a fresh admin CA, **re-copy**
|
||||||
|
it to the manager — a stale copy won't verify. Verify the mTLS end to end:
|
||||||
|
```bash
|
||||||
|
curl -s --cert ./pki/broker/clients/fleet-manager.crt \
|
||||||
|
--key ./pki/broker/clients/fleet-manager.key \
|
||||||
|
--cacert ./pki/broker/admin-ca.crt \
|
||||||
|
https://127.0.0.1:9443/pending ; echo # expect [] or null (empty queue)
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2c. Admin UI cert (Let's Encrypt DNS-01)
|
### 2c. Admin UI cert (Let's Encrypt DNS-01)
|
||||||
@@ -146,6 +174,31 @@ and the `influx-data` volume. Losing `admins.json` locks every admin out — you
|
|||||||
re-bootstrap with `register-admin`. Losing `enroll.key` invalidates any
|
re-bootstrap with `register-admin`. Losing `enroll.key` invalidates any
|
||||||
outstanding (unused) enrol links.
|
outstanding (unused) enrol links.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
- **`permission denied` writing the inventory on approve** — the runtime image is
|
||||||
|
distroless `nonroot` (uid **65532**), but a named volume is created root-owned,
|
||||||
|
so `/data` isn't writable. A rebuilt image pre-creates `/data` owned by nonroot
|
||||||
|
(a *fresh* volume inherits it), but an **already-created** volume must be fixed
|
||||||
|
once:
|
||||||
|
```bash
|
||||||
|
docker volume ls | grep fleet # find the name, e.g. fleet-manager_fleet-data
|
||||||
|
docker run --rm -v <vol>:/data alpine chown -R 65532:65532 /data
|
||||||
|
```
|
||||||
|
- **`authorise plane not configured (no enrol-broker wired)` in the SPA** — the
|
||||||
|
manager has the pillar disabled (`s.broker == nil`). Check
|
||||||
|
`docker compose logs fleet-manager | grep -i pillar`: a `DISABLED`/`WARNING`
|
||||||
|
line names the missing URL or cert file. After staging the broker PKI, the
|
||||||
|
container must be **recreated** (`docker compose up -d --force-recreate
|
||||||
|
fleet-manager`) — a plain restart won't re-enable a plane that soft-disabled at
|
||||||
|
boot. Also make sure only **one** manager instance is bound to `:9500`
|
||||||
|
(`docker ps | grep manager`); a stale parallel instance serves the old state.
|
||||||
|
- **`TLS connect error: wrong version number` to the broker** — the broker admin
|
||||||
|
plane is answering in **plaintext**: it's running with `ENROL_ADMIN_INSECURE=true`
|
||||||
|
(the main serving-stack default). Set it `false`, ensure the broker's
|
||||||
|
`server.crt`/`server.key`/`admin-ca.crt` exist in its PKI mount, and restart it
|
||||||
|
(see step 2b).
|
||||||
|
|
||||||
## What is NOT yet wired (recorded intent)
|
## What is NOT yet wired (recorded intent)
|
||||||
|
|
||||||
Per-host **boot-version assignment** (`desired.target_version`) is stored and
|
Per-host **boot-version assignment** (`desired.target_version`) is stored and
|
||||||
|
|||||||
@@ -8,7 +8,7 @@
|
|||||||
# docker build -t fleet-manager serving/fleet-manager
|
# docker build -t fleet-manager serving/fleet-manager
|
||||||
|
|
||||||
# ---- 1. build the SPA ------------------------------------------------------
|
# ---- 1. build the SPA ------------------------------------------------------
|
||||||
FROM node:24-alpine AS ui
|
FROM harbor.bitasylum.org/dockerhub/library/node:24-alpine AS ui
|
||||||
WORKDIR /ui
|
WORKDIR /ui
|
||||||
COPY ui/package.json ui/package-lock.json ./
|
COPY ui/package.json ui/package-lock.json ./
|
||||||
RUN npm ci --no-audit --no-fund
|
RUN npm ci --no-audit --no-fund
|
||||||
@@ -16,7 +16,7 @@ COPY ui/ ./
|
|||||||
RUN npm run build # -> /ui/dist
|
RUN npm run build # -> /ui/dist
|
||||||
|
|
||||||
# ---- 2. build the Go binary (SPA embedded) --------------------------------
|
# ---- 2. build the Go binary (SPA embedded) --------------------------------
|
||||||
FROM golang:1.26-alpine AS build
|
FROM harbor.bitasylum.org/dockerhub/library/golang:1.26-alpine AS build
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN go mod download
|
RUN go mod download
|
||||||
@@ -25,11 +25,19 @@ COPY --from=ui /ui/dist ./ui/dist
|
|||||||
ENV CGO_ENABLED=0 GOTOOLCHAIN=local
|
ENV CGO_ENABLED=0 GOTOOLCHAIN=local
|
||||||
RUN go build -tags embedspa -trimpath -buildvcs=false \
|
RUN go build -tags embedspa -trimpath -buildvcs=false \
|
||||||
-ldflags '-s -w -buildid=' -o /fleet-manager .
|
-ldflags '-s -w -buildid=' -o /fleet-manager .
|
||||||
|
# Seed dir whose ownership a FRESH named volume inherits via Docker's copy-up, so
|
||||||
|
# the nonroot runtime (uid 65532) can write inventory/admins/enroll.key under
|
||||||
|
# /data without a manual chown. (An ALREADY-created root-owned volume must still
|
||||||
|
# be chowned or recreated once — see DEPLOY.md.)
|
||||||
|
RUN mkdir -p /data-seed && touch /data-seed/.keep
|
||||||
|
|
||||||
# ---- 3. minimal runtime ----------------------------------------------------
|
# ---- 3. minimal runtime ----------------------------------------------------
|
||||||
# distroless/static: static binary + ca-certificates + tzdata, non-root by
|
# distroless/static: static binary + ca-certificates + tzdata, non-root by
|
||||||
# default. The data dir (inventory + admin store) is a mounted, writable volume.
|
# default (uid 65532). The data dir (inventory + admin store + enrol key) is a
|
||||||
|
# mounted, writable volume — pre-created here owned by nonroot so a fresh volume
|
||||||
|
# inherits that ownership.
|
||||||
FROM gcr.io/distroless/static-debian12:nonroot
|
FROM gcr.io/distroless/static-debian12:nonroot
|
||||||
COPY --from=build /fleet-manager /usr/local/bin/fleet-manager
|
COPY --from=build /fleet-manager /usr/local/bin/fleet-manager
|
||||||
|
COPY --from=build --chown=65532:65532 /data-seed /data
|
||||||
USER nonroot:nonroot
|
USER nonroot:nonroot
|
||||||
ENTRYPOINT ["/usr/local/bin/fleet-manager"]
|
ENTRYPOINT ["/usr/local/bin/fleet-manager"]
|
||||||
|
|||||||
Reference in New Issue
Block a user