fleet-manager: fix nonroot data-volume ownership + document broker wiring

The distroless nonroot runtime (uid 65532) couldn't write /data/inventory.json
because a named volume is created root-owned — approve failed with permission
denied on the inventory birth. Pre-create /data owned by nonroot in the image so
a fresh volume inherits it via Docker copy-up; an already-created root-owned
volume still needs a one-time chown (documented).

DEPLOY.md:
- 2b now covers BOTH broker deployments — the main serving stack
  (enrol-pki/broker, ENROL_ADMIN_INSECURE=true by default → must flip to false)
  and the bench stack — with the mint-once-into-broker-dir, copy-CA-to-manager
  flow and an mTLS verify curl.
- Troubleshooting section: the data-volume permission-denied fix, the
  "authorise plane not configured" (recreate + single-instance) fix, and the
  "wrong version number" (broker plaintext) fix.
This commit is contained in:
Audiobuzz
2026-07-04 09:56:09 +00:00
parent 014f43cb5a
commit 0360c1c6d1
2 changed files with 68 additions and 7 deletions
+57 -4
View File
@@ -57,12 +57,40 @@ AGENT_SANS=diskless-server.bitasylum.org,10.0.0.1 \
The manager is an mTLS **client** of the enrol-broker's admin plane. Mint its
client cert from the broker's **dedicated admin CA** (NOT the fleet CA — reusing
the fleet CA would let any enrolled host authenticate to the admin plane):
the fleet CA would let any enrolled host authenticate to the admin plane). Mint
the whole set **once** into the broker's own PKI dir (so the broker gets its
`server.crt`/`server.key` too), then copy the CA + client cert to the manager so
**both sides share one admin CA**.
The broker's PKI dir + its "serve mTLS" switch differ by which stack runs it:
- **Main serving stack** (`serving/docker-compose.yml`) — broker mounts
`./enrol-pki:/pki`, and the admin plane is **plaintext by default**
(`ENROL_ADMIN_INSECURE=true` in `serving/.env`). Set it `false`, then:
```bash
cd .. # serving/
./enrol-broker/admin-pki.sh ./enrol-pki/broker fleet-manager
# -> enrol-pki/broker/{admin-ca.crt, server.crt, server.key, clients/fleet-manager.*}
docker compose up -d enrol-broker # restart so it serves mTLS (no cert error in logs)
mkdir -p fleet-manager/pki/broker
cp enrol-pki/broker/admin-ca.crt fleet-manager/pki/broker/
cp -r enrol-pki/broker/clients fleet-manager/pki/broker/
```
- **Bench stack** (`serving/dot1x-bench/`) — broker mounts `./pki:/pki` and is
mTLS-by-default (cert paths hardcoded in that compose):
```bash
../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager
cp -r ../dot1x-bench/pki/broker ./pki/broker
```
Either way the manager ends up with `./pki/broker/{admin-ca.crt,
clients/fleet-manager.{crt,key}}`. If you (re)generate a fresh admin CA, **re-copy**
it to the manager — a stale copy won't verify. Verify the mTLS end to end:
```bash
../enrol-broker/admin-pki.sh ../dot1x-bench/pki/broker fleet-manager
cp -r ../dot1x-bench/pki/broker ./pki/broker
# -> ./pki/broker/{admin-ca.crt, clients/fleet-manager.{crt,key}}
curl -s --cert ./pki/broker/clients/fleet-manager.crt \
--key ./pki/broker/clients/fleet-manager.key \
--cacert ./pki/broker/admin-ca.crt \
https://127.0.0.1:9443/pending ; echo # expect [] or null (empty queue)
```
### 2c. Admin UI cert (Let's Encrypt DNS-01)
@@ -146,6 +174,31 @@ and the `influx-data` volume. Losing `admins.json` locks every admin out — you
re-bootstrap with `register-admin`. Losing `enroll.key` invalidates any
outstanding (unused) enrol links.
## Troubleshooting
- **`permission denied` writing the inventory on approve** — the runtime image is
distroless `nonroot` (uid **65532**), but a named volume is created root-owned,
so `/data` isn't writable. A rebuilt image pre-creates `/data` owned by nonroot
(a *fresh* volume inherits it), but an **already-created** volume must be fixed
once:
```bash
docker volume ls | grep fleet # find the name, e.g. fleet-manager_fleet-data
docker run --rm -v <vol>:/data alpine chown -R 65532:65532 /data
```
- **`authorise plane not configured (no enrol-broker wired)` in the SPA** — the
manager has the pillar disabled (`s.broker == nil`). Check
`docker compose logs fleet-manager | grep -i pillar`: a `DISABLED`/`WARNING`
line names the missing URL or cert file. After staging the broker PKI, the
container must be **recreated** (`docker compose up -d --force-recreate
fleet-manager`) — a plain restart won't re-enable a plane that soft-disabled at
boot. Also make sure only **one** manager instance is bound to `:9500`
(`docker ps | grep manager`); a stale parallel instance serves the old state.
- **`TLS connect error: wrong version number` to the broker** — the broker admin
plane is answering in **plaintext**: it's running with `ENROL_ADMIN_INSECURE=true`
(the main serving-stack default). Set it `false`, ensure the broker's
`server.crt`/`server.key`/`admin-ca.crt` exist in its PKI mount, and restart it
(see step 2b).
## What is NOT yet wired (recorded intent)
Per-host **boot-version assignment** (`desired.target_version`) is stored and