The steady-state host<->manager transport, resolving the design's OPEN FORK to
bespoke gRPC (not OTLP+Wazuh). One long-lived, host-dialed, mTLS bidi stream:
health/metrics up, desired-state/commands down — no polling, real-time control,
still no inbound host port.
- proto/fleet_agent.proto: FleetAgent.Session(stream AgentUp) -> stream ManagerDown.
AgentUp = Hello | HealthSnapshot | CommandResult; ManagerDown = DesiredState |
Command(reboot/drain/resync). int64 unix times (no well-known-type dep).
Generated stubs committed (CI needs no protoc).
- agent.go: gRPC server with mTLS verified by the FLEET CA (the host's reused TPM
identity cert — distinct from the admin plane's admin CA). Identity is the peer
cert SERIAL, taken from the verified cert = inventory key, never self-asserted;
an unenrolled serial is refused (enrolment gates the stream). Per-conn send
channel decouples admin pushes from stream.Send. Stream registry (serial->conn)
with reconnect eviction. On connect + on change, desired-state is pushed live;
inventory stays authoritative so offline hosts reconcile on reconnect. Metrics
go through a MetricsSink seam (no-op now; InfluxDB in ckpt 4).
- main.go: agent plane starts only when its cert/key/CA are set; admin can run
standalone. Lifecycle PUT now live-pushes and reports pushed_live.
Full mTLS integration test (in-process fleet CA + host client cert): initial
desired-state on connect, Hello+snapshot -> inventory + online presence, live
desired-state push, and unenrolled-serial rejection. go build/test/vet/gofmt clean.
Deps (server-side only): google.golang.org/grpc + protobuf.
The checkpoint-1 commit included the 9.1M compiled 'fleet-manager' binary. Remove
it from tracking and gitignore it (/fleet-manager + *.test), matching the
enrol-broker convention. Build artifacts don't belong in git.
First slice of the server-side fleet control plane (serving/fleet-manager/), the
backend for the admin SPA (monitor / authorise / lifecycle). See
docs/fleet-management-design.md + tpm-enrolment-design.md §5.
This checkpoint (stdlib-only):
- Inventory record (identity/binding/enrolled/version/health/usage/desired),
keyed by TPM cert serial = hostnqn key, per the design's 'inventory record born
at enrolment'. JSON-file Store (atomic writes, mutex) behind an interface so a
DB swap is a drop-in.
- Admin plane: mTLS server verified by the DEDICATED admin CA (never the fleet CA,
so a host cert can't reach admin) + a plaintext dev mode. JSON API:
GET /api/hosts, GET /api/hosts/{serial}, PUT /api/hosts/{serial}/desired
(desired-state audited to the admin client-cert CN).
- Dev seed (FLEET_MGR_SEED) so the SPA has data before real enrolment/agent flow.
Smoke-tested: list/get/desired-PUT (incl. version-drift + drain), 404s, atomic
persistence. go build + go vet + gofmt clean.
Agent gRPC plane, broker-fold-in authorise, InfluxDB metrics, and the themed SPA
follow in checkpoints 2-6.
The Dell 820 kept "finding a cert" and bypassing enrolment despite repeated BIOS
TPM clears. tpm2_nvreadpublic revealed 0x01800001 on that hardware is a FIRMWARE
platform index (platformcreate|policydelete|writelocked, 70 bytes — not our
cert; siblings at 0x01800003/4 + 0x01810000). platformcreate survives TPM2_Clear
and policydelete needs the platform's secret policy, so it's permanent and
unusable by us — the blind check false-positived on it, and enrolment could
never write there anyway. Same class of bug as the EK-handle collision.
Move the NV cert index default 0x01800001 -> 0x01800100 (free on the 820, owner
range, overridable via rd.dot1x.nvindex): cmdline (extract-artifacts.sh),
fleet-enrol -nvindex, the dracut hooks, bench scripts, CLAUDE.md invariant #1,
and design/SSP docs. The signing-key handle (0x81000100) is unchanged.
Harden WriteNVCert: refuse a platformcreate index with a clear "pick a free
owner index via rd.dot1x.nvindex" error instead of a confusing undefine
failure. Regression simtest/TestWriteNVCertRefusesPlatformIndex; NV roundtrip +
ReadNVCert tests moved to 0x01800100. All go tests + packer validate pass.
The nvmeof (spdk-target) service carried a versioned bootstrap mount
(type: image, source ${OCI_IMAGE}:${IMAGE_VERSION} -> /artifacts) plus an
IMAGE_VERSION env. Bumping IMAGE_VERSION therefore changed the service config,
so `docker compose up -d` recreated the container -> nvmf_tgt restarted ->
every connected host's NVMe/TCP root dropped at once. That is precisely the
fleet-wide outage the long-lived-target design exists to prevent.
Remove the /artifacts image-volume and IMAGE_VERSION env from nvmeof so its
config no longer depends on the version — compose leaves it running across a
bump. Bootstrap and rolling updates are already fully covered by spdk-loader
(stages the current release into ./spdk/releases and adds it online via
JSON-RPC); spdk-target.sh's seed block is guarded, so it just no-ops when
/artifacts isn't mounted. Verified nvmeof no longer references IMAGE_VERSION
(only spdk-loader does); compose YAML validates.
BENCH-VERIFY: bump IMAGE_VERSION + up -d with a host connected -> nvmeof
container not recreated, host root survives, loader adds the new version.
Bench (Proxmox vTPM, host cygnus-cruiser-99) confirmed via tpm2_readpublic that
0x81010001 — the handle our design used for the fleet signing key — is the
TCG-standard RSA Endorsement Key slot: the object there is the swtpm-provisioned
EK (restricted|decrypt|adminwithpolicy, RSA-2048/AES128CFB, canonical EK
authPolicy 837197…, EK cert at NV 0x1c00002). Most firmware/swtpm TPMs occupy
that slot, so EnsureKey could never create our key there — it errored reading
the EK as ECC and enrolment died before the CSR/POST.
Move the fleet signing key to 0x81000100, a non-reserved owner persistent
handle (clear of EK 0x81010001/2 and SRK 0x81000001/2). Threaded through:
- fleet-enrol -handle default + new rd.dot1x.handle= cmdline knob (baked by
extract-artifacts.sh, overridable via SMBIOS)
- files/etc/dot1x/wpa_supplicant-dot1x.conf private_key handle
- the dracut hook, bench tpm-provision.sh / tpm-selftest.sh
- CLAUDE.md invariant #1 + design/SSP docs + TODO #11
NV index 0x01800001 (owner range) is unchanged.
EnsureKey hardening: refuse to evict any restricted-decrypt object (EK/SRK) as
a safety net — never clobber the TPM endorsement/storage identity (also
protects the future EK/AK attestation path). Still self-heals a genuine stale
signing key. New regression TestEnsureKeyRefusesToEvictProtectedKey; the
stale-key regression and handle constants moved to 0x81000100.
Design note records the IDevID (0x81020001) option for the later EK/AK
attestation work. Follow-up items logged: content-aware NV presence check
(verify the cert chains to the fleet CA, so leftover/foreign NV can't suppress
enrolment) and a de-enrol/TPM-reset procedure.
The `step` CLI ran against --ca-url https://ca.fleet.lab:8443, but that name is
only served by the fleet dnsmasq to booting clients — the serving host's own
resolver can't resolve it ("no such host on 127.0.0.53"), so provisioner list +
RADIUS cert issuance failed. Connect over CA_CONNECT (default localhost, which
the compose puts in the CA cert SAN); CA_DNS stays the cert/SAN + RADIUS CN name.
Verifying the step signer against a real step-ca surfaced two real bugs the
mock signer had hidden, plus made the bench CA actually initialise:
- signer.go: move --not-after from `step ca token` (validity caps ~1h ->
"token validity too long") to `step ca sign` (it is the CERT lifetime).
Without this the step signer never issues a cert.
- dot1x-bench compose: DOCKER_STEPCA_INIT_PASSWORD_FILE pointed at a file that
does not exist at first init (it is an INPUT path), so `step ca init` failed
and never wrote secrets/password. Use DOCKER_STEPCA_INIT_PASSWORD (the value)
which the entrypoint writes to secrets/password, where bootstrap + broker read.
- bootstrap.sh: self-heal the root-owned ./pki Docker auto-creates; detect the
CA password at $STEPPATH/password or secrets/password (varies by image
version); raise authority.claims.maxTLSCertDuration so 720h fleet certs are
allowed (stock cap 24h, which also blocked bootstrap's own RADIUS cert); fix
RADIUS cert paths to write via the /export mount (were host paths run inside
the container, so they never resolved).
- serving stack: add the enrol-broker service (temporary home; nginx already
proxies /enrol/) defaulting to the zero-dependency lab path (mock signer +
insecure loopback admin), with ENROL_* knobs to switch to step + mTLS.
- verify-serving.sh: exercise request -> approve -> cert through the deployed
nginx /enrol/ proxy.
- TODO #11: record the step-signer verification + the two fixes.
Step signer VERIFIED end-to-end against a real step-ca: issued cert chains to
the fleet root, EKU clientAuth, hostnqn derived from serial.
Audiobuzz