Split-horizon DNS made lego's propagation check query the container's internal
resolver, which doesn't serve the public _acme-challenge TXT record, so the
challenge timed out. Add ACME_RESOLVERS (comma-separated host:port, default
8.8.8.8:53) → one --dns.resolvers flag each. Documented in .env + DEPLOY.md.
The goacme/lego image ships the binary at /lego, not on PATH, so the bare `lego`
call failed ("not found") once the entrypoint was overridden with the shell.
Detect it (command -v, then /lego fallback) and invoke via $LEGO.
lego auto-discovers the Route 53 zone via ListHostedZonesByName, so the zone ID
is optional — but pinning it disambiguates overlapping public/private zones and
lets the IAM policy drop the list permission. Thread it through compose + .env
+ DEPLOY.md (no script change; lego reads it from the environment).
Auto-issue + auto-renew the admin-UI TLS cert with a long-running compose
side-car instead of a host cron. The `acme` service (goacme/lego image, running
acme/renew-loop.sh) obtains the cert over DNS-01 via the AWS Route 53 API,
writes fullchain/privkey into ./pki/admin/ — the SAME dir the manager reads
(./pki:/pki:ro) — and chowns them to the nonroot runtime uid so the read-only
container can load the key. The manager's certReloader hot-swaps on the mtime
bump, no restart. Loop renews within 30 days of expiry; ACME_STAGING=1 gives a
dry-run vs LE staging.
- docker-compose.yml: `acme` service + `acme-state` volume (lego account/cert
store). ACME_DOMAIN reuses FLEET_MGR_RP_ID so cert host == passkey origin.
- .env.example: ACME_EMAIL + AWS creds (or instance role) + ACME_STAGING.
- DEPLOY.md §2c: rewritten to the side-car flow — scoped Route 53 IAM policy,
bring-acme-up-first ordering, flip off insecure, and an origin-match note.
(The docker-compose.yml commit also carries the local harbor.bitasylum.org
influxdb image mirror edit.)
A just-approved inventory record has a zero-value last_seen (Go 0001-01-01),
which parsed to a pre-epoch instant and slipped past the !t guard, rendering as
"739800d ago". Treat any at-or-before-epoch timestamp as "never" — the honest
state for a born-but-never-connected host (its health/version stay "—" until an
agent dials the gRPC plane).
The distroless nonroot runtime (uid 65532) couldn't write /data/inventory.json
because a named volume is created root-owned — approve failed with permission
denied on the inventory birth. Pre-create /data owned by nonroot in the image so
a fresh volume inherits it via Docker copy-up; an already-created root-owned
volume still needs a one-time chown (documented).
DEPLOY.md:
- 2b now covers BOTH broker deployments — the main serving stack
(enrol-pki/broker, ENROL_ADMIN_INSECURE=true by default → must flip to false)
and the bench stack — with the mint-once-into-broker-dir, copy-CA-to-manager
flow and an mTLS verify curl.
- Troubleshooting section: the data-volume permission-denied fix, the
"authorise plane not configured" (recreate + single-instance) fix, and the
"wrong version number" (broker plaintext) fix.
Every plane logged its state EXCEPT the broker success path, which was silent —
so "authorise pillar working" was indistinguishable from "not logged". Add an
ENABLED line matching the InfluxDB/agent/admin convention.
The compose wires the authorise (broker) and agent planes to fixed cert paths,
so a first bring-up that hasn't run the PKI steps yet crash-looped on the first
missing file (e.g. /pki/broker/clients/fleet-manager.crt) instead of booting.
Treat a configured-but-absent cert as "plane not yet provisioned": log a loud
WARNING and disable that plane, so the manager boots and each plane lights up
as its PKI appears. A malformed/unreadable cert still hard-fails (real
misconfig). The admin plane stays a hard requirement (FLEET_MGR_ADMIN_INSECURE
is its bench escape). Adds a missingFiles() helper; hub push path already
nil-guards a disabled agent plane.
Complete the deployment story for the server-side control plane:
- docker-compose.yml: private InfluxDB (published to loopback only) + a
host-networked fleet-manager (reaches the loopback enrol-broker on
127.0.0.1:9443 and binds a routable gRPC agent listener, like the broker).
- .env.example: RP origin (diskless-server.bitasylum.org), Influx token/DB
password, cert/PKI paths; the Influx token wires influxdb + manager with
one value.
- server-pki.sh: issue the agent-plane server cert from the fleet CA (step)
and stage the fleet root as the client-verify CA.
- DEPLOY.md: end-to-end runbook — three-cert PKI (fleet-CA agent cert, broker
admin-client cert, Let's Encrypt DNS-01 UI cert), register-admin passkey
bootstrap, compose up, hardware-free smoke test, backups.
- .dockerignore: keep host node_modules/dist/pki/secrets out of the build
context so the embedded SPA is the one built in-image.
- .gitignore: never commit .env / issued PKI / data.
Design docs marked as-built: fleet-management-design.md as-built note
(resolved forks — gRPC bidi transport, InfluxDB metrics, SPA graphs, passkey
admin auth, broker fold-in "alongside"); tpm-enrolment-design.md (broker
fold-in + agent transport fork resolved); TODO.md #14 → partly built, with
the on-host agent + per-host boot-version assignment (#6) called out as the
remaining work.
Two-stage build: node builds the Svelte SPA to dist, then golang compiles the
binary with the SPA embedded (-tags embedspa). Committed .pb.go stubs mean no
protoc. Ships as a small static distroless/nonroot image; same binary serves the
register-admin subcommand. (Not build-verified here — no docker in the
devcontainer — but the go build -tags embedspa + npm build are both proven.)
Per the option-A decision (deploy current design without per-host versioning):
add an honest note on the Lifecycle panel that kiosk-url/reboot/drain reconcile
via the agent but per-host boot-version assignment (dnsmasq cohort steering) is
not yet wired, so a version pin is stored + pushed but does not change what the
host boots. Avoids implying the pin takes effect.
Replace the free-text target-version field with a select populated from versions
known across the fleet (each host's running + targeted version, newest first),
plus a 'custom...' escape to a text input for a version not yet seen. Avoids
typos on the pin. (Authoritative serving-side source is a follow-up.)
- Contrast: lighten the muted/faint text tokens (AA on the hull panels) and make
Monitor host-card data values bright ink against dim faint labels, so version/
tmpfs/seen details read clearly on the blue-grey panels (was grey-on-grey).
- Lifecycle: reboot selector now offers none | now | at... — 'now' sets
reboot_at to the current time (agent reboots on reconcile); still confirm-gated
as a high-consequence action.
The themed single-page admin UI (serving/fleet-manager/ui/), embedded into the
binary and served same-origin over the admin plane.
- Svelte 5 + Tailwind 4 + Vite; a sibling of serving/kiosk-app. Theme = LYSANDRA
'restricted-violet' (high-security evolution of the kiosk's flight-deck DNA):
amber deck, VIOLET reserved for privileged/destructive + auth surfaces, teal
telemetry, red alert, clearance chrome (corner-reticle panels, scanlines, mono
clearance labels, secure-link status bar).
- Passkey gate: Login (usernameless discoverable WebAuthn) + Enrol (from the
register-admin one-time link) via navigator.credentials, with base64url<->buffer
helpers and the go-webauthn response shapes.
- Three pillars over the JSON API: Monitor (fleet roster + version-drift + health
+ per-host detail with bespoke SVG telemetry charts), Authorise (pending queue:
fingerprint cross-check, approve+bind in violet, deny), Lifecycle (desired-state
form; drain/reboot confirm-gated). Session-aware fetch client (cookie + CSRF).
- go:embed behind the build tag (spa_embed.go / spa_stub.go) so the
default go build/test need no dist/; mounted at / (api/auth/healthz win).
Verified: npm build clean (71KB JS / 33KB CSS + self-hosted fonts); default +
-tags embedspa go builds; smoke test serves index + assets + API through the SPA
origin. Against an insecure+seeded backend the deck renders passkey-free for
theme work. dist/ + node_modules gitignored.
The human->UI hop now authenticates with a hardware passkey (YubiKey), not an
mTLS client cert: phishing-resistant, origin-bound, better UX. (manager->broker
stays mTLS — service-to-service.)
- webauthn.go: credential store (registered admins + passkeys + a one-time-token
jti replay ledger; JSON file, atomic writes) + the webauthn.User impl + RP setup.
- cli.go: 'fleet-manager register-admin <name>' mints a short-lived HMAC-signed
enrolment link over a shared enrol key; the browser completes the WebAuthn
create ceremony. No anyone-can-register endpoint; single-use via the ledger.
- authapi.go: register/login ceremonies (usernameless discoverable login,
userVerification required) + logout + whoami; short-lived in-memory ceremony
store between begin/finish.
- session.go: opaque server-side sessions (Secure/HttpOnly/SameSite=Strict cookie)
+ requireSession middleware gating /api/*, with a CSRF header check on mutations.
adminName now comes from the session (audit attribution).
- main.go: register-admin subcommand dispatch; admin TLS switched to SERVER-AUTH
(no client cert) with a HOT-RELOADING cert (certReloader) so a Let's Encrypt
DNS-01 renewal is picked up without a restart. RP = diskless-server.bitasylum.org.
/api/* mounted behind requireSession; /auth/* + /healthz public.
Cert strategy (per design decision): the UI HTTPS cert is a publicly-trusted LE
cert via DNS-01 (mgmt-network UI isn't internet-reachable, so HTTP-01/TLS-ALPN-01
can't validate; DNS-01 proves control of bitasylum.org via TXT + supports
wildcards). External ACME client issues+renews to the cert path; manager reloads.
Deps: github.com/go-webauthn/webauthn (server-side). Tests: enrol-token
mint/verify/tamper/expiry, session middleware (401/403/CSRF/expiry), cred-store
persistence + jti ledger. build/test/vet/gofmt clean. Live passkey ceremony
(virtual authenticator / real YubiKey) is a bench item.
The manager is the ONLY thing that talks to InfluxDB, both directions; hosts hold
no token and never reach it.
- influx.go: influxSink implements MetricsSink — each agent HealthSnapshot is
written as a fleet_health point, identity (serial/hostnqn/starship) as TAGS
from the AUTHENTICATED stream (a host can only write its own series), telemetry
as fields. QueryField answers the SPA's read: a Flux query built with a strict
FIELD WHITELIST + hex-serial guard + parsed-duration range, so no caller input
is interpolated raw (injection-safe). Server-side influxdb-client-go/v2.
- admin.go: GET /api/hosts/{serial}/metrics?field=&range= -> time-series JSON;
503 when InfluxDB isn't wired, 400 on a bad range, 404 for unknown host.
- main.go: sink created when FLEET_MGR_INFLUX_URL+_TOKEN set, fed to the agent
hub; nil -> no-op sink so the manager runs without metrics.
Tests: snapshotPoint tags/fields/time, fluxQuery contents + %q quoting, and
QueryField rejects a non-hex serial + non-whitelisted field. build/test/vet/gofmt
clean. Live InfluxDB path validated in the compose bench (ckpt 6).
The AUTHORISE pillar: the manager owns the enrolment queue UX, the enrol-broker
stays the state machine + step-ca signer (v1 'alongside' fold-in).
enrol-broker (minimal, backward-compatible change):
- admin approve/deny now return the decided Enrolment as JSON when the caller
sends Accept: application/json. Browsers still redirect; other API callers
(verify-serving.sh) still get 204. The cert PEM stays unexported (only the
requesting host gets it via /status) — the JSON carries cert_serial, hostnqn,
context, fingerprint, cn.
fleet-manager:
- broker.go: mTLS client for the broker admin plane (Pending/Approve/Deny),
nil when unconfigured so the manager runs standalone.
- admin.go: GET /api/enrol/pending (proxy), POST /api/enrol/{id}/approve
(drives broker, then BIRTHS the inventory record — identity from the returned
serial/hostnqn/context, binding from the operator's optional body, enrolled
with the HUMAN approver = the manager's admin mTLS CN), POST .../deny.
503 when the broker isn't wired.
Test: httptest mock broker asserts the Accept:application/json approve, pending
proxy, inventory birth (identity+binding+enrolled), and the disabled-503 path.
Broker + manager both build/vet/test/gofmt clean.
The steady-state host<->manager transport, resolving the design's OPEN FORK to
bespoke gRPC (not OTLP+Wazuh). One long-lived, host-dialed, mTLS bidi stream:
health/metrics up, desired-state/commands down — no polling, real-time control,
still no inbound host port.
- proto/fleet_agent.proto: FleetAgent.Session(stream AgentUp) -> stream ManagerDown.
AgentUp = Hello | HealthSnapshot | CommandResult; ManagerDown = DesiredState |
Command(reboot/drain/resync). int64 unix times (no well-known-type dep).
Generated stubs committed (CI needs no protoc).
- agent.go: gRPC server with mTLS verified by the FLEET CA (the host's reused TPM
identity cert — distinct from the admin plane's admin CA). Identity is the peer
cert SERIAL, taken from the verified cert = inventory key, never self-asserted;
an unenrolled serial is refused (enrolment gates the stream). Per-conn send
channel decouples admin pushes from stream.Send. Stream registry (serial->conn)
with reconnect eviction. On connect + on change, desired-state is pushed live;
inventory stays authoritative so offline hosts reconcile on reconnect. Metrics
go through a MetricsSink seam (no-op now; InfluxDB in ckpt 4).
- main.go: agent plane starts only when its cert/key/CA are set; admin can run
standalone. Lifecycle PUT now live-pushes and reports pushed_live.
Full mTLS integration test (in-process fleet CA + host client cert): initial
desired-state on connect, Hello+snapshot -> inventory + online presence, live
desired-state push, and unenrolled-serial rejection. go build/test/vet/gofmt clean.
Deps (server-side only): google.golang.org/grpc + protobuf.
The checkpoint-1 commit included the 9.1M compiled 'fleet-manager' binary. Remove
it from tracking and gitignore it (/fleet-manager + *.test), matching the
enrol-broker convention. Build artifacts don't belong in git.
First slice of the server-side fleet control plane (serving/fleet-manager/), the
backend for the admin SPA (monitor / authorise / lifecycle). See
docs/fleet-management-design.md + tpm-enrolment-design.md §5.
This checkpoint (stdlib-only):
- Inventory record (identity/binding/enrolled/version/health/usage/desired),
keyed by TPM cert serial = hostnqn key, per the design's 'inventory record born
at enrolment'. JSON-file Store (atomic writes, mutex) behind an interface so a
DB swap is a drop-in.
- Admin plane: mTLS server verified by the DEDICATED admin CA (never the fleet CA,
so a host cert can't reach admin) + a plaintext dev mode. JSON API:
GET /api/hosts, GET /api/hosts/{serial}, PUT /api/hosts/{serial}/desired
(desired-state audited to the admin client-cert CN).
- Dev seed (FLEET_MGR_SEED) so the SPA has data before real enrolment/agent flow.
Smoke-tested: list/get/desired-PUT (incl. version-drift + drain), 404s, atomic
persistence. go build + go vet + gofmt clean.
Agent gRPC plane, broker-fold-in authorise, InfluxDB metrics, and the themed SPA
follow in checkpoints 2-6.
The '#18 render should be safe' call on the nss-mdns drop (5cc72cc) was wrong: it
bricked the kiosk on trim-rpm-48dc3e1 (chromium hard-Requires nss-mdns, so the 2d
remove cascaded chromium + its GUI stack out → 500 MB browserless image, flashing
cursor). Fixed in 2bc2daf (keep nss-mdns, mask avahi daemon; final build guard
asserts the kiosk stack survived). Re-verified booting into the kiosk on the 820;
container image measured 1.8 GiB → 755 MiB (~58%).
Root cause of the trim-rpm-48dc3e1 'flashing cursor, no kiosk' + mysteriously
small (~500 MB) image: the 2d hygiene step ran 'dnf remove -y nss-mdns', but the
EPEL chromium package HARD-Requires nss-mdns (Chromium uses .local mDNS hostnames
for WebRTC ICE candidates). So dnf removed chromium as a dependent package and
then reaped its orphaned GUI stack (avahi, at-spi2, atk, adwaita, chromium-common
...) — a ~300 MB drop. The shipped image had no browser, so kiosk-session died at
the ${BROWSER:?} guard before ever starting weston: a bare KMS cursor.
The earlier nss-mdns commit checked what nss-mdns NEEDS (avahi) but not what NEEDS
nss-mdns (chromium).
Fix:
- Keep nss-mdns + avahi (satisfy chromium's Requires); instead MASK the avahi
mDNS daemon/socket so nothing listens on :5353 — that was the real
attack-surface concern, and nss-mdns is only an NSS module (no daemon needed).
- Add a FINAL GUARD at the end of 2d: assert chromium/chromium-common/weston/mesa
are still installed in the final image, else fail the BUILD. A missing browser
must brick the build, never the fleet. Catches any future silent removal
cascade, oscap package_*_removed rule, or autoremove.
0bfe872 builds, deploys, and the host now mounts ONLY its own version's image
instead of connect-all'ing every served version — the targeted 'nvme connect -n
rd.nvmf.nqn' works. Each host = 1 controller on its own subsystem, which is what
lets old versions drain to 0 and spdk-loader auto-reap. Still to confirm: the reap
payoff (migrate a host, watch its old version drain + reap), and the secure path's
new step-6 connect over macsec0.
853ae20's diagnostic did 'awk … | sort -rn | head -25'. head closes the pipe
after 25 lines, sort dies of SIGPIPE (141), and step 2's set -euxo pipefail turns
that into a build failure (the exact trap the lsinitrd '| grep -q' comment warns
about). The kiosk step's rpm-manifest '| head' is safe only because that step
runs plain set -e, no pipefail.
Fix with the codebase convention: sort into a file, then head the file — no live
pipe for head to break. (Bonus: the crash log confirms omit_drivers worked —
amdgpu/radeon/nouveau kmods are gone from the top-25; the new biggest initramfs
item is GenuineIntel.bin CPU microcode at 22 MB, a future trim candidate.)
The dry-run failed with HTTP 403 on the branches API: PACKAGE_TOKEN is a
package-scoped token (registry push/delete) and can't read repo branches. Split
by capability: branch reads now use the automatic Actions token (github.token,
repo read via 'permissions: contents: read'); package list + delete keep
PACKAGE_TOKEN. Also replaced the bare 'curl -f' failures with a get() helper that
reports the HTTP code + a token-scope hint, so an auth problem can't look like
'no results' (which would otherwise skew the reconcile).
Reconciliation workflow (schedule + workflow_dispatch): lists live branches and
every container version across the four images this repo builds (fleet-boot,
fleet-ipxe, fleet-dnsmasq, spdk-nvmf) and deletes dev/branch builds whose branch
no longer exists, via the Gitea Packages API. Not event-driven, so it catches
branches deleted any way and never depends on a 'delete' webhook.
Safety (per request — never touch release tags or latest):
- Only tags ending in the dev commit-sha suffix '-<7 hex>' are ever candidates.
Release/version tags (1.2.3, v1.2.3, spdk v24.09, dnsmasq main-<sha>) and
'latest' never carry that suffix, so they're structurally out of scope; there's
also an explicit 'latest' skip. Verified by simulation across representative
tags incl. spdk's <ver>-<branch>-<sha> form and pre-release semver.
- Keeps any image whose sanitized branch is still live (+ always 'main'); branch
sanitization mirrors the build workflows.
- Fail-safe: aborts before deleting anything if the live-branch list can't be
fetched (0 branches), so an API/auth blip can't trigger mass deletion.
- Conservative matching: a suffix collision only ever causes a KEEP.
DRY RUN by default: workflow_dispatch previews unless dry_run=false; scheduled
runs dry-run until armed via repo variable REGISTRY_GC_ENABLE=true (so a
scheduled dry-run can be eyeballed first). Uses the existing PACKAGE_TOKEN
(write:package). shell: bash (runner defaults to sh).
f59eb79 now fully bench-verified: the fleet SPA renders on both the VM
(virtio-gpu) and the 820 (i915), so the i915 DMC/GuC firmware loads on real
hardware and the firmware keep-list is validated end-to-end. The two later
commits (nss-mdns drop, omit_drivers + keep-list tightening) don't touch i915,
so render should be safe, but still need their own build+boot.
The ~800 MB trim (glibc langpacks en-only + linux-firmware strip to the i915
keep-list, run pre-dracut) builds and boots — the boot/root path survives it
(firmware 678->147 MB, initramfs bundles only the keep-list). Notes still-open
checks: kiosk RENDER on real i915 hardware ('booted' != 'display up'), and that
the two later commits (nss-mdns drop 5cc72cc; omit_drivers + keep-list tightening
853ae20) are NOT in f59eb79 and need separate bench-verify.
First trim build confirmed firmware 678 MB -> 147 MB, but the initramfs came
out 97 MB and is now kernel-MODULE-bound, not firmware: --no-hostonly +
plymouth's drm dep instmods every GPU driver (amdgpu.ko etc.) though the fleet
is i915-only.
- omit_drivers += amdgpu radeon nouveau (90-diskless.conf): drop the discrete-GPU
kmods (amdgpu.ko is the biggest single module in the initramfs); keep i915 +
virtio-gpu/bochs for the splash. Shrinks the initramfs/UKI pulled over PXE on
every boot.
- Firmware keep-list tightened: drop mellanox/ (mlxsw_spectrum SWITCH-ASIC blobs,
not NIC fw — mlx5 NIC fw is on-card) and narrow intel/ to just ice (drop
ish/sof/ibt sensor-hub/audio/bluetooth, none loaded at boot per the audit).
- Add an 'initramfs largest files' diagnostic (sort lsinitrd.txt by size) so the
next build shows what dominates the .img.
BENCH-VERIFY: kiosk still renders on the 820 (i915 kept); watch the new initramfs
largest-files log line for the post-omit size.
Hosts connected with 'nvme connect-all', which attaches EVERY subsystem in the
discovery log. Each host then mounts only its own by-UUID verity namespaces but
leaves a live controller on every other served version's subsystem. Observed via
spdk-release list: more controllers than workstations, each box on every version
(1/3/3/1/3 across 5 versions for ~3 boxes). Beyond wasted connections
(O(hosts x versions)), it defeated the rolling-update drain/reap: no version ever
reached 0 connections while any host was up, so spdk-loader's auto-reap could
never fire and old versions accumulated without bound.
Fix — targeted connect to this build's subsystem only:
- extract-artifacts.sh bakes the per-version SUBNQN (already in verity.json
.subnqn, matched by the serving side) into the signed UKI cmdline as
rd.nvmf.nqn=. NQN is version identity, not per-host state, and the cmdline is
already per-version + immutable-per-signed-UKI, so invariant #1 holds.
- nvmf-connect.sh (test path) uses 'nvme connect -n <nqn>'; legacy UKIs without
rd.nvmf.nqn fall back to connect-all with a warning so they still boot.
- dot1x-macsec-up.sh (secure path) gains an explicit step-6 targeted connect over
macsec0 (after it's addressed, per invariant #2), replacing the delegation to
the dracut nvmf module's discovery connect-all.
Now each host = 1 controller on its own subsystem; versions drain to 0 when their
hosts migrate, so auto-reap works and connections scale O(hosts). Requires a
rebuild + re-sign (cmdline change, invariant #6). BENCH-VERIFY both paths;
confirm the dracut nvmf module isn't also connect-all-ing on the secure path.
CLAUDE.md invariant #2 + TODO #6 updated.
Investigating avahi/samba/rabbitmq in the image: ldd on libavformat.so confirms
samba (libsmbclient), librabbitmq, librist, libsrt and libzmq are all DT_NEEDED
by FFmpeg's libavformat-free — removing any breaks libavformat and thus future
HTML5 media in the kiosk, so they stay. avahi is the exception: it's absent from
libavformat's links and 'rpm -e --test avahi' shows its only reverse-dep is
nss-mdns (the mDNS/.local NSS resolver).
The fleet resolves via DHCP/DNS (boot.fleet.lab), never .local, so mDNS is dead
weight and a network-listening service unwanted on a locked-down kiosk. Remove
nss-mdns; dnf's clean_requirements_on_remove reaps avahi/avahi-glib/avahi-libs as
orphans. Belt-and-braces alongside install_weak_deps=False (which already drops
nss-mdns if it came in as a Recommends).
Follow-up to the linux-firmware strip (prev commit put it in the 2d hygiene
block, which runs AFTER the dracut build — so it only shrank the rootfs erofs,
not the initramfs). Move the strip to step 2, before the dracut build: with
--no-hostonly, plymouth's drm dep instmods a broad DRM set and dracut bundles
each module's firmware into the initramfs (amdgpu being the classic UKI
bloater). Trimming first means dracut never sees it, so both the rootfs erofs
AND the UKI initramfs shrink. Keep-list is unchanged (i915 + intel/ + mellanox
+ intel-ucode), so the fleet splash GPU (i915) and ice DDP firmware the
initramfs actually needs are retained; dracut just warns 'firmware not found'
for the removed drm modules and continues.
Add a post-build diagnostic (initramfs .img size + bundled firmware, expected
i915 only) so the initrd win is measured in the build log, not estimated.
The 2b rpm-size manifest revealed the biggest packages aren't the kiosk but
base bloat: linux-firmware 678 MB and glibc-all-langpacks 228 MB (vs chromium
370 + llvm 129). The uncompressed erofs is read over NVMe/TCP on every boot,
so this is fleet-wide dead weight + attack surface.
glibc: swap glibc-all-langpacks (every language) for glibc-minimal-langpack +
glibc-langpack-en. ~220 MB; image is English-only.
linux-firmware: a live fleet audit (modinfo -F firmware over loaded modules +
sysfs/lspci driver bindings on the real 820-class hosts and the VMs) proved the
only firmware any host loads is Intel i915 (GPU). Every NIC is firmware-free
(e1000e reports on-chip NVM, not a fw file; virtio_net loads nothing), so the
catastrophic missing-NIC-firmware case is off the table. Keep i915 + intel/
(ice DDP insurance) + mellanox + intel-ucode; strip the ~600 MB of foreign
vendor blobs (amdgpu/nvidia/radeon/ath/qcom/...). Remove iwl*-firmware (Wi-Fi,
~83 MB) on a wired-only fleet. Strip-in-place since the package can't be
removed without losing i915; du reports the result in the build log.
BENCH-VERIFY: kiosk still renders on the 820 (i915 DMC/GuC load) — the one real
risk; widen the keep-list if any host has a non-Intel GPU or an ice/mlx5 NIC.
The kiosk dnf transaction was measured at 229 pkgs / 1.0 GB. Nearly all
of the non-core packages are Recommends: a locked-down single-URL kiosk
never uses flatpak + its ostree/bubblewrap sandbox runtime, tracker-miners
(a filesystem indexer — dead weight and a threat surface), the pipewire/
pulseaudio audio stack, or libproxy-webkitgtk4 -> webkit2gtk3-jsc (an 8.9M
second browser engine dragged into a Chromium kiosk for libproxy's PAC
runner).
Add --setopt=install_weak_deps=False to cut them, and an rpm -qa size
manifest to the build log so the next CI build reports the exact new
footprint (turning 'seems unnecessary' into data for the next trim round:
gstreamer/media codecs, or mesa-dri-drivers if the GPU path is abandoned).
The immovable floor is chromium + chromium-common + llvm-libs (mesa sw-GL)
+ gtk3/adwaita + libicu, all hard deps. BENCH-VERIFY the kiosk still
renders with weak deps off (watch for a missing font or libva).
The build failed: checkmodule rejected `policy_module(chrony_dhcp_dispatcher,
1.0.0)` with "Building a policy module, but no module specification found".
policy_module() is a refpolicy m4 macro expanded only by the refpolicy Makefile
(/usr/share/selinux/devel), not by checkmodule. Use the raw module-policy
declaration `module chrony_dhcp_dispatcher 1.0.0;`, which is exactly what
checkmodule -M -m parses. require{} + allow rule unchanged.
The Dell 820 kept "finding a cert" and bypassing enrolment despite repeated BIOS
TPM clears. tpm2_nvreadpublic revealed 0x01800001 on that hardware is a FIRMWARE
platform index (platformcreate|policydelete|writelocked, 70 bytes — not our
cert; siblings at 0x01800003/4 + 0x01810000). platformcreate survives TPM2_Clear
and policydelete needs the platform's secret policy, so it's permanent and
unusable by us — the blind check false-positived on it, and enrolment could
never write there anyway. Same class of bug as the EK-handle collision.
Move the NV cert index default 0x01800001 -> 0x01800100 (free on the 820, owner
range, overridable via rd.dot1x.nvindex): cmdline (extract-artifacts.sh),
fleet-enrol -nvindex, the dracut hooks, bench scripts, CLAUDE.md invariant #1,
and design/SSP docs. The signing-key handle (0x81000100) is unchanged.
Harden WriteNVCert: refuse a platformcreate index with a clear "pick a free
owner index via rd.dot1x.nvindex" error instead of a confusing undefine
failure. Regression simtest/TestWriteNVCertRefusesPlatformIndex; NV roundtrip +
ReadNVCert tests moved to 0x01800100. All go tests + packer validate pass.
Captures the decision to re-enable a CONFIGURED firewalld (currently removed),
the required .pyc/-T0 root-cause fix (compileall --unchecked-hash), the 4 STIG
waivers it converts, and the load-bearing NVMe/TCP-survives-firewalld bench gate.
http/ks.cfg excludes firewalld (-firewalld) and sets firewall --disabled — the
fleet's security boundary is the network (802.1X/MACsec + switch); the host
firewall is a documented OFF item (stig-deviations #4). But a provisioner dnf
pulled firewalld back as a weak dep (Recommends) and it starts by default. Its
Python then tries to rewrite site-package .pyc bytecode, which the reproducible
erofs -T 0 mtime reset makes look stale, so firewalld_t gets write/unlink AVCs
on lib_t that flood the SIEM.
Remove firewalld in 2d hygiene (weak dep, nothing hard-needs it) and mask it as
a backstop against future weak-dep reinstalls. No firewalld => no daemon, no
AVC, matches the kickstart intent.
The NM dispatcher 10-chrony-dhcp-ntp runs `chronyc reload sources` to load DHCP
option-42 NTP servers. That chronyc runs in the confined custom-dispatcher
domain (NetworkManager_dispatcher_custom_t), so its reply socket carries that
label and the shipped policy has no rule for chronyd_t to sendto it. Under
enforcing, chronyd does the reload but its ack is denied — chronyc reports
failure and the AVC floods the SIEM (seen 2026-07-03; permissive=1 only because
the box was booted enforcing=0).
Ship a minimal local allow module (files/stig/chrony-dhcp-dispatcher.te):
allow chronyd_t NetworkManager_dispatcher_custom_t:unix_dgram_socket sendto;
Built + semodule -i'd in the 2c-STIG provisioner (checkpolicy installed if
absent), with an install assertion. `allow` not `dontaudit` — it also lets the
reload ack succeed so chronyc returns cleanly. Documented as
docs/stig-deviations.md #14 (not a rule waiver — a local functional allow).
BENCH-VERIFY: enforcing boot shows no AVC, chronyc reload returns 0, a
DHCP-handed NTP server appears in `chronyc sources`; and re-check verity-hash
reproducibility (semodule rebuilds the policy store, like the semanage step).
The nvmeof (spdk-target) service carried a versioned bootstrap mount
(type: image, source ${OCI_IMAGE}:${IMAGE_VERSION} -> /artifacts) plus an
IMAGE_VERSION env. Bumping IMAGE_VERSION therefore changed the service config,
so `docker compose up -d` recreated the container -> nvmf_tgt restarted ->
every connected host's NVMe/TCP root dropped at once. That is precisely the
fleet-wide outage the long-lived-target design exists to prevent.
Remove the /artifacts image-volume and IMAGE_VERSION env from nvmeof so its
config no longer depends on the version — compose leaves it running across a
bump. Bootstrap and rolling updates are already fully covered by spdk-loader
(stages the current release into ./spdk/releases and adds it online via
JSON-RPC); spdk-target.sh's seed block is guarded, so it just no-ops when
/artifacts isn't mounted. Verified nvmeof no longer references IMAGE_VERSION
(only spdk-loader does); compose YAML validates.
BENCH-VERIFY: bump IMAGE_VERSION + up -d with a host connected -> nvmeof
container not recreated, host root survives, loader adds the new version.
The enrol trigger keyed on "is NV 0x01800001 defined/readable", so leftover
state (an earlier tpm-provision run — bench-seen on the 820: "cert present —
normal boot" with no real identity) or random bytes (tpm-selftest) silently
suppressed enrolment. Now content-aware:
- tpmkey.ReadNVCert reads the full NV contents (chunked).
- classifyNVCert (TPM-agnostic, unit-tested): parses X.509, checks validity
window, and when a real fleet CA bundle is present verifies the cert chains
to it. Only a valid fleet cert = enrolled.
- New `fleet-enrol -check-only` (no network/no key-gen) exits 0=valid cert,
20=enrol needed, 1=error. The full enrol path uses the same classification
instead of NVPresent.
- The dracut hook calls -check-only as the single gate, replacing the blind
tpm2_nvread (and its now-dead TPM2TOOLS_TCTI export). tpm2_nvread stays in
the initramfs for the secure path (dot1x-macsec-up.sh).
Lab fallback: the shipped radius-ca.pem is a placeholder (not a cert) ->
loadCAPool returns nil -> chain-less "parseable + unexpired" acceptance, so
reboot-converge doesn't loop; production replaces radius-ca.pem with the real
fleet CA for strict chaining.
Tests: classifyNVCert table (valid/expired/garbage/foreign/no-CA) + simtest
ReadNVCert roundtrip across the 512B chunk. All pass; packer validate clean.
Bench (Proxmox vTPM, host cygnus-cruiser-99) confirmed via tpm2_readpublic that
0x81010001 — the handle our design used for the fleet signing key — is the
TCG-standard RSA Endorsement Key slot: the object there is the swtpm-provisioned
EK (restricted|decrypt|adminwithpolicy, RSA-2048/AES128CFB, canonical EK
authPolicy 837197…, EK cert at NV 0x1c00002). Most firmware/swtpm TPMs occupy
that slot, so EnsureKey could never create our key there — it errored reading
the EK as ECC and enrolment died before the CSR/POST.
Move the fleet signing key to 0x81000100, a non-reserved owner persistent
handle (clear of EK 0x81010001/2 and SRK 0x81000001/2). Threaded through:
- fleet-enrol -handle default + new rd.dot1x.handle= cmdline knob (baked by
extract-artifacts.sh, overridable via SMBIOS)
- files/etc/dot1x/wpa_supplicant-dot1x.conf private_key handle
- the dracut hook, bench tpm-provision.sh / tpm-selftest.sh
- CLAUDE.md invariant #1 + design/SSP docs + TODO #11
NV index 0x01800001 (owner range) is unchanged.
EnsureKey hardening: refuse to evict any restricted-decrypt object (EK/SRK) as
a safety net — never clobber the TPM endorsement/storage identity (also
protects the future EK/AK attestation path). Still self-heals a genuine stale
signing key. New regression TestEnsureKeyRefusesToEvictProtectedKey; the
stale-key regression and handle constants moved to 0x81000100.
Design note records the IDevID (0x81020001) option for the later EK/AK
attestation work. Follow-up items logged: content-aware NV presence check
(verify the cert chains to the fleet CA, so leftover/foreign NV can't suppress
enrolment) and a de-enrol/TPM-reset procedure.
Bench boot failed with "fleet-enrol: ensure key at 0x81010001: key at handle
is not ECC: did not contain eccDetail (selector value was 1)": a real host had
an RSA key persisted at 0x81010001 (left by an older tpm2-tools/openssl
provisioning path), and EnsureKey fatally errored trying to read it as our ECC
signing key — so enrolment died before the CSR/POST and the host fell through
to a normal boot (no fingerprint, nothing at the broker).
EnsureKey now evicts a wrong-type/unusable key at the handle and creates a
fresh ECC key. Safe on the enrol path: it only runs on an empty-NV host, so no
issued cert references the old key. A valid ECC key is still reused (idempotent
retried enrol).
Regression: simtest TestEnsureKeyEvictsStaleNonECC pre-plants an RSA key and
asserts EnsureKey recreates a usable ECC signer (passes against the go-tpm
simulator).
Audiobuzz