The SSP (v0.2, 2026-06-27) and annex predated the entire management/enrolment
plane and the July hardening work. Bring both to v0.3 (2026-07-10):
SSP:
- Scope/boundary: on-host fleet-agent now IN scope; server-side manager/broker/
step-ca/InfluxDB named as connected systems; new host interface (g) = outbound
mTLS to the management plane. Boot chain gains step 8 (post-switch-root agent).
- Data: management telemetry (operational metadata, mTLS-encrypted, no per-host
secret); the agent stream is the one non-plaintext transit path.
- §5.2 Protect: agent reuses the TPM identity for mTLS; WebAuthn-passkey admin
MFA (first human-admin MFA); 3 local SELinux modules; the 2026-07-04 fapolicyd
overlay watch_fs fix (app-control was inactive on the root until then).
- §5.3 Detect: operational health telemetry signal (distinct from auditd/Wazuh).
- §6 Essential Eight: MFA + admin rows updated (passkeys now enforced on mgmt UI).
- §7: new R-14 (mgmt plane = connected system pending authorisation) + R-15
(deferred agent actuators = future remote-mutation channel); R-05 MFA clause met.
Annex:
- §2.3 hardening: fapolicyd overlay-effectiveness fix + residual; 3 local SELinux
allow modules on the MAC row.
- §2.4 auth: agent mTLS on the machine-auth row; MFA row now Implemented for the
mgmt UI (passkeys), Partial for host SSH.
- §2.5 mgmt: new rows for TPM enrolment lifecycle (broker) and server-authoritative
management plane. §2.6: operational health-monitoring row. §2.7: mgmt-plane
segmentation (OPNsense L3, outbound-only). §2.9: agent mTLS, fleet CA (step-ca),
cert-lifecycle auto-renewal.
- §3 register: R-14, R-15 added; R-05 updated; heatmap + note + 3 new assessor
notes (scope the mgmt plane, witness fapolicyd deny, confirm actuators inert).
Recent work added telemetry fields and a cert path not yet in the docs:
- files/agent/README.md: add uptime to the health list; document that chrony is
read from /var/log/chrony/tracking.log (not the command socket) and needs BOTH
the chrony supplementary group (DAC) and the fleet_agent_chronylog SELinux allow
(TE), plus a reachable lab NTP (123/udp firewall); note the chrony group on the
unit row.
- serving/fleet-manager/README.md: add uptime + chrony detail to the health list;
note the cert-auto agent-cert-renewer sidecar + GetCertificate hot-reload
(certreload.go) in the bring-up and checkpoint 7.
- docs/TODO.md #14: probe list gains uptime + chrony-from-tracking.log; new dated
block records uptime/reboot-pending, the 4-layer chrony-access chase (all
silent-failing; inventory.json is the isolator), and the agent-plane cert
auto-renewal, all bench-verified 2026-07-10.
- docs/fleet-management-design.md: health record gains uptime + chrony
source/stratum/offset detail.
The tracking.log read was still returning empty under enforcing with NO AVC —
the tell of a DAC (not SELinux) denial. Root cause: /var/log/chrony is mode
0750 chrony:chrony (others have no access), so the agent's DynamicUser (group
tss only) can't even traverse the directory to reach the 0644 tracking.log —
EACCES on open, which produces no audit record. Manager confirmed genuinely on
fleet-manager-1db3977 (chrony-capable), and uptime flowed while chrony didn't,
proving the agent was sending empty chrony (failed read), not a manager gap.
Earlier manual reads worked only as root (sudo) or because fleetadmin had been
usermod'd into chrony during the socket test (reverted on reboot — /etc/group
is on the volatile overlay).
Fix: SupplementaryGroups=tss chrony on the unit — the chrony group has r-x on
the 0750 dir so the agent can traverse it; the log file is world-readable. This
is the DAC half; the fleet_agent_chronylog SELinux module (02a6621) is the TE
half — BOTH are required (verified: group-only -> TE-denied; module-only ->
DAC-denied silently). Corrected the now-stale 'no group' code comment; recorded
the two-layer requirement in the .te and stig-deviations #16. Needs a
workstation image rebuild + reboot.
The tracking.log approach (4ca0b6d) works in permissive but is blocked under
enforcing: chrony's log is labeled chronyd_var_log_t, confined by the shipped
policy to chronyd_t despite the 0644 mode. Verified on comet-beacon-54 — a
non-root read succeeds permissive, but under enforcing even a STIG-confined
admin (staff_t) gets Permission denied (ls -Z on the file is itself blocked).
The agent runs unconfined_service_t (confirmed via ps -eZ) and is denied the
same way.
Ship a minimal read-only allow module (files/stig/fleet-agent-chronylog.te):
allow unconfined_service_t chronyd_var_log_t:file { open read getattr } + dir
search to reach it. No write, no other chrony access, and no domain transition
(the agent reads a file; it does not exec chronyc) — narrower than the
abandoned nnp-transition module. Built + semodule -i'd in 2c-STIG alongside the
existing chrony_dhcp_dispatcher / fapolicyd_tmpfs modules. docs/stig-deviations
#16. Needs a workstation image rebuild + reboot; same policy-store-rebuild
reproducibility re-check as #14/#15.
The ntp service is host-networked, so (like the audit collector's 514/tcp)
Docker doesn't open the port — the serving host's own ufw/firewalld governs
123/udp. Without a rule, workstations are handed 192.168.50.10 as their NTP
server via DHCP opt 42 but their packets are dropped: chrony logs 'Added source
… / unreachable' and never syncs (seen on comet-beacon-54). Add the boot-LAN
firewall rule to the ntp service NOTE, mirroring the audit-collector README.
Bench on comet-beacon-54 proved every command-socket route is blocked for the
hardened agent: chronyd's unix socket is owner-write-only (0755, so even a
chrony-group member can't connect), and STIG closes the UDP command port
(cmdport 0). Rather than re-open a port or expose chrony's privileged socket,
enable chronyd's tracking log and have the agent READ it — no port, no socket,
no group, no SELinux domain transition, full hardening intact.
- chrony.conf: enable 'log tracking' (idempotent) so chronyd appends
source/stratum/offset to /var/log/chrony/tracking.log. Write volume is a ~130B
line per clock update (backing off to ~maxpoll when stable) and capped by the
chrony RPM's logrotate — negligible on the RAM overlay. cmdport 0 stays, so no
STIG deviation.
- agent: chronyTracking() now reads the last data line of tracking.log via
parseTrackingLog() (source f[2], stratum f[3], offset f[6], leap/sync f[7]);
replaces the chronyc exec + parseChrony. Unit test swapped accordingly.
- revert the two abandoned approaches: drop the bindcmdaddress line (394ad10)
and the fleet_agent_chronyc SELinux nnp_transition module + provisioner block
(e6fd4a2, .te deleted) — no chronyc exec means no transition to allow. Removed
stig-deviations #16 and #17 (both waivers no longer needed).
Needs a workstation image rebuild + reboot. BENCH-VERIFY: tracking.log is
world-readable so the DynamicUser agent can read it; under enforcing
unconfined_service_t can read chronyd_var_log_t (add a minimal file-read allow
if denied); and the offset sign matches chronyc on a synced host.
Root cause of the blank chrony fields (found on comet-beacon-54): chrony 4.x
(RHEL 9) defaults its command interface to the UNIX socket only
(/run/chrony/chronyd.sock), usable by root or the chrony group with a writable
/run/chrony. The fleet-agent unit is hardened — DynamicUser (group tss, not
chrony) + ProtectSystem=strict (/run read-only) — so it cannot open the unix
socket nor create chronyc's reply socket. Bench proof: normal-user 'chronyc
tracking' -> 506 Cannot talk to daemon; only sudo worked; 'chronyc -h 127.0.0.1'
failed and ss showed no UDP :323. So the probe always failed and every chrony_*
field stayed empty. SELinux permissive did NOT help — it was never (only) an
SELinux problem; the nnp_transition module (prev commit) is still needed for the
enforcing exec transition.
Fix: bind chronyd's command socket on 127.0.0.1 (bindcmdaddress in chrony.conf,
idempotent in the 2c provisioner) and point the probe at it (chronyc -n -h
127.0.0.1 tracking). UDP needs no fs write or group and rides the unit's
already-allowed AF_INET. Loopback + monitoring-only (state changes still need the
root unix socket), so exposure is minimal. Rejected the unix-socket route (chrony
group + ReadWritePaths=/run/chrony) because /run/chrony is 0750 (group has no
write). Documented in docs/stig-deviations.md #17; .te bench note updated for the
enforcing UDP path. Needs a workstation image rebuild + reboot.
fleet-agent reports clock health via 'chronyc -n tracking', but the hardened
unit runs with NoNewPrivileges (explicit + forced by DynamicUser/SystemCallFilter/
MemoryDenyWriteExecute). SELinux transitions the exec'd chronyc into chronyc_t,
and NNP forbids that domain change without process2:nnp_transition — which the
base policy doesn't grant for unconfined_service_t -> chronyc_t. Under enforcing
the exec is denied (avc: denied { nnp_transition } comm=fleet-agent, seen on
comet-beacon-54), so chronyc never runs and every chrony_* health field stays
empty (Monitor chrony row shows '—'). uptime/cpu/mem are unaffected (no exec).
Ship a minimal local module (files/stig/fleet-agent-chronyc.te) granting only
that one nnp_transition to the MORE-confined chronyc_t domain — the safe
direction the permission exists to gate — loosening nothing else in the agent
sandbox. Built + semodule -i'd in 2c-STIG alongside the existing
chrony_dhcp_dispatcher / fapolicyd_tmpfs modules. Justified in
docs/stig-deviations.md #16. Needs a workstation image rebuild + reboot; same
policy-store-rebuild reproducibility re-check as #14/#15.
The agent-plane server cert was issued by hand (server-pki.sh / docker exec
step ca certificate), which needs the step CLI on the host, re-specifies SANs
each time, and silently expires at the CA's 90-day cap — the exact failure that
just broke agent TLS (cert minted with a stale SAN, missing manager.fleet.lab).
Automate it:
- agent-cert-renewer sidecar (smallstep step-cli image, compose profile
cert-auto): issues the cert once from the co-located fleet step-ca, then
'step ca renew --daemon' rotates it before the cap. Renewal preserves the
subject + SANs, so SAN drift cannot recur. Chowns to the manager's nonroot
uid so the distroless manager reads the key. Opt-in + only valid where the CA
is co-located; server-pki.sh stays as the manual/remote-CA path.
- manager hot-reloads via GetCertificate (certreload.go): a rotated cert on
disk is picked up on the next handshake with no restart and no dropped agent
streams; a half-written pair keeps the last good cert serving. Unit-tested.
- defaults.env documents the cert-auto knobs; DEPLOY.md §2a recommends the
renewer and flags the mandatory manager.fleet.lab SAN.
The agent only reported chrony as synced|unsynced — nothing about how well the
clock tracks. Parse `chronyc tracking` for the source-relative metrics.
- proto: HealthSnapshot.chrony_offset_seconds (16, signed system-time offset,
fast=+), chrony_source (17, reference name/IP), chrony_stratum (18); regenerated.
- agent: replace chronyState() with chronyTracking()/parseChrony() — one exec,
parses Leap status + Reference ID + Stratum + System time offset (+ test).
- manager: Health.{ChronyOffset,ChronySource,ChronyStratum}; snapshotToHealth maps
them; influx writes chrony_offset_seconds/_stratum/_source and whitelists the
offset + stratum as graphable fields.
- UI: chronySummary() + offsetMs() — the chrony row now shows
"synced · 192.168.50.10 · s3 · +0.01ms" instead of just "synced".
The manager lets an admin schedule a reboot (desired.reboot_at) but never showed
whether it had happened. Add host uptime to the health snapshot and derive a
reboot status from it.
- proto: HealthSnapshot.uptime_seconds (15); regenerated pb.go (manager + agent copy).
- agent: probe /proc/uptime -> uptime_seconds (with parser + test).
- manager: Health.UptimeSeconds cached in inventory; snapshotToHealth maps it;
influx writes + whitelists uptime_seconds as a queryable field.
- UI: fmt.js dur() + rebootStatus() — boot_time = last_seen - uptime (manager clock
frame, no host/manager sync needed), compared to reboot_at. Monitor host detail
shows an "uptime" row and a "reboot" row: pending in X / overdue by X / rebooted.
Observability-only — works before the reboot ACTUATOR lands (deferred). Once a host
boots after the scheduled time (BMC/manual now, agent later), the badge flips to
"rebooted" automatically.
The Monitor tab polled /api/hosts every 10s (roster + host-detail refresh) but
only fetched a host's metric series on click, so the tmpfs/cpu/temp graphs went
stale until you re-selected. Extract loadMetrics() and call it from the same 10s
tick when a host is selected, so the graphs stay live.
Completes the rename commit (which carried only the file move): defaults.env
layering header, docker-compose usage comment, serving/Makefile (sets
COMPOSE_ENV_FILES), serving/.gitignore (.env), and the README/TODO references.
Same layering as the fleet-manager stack: defaults.env holds the committed
defaults, .env holds only per-LAN overrides + secrets, layered via
COMPOSE_ENV_FILES=defaults.env,.env. Changing a default here reaches every run
without re-copying.
- git mv serving/.env.example -> serving/defaults.env; header explains the layering.
- docker-compose.yml usage comment -> the COMPOSE_ENV_FILES / Makefile flow.
- serving/Makefile: sets COMPOSE_ENV_FILES; help/up/down/restart/pull/logs/ps/config
with a clear .env-missing guard (mirrors serving/fleet-manager/Makefile).
- serving/.gitignore: ignore .env (defaults.env is committed).
- README + TODO references updated. serving/siem/.env.example is a separate
wazuh1-node file, left as-is.
The agent's baked default pointed at the PUBLIC admin-UI name
(diskless-server.bitasylum.org), which resolves to the serving host's outbound IP
— unreachable from the fleet LAN. There's no split-horizon to "automate": the
manager agent plane is just another *.fleet.lab, served by the fleet dnsmasq like
boot/nvme/audit.
- serving/.env.example: MGMT_HOST=manager.fleet.lab.
- serving/docker-compose.yml: dnsmasq --address=/${MGMT_HOST}/${SERVER_IP}.
- files/agent/agent.env + main.go default: FLEET_AGENT_MANAGER=manager.fleet.lab:9600.
- server-pki.sh: AGENT_SANS default -> manager.fleet.lab (cert SAN matches the
baked name, so no FLEET_AGENT_SERVER_NAME override).
- docs/TODO #14 gotcha #4 updated: fleet name, not the IP+server-name workaround.
The public diskless-server.bitasylum.org stays the ADMIN-UI name (Traefik/:443).
Record the 2026-07-10 end-to-end verification (Dell 820 comet-beacon-54, real
TPM): TPM-identity mTLS -> enrolled -> streaming health + receiving desired-state,
manager logs "agent connected". Add the "deploying the agent plane" gotcha list
(placeholder CA, root+intermediate client CA, distroless PKI perms, split-horizon
DNS, out-of-band inventory seeding) — all automated by commit a89c3d2 — so the
next deploy doesn't rediscover the debug chain.
Four fixes from the on-host agent bring-up:
1. Bake the real fleet CA into the image. New `fleet_ca` build var (default =
the committed placeholder) file-provisions the trust anchor to
/etc/dot1x/radius-ca.pem — used by wpa_supplicant (RADIUS) AND the fleet-agent
(verify the manager). CI materializes a FLEET_CA secret to a runner-temp file
and passes -var fleet_ca; unset => placeholder (lab). No CA material committed.
2. server-pki.sh installs the agent CLIENT CA as root+intermediate (not root
only). Host TPM certs are leaf-only in NV and signed by the step-ca
intermediate, so a root-only client CA makes the manager advertise only the
root in its CertificateRequest -> Go clients send no cert ("tls: certificate
required") and it couldn't verify a leaf-only cert anyway. bootstrap.sh now
exports intermediate_ca.crt so FLEET_CA_INTERMEDIATE resolves.
3. New `fleet-manager register-host <serial> [-name] [-hostnqn]` subcommand:
seeds/updates an inventory record for a host enrolled out of band (bench
broker, migrated fleet) through the real store, so its agent stream is
accepted. The manual counterpart to the authorise-approve flow.
4. The agent surfaces the authoritative recv-side gRPC status instead of the
bare io.EOF gRPC returns on the send side when the server ends the stream —
so "host is not enrolled" (PermissionDenied) shows up instead of "EOF".
Companion to the previous commit's file move: point the 2b-agent install at
/tmp/agent/agent.env (the agent's own upload) instead of the deleted
/tmp/etc-stage path, update the file-provisioner comment, the enable_agent var
description, and the TODO source-path reference. packer validate passes.
The 2b-agent provisioner installed agent.env from /tmp/etc-stage, but the 2a
package step deletes /tmp/etc-stage before 2b runs, so the build failed with
"cannot stat '/tmp/etc-stage/fleet-agent/agent.env'". 2b-hostapi avoids this by
installing from its own upload dir; do the same: move agent.env into files/agent/
so it rides the agent's file upload (/tmp/agent), which survives until 2b-agent
cleans it. Runtime path (/etc/fleet-agent/agent.env) is unchanged.
Build the on-host agent that pairs with fleet-hostapi and feeds the
fleet-manager control plane (TODO #14 "STILL TO BUILD (a)"). Nothing on the
workstation talked to the manager before this; Monitor + InfluxDB stayed empty.
Scope: telemetry only. The agent reuses the host TPM identity for mTLS (read-only
tpmkey.LoadSigner at 0x81000100 + cert from NV 0x01800100; the private key never
leaves the TPM), opens the long-lived FleetAgent.Session bidi stream, sends Hello
+ periodic HealthSnapshot (load/cpu/temp/mem/tmpfs/nvme/auth/selinux/chrony/kiosk,
probed read-only from /proc+/sys plus hostapi facts), and LOGS but does not act on
the DesiredState/Command it receives. Reconcile actuators (kiosk URL, reboot,
drain) are deferred until the telemetry half is bench-verified.
- files/agent/: new module (go-tpm + grpc), static CGO_ENABLED=0 like enrol/hostapi.
Copies the enrol tpmkey (+ read-only LoadSigner) and the generated fleet.agent.v1
proto so it stays self-contained without importing the initramfs enrol module.
- files/agent/simtest/: proves the TPM-signed cert completes a real mTLS client-auth
handshake (RequireAndVerifyClientCert) against the go-tpm software simulator.
- files/etc/fleet-agent/agent.env: fleet-generic config (manager DNS name, fleet CA
reused from radius-ca.pem, TPM handle/nvindex, interval). No per-host state.
- fleet-agent.service: hardened unit (tss group for /dev/tpmrm0, INET egress).
- CI builds fleet-agent out-of-VM + runs vet/test + the simulator handshake test.
- Packer: enable_agent flag (default on) + 2b-agent provisioner + fapolicyd trust.
Post-switch-root only, so no dracut/initramfs changes.
- docs: TODO #14 + fleet-management-design as-built updated.
Requires the host already enrolled (broker approve births the inventory row) —
else the manager returns PermissionDenied, the correct "not enrolled yet" signal.
`make up` / `logs` / `restart` / `pull` / `build` / `config` / `register-admin`
/ `backup` — each exports COMPOSE_ENV_FILES=defaults.env,.env so plain `make ...`
layers the committed defaults under .env with no manual export or --env-file.
Guards on .env existing (COMPOSE_ENV_FILES requires it) with a clear message.
`make help` lists targets. DEPLOY §1 points at it.
Rename .env.example -> defaults.env and make it the committed defaults layer.
Compose reads defaults.env then .env (later wins) via COMPOSE_ENV_FILES, feeding
BOTH container env and ${...} interpolation — so a changed default here reaches
every run without re-copying, and .env shrinks to just overrides + secrets.
No compose service restructure: the existing interpolated environment: blocks
already build the container env; the layering just supplies their values from two
files. (env_file: directive alone wouldn't work — it doesn't feed the image-tag /
cross-service Influx / ACME_EMAIL interpolation this compose relies on.)
Docs (DEPLOY §1, compose header, README) updated: export
COMPOSE_ENV_FILES=defaults.env,.env, keep only secrets/overrides in .env.
Route the gcr.io distroless runtime image through the Harbor mirror
(harbor.bitasylum.org/gcr.io/distroless/static-debian12:nonroot) per the standing
rule, completing the fleet-manager Dockerfile's base-image mirroring.
Follow the code removal through the rest of the tree now that Traefik owns UI TLS:
- compose: drop FLEET_MGR_ADMIN_BEHIND_PROXY + ADMIN_CERT/_KEY from the manager
env; correct the ./pki mount comment (agent + broker certs only).
- .env.example: drop the BEHIND_PROXY + cert vars + self-TLS note.
- DEPLOY.md / README: remove the self-TLS "alternative" and the BEHIND_PROXY /
ADMIN_CERT config rows.
- server-pki.sh: point the admin UI cert at Traefik, not ./pki/admin.
- delete serving/fleet-manager/acme/ (the retired ACME side-car).
Traefik (and the k8s Ingress later) owns UI TLS, so remove the vestigial
self-TLS path: adminTLS(), the certReloader type, the adminCert/adminKey config
+ FLEET_MGR_ADMIN_CERT/_KEY env, and the now-redundant adminBehindProxy flag.
The admin plane always serves plaintext now — two modes: INSECURE (dev, no auth)
or default (passkey behind a trusted TLS proxy). Drops the crypto/tls, fmt, sync
imports and the errString helper.
Route the ad-hoc images I introduced through harbor.bitasylum.org (the standing
rule): backup.sh uses harbor.bitasylum.org/dockerhub/library/alpine:3.20 (via an
overridable ALPINE_IMAGE); the DEPLOY.md volume-chown fix + the goacme/lego prose
use the Harbor paths too. Compose (influxdb/traefik) + Dockerfile bases are already
harbor-mirrored; git.bitasylum.org/audiobuzz/* stays (CI-built).
- .env.example: RP_ORIGIN port-less, FLEET_MGR_ADMIN_BEHIND_PROXY=true, cert vars
labelled self-TLS-only; ACME vars are now Traefik's (dropped sidecar-only ones).
- DEPLOY.md §2c rewritten for Traefik (443 + Route 53 DNS-01), plus a "Network
paths" table answering how each plane is reached (enrolment via boot nginx,
agent direct mTLS via OPNsense L3, admin via Traefik). Start sequence + hardening
status updated (admin-exposure item resolved).
- README: admin transport = Traefik; behind-proxy explained; config table gains
BEHIND_PROXY; checkpoint 8 (hardening + Traefik ingress).
Management-plane ingress: a host-networked Traefik terminates the LE cert on :443
(Route 53 DNS-01, public-resolver propagation check) and forwards plaintext to
the manager running FLEET_MGR_ADMIN_BEHIND_PROXY=true on loopback :9500. Only the
admin plane rides Traefik — the agent gRPC/mTLS plane (:9600) stays direct (Traefik
can't terminate fleet-CA mTLS; hosts reach it via OPNsense L3 routing). This is the
k8s ingress-controller + cert-manager role in miniature.
- traefik/dynamic/fleet-manager.yml: file-provider router (Host rule -> manager),
passHostHeader so the WebAuthn RP origin matches.
- compose: traefik service (static config as CLI flags so ${ACME_EMAIL} interpolates)
+ traefik-acme volume; manager switched to behind-proxy defaults, RP_ORIGIN now
the port-less https origin.
- Retire the acme side-car (Traefik owns ACME now): remove the service + acme-state
volume; renew-loop.sh kept + relabelled as the self-TLS-mode cert helper.
- :443 only (admin UI is https-only) so it can't collide with a boot-LAN nginx :80.
Decouples "passkey auth ON" from "the listener terminates TLS". A new
FLEET_MGR_ADMIN_BEHIND_PROXY=true keeps WebAuthn/session auth fully on but serves
plaintext, for running behind a trusted TLS-terminating ingress (Traefik now, a
k8s Ingress later) that owns the LE cert. WebAuthn only needs RP_ORIGIN to match
the browser's HTTPS origin, not our transport; session cookies stay Secure. The
listener must be bound private (loopback/mgmt) so only the proxy reaches it —
logged as a warning. Insecure (dev, no auth) and self-TLS (default) modes
unchanged.
DEPLOY.md gains a "Hardening status" section (done vs operator-to-close), and
TODO #14 records the deployment as CI-built + acme-cert + backup, with the open
items called out: network-segment the :9500/:9600 listeners to the mgmt net, run
server-pki.sh when the agent lands, choose :9500 vs a 443 proxy, and digest-pin
the images.
Snapshot fleet-data (inventory + admins.json + enroll.key — the admin-lockout
risk) and influx-data to timestamped tarballs via a throwaway alpine container
(read-only, live-safe), with BACKUP_KEEP retention. Documented in DEPLOY.md
Backups with a restore pointer.
Point the compose service at the registry image the new CI workflow pushes
(git.bitasylum.org/audiobuzz/fleet-manager:${FLEET_MANAGER_VERSION:-latest}) so
mgmt hosts `docker compose pull` instead of building at deploy time — matching
dnsmasq/spdk/kiosk. `build: .` stays for local dev. Add FLEET_MANAGER_VERSION to
.env.example.
Add .gitea/workflows/fleet-manager-image.yaml, modelled on dnsmasq-image.yaml:
buildah bud of serving/fleet-manager (multi-stage SPA+Go→distroless) on a
path filter, with the same branch/tag tagging scheme (semver on tags, main-<sha>
+ latest on main, <branch>-<sha> dev images). Wires it into the shared
conventions:
- gitea-main-ci.yaml paths-ignore: list the new workflow so a fleet-manager
image change doesn't trigger the ~17-min UKI build (serving/** already covers
the dir; the .gitea/ workflow file needed listing).
- registry-gc.yaml IMAGES: add fleet-manager so its dev-branch builds are GC'd.
Follow-up to comma-separated RP_ORIGIN support: the enrolment URL builder used
the raw origin string, which would embed the whole comma list. Take the first
origin for the link.
Passkeys bind to the exact scheme://host:port the browser used, so a single
configured origin fails "Error validating origin" when reached via a different
form (e.g. :9500 direct vs a 443 proxy). Split FLEET_MGR_RP_ORIGIN on commas so
several exact origins can be allowed. Single-value config is unchanged.
Flipping FLEET_MGR_ADMIN_INSECURE=false before bootstrapping an admin crash-looped
on a cryptic "enrol key: open /data/enroll.key: no such file or directory". The
key is created by `register-admin`, so on ErrNotExist say exactly that (and note
the insecure escape hatch) instead of surfacing the raw errno.
Split-horizon DNS made lego's propagation check query the container's internal
resolver, which doesn't serve the public _acme-challenge TXT record, so the
challenge timed out. Add ACME_RESOLVERS (comma-separated host:port, default
8.8.8.8:53) → one --dns.resolvers flag each. Documented in .env + DEPLOY.md.
The goacme/lego image ships the binary at /lego, not on PATH, so the bare `lego`
call failed ("not found") once the entrypoint was overridden with the shell.
Detect it (command -v, then /lego fallback) and invoke via $LEGO.
lego auto-discovers the Route 53 zone via ListHostedZonesByName, so the zone ID
is optional — but pinning it disambiguates overlapping public/private zones and
lets the IAM policy drop the list permission. Thread it through compose + .env
+ DEPLOY.md (no script change; lego reads it from the environment).
Auto-issue + auto-renew the admin-UI TLS cert with a long-running compose
side-car instead of a host cron. The `acme` service (goacme/lego image, running
acme/renew-loop.sh) obtains the cert over DNS-01 via the AWS Route 53 API,
writes fullchain/privkey into ./pki/admin/ — the SAME dir the manager reads
(./pki:/pki:ro) — and chowns them to the nonroot runtime uid so the read-only
container can load the key. The manager's certReloader hot-swaps on the mtime
bump, no restart. Loop renews within 30 days of expiry; ACME_STAGING=1 gives a
dry-run vs LE staging.
- docker-compose.yml: `acme` service + `acme-state` volume (lego account/cert
store). ACME_DOMAIN reuses FLEET_MGR_RP_ID so cert host == passkey origin.
- .env.example: ACME_EMAIL + AWS creds (or instance role) + ACME_STAGING.
- DEPLOY.md §2c: rewritten to the side-car flow — scoped Route 53 IAM policy,
bring-acme-up-first ordering, flip off insecure, and an origin-match note.
(The docker-compose.yml commit also carries the local harbor.bitasylum.org
influxdb image mirror edit.)
A just-approved inventory record has a zero-value last_seen (Go 0001-01-01),
which parsed to a pre-epoch instant and slipped past the !t guard, rendering as
"739800d ago". Treat any at-or-before-epoch timestamp as "never" — the honest
state for a born-but-never-connected host (its health/version stay "—" until an
agent dials the gRPC plane).
The distroless nonroot runtime (uid 65532) couldn't write /data/inventory.json
because a named volume is created root-owned — approve failed with permission
denied on the inventory birth. Pre-create /data owned by nonroot in the image so
a fresh volume inherits it via Docker copy-up; an already-created root-owned
volume still needs a one-time chown (documented).
DEPLOY.md:
- 2b now covers BOTH broker deployments — the main serving stack
(enrol-pki/broker, ENROL_ADMIN_INSECURE=true by default → must flip to false)
and the bench stack — with the mint-once-into-broker-dir, copy-CA-to-manager
flow and an mTLS verify curl.
- Troubleshooting section: the data-volume permission-denied fix, the
"authorise plane not configured" (recreate + single-instance) fix, and the
"wrong version number" (broker plaintext) fix.
Every plane logged its state EXCEPT the broker success path, which was silent —
so "authorise pillar working" was indistinguishable from "not logged". Add an
ENABLED line matching the InfluxDB/agent/admin convention.
The compose wires the authorise (broker) and agent planes to fixed cert paths,
so a first bring-up that hasn't run the PKI steps yet crash-looped on the first
missing file (e.g. /pki/broker/clients/fleet-manager.crt) instead of booting.
Treat a configured-but-absent cert as "plane not yet provisioned": log a loud
WARNING and disable that plane, so the manager boots and each plane lights up
as its PKI appears. A malformed/unreadable cert still hard-fails (real
misconfig). The admin plane stays a hard requirement (FLEET_MGR_ADMIN_INSECURE
is its bench escape). Adds a missingFiles() helper; hub push path already
nil-guards a disabled agent plane.
Complete the deployment story for the server-side control plane:
- docker-compose.yml: private InfluxDB (published to loopback only) + a
host-networked fleet-manager (reaches the loopback enrol-broker on
127.0.0.1:9443 and binds a routable gRPC agent listener, like the broker).
- .env.example: RP origin (diskless-server.bitasylum.org), Influx token/DB
password, cert/PKI paths; the Influx token wires influxdb + manager with
one value.
- server-pki.sh: issue the agent-plane server cert from the fleet CA (step)
and stage the fleet root as the client-verify CA.
- DEPLOY.md: end-to-end runbook — three-cert PKI (fleet-CA agent cert, broker
admin-client cert, Let's Encrypt DNS-01 UI cert), register-admin passkey
bootstrap, compose up, hardware-free smoke test, backups.
- .dockerignore: keep host node_modules/dist/pki/secrets out of the build
context so the embedded SPA is the one built in-image.
- .gitignore: never commit .env / issued PKI / data.
Design docs marked as-built: fleet-management-design.md as-built note
(resolved forks — gRPC bidi transport, InfluxDB metrics, SPA graphs, passkey
admin auth, broker fold-in "alongside"); tpm-enrolment-design.md (broker
fold-in + agent transport fork resolved); TODO.md #14 → partly built, with
the on-host agent + per-host boot-version assignment (#6) called out as the
remaining work.
Two-stage build: node builds the Svelte SPA to dist, then golang compiles the
binary with the SPA embedded (-tags embedspa). Committed .pb.go stubs mean no
protoc. Ships as a small static distroless/nonroot image; same binary serves the
register-admin subcommand. (Not build-verified here — no docker in the
devcontainer — but the go build -tags embedspa + npm build are both proven.)
Per the option-A decision (deploy current design without per-host versioning):
add an honest note on the Lifecycle panel that kiosk-url/reboot/drain reconcile
via the agent but per-host boot-version assignment (dnsmasq cohort steering) is
not yet wired, so a version pin is stored + pushed but does not change what the
host boots. Avoids implying the pin takes effect.
Replace the free-text target-version field with a select populated from versions
known across the fleet (each host's running + targeted version, newest first),
plus a 'custom...' escape to a text input for a version not yet seen. Avoids
typos on the pin. (Authoritative serving-side source is a follow-up.)
- Contrast: lighten the muted/faint text tokens (AA on the hull panels) and make
Monitor host-card data values bright ink against dim faint labels, so version/
tmpfs/seen details read clearly on the blue-grey panels (was grey-on-grey).
- Lifecycle: reboot selector now offers none | now | at... — 'now' sets
reboot_at to the current time (agent reboots on reconcile); still confirm-gated
as a high-consequence action.
The themed single-page admin UI (serving/fleet-manager/ui/), embedded into the
binary and served same-origin over the admin plane.
- Svelte 5 + Tailwind 4 + Vite; a sibling of serving/kiosk-app. Theme = LYSANDRA
'restricted-violet' (high-security evolution of the kiosk's flight-deck DNA):
amber deck, VIOLET reserved for privileged/destructive + auth surfaces, teal
telemetry, red alert, clearance chrome (corner-reticle panels, scanlines, mono
clearance labels, secure-link status bar).
- Passkey gate: Login (usernameless discoverable WebAuthn) + Enrol (from the
register-admin one-time link) via navigator.credentials, with base64url<->buffer
helpers and the go-webauthn response shapes.
- Three pillars over the JSON API: Monitor (fleet roster + version-drift + health
+ per-host detail with bespoke SVG telemetry charts), Authorise (pending queue:
fingerprint cross-check, approve+bind in violet, deny), Lifecycle (desired-state
form; drain/reboot confirm-gated). Session-aware fetch client (cookie + CSRF).
- go:embed behind the build tag (spa_embed.go / spa_stub.go) so the
default go build/test need no dist/; mounted at / (api/auth/healthz win).
Verified: npm build clean (71KB JS / 33KB CSS + self-hosted fonts); default +
-tags embedspa go builds; smoke test serves index + assets + API through the SPA
origin. Against an insecure+seeded backend the deck renders passkey-free for
theme work. dist/ + node_modules gitignored.
The human->UI hop now authenticates with a hardware passkey (YubiKey), not an
mTLS client cert: phishing-resistant, origin-bound, better UX. (manager->broker
stays mTLS — service-to-service.)
- webauthn.go: credential store (registered admins + passkeys + a one-time-token
jti replay ledger; JSON file, atomic writes) + the webauthn.User impl + RP setup.
- cli.go: 'fleet-manager register-admin <name>' mints a short-lived HMAC-signed
enrolment link over a shared enrol key; the browser completes the WebAuthn
create ceremony. No anyone-can-register endpoint; single-use via the ledger.
- authapi.go: register/login ceremonies (usernameless discoverable login,
userVerification required) + logout + whoami; short-lived in-memory ceremony
store between begin/finish.
- session.go: opaque server-side sessions (Secure/HttpOnly/SameSite=Strict cookie)
+ requireSession middleware gating /api/*, with a CSRF header check on mutations.
adminName now comes from the session (audit attribution).
- main.go: register-admin subcommand dispatch; admin TLS switched to SERVER-AUTH
(no client cert) with a HOT-RELOADING cert (certReloader) so a Let's Encrypt
DNS-01 renewal is picked up without a restart. RP = diskless-server.bitasylum.org.
/api/* mounted behind requireSession; /auth/* + /healthz public.
Cert strategy (per design decision): the UI HTTPS cert is a publicly-trusted LE
cert via DNS-01 (mgmt-network UI isn't internet-reachable, so HTTP-01/TLS-ALPN-01
can't validate; DNS-01 proves control of bitasylum.org via TXT + supports
wildcards). External ACME client issues+renews to the cert path; manager reloads.
Deps: github.com/go-webauthn/webauthn (server-side). Tests: enrol-token
mint/verify/tamper/expiry, session middleware (401/403/CSRF/expiry), cred-store
persistence + jti ledger. build/test/vet/gofmt clean. Live passkey ceremony
(virtual authenticator / real YubiKey) is a bench item.
The manager is the ONLY thing that talks to InfluxDB, both directions; hosts hold
no token and never reach it.
- influx.go: influxSink implements MetricsSink — each agent HealthSnapshot is
written as a fleet_health point, identity (serial/hostnqn/starship) as TAGS
from the AUTHENTICATED stream (a host can only write its own series), telemetry
as fields. QueryField answers the SPA's read: a Flux query built with a strict
FIELD WHITELIST + hex-serial guard + parsed-duration range, so no caller input
is interpolated raw (injection-safe). Server-side influxdb-client-go/v2.
- admin.go: GET /api/hosts/{serial}/metrics?field=&range= -> time-series JSON;
503 when InfluxDB isn't wired, 400 on a bad range, 404 for unknown host.
- main.go: sink created when FLEET_MGR_INFLUX_URL+_TOKEN set, fed to the agent
hub; nil -> no-op sink so the manager runs without metrics.
Tests: snapshotPoint tags/fields/time, fluxQuery contents + %q quoting, and
QueryField rejects a non-hex serial + non-whitelisted field. build/test/vet/gofmt
clean. Live InfluxDB path validated in the compose bench (ckpt 6).
The AUTHORISE pillar: the manager owns the enrolment queue UX, the enrol-broker
stays the state machine + step-ca signer (v1 'alongside' fold-in).
enrol-broker (minimal, backward-compatible change):
- admin approve/deny now return the decided Enrolment as JSON when the caller
sends Accept: application/json. Browsers still redirect; other API callers
(verify-serving.sh) still get 204. The cert PEM stays unexported (only the
requesting host gets it via /status) — the JSON carries cert_serial, hostnqn,
context, fingerprint, cn.
fleet-manager:
- broker.go: mTLS client for the broker admin plane (Pending/Approve/Deny),
nil when unconfigured so the manager runs standalone.
- admin.go: GET /api/enrol/pending (proxy), POST /api/enrol/{id}/approve
(drives broker, then BIRTHS the inventory record — identity from the returned
serial/hostnqn/context, binding from the operator's optional body, enrolled
with the HUMAN approver = the manager's admin mTLS CN), POST .../deny.
503 when the broker isn't wired.
Test: httptest mock broker asserts the Accept:application/json approve, pending
proxy, inventory birth (identity+binding+enrolled), and the disabled-503 path.
Broker + manager both build/vet/test/gofmt clean.
The steady-state host<->manager transport, resolving the design's OPEN FORK to
bespoke gRPC (not OTLP+Wazuh). One long-lived, host-dialed, mTLS bidi stream:
health/metrics up, desired-state/commands down — no polling, real-time control,
still no inbound host port.
- proto/fleet_agent.proto: FleetAgent.Session(stream AgentUp) -> stream ManagerDown.
AgentUp = Hello | HealthSnapshot | CommandResult; ManagerDown = DesiredState |
Command(reboot/drain/resync). int64 unix times (no well-known-type dep).
Generated stubs committed (CI needs no protoc).
- agent.go: gRPC server with mTLS verified by the FLEET CA (the host's reused TPM
identity cert — distinct from the admin plane's admin CA). Identity is the peer
cert SERIAL, taken from the verified cert = inventory key, never self-asserted;
an unenrolled serial is refused (enrolment gates the stream). Per-conn send
channel decouples admin pushes from stream.Send. Stream registry (serial->conn)
with reconnect eviction. On connect + on change, desired-state is pushed live;
inventory stays authoritative so offline hosts reconcile on reconnect. Metrics
go through a MetricsSink seam (no-op now; InfluxDB in ckpt 4).
- main.go: agent plane starts only when its cert/key/CA are set; admin can run
standalone. Lifecycle PUT now live-pushes and reports pushed_live.
Full mTLS integration test (in-process fleet CA + host client cert): initial
desired-state on connect, Hello+snapshot -> inventory + online presence, live
desired-state push, and unenrolled-serial rejection. go build/test/vet/gofmt clean.
Deps (server-side only): google.golang.org/grpc + protobuf.
The checkpoint-1 commit included the 9.1M compiled 'fleet-manager' binary. Remove
it from tracking and gitignore it (/fleet-manager + *.test), matching the
enrol-broker convention. Build artifacts don't belong in git.
First slice of the server-side fleet control plane (serving/fleet-manager/), the
backend for the admin SPA (monitor / authorise / lifecycle). See
docs/fleet-management-design.md + tpm-enrolment-design.md §5.
This checkpoint (stdlib-only):
- Inventory record (identity/binding/enrolled/version/health/usage/desired),
keyed by TPM cert serial = hostnqn key, per the design's 'inventory record born
at enrolment'. JSON-file Store (atomic writes, mutex) behind an interface so a
DB swap is a drop-in.
- Admin plane: mTLS server verified by the DEDICATED admin CA (never the fleet CA,
so a host cert can't reach admin) + a plaintext dev mode. JSON API:
GET /api/hosts, GET /api/hosts/{serial}, PUT /api/hosts/{serial}/desired
(desired-state audited to the admin client-cert CN).
- Dev seed (FLEET_MGR_SEED) so the SPA has data before real enrolment/agent flow.
Smoke-tested: list/get/desired-PUT (incl. version-drift + drain), 404s, atomic
persistence. go build + go vet + gofmt clean.
Agent gRPC plane, broker-fold-in authorise, InfluxDB metrics, and the themed SPA
follow in checkpoints 2-6.
Bench pass on fapolicyd-tmpfs-12043d3 / comet-beacon-54: permissive=0, watch_fs has
overlay, / and /tmp both overlayfs; novel-hash /tmp binary DENIED (FANOTIFY resp=2,
obj_trust=0, exit=126); empty deny sweep = zero false denials (kiosk/weston/chromium/
NM-dispatcher all trusted, admin SSH+sudo work under staff_t). The exec guard went
from effectively OFF to enforcing on the whole overlay root, nothing legit broken.
Also records the scope caveat: integrity=none path-trust closes the UNTRUSTED-PATH
half (proven) but not the MODIFIED-TRUSTED-PATH half — an overlayfs copy-up of a
trusted path (e.g. /usr/bin/bash) lands a same-path tmpfs copy outside dm-verity that
path-trust still trusts. Closing it needs integrity=sha256 or an un-shadowable /usr
(TODO #15, preferred).
Combines the image trim (#18: langpack/firmware slim, kiosk weak-deps, nss-mdns
KEEP+mask-avahi fix, final kiosk-stack build guard) with optimise-ram's overlay-RAM
work (#17 fapolicyd db_max_size cap, .pyc unchecked-hash recompile).
Conflict resolutions:
- diskless.pkr.hcl (2d hygiene tail): kept BOTH additions — the .pyc recompile
runs first (the 'must run last after every dnf' mutation), then the read-only
FINAL GUARD (rpm -q kiosk stack) as the absolute last step. The nss-mdns fix
(mask avahi, keep the package) auto-merged above the conflict — no regression.
- docs/TODO.md: kept both #17 (fapolicyd db_max_size) and #18 (image trim), as the
#18 stack-note predicted.
The '#18 render should be safe' call on the nss-mdns drop (5cc72cc) was wrong: it
bricked the kiosk on trim-rpm-48dc3e1 (chromium hard-Requires nss-mdns, so the 2d
remove cascaded chromium + its GUI stack out → 500 MB browserless image, flashing
cursor). Fixed in 2bc2daf (keep nss-mdns, mask avahi daemon; final build guard
asserts the kiosk stack survived). Re-verified booting into the kiosk on the 820;
container image measured 1.8 GiB → 755 MiB (~58%).
Root cause of the trim-rpm-48dc3e1 'flashing cursor, no kiosk' + mysteriously
small (~500 MB) image: the 2d hygiene step ran 'dnf remove -y nss-mdns', but the
EPEL chromium package HARD-Requires nss-mdns (Chromium uses .local mDNS hostnames
for WebRTC ICE candidates). So dnf removed chromium as a dependent package and
then reaped its orphaned GUI stack (avahi, at-spi2, atk, adwaita, chromium-common
...) — a ~300 MB drop. The shipped image had no browser, so kiosk-session died at
the ${BROWSER:?} guard before ever starting weston: a bare KMS cursor.
The earlier nss-mdns commit checked what nss-mdns NEEDS (avahi) but not what NEEDS
nss-mdns (chromium).
Fix:
- Keep nss-mdns + avahi (satisfy chromium's Requires); instead MASK the avahi
mDNS daemon/socket so nothing listens on :5353 — that was the real
attack-surface concern, and nss-mdns is only an NSS module (no daemon needed).
- Add a FINAL GUARD at the end of 2d: assert chromium/chromium-common/weston/mesa
are still installed in the final image, else fail the BUILD. A missing browser
must brick the build, never the fleet. Catches any future silent removal
cascade, oscap package_*_removed rule, or autoremove.
Bench-verified the getattr allow (620a604) alone did NOT close the exec gap: with
the fapolicyd_tmpfs module installed (AVC gone, fapolicyd active, permissive=0), a
NOVEL non-RPM binary in /tmp still ran. Root cause: the diskless root is overlayfs
(stat -f / -> overlayfs), but the shipped watch_fs is
ext2,ext3,ext4,tmpfs,xfs,vfat,iso9660,btrfs — no 'overlay'. fapolicyd marks mounts
by fs type, so it never marked the overlay and intercepted no exec on / at all.
The exec guard we keep fapolicyd for was effectively OFF at runtime on every
diskless host.
Fix: the 2c fapolicyd step now appends 'overlay' to watch_fs (idempotent) so
fapolicyd watches the overlay root. Trust is path-based (integrity=none) so overlay
paths resolve against the rpmdb trust normally; the fapolicyd_tmpfs .te still
supports fapolicyd statfs'ing the overlay's tmpfs upper. Deviation #15 updated.
This turns app-allowlisting ON for the whole root at runtime for the first time.
BENCH-VERIFY HARD: boot + kiosk render + admin SSH + sudo + TPM enrolment all still
work (nothing legit untrusted), a novel /tmp binary is now BLOCKED (EPERM), no new
AVCs. enable_stig=false is the escape hatch if it regresses.
0bfe872 builds, deploys, and the host now mounts ONLY its own version's image
instead of connect-all'ing every served version — the targeted 'nvme connect -n
rd.nvmf.nqn' works. Each host = 1 controller on its own subsystem, which is what
lets old versions drain to 0 and spdk-loader auto-reap. Still to confirm: the reap
payoff (migrate a host, watch its old version drain + reap), and the secure path's
new step-6 connect over macsec0.
853ae20's diagnostic did 'awk … | sort -rn | head -25'. head closes the pipe
after 25 lines, sort dies of SIGPIPE (141), and step 2's set -euxo pipefail turns
that into a build failure (the exact trap the lsinitrd '| grep -q' comment warns
about). The kiosk step's rpm-manifest '| head' is safe only because that step
runs plain set -e, no pipefail.
Fix with the codebase convention: sort into a file, then head the file — no live
pipe for head to break. (Bonus: the crash log confirms omit_drivers worked —
amdgpu/radeon/nouveau kmods are gone from the top-25; the new biggest initramfs
item is GenuineIntel.bin CPU microcode at 22 MB, a future trim candidate.)
The dry-run failed with HTTP 403 on the branches API: PACKAGE_TOKEN is a
package-scoped token (registry push/delete) and can't read repo branches. Split
by capability: branch reads now use the automatic Actions token (github.token,
repo read via 'permissions: contents: read'); package list + delete keep
PACKAGE_TOKEN. Also replaced the bare 'curl -f' failures with a get() helper that
reports the HTTP code + a token-scope hint, so an auth problem can't look like
'no results' (which would otherwise skew the reconcile).
Reconciliation workflow (schedule + workflow_dispatch): lists live branches and
every container version across the four images this repo builds (fleet-boot,
fleet-ipxe, fleet-dnsmasq, spdk-nvmf) and deletes dev/branch builds whose branch
no longer exists, via the Gitea Packages API. Not event-driven, so it catches
branches deleted any way and never depends on a 'delete' webhook.
Safety (per request — never touch release tags or latest):
- Only tags ending in the dev commit-sha suffix '-<7 hex>' are ever candidates.
Release/version tags (1.2.3, v1.2.3, spdk v24.09, dnsmasq main-<sha>) and
'latest' never carry that suffix, so they're structurally out of scope; there's
also an explicit 'latest' skip. Verified by simulation across representative
tags incl. spdk's <ver>-<branch>-<sha> form and pre-release semver.
- Keeps any image whose sanitized branch is still live (+ always 'main'); branch
sanitization mirrors the build workflows.
- Fail-safe: aborts before deleting anything if the live-branch list can't be
fetched (0 branches), so an API/auth blip can't trigger mass deletion.
- Conservative matching: a suffix collision only ever causes a KEEP.
DRY RUN by default: workflow_dispatch previews unless dry_run=false; scheduled
runs dry-run until armed via repo variable REGISTRY_GC_ENABLE=true (so a
scheduled dry-run can be eyeballed first). Uses the existing PACKAGE_TOKEN
(write:package). shell: bash (runner defaults to sh).
fapolicyd is the app-allowlisting exec guard for the writable RAM overlay
(dm-verity covers only the read-only erofs lower). It statfs's mounts to decide
which to watch, and tmpfs is in its default watch_fs. This image's root is
systemd.volatile=overlay, so statfs("/") returns the overlay upper — a tmpfs —
and the shipped policy has no 'fapolicyd_t tmpfs_t:filesystem getattr' (no
ordinary host roots on tmpfs). Under enforcing the getattr is DENIED (audit:
dev=tmpfs ino=1 name="/", permissive=0, seen 2026-07-03 on comet-beacon-54),
so fapolicyd can't fully mark the tmpfs mounts (/, /tmp, /dev/shm) and its guard
over the writable overlay is incomplete — a real exec-guard gap, not just SIEM
noise.
Ship a minimal local allow module (allow, not dontaudit — we want fapolicyd to
see the tmpfs), compiled + semodule -i'd in 2c-STIG right after the chrony one.
Registered as deviation #15.
BENCH-VERIFY: the AVC is gone on an enforcing boot AND fapolicyd blocks a non-RPM
executable dropped in /tmp (proves the guard now covers the overlay), trusted
kiosk/appliance binaries still run, and the verity hash is reproducible across
two builds (semodule rebuilds the policy store).
f59eb79 now fully bench-verified: the fleet SPA renders on both the VM
(virtio-gpu) and the 820 (i915), so the i915 DMC/GuC firmware loads on real
hardware and the firmware keep-list is validated end-to-end. The two later
commits (nss-mdns drop, omit_drivers + keep-list tightening) don't touch i915,
so render should be safe, but still need their own build+boot.
The ~800 MB trim (glibc langpacks en-only + linux-firmware strip to the i915
keep-list, run pre-dracut) builds and boots — the boot/root path survives it
(firmware 678->147 MB, initramfs bundles only the keep-list). Notes still-open
checks: kiosk RENDER on real i915 hardware ('booted' != 'display up'), and that
the two later commits (nss-mdns drop 5cc72cc; omit_drivers + keep-list tightening
853ae20) are NOT in f59eb79 and need separate bench-verify.
First trim build confirmed firmware 678 MB -> 147 MB, but the initramfs came
out 97 MB and is now kernel-MODULE-bound, not firmware: --no-hostonly +
plymouth's drm dep instmods every GPU driver (amdgpu.ko etc.) though the fleet
is i915-only.
- omit_drivers += amdgpu radeon nouveau (90-diskless.conf): drop the discrete-GPU
kmods (amdgpu.ko is the biggest single module in the initramfs); keep i915 +
virtio-gpu/bochs for the splash. Shrinks the initramfs/UKI pulled over PXE on
every boot.
- Firmware keep-list tightened: drop mellanox/ (mlxsw_spectrum SWITCH-ASIC blobs,
not NIC fw — mlx5 NIC fw is on-card) and narrow intel/ to just ice (drop
ish/sof/ibt sensor-hub/audio/bluetooth, none loaded at boot per the audit).
- Add an 'initramfs largest files' diagnostic (sort lsinitrd.txt by size) so the
next build shows what dominates the .img.
BENCH-VERIFY: kiosk still renders on the 820 (i915 kept); watch the new initramfs
largest-files log line for the post-omit size.
Hosts connected with 'nvme connect-all', which attaches EVERY subsystem in the
discovery log. Each host then mounts only its own by-UUID verity namespaces but
leaves a live controller on every other served version's subsystem. Observed via
spdk-release list: more controllers than workstations, each box on every version
(1/3/3/1/3 across 5 versions for ~3 boxes). Beyond wasted connections
(O(hosts x versions)), it defeated the rolling-update drain/reap: no version ever
reached 0 connections while any host was up, so spdk-loader's auto-reap could
never fire and old versions accumulated without bound.
Fix — targeted connect to this build's subsystem only:
- extract-artifacts.sh bakes the per-version SUBNQN (already in verity.json
.subnqn, matched by the serving side) into the signed UKI cmdline as
rd.nvmf.nqn=. NQN is version identity, not per-host state, and the cmdline is
already per-version + immutable-per-signed-UKI, so invariant #1 holds.
- nvmf-connect.sh (test path) uses 'nvme connect -n <nqn>'; legacy UKIs without
rd.nvmf.nqn fall back to connect-all with a warning so they still boot.
- dot1x-macsec-up.sh (secure path) gains an explicit step-6 targeted connect over
macsec0 (after it's addressed, per invariant #2), replacing the delegation to
the dracut nvmf module's discovery connect-all.
Now each host = 1 controller on its own subsystem; versions drain to 0 when their
hosts migrate, so auto-reap works and connections scale O(hosts). Requires a
rebuild + re-sign (cmdline change, invariant #6). BENCH-VERIFY both paths;
confirm the dracut nvmf module isn't also connect-all-ing on the secure path.
CLAUDE.md invariant #2 + TODO #6 updated.
Investigating avahi/samba/rabbitmq in the image: ldd on libavformat.so confirms
samba (libsmbclient), librabbitmq, librist, libsrt and libzmq are all DT_NEEDED
by FFmpeg's libavformat-free — removing any breaks libavformat and thus future
HTML5 media in the kiosk, so they stay. avahi is the exception: it's absent from
libavformat's links and 'rpm -e --test avahi' shows its only reverse-dep is
nss-mdns (the mDNS/.local NSS resolver).
The fleet resolves via DHCP/DNS (boot.fleet.lab), never .local, so mDNS is dead
weight and a network-listening service unwanted on a locked-down kiosk. Remove
nss-mdns; dnf's clean_requirements_on_remove reaps avahi/avahi-glib/avahi-libs as
orphans. Belt-and-braces alongside install_weak_deps=False (which already drops
nss-mdns if it came in as a Recommends).
Follow-up to the linux-firmware strip (prev commit put it in the 2d hygiene
block, which runs AFTER the dracut build — so it only shrank the rootfs erofs,
not the initramfs). Move the strip to step 2, before the dracut build: with
--no-hostonly, plymouth's drm dep instmods a broad DRM set and dracut bundles
each module's firmware into the initramfs (amdgpu being the classic UKI
bloater). Trimming first means dracut never sees it, so both the rootfs erofs
AND the UKI initramfs shrink. Keep-list is unchanged (i915 + intel/ + mellanox
+ intel-ucode), so the fleet splash GPU (i915) and ice DDP firmware the
initramfs actually needs are retained; dracut just warns 'firmware not found'
for the removed drm modules and continues.
Add a post-build diagnostic (initramfs .img size + bundled firmware, expected
i915 only) so the initrd win is measured in the build log, not estimated.
The 2b rpm-size manifest revealed the biggest packages aren't the kiosk but
base bloat: linux-firmware 678 MB and glibc-all-langpacks 228 MB (vs chromium
370 + llvm 129). The uncompressed erofs is read over NVMe/TCP on every boot,
so this is fleet-wide dead weight + attack surface.
glibc: swap glibc-all-langpacks (every language) for glibc-minimal-langpack +
glibc-langpack-en. ~220 MB; image is English-only.
linux-firmware: a live fleet audit (modinfo -F firmware over loaded modules +
sysfs/lspci driver bindings on the real 820-class hosts and the VMs) proved the
only firmware any host loads is Intel i915 (GPU). Every NIC is firmware-free
(e1000e reports on-chip NVM, not a fw file; virtio_net loads nothing), so the
catastrophic missing-NIC-firmware case is off the table. Keep i915 + intel/
(ice DDP insurance) + mellanox + intel-ucode; strip the ~600 MB of foreign
vendor blobs (amdgpu/nvidia/radeon/ath/qcom/...). Remove iwl*-firmware (Wi-Fi,
~83 MB) on a wired-only fleet. Strip-in-place since the package can't be
removed without losing i915; du reports the result in the build log.
BENCH-VERIFY: kiosk still renders on the 820 (i915 DMC/GuC load) — the one real
risk; widen the keep-list if any host has a non-Intel GPU or an ice/mlx5 NIC.
Test-path enrolment (rd.dot1x=0) succeeded end-to-end on the Dell 820 — the
hard case that previously found the un-clearable OEM platform NV at 0x01800001
and silently bypassed enrolment every boot. Confirms BOTH the NV-index move
(0x01800001 -> 0x01800100) and the signing-key handle move (EK slot -> 0x81000100)
on a physical TPM, not just the Proxmox swtpm: ensure ECDSA key -> CSR -> broker
fingerprint -> approve -> leaf cert -> NV write all run on the metal.
Mark #11 in-progress ([~]). Still to verify: the enrolled cert authenticating
EAP-TLS on the secure 802.1X/MACsec path (this run was test-mode), and
tpm2_nvreadpublic on other fleet hardware models to confirm 0x01800100 is free
fleet-wide.
Two fixes from the booted-host overlay RAM audit (2026-07-03). The writable
root overlay (systemd.volatile=overlay, tmpfs) held 124.5 MiB of runtime
writes; almost all of it is runtime-regenerated state, not real fs deltas.
1. fapolicyd db_max_size 100 -> 50 (TODO #17). /var/lib/fapolicyd/data.mdb was
100 MiB fully allocated (~80% of the overlay). fapolicyd's LMDB trust DB is
rebuilt at boot onto the overlay, so db_max_size is a fixed RAM reservation.
Sed it to 50 before the fapolicyd-cli --update rebuild. 50 was fapolicyd's
long-time upstream default (covered a full workstation rpmdb), so it has
headroom for this smaller image. Reclaims ~50 MiB.
BENCH-VERIFY (STIG-critical exec guard): boot + kiosk + admin SSH + sudo +
custom-binary exec must work (undersize -> MDB_MAP_FULL bricks exec).
2. .pyc invalidation fix (TODO #16 item 2). erofs -T 0 zeroes mtimes, so
Python thinks every baked .pyc is stale and rewrites it into the overlay at
runtime (wasting RAM + throwing lib_t write/unlink AVCs that flood the SIEM).
Observed trigger: dnf-makecache.timer (~10 min post-boot) re-importing the
dnf Python stack. Fix both ends: (a) compileall --invalidation-mode
unchecked-hash over /usr/lib*/python3.* at the end of 2d (mtime-independent
.pyc, no rewrite, also more reproducible); (b) mask dnf-makecache.timer next
to the existing rhsmcertd mask (removes the trigger). Firewalld enablement
(#16 items 1 & 3) still open.
The kiosk dnf transaction was measured at 229 pkgs / 1.0 GB. Nearly all
of the non-core packages are Recommends: a locked-down single-URL kiosk
never uses flatpak + its ostree/bubblewrap sandbox runtime, tracker-miners
(a filesystem indexer — dead weight and a threat surface), the pipewire/
pulseaudio audio stack, or libproxy-webkitgtk4 -> webkit2gtk3-jsc (an 8.9M
second browser engine dragged into a Chromium kiosk for libproxy's PAC
runner).
Add --setopt=install_weak_deps=False to cut them, and an rpm -qa size
manifest to the build log so the next CI build reports the exact new
footprint (turning 'seems unnecessary' into data for the next trim round:
gstreamer/media codecs, or mesa-dri-drivers if the GPU path is abandoned).
The immovable floor is chromium + chromium-common + llvm-libs (mesa sw-GL)
+ gtk3/adwaita + libicu, all hard deps. BENCH-VERIFY the kiosk still
renders with weak deps off (watch for a missing font or libva).
The build failed: checkmodule rejected `policy_module(chrony_dhcp_dispatcher,
1.0.0)` with "Building a policy module, but no module specification found".
policy_module() is a refpolicy m4 macro expanded only by the refpolicy Makefile
(/usr/share/selinux/devel), not by checkmodule. Use the raw module-policy
declaration `module chrony_dhcp_dispatcher 1.0.0;`, which is exactly what
checkmodule -M -m parses. require{} + allow rule unchanged.
The Dell 820 kept "finding a cert" and bypassing enrolment despite repeated BIOS
TPM clears. tpm2_nvreadpublic revealed 0x01800001 on that hardware is a FIRMWARE
platform index (platformcreate|policydelete|writelocked, 70 bytes — not our
cert; siblings at 0x01800003/4 + 0x01810000). platformcreate survives TPM2_Clear
and policydelete needs the platform's secret policy, so it's permanent and
unusable by us — the blind check false-positived on it, and enrolment could
never write there anyway. Same class of bug as the EK-handle collision.
Move the NV cert index default 0x01800001 -> 0x01800100 (free on the 820, owner
range, overridable via rd.dot1x.nvindex): cmdline (extract-artifacts.sh),
fleet-enrol -nvindex, the dracut hooks, bench scripts, CLAUDE.md invariant #1,
and design/SSP docs. The signing-key handle (0x81000100) is unchanged.
Harden WriteNVCert: refuse a platformcreate index with a clear "pick a free
owner index via rd.dot1x.nvindex" error instead of a confusing undefine
failure. Regression simtest/TestWriteNVCertRefusesPlatformIndex; NV roundtrip +
ReadNVCert tests moved to 0x01800100. All go tests + packer validate pass.
Captures the decision to re-enable a CONFIGURED firewalld (currently removed),
the required .pyc/-T0 root-cause fix (compileall --unchecked-hash), the 4 STIG
waivers it converts, and the load-bearing NVMe/TCP-survives-firewalld bench gate.
http/ks.cfg excludes firewalld (-firewalld) and sets firewall --disabled — the
fleet's security boundary is the network (802.1X/MACsec + switch); the host
firewall is a documented OFF item (stig-deviations #4). But a provisioner dnf
pulled firewalld back as a weak dep (Recommends) and it starts by default. Its
Python then tries to rewrite site-package .pyc bytecode, which the reproducible
erofs -T 0 mtime reset makes look stale, so firewalld_t gets write/unlink AVCs
on lib_t that flood the SIEM.
Remove firewalld in 2d hygiene (weak dep, nothing hard-needs it) and mask it as
a backstop against future weak-dep reinstalls. No firewalld => no daemon, no
AVC, matches the kickstart intent.
The NM dispatcher 10-chrony-dhcp-ntp runs `chronyc reload sources` to load DHCP
option-42 NTP servers. That chronyc runs in the confined custom-dispatcher
domain (NetworkManager_dispatcher_custom_t), so its reply socket carries that
label and the shipped policy has no rule for chronyd_t to sendto it. Under
enforcing, chronyd does the reload but its ack is denied — chronyc reports
failure and the AVC floods the SIEM (seen 2026-07-03; permissive=1 only because
the box was booted enforcing=0).
Ship a minimal local allow module (files/stig/chrony-dhcp-dispatcher.te):
allow chronyd_t NetworkManager_dispatcher_custom_t:unix_dgram_socket sendto;
Built + semodule -i'd in the 2c-STIG provisioner (checkpolicy installed if
absent), with an install assertion. `allow` not `dontaudit` — it also lets the
reload ack succeed so chronyc returns cleanly. Documented as
docs/stig-deviations.md #14 (not a rule waiver — a local functional allow).
BENCH-VERIFY: enforcing boot shows no AVC, chronyc reload returns 0, a
DHCP-handed NTP server appears in `chronyc sources`; and re-check verity-hash
reproducibility (semodule rebuilds the policy store, like the semanage step).
Audiobuzz