NTS authenticated time is now bench-verified end-to-end unattended on a clean
build (comet-beacon-54, 5294860): chronyc -N authdata shows ntp.fleet.lab in NTS
mode with cookies > 0, sources read under enforcing SELinux, no lo.sources, no
manual step. This clears the two dispatcher-integration defects found on the way
(distro 20-chrony-dhcp clobber; /run/chrony-dhcp SELinux label).
Flips F-03/P-18 / R-17 from "bench-verify pending" to bench-verified in the SSP
status line + register and the Annex §2.9 control row + R-17 register + summary,
mirroring the F-02 treatment. Rated Low. Residual (defence-in-depth, not blocking):
issue the per-environment NTS server cert, and exercise the cold-clock
nocerttimecheck bootstrap + active-spoof rejection.
Second bench defect after removing the distro 20-chrony-dhcp clobberer
(secure-ntp-5a3eaed, comet-beacon-54): the dispatcher wrote the correct
`server ntp.fleet.lab iburst nts` line, but chrony loaded ZERO sources --
`chronyc reload sources` returned 200 OK yet authdata/sources stayed empty.
Root cause (AVC): chronyd_t was DENIED read on the .sources files because they and
/run/chrony-dhcp itself were labeled generic var_run_t; chronyd_t may only read its
own chronyd_var_run_t. The dispatcher's `mkdir -p` creates the dir as var_run_t and
files inherit it. The distro 20-chrony-dhcp we removed had been restorecon-ing them
to the readable label, so removing it exposed a latent labelling gap.
Stock RHEL policy already maps /run/chrony-dhcp -> chronyd_var_run_t (a restorecon
relabels it and chrony immediately reaches NTS Cook 8). Fix: ship
/etc/tmpfiles.d/chrony-dhcp.conf so systemd-tmpfiles pre-creates the dir with that
fcontext at sysinit, before any NM event -- the dispatcher's files then inherit
chronyd_var_run_t and chronyd can read them. No SELinux allow rules needed
(bench-verified: after the fix a real NM event wrote the source as chronyd_var_run_t,
no AVC, authdata -> NTS Cook 8). Applies in both secure_ntp modes.
Also: the dispatcher now skips `lo` -- it was writing a spurious lo.sources
duplicate of the same server (the distro script had skipped loopback).
BENCH-VERIFY on the next clean build: a normal boot leaves the source file labeled
chronyd_var_run_t (ls -Z), no lo.sources, and authdata reaches NTS/cookies>0 with no
manual step.
NTS was inactive on the clean-build deploy (secure-ntp-d1621e9, comet-beacon-54):
chronyc authdata showed a plain IP source in Mode '-', Cook 0, with no NTS-KE
attempt at all -- chrony was on plain NTP.
Root cause: NetworkManager runs dispatcher scripts from BOTH
/etc/NetworkManager/dispatcher.d and /usr/lib/NetworkManager/dispatcher.d, merged
and sorted by filename. The chrony RPM ships
/usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp, which writes the DHCP
option-42 server to the SAME /run/chrony-dhcp/<iface>.sources file our
10-chrony-dhcp-ntp uses. Sorting after 10-, it ran last and overwrote our
`server ntp.fleet.lab iburst nts` line with `server 192.168.50.10 iburst`, then
reloaded chrony -- last writer wins on the shared filename, silently defeating
authenticated time.
Isolation proof: removing the distro script + hand-writing the NTS source brought
NTS up immediately (ntp.fleet.lab NTS, Cook 8, zero NAKs), confirming the host
image, the fleet-signed NTS-KE cert, DNS, and tcp/4460 were all already correct --
the collision was the sole defect.
Fix: the provisioner removes /usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp
at build (our dispatcher supersedes it in both modes); 20-chrony-onoffline is left
in place (harmless). Documents the finding in the dispatcher header,
docs/secure-ntp-design.md, and the CLAUDE.md secure_ntp note. BENCH-VERIFY on the
next clean build: a normal boot reaches NTS/cookies>0 with no manual step.
F-02/P-17 SSH host certs re-verified on the rebuilt 28d31e4 image (not the
hand-fixed host): the auto-derived FQDN principal is baked, so by-FQDN no-TOFU
admin login survives a reboot with zero manual steps. Cite the digest in the
design-doc status + bench-outcome, SSP v0.4 status line + R-16 row, and Annex A
2.4 row + R-16 register row.
NTS proven end-to-end on comet-beacon-54 (chronyc authdata cookies > 0); these
close the gaps the bench found so a clean build/serving run reaches that state
without manual patching:
- dispatcher (10-chrony-dhcp-ntp): fall back to resolv.conf search/domain when NM
doesn't export DHCP4_DOMAIN_NAME on the firing event, so the NTS source is
written reliably at boot (tonight it needed a manual dispatcher run).
- STIG vs secure_ntp (P2, security): STIG's chronyd_specify_remote_server re-injects
an UNAUTHENTICATED 'server 0.us.pool.ntp.mil maxpoll 16' AFTER the secure_ntp
chrony step commented servers out -> a silent fallback that defeats NTS's
DoS-only guarantee (host not fail-closed). Unselect chronyd_specify_remote_server
+ chronyd_or_ntpd_set_maxpoll in diskless-tailoring.xml, and re-strip any
server/pool in 2c-STIG-postfix when secure_ntp=true. Documented: stig-deviations #17.
- nts-cert.sh (P3): the container's chrony user (non-root) can't read a 0640
host-owned key -> NTS-KE TLS aborts ('non-properly terminated'). chown the key to
the ntp container's chrony uid + keep 0640 (0644 works but chrony warns
world-readable); 0644 fallback if the uid can't be found.
Bench-verified 2026-07-11; docs updated in docs/ssh-host-certificate-design.md's
sibling secure-ntp-design.md flow. F-03/P-18 stays open until a clean build shows
chrony.conf has no plain server and authdata cookies>0 as the only source.
F-02/P-17 (SSH host certificates, R-16) bench-verified end-to-end on
comet-beacon-54: admin login by FQDN with no TOFU, per-boot auto-re-sign across
reboot, SELinux confinement clean. Updates the SSP v0.4 status line + R-16
summary row and Annex A version note, the 2.4 host-key-trust row, the R-16
register row (treatment -> roll anchor + broker-side principal validation TODO)
and the 3.1 narrative. F-03/P-18 (NTS, R-17) stays bench-verify pending -- not
yet tested. Bench-surfaced code fixes are on branch poam-hardening (28d31e4).
F-02/P-17 SSH host-cert flow was proven end-to-end on comet-beacon-54
(2026-07-11); these fix the correctness/scoping/cleanup defects the bench
surfaced:
- ssh-hostcert: derive the FQDN principal <hostname>.<domain> from the DHCP
search domain at runtime (resolv.conf), alongside the short hostname, so an
admin `@cert-authority *.<domain>` line accepts by-FQDN logins instead of
"name is not a listed principal". Never baked (invariant #1). + tests.
- ssh-ca-bootstrap.sh: export the SSH host CA public key in OpenSSH wire format
(ssh-keygen -i -m PKCS8), not the PEM/PKCS8 that `step crypto keypair` writes
-- a PEM @cert-authority anchor is unparseable and silently falls back to TOFU.
- ssh-ca-bootstrap.sh: fail loudly (and verify enableSSHCA=true) if the
provisioner SSH-enable step fails, instead of swallowing it and shipping a CA
that 502s every runtime sign.
- dot1x-bench: point the co-located broker at https://localhost:8443, not the
fleet dnsmasq name ca.fleet.lab (unresolvable from the broker's host netns ->
"no such host" at sign time). localhost is in the CA cert SAN.
- serving: remove the mock-signer enrol-broker copy -- two host-networked
brokers collide on 9090/9443 and the mock issued untrusted certs. The real
broker lives with step-ca (dot1x-bench); nginx /enrol/ still proxies to it on
loopback.
- enrol-broker: retire the HTML admin console; the fleet-manager passkey SPA now
drives the mTLS JSON API (/pending, /approve|deny). Removes a weaker,
client-cert-only parallel human-approve path.
Design-doc bench note deferred: docs/ssh-host-certificate-design.md lives on the
secure-ntp branch (fbb8100), not here -- to be added there / on reconcile.
The SSP (v0.2, 2026-06-27) and annex predated the entire management/enrolment
plane and the July hardening work. Bring both to v0.3 (2026-07-10):
SSP:
- Scope/boundary: on-host fleet-agent now IN scope; server-side manager/broker/
step-ca/InfluxDB named as connected systems; new host interface (g) = outbound
mTLS to the management plane. Boot chain gains step 8 (post-switch-root agent).
- Data: management telemetry (operational metadata, mTLS-encrypted, no per-host
secret); the agent stream is the one non-plaintext transit path.
- §5.2 Protect: agent reuses the TPM identity for mTLS; WebAuthn-passkey admin
MFA (first human-admin MFA); 3 local SELinux modules; the 2026-07-04 fapolicyd
overlay watch_fs fix (app-control was inactive on the root until then).
- §5.3 Detect: operational health telemetry signal (distinct from auditd/Wazuh).
- §6 Essential Eight: MFA + admin rows updated (passkeys now enforced on mgmt UI).
- §7: new R-14 (mgmt plane = connected system pending authorisation) + R-15
(deferred agent actuators = future remote-mutation channel); R-05 MFA clause met.
Annex:
- §2.3 hardening: fapolicyd overlay-effectiveness fix + residual; 3 local SELinux
allow modules on the MAC row.
- §2.4 auth: agent mTLS on the machine-auth row; MFA row now Implemented for the
mgmt UI (passkeys), Partial for host SSH.
- §2.5 mgmt: new rows for TPM enrolment lifecycle (broker) and server-authoritative
management plane. §2.6: operational health-monitoring row. §2.7: mgmt-plane
segmentation (OPNsense L3, outbound-only). §2.9: agent mTLS, fleet CA (step-ca),
cert-lifecycle auto-renewal.
- §3 register: R-14, R-15 added; R-05 updated; heatmap + note + 3 new assessor
notes (scope the mgmt plane, witness fapolicyd deny, confirm actuators inert).
Recent work added telemetry fields and a cert path not yet in the docs:
- files/agent/README.md: add uptime to the health list; document that chrony is
read from /var/log/chrony/tracking.log (not the command socket) and needs BOTH
the chrony supplementary group (DAC) and the fleet_agent_chronylog SELinux allow
(TE), plus a reachable lab NTP (123/udp firewall); note the chrony group on the
unit row.
- serving/fleet-manager/README.md: add uptime + chrony detail to the health list;
note the cert-auto agent-cert-renewer sidecar + GetCertificate hot-reload
(certreload.go) in the bring-up and checkpoint 7.
- docs/TODO.md #14: probe list gains uptime + chrony-from-tracking.log; new dated
block records uptime/reboot-pending, the 4-layer chrony-access chase (all
silent-failing; inventory.json is the isolator), and the agent-plane cert
auto-renewal, all bench-verified 2026-07-10.
- docs/fleet-management-design.md: health record gains uptime + chrony
source/stratum/offset detail.
The tracking.log read was still returning empty under enforcing with NO AVC —
the tell of a DAC (not SELinux) denial. Root cause: /var/log/chrony is mode
0750 chrony:chrony (others have no access), so the agent's DynamicUser (group
tss only) can't even traverse the directory to reach the 0644 tracking.log —
EACCES on open, which produces no audit record. Manager confirmed genuinely on
fleet-manager-1db3977 (chrony-capable), and uptime flowed while chrony didn't,
proving the agent was sending empty chrony (failed read), not a manager gap.
Earlier manual reads worked only as root (sudo) or because fleetadmin had been
usermod'd into chrony during the socket test (reverted on reboot — /etc/group
is on the volatile overlay).
Fix: SupplementaryGroups=tss chrony on the unit — the chrony group has r-x on
the 0750 dir so the agent can traverse it; the log file is world-readable. This
is the DAC half; the fleet_agent_chronylog SELinux module (02a6621) is the TE
half — BOTH are required (verified: group-only -> TE-denied; module-only ->
DAC-denied silently). Corrected the now-stale 'no group' code comment; recorded
the two-layer requirement in the .te and stig-deviations #16. Needs a
workstation image rebuild + reboot.
The tracking.log approach (4ca0b6d) works in permissive but is blocked under
enforcing: chrony's log is labeled chronyd_var_log_t, confined by the shipped
policy to chronyd_t despite the 0644 mode. Verified on comet-beacon-54 — a
non-root read succeeds permissive, but under enforcing even a STIG-confined
admin (staff_t) gets Permission denied (ls -Z on the file is itself blocked).
The agent runs unconfined_service_t (confirmed via ps -eZ) and is denied the
same way.
Ship a minimal read-only allow module (files/stig/fleet-agent-chronylog.te):
allow unconfined_service_t chronyd_var_log_t:file { open read getattr } + dir
search to reach it. No write, no other chrony access, and no domain transition
(the agent reads a file; it does not exec chronyc) — narrower than the
abandoned nnp-transition module. Built + semodule -i'd in 2c-STIG alongside the
existing chrony_dhcp_dispatcher / fapolicyd_tmpfs modules. docs/stig-deviations
#16. Needs a workstation image rebuild + reboot; same policy-store-rebuild
reproducibility re-check as #14/#15.
The ntp service is host-networked, so (like the audit collector's 514/tcp)
Docker doesn't open the port — the serving host's own ufw/firewalld governs
123/udp. Without a rule, workstations are handed 192.168.50.10 as their NTP
server via DHCP opt 42 but their packets are dropped: chrony logs 'Added source
… / unreachable' and never syncs (seen on comet-beacon-54). Add the boot-LAN
firewall rule to the ntp service NOTE, mirroring the audit-collector README.
Bench on comet-beacon-54 proved every command-socket route is blocked for the
hardened agent: chronyd's unix socket is owner-write-only (0755, so even a
chrony-group member can't connect), and STIG closes the UDP command port
(cmdport 0). Rather than re-open a port or expose chrony's privileged socket,
enable chronyd's tracking log and have the agent READ it — no port, no socket,
no group, no SELinux domain transition, full hardening intact.
- chrony.conf: enable 'log tracking' (idempotent) so chronyd appends
source/stratum/offset to /var/log/chrony/tracking.log. Write volume is a ~130B
line per clock update (backing off to ~maxpoll when stable) and capped by the
chrony RPM's logrotate — negligible on the RAM overlay. cmdport 0 stays, so no
STIG deviation.
- agent: chronyTracking() now reads the last data line of tracking.log via
parseTrackingLog() (source f[2], stratum f[3], offset f[6], leap/sync f[7]);
replaces the chronyc exec + parseChrony. Unit test swapped accordingly.
- revert the two abandoned approaches: drop the bindcmdaddress line (394ad10)
and the fleet_agent_chronyc SELinux nnp_transition module + provisioner block
(e6fd4a2, .te deleted) — no chronyc exec means no transition to allow. Removed
stig-deviations #16 and #17 (both waivers no longer needed).
Needs a workstation image rebuild + reboot. BENCH-VERIFY: tracking.log is
world-readable so the DynamicUser agent can read it; under enforcing
unconfined_service_t can read chronyd_var_log_t (add a minimal file-read allow
if denied); and the offset sign matches chronyc on a synced host.
Root cause of the blank chrony fields (found on comet-beacon-54): chrony 4.x
(RHEL 9) defaults its command interface to the UNIX socket only
(/run/chrony/chronyd.sock), usable by root or the chrony group with a writable
/run/chrony. The fleet-agent unit is hardened — DynamicUser (group tss, not
chrony) + ProtectSystem=strict (/run read-only) — so it cannot open the unix
socket nor create chronyc's reply socket. Bench proof: normal-user 'chronyc
tracking' -> 506 Cannot talk to daemon; only sudo worked; 'chronyc -h 127.0.0.1'
failed and ss showed no UDP :323. So the probe always failed and every chrony_*
field stayed empty. SELinux permissive did NOT help — it was never (only) an
SELinux problem; the nnp_transition module (prev commit) is still needed for the
enforcing exec transition.
Fix: bind chronyd's command socket on 127.0.0.1 (bindcmdaddress in chrony.conf,
idempotent in the 2c provisioner) and point the probe at it (chronyc -n -h
127.0.0.1 tracking). UDP needs no fs write or group and rides the unit's
already-allowed AF_INET. Loopback + monitoring-only (state changes still need the
root unix socket), so exposure is minimal. Rejected the unix-socket route (chrony
group + ReadWritePaths=/run/chrony) because /run/chrony is 0750 (group has no
write). Documented in docs/stig-deviations.md #17; .te bench note updated for the
enforcing UDP path. Needs a workstation image rebuild + reboot.
fleet-agent reports clock health via 'chronyc -n tracking', but the hardened
unit runs with NoNewPrivileges (explicit + forced by DynamicUser/SystemCallFilter/
MemoryDenyWriteExecute). SELinux transitions the exec'd chronyc into chronyc_t,
and NNP forbids that domain change without process2:nnp_transition — which the
base policy doesn't grant for unconfined_service_t -> chronyc_t. Under enforcing
the exec is denied (avc: denied { nnp_transition } comm=fleet-agent, seen on
comet-beacon-54), so chronyc never runs and every chrony_* health field stays
empty (Monitor chrony row shows '—'). uptime/cpu/mem are unaffected (no exec).
Ship a minimal local module (files/stig/fleet-agent-chronyc.te) granting only
that one nnp_transition to the MORE-confined chronyc_t domain — the safe
direction the permission exists to gate — loosening nothing else in the agent
sandbox. Built + semodule -i'd in 2c-STIG alongside the existing
chrony_dhcp_dispatcher / fapolicyd_tmpfs modules. Justified in
docs/stig-deviations.md #16. Needs a workstation image rebuild + reboot; same
policy-store-rebuild reproducibility re-check as #14/#15.
The agent-plane server cert was issued by hand (server-pki.sh / docker exec
step ca certificate), which needs the step CLI on the host, re-specifies SANs
each time, and silently expires at the CA's 90-day cap — the exact failure that
just broke agent TLS (cert minted with a stale SAN, missing manager.fleet.lab).
Automate it:
- agent-cert-renewer sidecar (smallstep step-cli image, compose profile
cert-auto): issues the cert once from the co-located fleet step-ca, then
'step ca renew --daemon' rotates it before the cap. Renewal preserves the
subject + SANs, so SAN drift cannot recur. Chowns to the manager's nonroot
uid so the distroless manager reads the key. Opt-in + only valid where the CA
is co-located; server-pki.sh stays as the manual/remote-CA path.
- manager hot-reloads via GetCertificate (certreload.go): a rotated cert on
disk is picked up on the next handshake with no restart and no dropped agent
streams; a half-written pair keeps the last good cert serving. Unit-tested.
- defaults.env documents the cert-auto knobs; DEPLOY.md §2a recommends the
renewer and flags the mandatory manager.fleet.lab SAN.
The agent only reported chrony as synced|unsynced — nothing about how well the
clock tracks. Parse `chronyc tracking` for the source-relative metrics.
- proto: HealthSnapshot.chrony_offset_seconds (16, signed system-time offset,
fast=+), chrony_source (17, reference name/IP), chrony_stratum (18); regenerated.
- agent: replace chronyState() with chronyTracking()/parseChrony() — one exec,
parses Leap status + Reference ID + Stratum + System time offset (+ test).
- manager: Health.{ChronyOffset,ChronySource,ChronyStratum}; snapshotToHealth maps
them; influx writes chrony_offset_seconds/_stratum/_source and whitelists the
offset + stratum as graphable fields.
- UI: chronySummary() + offsetMs() — the chrony row now shows
"synced · 192.168.50.10 · s3 · +0.01ms" instead of just "synced".
The manager lets an admin schedule a reboot (desired.reboot_at) but never showed
whether it had happened. Add host uptime to the health snapshot and derive a
reboot status from it.
- proto: HealthSnapshot.uptime_seconds (15); regenerated pb.go (manager + agent copy).
- agent: probe /proc/uptime -> uptime_seconds (with parser + test).
- manager: Health.UptimeSeconds cached in inventory; snapshotToHealth maps it;
influx writes + whitelists uptime_seconds as a queryable field.
- UI: fmt.js dur() + rebootStatus() — boot_time = last_seen - uptime (manager clock
frame, no host/manager sync needed), compared to reboot_at. Monitor host detail
shows an "uptime" row and a "reboot" row: pending in X / overdue by X / rebooted.
Observability-only — works before the reboot ACTUATOR lands (deferred). Once a host
boots after the scheduled time (BMC/manual now, agent later), the badge flips to
"rebooted" automatically.
The Monitor tab polled /api/hosts every 10s (roster + host-detail refresh) but
only fetched a host's metric series on click, so the tmpfs/cpu/temp graphs went
stale until you re-selected. Extract loadMetrics() and call it from the same 10s
tick when a host is selected, so the graphs stay live.
Completes the rename commit (which carried only the file move): defaults.env
layering header, docker-compose usage comment, serving/Makefile (sets
COMPOSE_ENV_FILES), serving/.gitignore (.env), and the README/TODO references.
Same layering as the fleet-manager stack: defaults.env holds the committed
defaults, .env holds only per-LAN overrides + secrets, layered via
COMPOSE_ENV_FILES=defaults.env,.env. Changing a default here reaches every run
without re-copying.
- git mv serving/.env.example -> serving/defaults.env; header explains the layering.
- docker-compose.yml usage comment -> the COMPOSE_ENV_FILES / Makefile flow.
- serving/Makefile: sets COMPOSE_ENV_FILES; help/up/down/restart/pull/logs/ps/config
with a clear .env-missing guard (mirrors serving/fleet-manager/Makefile).
- serving/.gitignore: ignore .env (defaults.env is committed).
- README + TODO references updated. serving/siem/.env.example is a separate
wazuh1-node file, left as-is.
The agent's baked default pointed at the PUBLIC admin-UI name
(diskless-server.bitasylum.org), which resolves to the serving host's outbound IP
— unreachable from the fleet LAN. There's no split-horizon to "automate": the
manager agent plane is just another *.fleet.lab, served by the fleet dnsmasq like
boot/nvme/audit.
- serving/.env.example: MGMT_HOST=manager.fleet.lab.
- serving/docker-compose.yml: dnsmasq --address=/${MGMT_HOST}/${SERVER_IP}.
- files/agent/agent.env + main.go default: FLEET_AGENT_MANAGER=manager.fleet.lab:9600.
- server-pki.sh: AGENT_SANS default -> manager.fleet.lab (cert SAN matches the
baked name, so no FLEET_AGENT_SERVER_NAME override).
- docs/TODO #14 gotcha #4 updated: fleet name, not the IP+server-name workaround.
The public diskless-server.bitasylum.org stays the ADMIN-UI name (Traefik/:443).
Record the 2026-07-10 end-to-end verification (Dell 820 comet-beacon-54, real
TPM): TPM-identity mTLS -> enrolled -> streaming health + receiving desired-state,
manager logs "agent connected". Add the "deploying the agent plane" gotcha list
(placeholder CA, root+intermediate client CA, distroless PKI perms, split-horizon
DNS, out-of-band inventory seeding) — all automated by commit a89c3d2 — so the
next deploy doesn't rediscover the debug chain.
Four fixes from the on-host agent bring-up:
1. Bake the real fleet CA into the image. New `fleet_ca` build var (default =
the committed placeholder) file-provisions the trust anchor to
/etc/dot1x/radius-ca.pem — used by wpa_supplicant (RADIUS) AND the fleet-agent
(verify the manager). CI materializes a FLEET_CA secret to a runner-temp file
and passes -var fleet_ca; unset => placeholder (lab). No CA material committed.
2. server-pki.sh installs the agent CLIENT CA as root+intermediate (not root
only). Host TPM certs are leaf-only in NV and signed by the step-ca
intermediate, so a root-only client CA makes the manager advertise only the
root in its CertificateRequest -> Go clients send no cert ("tls: certificate
required") and it couldn't verify a leaf-only cert anyway. bootstrap.sh now
exports intermediate_ca.crt so FLEET_CA_INTERMEDIATE resolves.
3. New `fleet-manager register-host <serial> [-name] [-hostnqn]` subcommand:
seeds/updates an inventory record for a host enrolled out of band (bench
broker, migrated fleet) through the real store, so its agent stream is
accepted. The manual counterpart to the authorise-approve flow.
4. The agent surfaces the authoritative recv-side gRPC status instead of the
bare io.EOF gRPC returns on the send side when the server ends the stream —
so "host is not enrolled" (PermissionDenied) shows up instead of "EOF".
Companion to the previous commit's file move: point the 2b-agent install at
/tmp/agent/agent.env (the agent's own upload) instead of the deleted
/tmp/etc-stage path, update the file-provisioner comment, the enable_agent var
description, and the TODO source-path reference. packer validate passes.
The 2b-agent provisioner installed agent.env from /tmp/etc-stage, but the 2a
package step deletes /tmp/etc-stage before 2b runs, so the build failed with
"cannot stat '/tmp/etc-stage/fleet-agent/agent.env'". 2b-hostapi avoids this by
installing from its own upload dir; do the same: move agent.env into files/agent/
so it rides the agent's file upload (/tmp/agent), which survives until 2b-agent
cleans it. Runtime path (/etc/fleet-agent/agent.env) is unchanged.
Build the on-host agent that pairs with fleet-hostapi and feeds the
fleet-manager control plane (TODO #14 "STILL TO BUILD (a)"). Nothing on the
workstation talked to the manager before this; Monitor + InfluxDB stayed empty.
Scope: telemetry only. The agent reuses the host TPM identity for mTLS (read-only
tpmkey.LoadSigner at 0x81000100 + cert from NV 0x01800100; the private key never
leaves the TPM), opens the long-lived FleetAgent.Session bidi stream, sends Hello
+ periodic HealthSnapshot (load/cpu/temp/mem/tmpfs/nvme/auth/selinux/chrony/kiosk,
probed read-only from /proc+/sys plus hostapi facts), and LOGS but does not act on
the DesiredState/Command it receives. Reconcile actuators (kiosk URL, reboot,
drain) are deferred until the telemetry half is bench-verified.
- files/agent/: new module (go-tpm + grpc), static CGO_ENABLED=0 like enrol/hostapi.
Copies the enrol tpmkey (+ read-only LoadSigner) and the generated fleet.agent.v1
proto so it stays self-contained without importing the initramfs enrol module.
- files/agent/simtest/: proves the TPM-signed cert completes a real mTLS client-auth
handshake (RequireAndVerifyClientCert) against the go-tpm software simulator.
- files/etc/fleet-agent/agent.env: fleet-generic config (manager DNS name, fleet CA
reused from radius-ca.pem, TPM handle/nvindex, interval). No per-host state.
- fleet-agent.service: hardened unit (tss group for /dev/tpmrm0, INET egress).
- CI builds fleet-agent out-of-VM + runs vet/test + the simulator handshake test.
- Packer: enable_agent flag (default on) + 2b-agent provisioner + fapolicyd trust.
Post-switch-root only, so no dracut/initramfs changes.
- docs: TODO #14 + fleet-management-design as-built updated.
Requires the host already enrolled (broker approve births the inventory row) —
else the manager returns PermissionDenied, the correct "not enrolled yet" signal.
`make up` / `logs` / `restart` / `pull` / `build` / `config` / `register-admin`
/ `backup` — each exports COMPOSE_ENV_FILES=defaults.env,.env so plain `make ...`
layers the committed defaults under .env with no manual export or --env-file.
Guards on .env existing (COMPOSE_ENV_FILES requires it) with a clear message.
`make help` lists targets. DEPLOY §1 points at it.
Rename .env.example -> defaults.env and make it the committed defaults layer.
Compose reads defaults.env then .env (later wins) via COMPOSE_ENV_FILES, feeding
BOTH container env and ${...} interpolation — so a changed default here reaches
every run without re-copying, and .env shrinks to just overrides + secrets.
No compose service restructure: the existing interpolated environment: blocks
already build the container env; the layering just supplies their values from two
files. (env_file: directive alone wouldn't work — it doesn't feed the image-tag /
cross-service Influx / ACME_EMAIL interpolation this compose relies on.)
Docs (DEPLOY §1, compose header, README) updated: export
COMPOSE_ENV_FILES=defaults.env,.env, keep only secrets/overrides in .env.
Route the gcr.io distroless runtime image through the Harbor mirror
(harbor.bitasylum.org/gcr.io/distroless/static-debian12:nonroot) per the standing
rule, completing the fleet-manager Dockerfile's base-image mirroring.
Follow the code removal through the rest of the tree now that Traefik owns UI TLS:
- compose: drop FLEET_MGR_ADMIN_BEHIND_PROXY + ADMIN_CERT/_KEY from the manager
env; correct the ./pki mount comment (agent + broker certs only).
- .env.example: drop the BEHIND_PROXY + cert vars + self-TLS note.
- DEPLOY.md / README: remove the self-TLS "alternative" and the BEHIND_PROXY /
ADMIN_CERT config rows.
- server-pki.sh: point the admin UI cert at Traefik, not ./pki/admin.
- delete serving/fleet-manager/acme/ (the retired ACME side-car).
Traefik (and the k8s Ingress later) owns UI TLS, so remove the vestigial
self-TLS path: adminTLS(), the certReloader type, the adminCert/adminKey config
+ FLEET_MGR_ADMIN_CERT/_KEY env, and the now-redundant adminBehindProxy flag.
The admin plane always serves plaintext now — two modes: INSECURE (dev, no auth)
or default (passkey behind a trusted TLS proxy). Drops the crypto/tls, fmt, sync
imports and the errString helper.
Route the ad-hoc images I introduced through harbor.bitasylum.org (the standing
rule): backup.sh uses harbor.bitasylum.org/dockerhub/library/alpine:3.20 (via an
overridable ALPINE_IMAGE); the DEPLOY.md volume-chown fix + the goacme/lego prose
use the Harbor paths too. Compose (influxdb/traefik) + Dockerfile bases are already
harbor-mirrored; git.bitasylum.org/audiobuzz/* stays (CI-built).
- .env.example: RP_ORIGIN port-less, FLEET_MGR_ADMIN_BEHIND_PROXY=true, cert vars
labelled self-TLS-only; ACME vars are now Traefik's (dropped sidecar-only ones).
- DEPLOY.md §2c rewritten for Traefik (443 + Route 53 DNS-01), plus a "Network
paths" table answering how each plane is reached (enrolment via boot nginx,
agent direct mTLS via OPNsense L3, admin via Traefik). Start sequence + hardening
status updated (admin-exposure item resolved).
- README: admin transport = Traefik; behind-proxy explained; config table gains
BEHIND_PROXY; checkpoint 8 (hardening + Traefik ingress).
Management-plane ingress: a host-networked Traefik terminates the LE cert on :443
(Route 53 DNS-01, public-resolver propagation check) and forwards plaintext to
the manager running FLEET_MGR_ADMIN_BEHIND_PROXY=true on loopback :9500. Only the
admin plane rides Traefik — the agent gRPC/mTLS plane (:9600) stays direct (Traefik
can't terminate fleet-CA mTLS; hosts reach it via OPNsense L3 routing). This is the
k8s ingress-controller + cert-manager role in miniature.
- traefik/dynamic/fleet-manager.yml: file-provider router (Host rule -> manager),
passHostHeader so the WebAuthn RP origin matches.
- compose: traefik service (static config as CLI flags so ${ACME_EMAIL} interpolates)
+ traefik-acme volume; manager switched to behind-proxy defaults, RP_ORIGIN now
the port-less https origin.
- Retire the acme side-car (Traefik owns ACME now): remove the service + acme-state
volume; renew-loop.sh kept + relabelled as the self-TLS-mode cert helper.
- :443 only (admin UI is https-only) so it can't collide with a boot-LAN nginx :80.
Decouples "passkey auth ON" from "the listener terminates TLS". A new
FLEET_MGR_ADMIN_BEHIND_PROXY=true keeps WebAuthn/session auth fully on but serves
plaintext, for running behind a trusted TLS-terminating ingress (Traefik now, a
k8s Ingress later) that owns the LE cert. WebAuthn only needs RP_ORIGIN to match
the browser's HTTPS origin, not our transport; session cookies stay Secure. The
listener must be bound private (loopback/mgmt) so only the proxy reaches it —
logged as a warning. Insecure (dev, no auth) and self-TLS (default) modes
unchanged.
DEPLOY.md gains a "Hardening status" section (done vs operator-to-close), and
TODO #14 records the deployment as CI-built + acme-cert + backup, with the open
items called out: network-segment the :9500/:9600 listeners to the mgmt net, run
server-pki.sh when the agent lands, choose :9500 vs a 443 proxy, and digest-pin
the images.
Snapshot fleet-data (inventory + admins.json + enroll.key — the admin-lockout
risk) and influx-data to timestamped tarballs via a throwaway alpine container
(read-only, live-safe), with BACKUP_KEEP retention. Documented in DEPLOY.md
Backups with a restore pointer.
Point the compose service at the registry image the new CI workflow pushes
(git.bitasylum.org/audiobuzz/fleet-manager:${FLEET_MANAGER_VERSION:-latest}) so
mgmt hosts `docker compose pull` instead of building at deploy time — matching
dnsmasq/spdk/kiosk. `build: .` stays for local dev. Add FLEET_MANAGER_VERSION to
.env.example.
Add .gitea/workflows/fleet-manager-image.yaml, modelled on dnsmasq-image.yaml:
buildah bud of serving/fleet-manager (multi-stage SPA+Go→distroless) on a
path filter, with the same branch/tag tagging scheme (semver on tags, main-<sha>
+ latest on main, <branch>-<sha> dev images). Wires it into the shared
conventions:
- gitea-main-ci.yaml paths-ignore: list the new workflow so a fleet-manager
image change doesn't trigger the ~17-min UKI build (serving/** already covers
the dir; the .gitea/ workflow file needed listing).
- registry-gc.yaml IMAGES: add fleet-manager so its dev-branch builds are GC'd.
Follow-up to comma-separated RP_ORIGIN support: the enrolment URL builder used
the raw origin string, which would embed the whole comma list. Take the first
origin for the link.
Passkeys bind to the exact scheme://host:port the browser used, so a single
configured origin fails "Error validating origin" when reached via a different
form (e.g. :9500 direct vs a 443 proxy). Split FLEET_MGR_RP_ORIGIN on commas so
several exact origins can be allowed. Single-value config is unchanged.
Flipping FLEET_MGR_ADMIN_INSECURE=false before bootstrapping an admin crash-looped
on a cryptic "enrol key: open /data/enroll.key: no such file or directory". The
key is created by `register-admin`, so on ErrNotExist say exactly that (and note
the insecure escape hatch) instead of surfacing the raw errno.
Split-horizon DNS made lego's propagation check query the container's internal
resolver, which doesn't serve the public _acme-challenge TXT record, so the
challenge timed out. Add ACME_RESOLVERS (comma-separated host:port, default
8.8.8.8:53) → one --dns.resolvers flag each. Documented in .env + DEPLOY.md.
The goacme/lego image ships the binary at /lego, not on PATH, so the bare `lego`
call failed ("not found") once the entrypoint was overridden with the shell.
Detect it (command -v, then /lego fallback) and invoke via $LEGO.
lego auto-discovers the Route 53 zone via ListHostedZonesByName, so the zone ID
is optional — but pinning it disambiguates overlapping public/private zones and
lets the IAM policy drop the list permission. Thread it through compose + .env
+ DEPLOY.md (no script change; lego reads it from the environment).
Auto-issue + auto-renew the admin-UI TLS cert with a long-running compose
side-car instead of a host cron. The `acme` service (goacme/lego image, running
acme/renew-loop.sh) obtains the cert over DNS-01 via the AWS Route 53 API,
writes fullchain/privkey into ./pki/admin/ — the SAME dir the manager reads
(./pki:/pki:ro) — and chowns them to the nonroot runtime uid so the read-only
container can load the key. The manager's certReloader hot-swaps on the mtime
bump, no restart. Loop renews within 30 days of expiry; ACME_STAGING=1 gives a
dry-run vs LE staging.
- docker-compose.yml: `acme` service + `acme-state` volume (lego account/cert
store). ACME_DOMAIN reuses FLEET_MGR_RP_ID so cert host == passkey origin.
- .env.example: ACME_EMAIL + AWS creds (or instance role) + ACME_STAGING.
- DEPLOY.md §2c: rewritten to the side-car flow — scoped Route 53 IAM policy,
bring-acme-up-first ordering, flip off insecure, and an origin-match note.
(The docker-compose.yml commit also carries the local harbor.bitasylum.org
influxdb image mirror edit.)
A just-approved inventory record has a zero-value last_seen (Go 0001-01-01),
which parsed to a pre-epoch instant and slipped past the !t guard, rendering as
"739800d ago". Treat any at-or-before-epoch timestamp as "never" — the honest
state for a born-but-never-connected host (its health/version stay "—" until an
agent dials the gRPC plane).
The distroless nonroot runtime (uid 65532) couldn't write /data/inventory.json
because a named volume is created root-owned — approve failed with permission
denied on the inventory birth. Pre-create /data owned by nonroot in the image so
a fresh volume inherits it via Docker copy-up; an already-created root-owned
volume still needs a one-time chown (documented).
DEPLOY.md:
- 2b now covers BOTH broker deployments — the main serving stack
(enrol-pki/broker, ENROL_ADMIN_INSECURE=true by default → must flip to false)
and the bench stack — with the mint-once-into-broker-dir, copy-CA-to-manager
flow and an mTLS verify curl.
- Troubleshooting section: the data-volume permission-denied fix, the
"authorise plane not configured" (recreate + single-instance) fix, and the
"wrong version number" (broker plaintext) fix.
Every plane logged its state EXCEPT the broker success path, which was silent —
so "authorise pillar working" was indistinguishable from "not logged". Add an
ENABLED line matching the InfluxDB/agent/admin convention.
The compose wires the authorise (broker) and agent planes to fixed cert paths,
so a first bring-up that hasn't run the PKI steps yet crash-looped on the first
missing file (e.g. /pki/broker/clients/fleet-manager.crt) instead of booting.
Treat a configured-but-absent cert as "plane not yet provisioned": log a loud
WARNING and disable that plane, so the manager boots and each plane lights up
as its PKI appears. A malformed/unreadable cert still hard-fails (real
misconfig). The admin plane stays a hard requirement (FLEET_MGR_ADMIN_INSECURE
is its bench escape). Adds a missingFiles() helper; hub push path already
nil-guards a disabled agent plane.
Complete the deployment story for the server-side control plane:
- docker-compose.yml: private InfluxDB (published to loopback only) + a
host-networked fleet-manager (reaches the loopback enrol-broker on
127.0.0.1:9443 and binds a routable gRPC agent listener, like the broker).
- .env.example: RP origin (diskless-server.bitasylum.org), Influx token/DB
password, cert/PKI paths; the Influx token wires influxdb + manager with
one value.
- server-pki.sh: issue the agent-plane server cert from the fleet CA (step)
and stage the fleet root as the client-verify CA.
- DEPLOY.md: end-to-end runbook — three-cert PKI (fleet-CA agent cert, broker
admin-client cert, Let's Encrypt DNS-01 UI cert), register-admin passkey
bootstrap, compose up, hardware-free smoke test, backups.
- .dockerignore: keep host node_modules/dist/pki/secrets out of the build
context so the embedded SPA is the one built in-image.
- .gitignore: never commit .env / issued PKI / data.
Design docs marked as-built: fleet-management-design.md as-built note
(resolved forks — gRPC bidi transport, InfluxDB metrics, SPA graphs, passkey
admin auth, broker fold-in "alongside"); tpm-enrolment-design.md (broker
fold-in + agent transport fork resolved); TODO.md #14 → partly built, with
the on-host agent + per-host boot-version assignment (#6) called out as the
remaining work.
Two-stage build: node builds the Svelte SPA to dist, then golang compiles the
binary with the SPA embedded (-tags embedspa). Committed .pb.go stubs mean no
protoc. Ships as a small static distroless/nonroot image; same binary serves the
register-admin subcommand. (Not build-verified here — no docker in the
devcontainer — but the go build -tags embedspa + npm build are both proven.)
Per the option-A decision (deploy current design without per-host versioning):
add an honest note on the Lifecycle panel that kiosk-url/reboot/drain reconcile
via the agent but per-host boot-version assignment (dnsmasq cohort steering) is
not yet wired, so a version pin is stored + pushed but does not change what the
host boots. Avoids implying the pin takes effect.
Replace the free-text target-version field with a select populated from versions
known across the fleet (each host's running + targeted version, newest first),
plus a 'custom...' escape to a text input for a version not yet seen. Avoids
typos on the pin. (Authoritative serving-side source is a follow-up.)
- Contrast: lighten the muted/faint text tokens (AA on the hull panels) and make
Monitor host-card data values bright ink against dim faint labels, so version/
tmpfs/seen details read clearly on the blue-grey panels (was grey-on-grey).
- Lifecycle: reboot selector now offers none | now | at... — 'now' sets
reboot_at to the current time (agent reboots on reconcile); still confirm-gated
as a high-consequence action.
The themed single-page admin UI (serving/fleet-manager/ui/), embedded into the
binary and served same-origin over the admin plane.
- Svelte 5 + Tailwind 4 + Vite; a sibling of serving/kiosk-app. Theme = LYSANDRA
'restricted-violet' (high-security evolution of the kiosk's flight-deck DNA):
amber deck, VIOLET reserved for privileged/destructive + auth surfaces, teal
telemetry, red alert, clearance chrome (corner-reticle panels, scanlines, mono
clearance labels, secure-link status bar).
- Passkey gate: Login (usernameless discoverable WebAuthn) + Enrol (from the
register-admin one-time link) via navigator.credentials, with base64url<->buffer
helpers and the go-webauthn response shapes.
- Three pillars over the JSON API: Monitor (fleet roster + version-drift + health
+ per-host detail with bespoke SVG telemetry charts), Authorise (pending queue:
fingerprint cross-check, approve+bind in violet, deny), Lifecycle (desired-state
form; drain/reboot confirm-gated). Session-aware fetch client (cookie + CSRF).
- go:embed behind the build tag (spa_embed.go / spa_stub.go) so the
default go build/test need no dist/; mounted at / (api/auth/healthz win).
Verified: npm build clean (71KB JS / 33KB CSS + self-hosted fonts); default +
-tags embedspa go builds; smoke test serves index + assets + API through the SPA
origin. Against an insecure+seeded backend the deck renders passkey-free for
theme work. dist/ + node_modules gitignored.
The human->UI hop now authenticates with a hardware passkey (YubiKey), not an
mTLS client cert: phishing-resistant, origin-bound, better UX. (manager->broker
stays mTLS — service-to-service.)
- webauthn.go: credential store (registered admins + passkeys + a one-time-token
jti replay ledger; JSON file, atomic writes) + the webauthn.User impl + RP setup.
- cli.go: 'fleet-manager register-admin <name>' mints a short-lived HMAC-signed
enrolment link over a shared enrol key; the browser completes the WebAuthn
create ceremony. No anyone-can-register endpoint; single-use via the ledger.
- authapi.go: register/login ceremonies (usernameless discoverable login,
userVerification required) + logout + whoami; short-lived in-memory ceremony
store between begin/finish.
- session.go: opaque server-side sessions (Secure/HttpOnly/SameSite=Strict cookie)
+ requireSession middleware gating /api/*, with a CSRF header check on mutations.
adminName now comes from the session (audit attribution).
- main.go: register-admin subcommand dispatch; admin TLS switched to SERVER-AUTH
(no client cert) with a HOT-RELOADING cert (certReloader) so a Let's Encrypt
DNS-01 renewal is picked up without a restart. RP = diskless-server.bitasylum.org.
/api/* mounted behind requireSession; /auth/* + /healthz public.
Cert strategy (per design decision): the UI HTTPS cert is a publicly-trusted LE
cert via DNS-01 (mgmt-network UI isn't internet-reachable, so HTTP-01/TLS-ALPN-01
can't validate; DNS-01 proves control of bitasylum.org via TXT + supports
wildcards). External ACME client issues+renews to the cert path; manager reloads.
Deps: github.com/go-webauthn/webauthn (server-side). Tests: enrol-token
mint/verify/tamper/expiry, session middleware (401/403/CSRF/expiry), cred-store
persistence + jti ledger. build/test/vet/gofmt clean. Live passkey ceremony
(virtual authenticator / real YubiKey) is a bench item.
The manager is the ONLY thing that talks to InfluxDB, both directions; hosts hold
no token and never reach it.
- influx.go: influxSink implements MetricsSink — each agent HealthSnapshot is
written as a fleet_health point, identity (serial/hostnqn/starship) as TAGS
from the AUTHENTICATED stream (a host can only write its own series), telemetry
as fields. QueryField answers the SPA's read: a Flux query built with a strict
FIELD WHITELIST + hex-serial guard + parsed-duration range, so no caller input
is interpolated raw (injection-safe). Server-side influxdb-client-go/v2.
- admin.go: GET /api/hosts/{serial}/metrics?field=&range= -> time-series JSON;
503 when InfluxDB isn't wired, 400 on a bad range, 404 for unknown host.
- main.go: sink created when FLEET_MGR_INFLUX_URL+_TOKEN set, fed to the agent
hub; nil -> no-op sink so the manager runs without metrics.
Tests: snapshotPoint tags/fields/time, fluxQuery contents + %q quoting, and
QueryField rejects a non-hex serial + non-whitelisted field. build/test/vet/gofmt
clean. Live InfluxDB path validated in the compose bench (ckpt 6).
The AUTHORISE pillar: the manager owns the enrolment queue UX, the enrol-broker
stays the state machine + step-ca signer (v1 'alongside' fold-in).
enrol-broker (minimal, backward-compatible change):
- admin approve/deny now return the decided Enrolment as JSON when the caller
sends Accept: application/json. Browsers still redirect; other API callers
(verify-serving.sh) still get 204. The cert PEM stays unexported (only the
requesting host gets it via /status) — the JSON carries cert_serial, hostnqn,
context, fingerprint, cn.
fleet-manager:
- broker.go: mTLS client for the broker admin plane (Pending/Approve/Deny),
nil when unconfigured so the manager runs standalone.
- admin.go: GET /api/enrol/pending (proxy), POST /api/enrol/{id}/approve
(drives broker, then BIRTHS the inventory record — identity from the returned
serial/hostnqn/context, binding from the operator's optional body, enrolled
with the HUMAN approver = the manager's admin mTLS CN), POST .../deny.
503 when the broker isn't wired.
Test: httptest mock broker asserts the Accept:application/json approve, pending
proxy, inventory birth (identity+binding+enrolled), and the disabled-503 path.
Broker + manager both build/vet/test/gofmt clean.
The steady-state host<->manager transport, resolving the design's OPEN FORK to
bespoke gRPC (not OTLP+Wazuh). One long-lived, host-dialed, mTLS bidi stream:
health/metrics up, desired-state/commands down — no polling, real-time control,
still no inbound host port.
- proto/fleet_agent.proto: FleetAgent.Session(stream AgentUp) -> stream ManagerDown.
AgentUp = Hello | HealthSnapshot | CommandResult; ManagerDown = DesiredState |
Command(reboot/drain/resync). int64 unix times (no well-known-type dep).
Generated stubs committed (CI needs no protoc).
- agent.go: gRPC server with mTLS verified by the FLEET CA (the host's reused TPM
identity cert — distinct from the admin plane's admin CA). Identity is the peer
cert SERIAL, taken from the verified cert = inventory key, never self-asserted;
an unenrolled serial is refused (enrolment gates the stream). Per-conn send
channel decouples admin pushes from stream.Send. Stream registry (serial->conn)
with reconnect eviction. On connect + on change, desired-state is pushed live;
inventory stays authoritative so offline hosts reconcile on reconnect. Metrics
go through a MetricsSink seam (no-op now; InfluxDB in ckpt 4).
- main.go: agent plane starts only when its cert/key/CA are set; admin can run
standalone. Lifecycle PUT now live-pushes and reports pushed_live.
Full mTLS integration test (in-process fleet CA + host client cert): initial
desired-state on connect, Hello+snapshot -> inventory + online presence, live
desired-state push, and unenrolled-serial rejection. go build/test/vet/gofmt clean.
Deps (server-side only): google.golang.org/grpc + protobuf.
The checkpoint-1 commit included the 9.1M compiled 'fleet-manager' binary. Remove
it from tracking and gitignore it (/fleet-manager + *.test), matching the
enrol-broker convention. Build artifacts don't belong in git.
First slice of the server-side fleet control plane (serving/fleet-manager/), the
backend for the admin SPA (monitor / authorise / lifecycle). See
docs/fleet-management-design.md + tpm-enrolment-design.md §5.
This checkpoint (stdlib-only):
- Inventory record (identity/binding/enrolled/version/health/usage/desired),
keyed by TPM cert serial = hostnqn key, per the design's 'inventory record born
at enrolment'. JSON-file Store (atomic writes, mutex) behind an interface so a
DB swap is a drop-in.
- Admin plane: mTLS server verified by the DEDICATED admin CA (never the fleet CA,
so a host cert can't reach admin) + a plaintext dev mode. JSON API:
GET /api/hosts, GET /api/hosts/{serial}, PUT /api/hosts/{serial}/desired
(desired-state audited to the admin client-cert CN).
- Dev seed (FLEET_MGR_SEED) so the SPA has data before real enrolment/agent flow.
Smoke-tested: list/get/desired-PUT (incl. version-drift + drain), 404s, atomic
persistence. go build + go vet + gofmt clean.
Agent gRPC plane, broker-fold-in authorise, InfluxDB metrics, and the themed SPA
follow in checkpoints 2-6.
Bench pass on fapolicyd-tmpfs-12043d3 / comet-beacon-54: permissive=0, watch_fs has
overlay, / and /tmp both overlayfs; novel-hash /tmp binary DENIED (FANOTIFY resp=2,
obj_trust=0, exit=126); empty deny sweep = zero false denials (kiosk/weston/chromium/
NM-dispatcher all trusted, admin SSH+sudo work under staff_t). The exec guard went
from effectively OFF to enforcing on the whole overlay root, nothing legit broken.
Also records the scope caveat: integrity=none path-trust closes the UNTRUSTED-PATH
half (proven) but not the MODIFIED-TRUSTED-PATH half — an overlayfs copy-up of a
trusted path (e.g. /usr/bin/bash) lands a same-path tmpfs copy outside dm-verity that
path-trust still trusts. Closing it needs integrity=sha256 or an un-shadowable /usr
(TODO #15, preferred).
Combines the image trim (#18: langpack/firmware slim, kiosk weak-deps, nss-mdns
KEEP+mask-avahi fix, final kiosk-stack build guard) with optimise-ram's overlay-RAM
work (#17 fapolicyd db_max_size cap, .pyc unchecked-hash recompile).
Conflict resolutions:
- diskless.pkr.hcl (2d hygiene tail): kept BOTH additions — the .pyc recompile
runs first (the 'must run last after every dnf' mutation), then the read-only
FINAL GUARD (rpm -q kiosk stack) as the absolute last step. The nss-mdns fix
(mask avahi, keep the package) auto-merged above the conflict — no regression.
- docs/TODO.md: kept both #17 (fapolicyd db_max_size) and #18 (image trim), as the
#18 stack-note predicted.
The '#18 render should be safe' call on the nss-mdns drop (5cc72cc) was wrong: it
bricked the kiosk on trim-rpm-48dc3e1 (chromium hard-Requires nss-mdns, so the 2d
remove cascaded chromium + its GUI stack out → 500 MB browserless image, flashing
cursor). Fixed in 2bc2daf (keep nss-mdns, mask avahi daemon; final build guard
asserts the kiosk stack survived). Re-verified booting into the kiosk on the 820;
container image measured 1.8 GiB → 755 MiB (~58%).
Root cause of the trim-rpm-48dc3e1 'flashing cursor, no kiosk' + mysteriously
small (~500 MB) image: the 2d hygiene step ran 'dnf remove -y nss-mdns', but the
EPEL chromium package HARD-Requires nss-mdns (Chromium uses .local mDNS hostnames
for WebRTC ICE candidates). So dnf removed chromium as a dependent package and
then reaped its orphaned GUI stack (avahi, at-spi2, atk, adwaita, chromium-common
...) — a ~300 MB drop. The shipped image had no browser, so kiosk-session died at
the ${BROWSER:?} guard before ever starting weston: a bare KMS cursor.
The earlier nss-mdns commit checked what nss-mdns NEEDS (avahi) but not what NEEDS
nss-mdns (chromium).
Fix:
- Keep nss-mdns + avahi (satisfy chromium's Requires); instead MASK the avahi
mDNS daemon/socket so nothing listens on :5353 — that was the real
attack-surface concern, and nss-mdns is only an NSS module (no daemon needed).
- Add a FINAL GUARD at the end of 2d: assert chromium/chromium-common/weston/mesa
are still installed in the final image, else fail the BUILD. A missing browser
must brick the build, never the fleet. Catches any future silent removal
cascade, oscap package_*_removed rule, or autoremove.
Bench-verified the getattr allow (620a604) alone did NOT close the exec gap: with
the fapolicyd_tmpfs module installed (AVC gone, fapolicyd active, permissive=0), a
NOVEL non-RPM binary in /tmp still ran. Root cause: the diskless root is overlayfs
(stat -f / -> overlayfs), but the shipped watch_fs is
ext2,ext3,ext4,tmpfs,xfs,vfat,iso9660,btrfs — no 'overlay'. fapolicyd marks mounts
by fs type, so it never marked the overlay and intercepted no exec on / at all.
The exec guard we keep fapolicyd for was effectively OFF at runtime on every
diskless host.
Fix: the 2c fapolicyd step now appends 'overlay' to watch_fs (idempotent) so
fapolicyd watches the overlay root. Trust is path-based (integrity=none) so overlay
paths resolve against the rpmdb trust normally; the fapolicyd_tmpfs .te still
supports fapolicyd statfs'ing the overlay's tmpfs upper. Deviation #15 updated.
This turns app-allowlisting ON for the whole root at runtime for the first time.
BENCH-VERIFY HARD: boot + kiosk render + admin SSH + sudo + TPM enrolment all still
work (nothing legit untrusted), a novel /tmp binary is now BLOCKED (EPERM), no new
AVCs. enable_stig=false is the escape hatch if it regresses.
0bfe872 builds, deploys, and the host now mounts ONLY its own version's image
instead of connect-all'ing every served version — the targeted 'nvme connect -n
rd.nvmf.nqn' works. Each host = 1 controller on its own subsystem, which is what
lets old versions drain to 0 and spdk-loader auto-reap. Still to confirm: the reap
payoff (migrate a host, watch its old version drain + reap), and the secure path's
new step-6 connect over macsec0.
853ae20's diagnostic did 'awk … | sort -rn | head -25'. head closes the pipe
after 25 lines, sort dies of SIGPIPE (141), and step 2's set -euxo pipefail turns
that into a build failure (the exact trap the lsinitrd '| grep -q' comment warns
about). The kiosk step's rpm-manifest '| head' is safe only because that step
runs plain set -e, no pipefail.
Fix with the codebase convention: sort into a file, then head the file — no live
pipe for head to break. (Bonus: the crash log confirms omit_drivers worked —
amdgpu/radeon/nouveau kmods are gone from the top-25; the new biggest initramfs
item is GenuineIntel.bin CPU microcode at 22 MB, a future trim candidate.)
The dry-run failed with HTTP 403 on the branches API: PACKAGE_TOKEN is a
package-scoped token (registry push/delete) and can't read repo branches. Split
by capability: branch reads now use the automatic Actions token (github.token,
repo read via 'permissions: contents: read'); package list + delete keep
PACKAGE_TOKEN. Also replaced the bare 'curl -f' failures with a get() helper that
reports the HTTP code + a token-scope hint, so an auth problem can't look like
'no results' (which would otherwise skew the reconcile).
Reconciliation workflow (schedule + workflow_dispatch): lists live branches and
every container version across the four images this repo builds (fleet-boot,
fleet-ipxe, fleet-dnsmasq, spdk-nvmf) and deletes dev/branch builds whose branch
no longer exists, via the Gitea Packages API. Not event-driven, so it catches
branches deleted any way and never depends on a 'delete' webhook.
Safety (per request — never touch release tags or latest):
- Only tags ending in the dev commit-sha suffix '-<7 hex>' are ever candidates.
Release/version tags (1.2.3, v1.2.3, spdk v24.09, dnsmasq main-<sha>) and
'latest' never carry that suffix, so they're structurally out of scope; there's
also an explicit 'latest' skip. Verified by simulation across representative
tags incl. spdk's <ver>-<branch>-<sha> form and pre-release semver.
- Keeps any image whose sanitized branch is still live (+ always 'main'); branch
sanitization mirrors the build workflows.
- Fail-safe: aborts before deleting anything if the live-branch list can't be
fetched (0 branches), so an API/auth blip can't trigger mass deletion.
- Conservative matching: a suffix collision only ever causes a KEEP.
DRY RUN by default: workflow_dispatch previews unless dry_run=false; scheduled
runs dry-run until armed via repo variable REGISTRY_GC_ENABLE=true (so a
scheduled dry-run can be eyeballed first). Uses the existing PACKAGE_TOKEN
(write:package). shell: bash (runner defaults to sh).
fapolicyd is the app-allowlisting exec guard for the writable RAM overlay
(dm-verity covers only the read-only erofs lower). It statfs's mounts to decide
which to watch, and tmpfs is in its default watch_fs. This image's root is
systemd.volatile=overlay, so statfs("/") returns the overlay upper — a tmpfs —
and the shipped policy has no 'fapolicyd_t tmpfs_t:filesystem getattr' (no
ordinary host roots on tmpfs). Under enforcing the getattr is DENIED (audit:
dev=tmpfs ino=1 name="/", permissive=0, seen 2026-07-03 on comet-beacon-54),
so fapolicyd can't fully mark the tmpfs mounts (/, /tmp, /dev/shm) and its guard
over the writable overlay is incomplete — a real exec-guard gap, not just SIEM
noise.
Ship a minimal local allow module (allow, not dontaudit — we want fapolicyd to
see the tmpfs), compiled + semodule -i'd in 2c-STIG right after the chrony one.
Registered as deviation #15.
BENCH-VERIFY: the AVC is gone on an enforcing boot AND fapolicyd blocks a non-RPM
executable dropped in /tmp (proves the guard now covers the overlay), trusted
kiosk/appliance binaries still run, and the verity hash is reproducible across
two builds (semodule rebuilds the policy store).
f59eb79 now fully bench-verified: the fleet SPA renders on both the VM
(virtio-gpu) and the 820 (i915), so the i915 DMC/GuC firmware loads on real
hardware and the firmware keep-list is validated end-to-end. The two later
commits (nss-mdns drop, omit_drivers + keep-list tightening) don't touch i915,
so render should be safe, but still need their own build+boot.
The ~800 MB trim (glibc langpacks en-only + linux-firmware strip to the i915
keep-list, run pre-dracut) builds and boots — the boot/root path survives it
(firmware 678->147 MB, initramfs bundles only the keep-list). Notes still-open
checks: kiosk RENDER on real i915 hardware ('booted' != 'display up'), and that
the two later commits (nss-mdns drop 5cc72cc; omit_drivers + keep-list tightening
853ae20) are NOT in f59eb79 and need separate bench-verify.
First trim build confirmed firmware 678 MB -> 147 MB, but the initramfs came
out 97 MB and is now kernel-MODULE-bound, not firmware: --no-hostonly +
plymouth's drm dep instmods every GPU driver (amdgpu.ko etc.) though the fleet
is i915-only.
- omit_drivers += amdgpu radeon nouveau (90-diskless.conf): drop the discrete-GPU
kmods (amdgpu.ko is the biggest single module in the initramfs); keep i915 +
virtio-gpu/bochs for the splash. Shrinks the initramfs/UKI pulled over PXE on
every boot.
- Firmware keep-list tightened: drop mellanox/ (mlxsw_spectrum SWITCH-ASIC blobs,
not NIC fw — mlx5 NIC fw is on-card) and narrow intel/ to just ice (drop
ish/sof/ibt sensor-hub/audio/bluetooth, none loaded at boot per the audit).
- Add an 'initramfs largest files' diagnostic (sort lsinitrd.txt by size) so the
next build shows what dominates the .img.
BENCH-VERIFY: kiosk still renders on the 820 (i915 kept); watch the new initramfs
largest-files log line for the post-omit size.
Audiobuzz