NTS authenticated time is now bench-verified end-to-end unattended on a clean
build (comet-beacon-54, 5294860): chronyc -N authdata shows ntp.fleet.lab in NTS
mode with cookies > 0, sources read under enforcing SELinux, no lo.sources, no
manual step. This clears the two dispatcher-integration defects found on the way
(distro 20-chrony-dhcp clobber; /run/chrony-dhcp SELinux label).
Flips F-03/P-18 / R-17 from "bench-verify pending" to bench-verified in the SSP
status line + register and the Annex §2.9 control row + R-17 register + summary,
mirroring the F-02 treatment. Rated Low. Residual (defence-in-depth, not blocking):
issue the per-environment NTS server cert, and exercise the cold-clock
nocerttimecheck bootstrap + active-spoof rejection.
Second bench defect after removing the distro 20-chrony-dhcp clobberer
(secure-ntp-5a3eaed, comet-beacon-54): the dispatcher wrote the correct
`server ntp.fleet.lab iburst nts` line, but chrony loaded ZERO sources --
`chronyc reload sources` returned 200 OK yet authdata/sources stayed empty.
Root cause (AVC): chronyd_t was DENIED read on the .sources files because they and
/run/chrony-dhcp itself were labeled generic var_run_t; chronyd_t may only read its
own chronyd_var_run_t. The dispatcher's `mkdir -p` creates the dir as var_run_t and
files inherit it. The distro 20-chrony-dhcp we removed had been restorecon-ing them
to the readable label, so removing it exposed a latent labelling gap.
Stock RHEL policy already maps /run/chrony-dhcp -> chronyd_var_run_t (a restorecon
relabels it and chrony immediately reaches NTS Cook 8). Fix: ship
/etc/tmpfiles.d/chrony-dhcp.conf so systemd-tmpfiles pre-creates the dir with that
fcontext at sysinit, before any NM event -- the dispatcher's files then inherit
chronyd_var_run_t and chronyd can read them. No SELinux allow rules needed
(bench-verified: after the fix a real NM event wrote the source as chronyd_var_run_t,
no AVC, authdata -> NTS Cook 8). Applies in both secure_ntp modes.
Also: the dispatcher now skips `lo` -- it was writing a spurious lo.sources
duplicate of the same server (the distro script had skipped loopback).
BENCH-VERIFY on the next clean build: a normal boot leaves the source file labeled
chronyd_var_run_t (ls -Z), no lo.sources, and authdata reaches NTS/cookies>0 with no
manual step.
NTS was inactive on the clean-build deploy (secure-ntp-d1621e9, comet-beacon-54):
chronyc authdata showed a plain IP source in Mode '-', Cook 0, with no NTS-KE
attempt at all -- chrony was on plain NTP.
Root cause: NetworkManager runs dispatcher scripts from BOTH
/etc/NetworkManager/dispatcher.d and /usr/lib/NetworkManager/dispatcher.d, merged
and sorted by filename. The chrony RPM ships
/usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp, which writes the DHCP
option-42 server to the SAME /run/chrony-dhcp/<iface>.sources file our
10-chrony-dhcp-ntp uses. Sorting after 10-, it ran last and overwrote our
`server ntp.fleet.lab iburst nts` line with `server 192.168.50.10 iburst`, then
reloaded chrony -- last writer wins on the shared filename, silently defeating
authenticated time.
Isolation proof: removing the distro script + hand-writing the NTS source brought
NTS up immediately (ntp.fleet.lab NTS, Cook 8, zero NAKs), confirming the host
image, the fleet-signed NTS-KE cert, DNS, and tcp/4460 were all already correct --
the collision was the sole defect.
Fix: the provisioner removes /usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp
at build (our dispatcher supersedes it in both modes); 20-chrony-onoffline is left
in place (harmless). Documents the finding in the dispatcher header,
docs/secure-ntp-design.md, and the CLAUDE.md secure_ntp note. BENCH-VERIFY on the
next clean build: a normal boot reaches NTS/cookies>0 with no manual step.
F-02/P-17 SSH host certs re-verified on the rebuilt 28d31e4 image (not the
hand-fixed host): the auto-derived FQDN principal is baked, so by-FQDN no-TOFU
admin login survives a reboot with zero manual steps. Cite the digest in the
design-doc status + bench-outcome, SSP v0.4 status line + R-16 row, and Annex A
2.4 row + R-16 register row.
NTS proven end-to-end on comet-beacon-54 (chronyc authdata cookies > 0); these
close the gaps the bench found so a clean build/serving run reaches that state
without manual patching:
- dispatcher (10-chrony-dhcp-ntp): fall back to resolv.conf search/domain when NM
doesn't export DHCP4_DOMAIN_NAME on the firing event, so the NTS source is
written reliably at boot (tonight it needed a manual dispatcher run).
- STIG vs secure_ntp (P2, security): STIG's chronyd_specify_remote_server re-injects
an UNAUTHENTICATED 'server 0.us.pool.ntp.mil maxpoll 16' AFTER the secure_ntp
chrony step commented servers out -> a silent fallback that defeats NTS's
DoS-only guarantee (host not fail-closed). Unselect chronyd_specify_remote_server
+ chronyd_or_ntpd_set_maxpoll in diskless-tailoring.xml, and re-strip any
server/pool in 2c-STIG-postfix when secure_ntp=true. Documented: stig-deviations #17.
- nts-cert.sh (P3): the container's chrony user (non-root) can't read a 0640
host-owned key -> NTS-KE TLS aborts ('non-properly terminated'). chown the key to
the ntp container's chrony uid + keep 0640 (0644 works but chrony warns
world-readable); 0644 fallback if the uid can't be found.
Bench-verified 2026-07-11; docs updated in docs/ssh-host-certificate-design.md's
sibling secure-ntp-design.md flow. F-03/P-18 stays open until a clean build shows
chrony.conf has no plain server and authdata cookies>0 as the only source.
F-02/P-17 (SSH host certificates, R-16) bench-verified end-to-end on
comet-beacon-54: admin login by FQDN with no TOFU, per-boot auto-re-sign across
reboot, SELinux confinement clean. Updates the SSP v0.4 status line + R-16
summary row and Annex A version note, the 2.4 host-key-trust row, the R-16
register row (treatment -> roll anchor + broker-side principal validation TODO)
and the 3.1 narrative. F-03/P-18 (NTS, R-17) stays bench-verify pending -- not
yet tested. Bench-surfaced code fixes are on branch poam-hardening (28d31e4).
F-02/P-17 SSH host-cert flow was proven end-to-end on comet-beacon-54
(2026-07-11); these fix the correctness/scoping/cleanup defects the bench
surfaced:
- ssh-hostcert: derive the FQDN principal <hostname>.<domain> from the DHCP
search domain at runtime (resolv.conf), alongside the short hostname, so an
admin `@cert-authority *.<domain>` line accepts by-FQDN logins instead of
"name is not a listed principal". Never baked (invariant #1). + tests.
- ssh-ca-bootstrap.sh: export the SSH host CA public key in OpenSSH wire format
(ssh-keygen -i -m PKCS8), not the PEM/PKCS8 that `step crypto keypair` writes
-- a PEM @cert-authority anchor is unparseable and silently falls back to TOFU.
- ssh-ca-bootstrap.sh: fail loudly (and verify enableSSHCA=true) if the
provisioner SSH-enable step fails, instead of swallowing it and shipping a CA
that 502s every runtime sign.
- dot1x-bench: point the co-located broker at https://localhost:8443, not the
fleet dnsmasq name ca.fleet.lab (unresolvable from the broker's host netns ->
"no such host" at sign time). localhost is in the CA cert SAN.
- serving: remove the mock-signer enrol-broker copy -- two host-networked
brokers collide on 9090/9443 and the mock issued untrusted certs. The real
broker lives with step-ca (dot1x-bench); nginx /enrol/ still proxies to it on
loopback.
- enrol-broker: retire the HTML admin console; the fleet-manager passkey SPA now
drives the mTLS JSON API (/pending, /approve|deny). Removes a weaker,
client-cert-only parallel human-approve path.
Design-doc bench note deferred: docs/ssh-host-certificate-design.md lives on the
secure-ntp branch (fbb8100), not here -- to be added there / on reconcile.
The SSP (v0.2, 2026-06-27) and annex predated the entire management/enrolment
plane and the July hardening work. Bring both to v0.3 (2026-07-10):
SSP:
- Scope/boundary: on-host fleet-agent now IN scope; server-side manager/broker/
step-ca/InfluxDB named as connected systems; new host interface (g) = outbound
mTLS to the management plane. Boot chain gains step 8 (post-switch-root agent).
- Data: management telemetry (operational metadata, mTLS-encrypted, no per-host
secret); the agent stream is the one non-plaintext transit path.
- §5.2 Protect: agent reuses the TPM identity for mTLS; WebAuthn-passkey admin
MFA (first human-admin MFA); 3 local SELinux modules; the 2026-07-04 fapolicyd
overlay watch_fs fix (app-control was inactive on the root until then).
- §5.3 Detect: operational health telemetry signal (distinct from auditd/Wazuh).
- §6 Essential Eight: MFA + admin rows updated (passkeys now enforced on mgmt UI).
- §7: new R-14 (mgmt plane = connected system pending authorisation) + R-15
(deferred agent actuators = future remote-mutation channel); R-05 MFA clause met.
Annex:
- §2.3 hardening: fapolicyd overlay-effectiveness fix + residual; 3 local SELinux
allow modules on the MAC row.
- §2.4 auth: agent mTLS on the machine-auth row; MFA row now Implemented for the
mgmt UI (passkeys), Partial for host SSH.
- §2.5 mgmt: new rows for TPM enrolment lifecycle (broker) and server-authoritative
management plane. §2.6: operational health-monitoring row. §2.7: mgmt-plane
segmentation (OPNsense L3, outbound-only). §2.9: agent mTLS, fleet CA (step-ca),
cert-lifecycle auto-renewal.
- §3 register: R-14, R-15 added; R-05 updated; heatmap + note + 3 new assessor
notes (scope the mgmt plane, witness fapolicyd deny, confirm actuators inert).
Recent work added telemetry fields and a cert path not yet in the docs:
- files/agent/README.md: add uptime to the health list; document that chrony is
read from /var/log/chrony/tracking.log (not the command socket) and needs BOTH
the chrony supplementary group (DAC) and the fleet_agent_chronylog SELinux allow
(TE), plus a reachable lab NTP (123/udp firewall); note the chrony group on the
unit row.
- serving/fleet-manager/README.md: add uptime + chrony detail to the health list;
note the cert-auto agent-cert-renewer sidecar + GetCertificate hot-reload
(certreload.go) in the bring-up and checkpoint 7.
- docs/TODO.md #14: probe list gains uptime + chrony-from-tracking.log; new dated
block records uptime/reboot-pending, the 4-layer chrony-access chase (all
silent-failing; inventory.json is the isolator), and the agent-plane cert
auto-renewal, all bench-verified 2026-07-10.
- docs/fleet-management-design.md: health record gains uptime + chrony
source/stratum/offset detail.
The tracking.log read was still returning empty under enforcing with NO AVC —
the tell of a DAC (not SELinux) denial. Root cause: /var/log/chrony is mode
0750 chrony:chrony (others have no access), so the agent's DynamicUser (group
tss only) can't even traverse the directory to reach the 0644 tracking.log —
EACCES on open, which produces no audit record. Manager confirmed genuinely on
fleet-manager-1db3977 (chrony-capable), and uptime flowed while chrony didn't,
proving the agent was sending empty chrony (failed read), not a manager gap.
Earlier manual reads worked only as root (sudo) or because fleetadmin had been
usermod'd into chrony during the socket test (reverted on reboot — /etc/group
is on the volatile overlay).
Fix: SupplementaryGroups=tss chrony on the unit — the chrony group has r-x on
the 0750 dir so the agent can traverse it; the log file is world-readable. This
is the DAC half; the fleet_agent_chronylog SELinux module (02a6621) is the TE
half — BOTH are required (verified: group-only -> TE-denied; module-only ->
DAC-denied silently). Corrected the now-stale 'no group' code comment; recorded
the two-layer requirement in the .te and stig-deviations #16. Needs a
workstation image rebuild + reboot.
The tracking.log approach (4ca0b6d) works in permissive but is blocked under
enforcing: chrony's log is labeled chronyd_var_log_t, confined by the shipped
policy to chronyd_t despite the 0644 mode. Verified on comet-beacon-54 — a
non-root read succeeds permissive, but under enforcing even a STIG-confined
admin (staff_t) gets Permission denied (ls -Z on the file is itself blocked).
The agent runs unconfined_service_t (confirmed via ps -eZ) and is denied the
same way.
Ship a minimal read-only allow module (files/stig/fleet-agent-chronylog.te):
allow unconfined_service_t chronyd_var_log_t:file { open read getattr } + dir
search to reach it. No write, no other chrony access, and no domain transition
(the agent reads a file; it does not exec chronyc) — narrower than the
abandoned nnp-transition module. Built + semodule -i'd in 2c-STIG alongside the
existing chrony_dhcp_dispatcher / fapolicyd_tmpfs modules. docs/stig-deviations
#16. Needs a workstation image rebuild + reboot; same policy-store-rebuild
reproducibility re-check as #14/#15.
Bench on comet-beacon-54 proved every command-socket route is blocked for the
hardened agent: chronyd's unix socket is owner-write-only (0755, so even a
chrony-group member can't connect), and STIG closes the UDP command port
(cmdport 0). Rather than re-open a port or expose chrony's privileged socket,
enable chronyd's tracking log and have the agent READ it — no port, no socket,
no group, no SELinux domain transition, full hardening intact.
- chrony.conf: enable 'log tracking' (idempotent) so chronyd appends
source/stratum/offset to /var/log/chrony/tracking.log. Write volume is a ~130B
line per clock update (backing off to ~maxpoll when stable) and capped by the
chrony RPM's logrotate — negligible on the RAM overlay. cmdport 0 stays, so no
STIG deviation.
- agent: chronyTracking() now reads the last data line of tracking.log via
parseTrackingLog() (source f[2], stratum f[3], offset f[6], leap/sync f[7]);
replaces the chronyc exec + parseChrony. Unit test swapped accordingly.
- revert the two abandoned approaches: drop the bindcmdaddress line (394ad10)
and the fleet_agent_chronyc SELinux nnp_transition module + provisioner block
(e6fd4a2, .te deleted) — no chronyc exec means no transition to allow. Removed
stig-deviations #16 and #17 (both waivers no longer needed).
Needs a workstation image rebuild + reboot. BENCH-VERIFY: tracking.log is
world-readable so the DynamicUser agent can read it; under enforcing
unconfined_service_t can read chronyd_var_log_t (add a minimal file-read allow
if denied); and the offset sign matches chronyc on a synced host.
Root cause of the blank chrony fields (found on comet-beacon-54): chrony 4.x
(RHEL 9) defaults its command interface to the UNIX socket only
(/run/chrony/chronyd.sock), usable by root or the chrony group with a writable
/run/chrony. The fleet-agent unit is hardened — DynamicUser (group tss, not
chrony) + ProtectSystem=strict (/run read-only) — so it cannot open the unix
socket nor create chronyc's reply socket. Bench proof: normal-user 'chronyc
tracking' -> 506 Cannot talk to daemon; only sudo worked; 'chronyc -h 127.0.0.1'
failed and ss showed no UDP :323. So the probe always failed and every chrony_*
field stayed empty. SELinux permissive did NOT help — it was never (only) an
SELinux problem; the nnp_transition module (prev commit) is still needed for the
enforcing exec transition.
Fix: bind chronyd's command socket on 127.0.0.1 (bindcmdaddress in chrony.conf,
idempotent in the 2c provisioner) and point the probe at it (chronyc -n -h
127.0.0.1 tracking). UDP needs no fs write or group and rides the unit's
already-allowed AF_INET. Loopback + monitoring-only (state changes still need the
root unix socket), so exposure is minimal. Rejected the unix-socket route (chrony
group + ReadWritePaths=/run/chrony) because /run/chrony is 0750 (group has no
write). Documented in docs/stig-deviations.md #17; .te bench note updated for the
enforcing UDP path. Needs a workstation image rebuild + reboot.
fleet-agent reports clock health via 'chronyc -n tracking', but the hardened
unit runs with NoNewPrivileges (explicit + forced by DynamicUser/SystemCallFilter/
MemoryDenyWriteExecute). SELinux transitions the exec'd chronyc into chronyc_t,
and NNP forbids that domain change without process2:nnp_transition — which the
base policy doesn't grant for unconfined_service_t -> chronyc_t. Under enforcing
the exec is denied (avc: denied { nnp_transition } comm=fleet-agent, seen on
comet-beacon-54), so chronyc never runs and every chrony_* health field stays
empty (Monitor chrony row shows '—'). uptime/cpu/mem are unaffected (no exec).
Ship a minimal local module (files/stig/fleet-agent-chronyc.te) granting only
that one nnp_transition to the MORE-confined chronyc_t domain — the safe
direction the permission exists to gate — loosening nothing else in the agent
sandbox. Built + semodule -i'd in 2c-STIG alongside the existing
chrony_dhcp_dispatcher / fapolicyd_tmpfs modules. Justified in
docs/stig-deviations.md #16. Needs a workstation image rebuild + reboot; same
policy-store-rebuild reproducibility re-check as #14/#15.
Completes the rename commit (which carried only the file move): defaults.env
layering header, docker-compose usage comment, serving/Makefile (sets
COMPOSE_ENV_FILES), serving/.gitignore (.env), and the README/TODO references.
The agent's baked default pointed at the PUBLIC admin-UI name
(diskless-server.bitasylum.org), which resolves to the serving host's outbound IP
— unreachable from the fleet LAN. There's no split-horizon to "automate": the
manager agent plane is just another *.fleet.lab, served by the fleet dnsmasq like
boot/nvme/audit.
- serving/.env.example: MGMT_HOST=manager.fleet.lab.
- serving/docker-compose.yml: dnsmasq --address=/${MGMT_HOST}/${SERVER_IP}.
- files/agent/agent.env + main.go default: FLEET_AGENT_MANAGER=manager.fleet.lab:9600.
- server-pki.sh: AGENT_SANS default -> manager.fleet.lab (cert SAN matches the
baked name, so no FLEET_AGENT_SERVER_NAME override).
- docs/TODO #14 gotcha #4 updated: fleet name, not the IP+server-name workaround.
The public diskless-server.bitasylum.org stays the ADMIN-UI name (Traefik/:443).
Record the 2026-07-10 end-to-end verification (Dell 820 comet-beacon-54, real
TPM): TPM-identity mTLS -> enrolled -> streaming health + receiving desired-state,
manager logs "agent connected". Add the "deploying the agent plane" gotcha list
(placeholder CA, root+intermediate client CA, distroless PKI perms, split-horizon
DNS, out-of-band inventory seeding) — all automated by commit a89c3d2 — so the
next deploy doesn't rediscover the debug chain.
Companion to the previous commit's file move: point the 2b-agent install at
/tmp/agent/agent.env (the agent's own upload) instead of the deleted
/tmp/etc-stage path, update the file-provisioner comment, the enable_agent var
description, and the TODO source-path reference. packer validate passes.
Build the on-host agent that pairs with fleet-hostapi and feeds the
fleet-manager control plane (TODO #14 "STILL TO BUILD (a)"). Nothing on the
workstation talked to the manager before this; Monitor + InfluxDB stayed empty.
Scope: telemetry only. The agent reuses the host TPM identity for mTLS (read-only
tpmkey.LoadSigner at 0x81000100 + cert from NV 0x01800100; the private key never
leaves the TPM), opens the long-lived FleetAgent.Session bidi stream, sends Hello
+ periodic HealthSnapshot (load/cpu/temp/mem/tmpfs/nvme/auth/selinux/chrony/kiosk,
probed read-only from /proc+/sys plus hostapi facts), and LOGS but does not act on
the DesiredState/Command it receives. Reconcile actuators (kiosk URL, reboot,
drain) are deferred until the telemetry half is bench-verified.
- files/agent/: new module (go-tpm + grpc), static CGO_ENABLED=0 like enrol/hostapi.
Copies the enrol tpmkey (+ read-only LoadSigner) and the generated fleet.agent.v1
proto so it stays self-contained without importing the initramfs enrol module.
- files/agent/simtest/: proves the TPM-signed cert completes a real mTLS client-auth
handshake (RequireAndVerifyClientCert) against the go-tpm software simulator.
- files/etc/fleet-agent/agent.env: fleet-generic config (manager DNS name, fleet CA
reused from radius-ca.pem, TPM handle/nvindex, interval). No per-host state.
- fleet-agent.service: hardened unit (tss group for /dev/tpmrm0, INET egress).
- CI builds fleet-agent out-of-VM + runs vet/test + the simulator handshake test.
- Packer: enable_agent flag (default on) + 2b-agent provisioner + fapolicyd trust.
Post-switch-root only, so no dracut/initramfs changes.
- docs: TODO #14 + fleet-management-design as-built updated.
Requires the host already enrolled (broker approve births the inventory row) —
else the manager returns PermissionDenied, the correct "not enrolled yet" signal.
DEPLOY.md gains a "Hardening status" section (done vs operator-to-close), and
TODO #14 records the deployment as CI-built + acme-cert + backup, with the open
items called out: network-segment the :9500/:9600 listeners to the mgmt net, run
server-pki.sh when the agent lands, choose :9500 vs a 443 proxy, and digest-pin
the images.
Complete the deployment story for the server-side control plane:
- docker-compose.yml: private InfluxDB (published to loopback only) + a
host-networked fleet-manager (reaches the loopback enrol-broker on
127.0.0.1:9443 and binds a routable gRPC agent listener, like the broker).
- .env.example: RP origin (diskless-server.bitasylum.org), Influx token/DB
password, cert/PKI paths; the Influx token wires influxdb + manager with
one value.
- server-pki.sh: issue the agent-plane server cert from the fleet CA (step)
and stage the fleet root as the client-verify CA.
- DEPLOY.md: end-to-end runbook — three-cert PKI (fleet-CA agent cert, broker
admin-client cert, Let's Encrypt DNS-01 UI cert), register-admin passkey
bootstrap, compose up, hardware-free smoke test, backups.
- .dockerignore: keep host node_modules/dist/pki/secrets out of the build
context so the embedded SPA is the one built in-image.
- .gitignore: never commit .env / issued PKI / data.
Design docs marked as-built: fleet-management-design.md as-built note
(resolved forks — gRPC bidi transport, InfluxDB metrics, SPA graphs, passkey
admin auth, broker fold-in "alongside"); tpm-enrolment-design.md (broker
fold-in + agent transport fork resolved); TODO.md #14 → partly built, with
the on-host agent + per-host boot-version assignment (#6) called out as the
remaining work.
Bench pass on fapolicyd-tmpfs-12043d3 / comet-beacon-54: permissive=0, watch_fs has
overlay, / and /tmp both overlayfs; novel-hash /tmp binary DENIED (FANOTIFY resp=2,
obj_trust=0, exit=126); empty deny sweep = zero false denials (kiosk/weston/chromium/
NM-dispatcher all trusted, admin SSH+sudo work under staff_t). The exec guard went
from effectively OFF to enforcing on the whole overlay root, nothing legit broken.
Also records the scope caveat: integrity=none path-trust closes the UNTRUSTED-PATH
half (proven) but not the MODIFIED-TRUSTED-PATH half — an overlayfs copy-up of a
trusted path (e.g. /usr/bin/bash) lands a same-path tmpfs copy outside dm-verity that
path-trust still trusts. Closing it needs integrity=sha256 or an un-shadowable /usr
(TODO #15, preferred).
Combines the image trim (#18: langpack/firmware slim, kiosk weak-deps, nss-mdns
KEEP+mask-avahi fix, final kiosk-stack build guard) with optimise-ram's overlay-RAM
work (#17 fapolicyd db_max_size cap, .pyc unchecked-hash recompile).
Conflict resolutions:
- diskless.pkr.hcl (2d hygiene tail): kept BOTH additions — the .pyc recompile
runs first (the 'must run last after every dnf' mutation), then the read-only
FINAL GUARD (rpm -q kiosk stack) as the absolute last step. The nss-mdns fix
(mask avahi, keep the package) auto-merged above the conflict — no regression.
- docs/TODO.md: kept both #17 (fapolicyd db_max_size) and #18 (image trim), as the
#18 stack-note predicted.
The '#18 render should be safe' call on the nss-mdns drop (5cc72cc) was wrong: it
bricked the kiosk on trim-rpm-48dc3e1 (chromium hard-Requires nss-mdns, so the 2d
remove cascaded chromium + its GUI stack out → 500 MB browserless image, flashing
cursor). Fixed in 2bc2daf (keep nss-mdns, mask avahi daemon; final build guard
asserts the kiosk stack survived). Re-verified booting into the kiosk on the 820;
container image measured 1.8 GiB → 755 MiB (~58%).
Bench-verified the getattr allow (620a604) alone did NOT close the exec gap: with
the fapolicyd_tmpfs module installed (AVC gone, fapolicyd active, permissive=0), a
NOVEL non-RPM binary in /tmp still ran. Root cause: the diskless root is overlayfs
(stat -f / -> overlayfs), but the shipped watch_fs is
ext2,ext3,ext4,tmpfs,xfs,vfat,iso9660,btrfs — no 'overlay'. fapolicyd marks mounts
by fs type, so it never marked the overlay and intercepted no exec on / at all.
The exec guard we keep fapolicyd for was effectively OFF at runtime on every
diskless host.
Fix: the 2c fapolicyd step now appends 'overlay' to watch_fs (idempotent) so
fapolicyd watches the overlay root. Trust is path-based (integrity=none) so overlay
paths resolve against the rpmdb trust normally; the fapolicyd_tmpfs .te still
supports fapolicyd statfs'ing the overlay's tmpfs upper. Deviation #15 updated.
This turns app-allowlisting ON for the whole root at runtime for the first time.
BENCH-VERIFY HARD: boot + kiosk render + admin SSH + sudo + TPM enrolment all still
work (nothing legit untrusted), a novel /tmp binary is now BLOCKED (EPERM), no new
AVCs. enable_stig=false is the escape hatch if it regresses.
0bfe872 builds, deploys, and the host now mounts ONLY its own version's image
instead of connect-all'ing every served version — the targeted 'nvme connect -n
rd.nvmf.nqn' works. Each host = 1 controller on its own subsystem, which is what
lets old versions drain to 0 and spdk-loader auto-reap. Still to confirm: the reap
payoff (migrate a host, watch its old version drain + reap), and the secure path's
new step-6 connect over macsec0.
fapolicyd is the app-allowlisting exec guard for the writable RAM overlay
(dm-verity covers only the read-only erofs lower). It statfs's mounts to decide
which to watch, and tmpfs is in its default watch_fs. This image's root is
systemd.volatile=overlay, so statfs("/") returns the overlay upper — a tmpfs —
and the shipped policy has no 'fapolicyd_t tmpfs_t:filesystem getattr' (no
ordinary host roots on tmpfs). Under enforcing the getattr is DENIED (audit:
dev=tmpfs ino=1 name="/", permissive=0, seen 2026-07-03 on comet-beacon-54),
so fapolicyd can't fully mark the tmpfs mounts (/, /tmp, /dev/shm) and its guard
over the writable overlay is incomplete — a real exec-guard gap, not just SIEM
noise.
Ship a minimal local allow module (allow, not dontaudit — we want fapolicyd to
see the tmpfs), compiled + semodule -i'd in 2c-STIG right after the chrony one.
Registered as deviation #15.
BENCH-VERIFY: the AVC is gone on an enforcing boot AND fapolicyd blocks a non-RPM
executable dropped in /tmp (proves the guard now covers the overlay), trusted
kiosk/appliance binaries still run, and the verity hash is reproducible across
two builds (semodule rebuilds the policy store).
f59eb79 now fully bench-verified: the fleet SPA renders on both the VM
(virtio-gpu) and the 820 (i915), so the i915 DMC/GuC firmware loads on real
hardware and the firmware keep-list is validated end-to-end. The two later
commits (nss-mdns drop, omit_drivers + keep-list tightening) don't touch i915,
so render should be safe, but still need their own build+boot.
The ~800 MB trim (glibc langpacks en-only + linux-firmware strip to the i915
keep-list, run pre-dracut) builds and boots — the boot/root path survives it
(firmware 678->147 MB, initramfs bundles only the keep-list). Notes still-open
checks: kiosk RENDER on real i915 hardware ('booted' != 'display up'), and that
the two later commits (nss-mdns drop 5cc72cc; omit_drivers + keep-list tightening
853ae20) are NOT in f59eb79 and need separate bench-verify.
First trim build confirmed firmware 678 MB -> 147 MB, but the initramfs came
out 97 MB and is now kernel-MODULE-bound, not firmware: --no-hostonly +
plymouth's drm dep instmods every GPU driver (amdgpu.ko etc.) though the fleet
is i915-only.
- omit_drivers += amdgpu radeon nouveau (90-diskless.conf): drop the discrete-GPU
kmods (amdgpu.ko is the biggest single module in the initramfs); keep i915 +
virtio-gpu/bochs for the splash. Shrinks the initramfs/UKI pulled over PXE on
every boot.
- Firmware keep-list tightened: drop mellanox/ (mlxsw_spectrum SWITCH-ASIC blobs,
not NIC fw — mlx5 NIC fw is on-card) and narrow intel/ to just ice (drop
ish/sof/ibt sensor-hub/audio/bluetooth, none loaded at boot per the audit).
- Add an 'initramfs largest files' diagnostic (sort lsinitrd.txt by size) so the
next build shows what dominates the .img.
BENCH-VERIFY: kiosk still renders on the 820 (i915 kept); watch the new initramfs
largest-files log line for the post-omit size.
Hosts connected with 'nvme connect-all', which attaches EVERY subsystem in the
discovery log. Each host then mounts only its own by-UUID verity namespaces but
leaves a live controller on every other served version's subsystem. Observed via
spdk-release list: more controllers than workstations, each box on every version
(1/3/3/1/3 across 5 versions for ~3 boxes). Beyond wasted connections
(O(hosts x versions)), it defeated the rolling-update drain/reap: no version ever
reached 0 connections while any host was up, so spdk-loader's auto-reap could
never fire and old versions accumulated without bound.
Fix — targeted connect to this build's subsystem only:
- extract-artifacts.sh bakes the per-version SUBNQN (already in verity.json
.subnqn, matched by the serving side) into the signed UKI cmdline as
rd.nvmf.nqn=. NQN is version identity, not per-host state, and the cmdline is
already per-version + immutable-per-signed-UKI, so invariant #1 holds.
- nvmf-connect.sh (test path) uses 'nvme connect -n <nqn>'; legacy UKIs without
rd.nvmf.nqn fall back to connect-all with a warning so they still boot.
- dot1x-macsec-up.sh (secure path) gains an explicit step-6 targeted connect over
macsec0 (after it's addressed, per invariant #2), replacing the delegation to
the dracut nvmf module's discovery connect-all.
Now each host = 1 controller on its own subsystem; versions drain to 0 when their
hosts migrate, so auto-reap works and connections scale O(hosts). Requires a
rebuild + re-sign (cmdline change, invariant #6). BENCH-VERIFY both paths;
confirm the dracut nvmf module isn't also connect-all-ing on the secure path.
CLAUDE.md invariant #2 + TODO #6 updated.
Follow-up to the linux-firmware strip (prev commit put it in the 2d hygiene
block, which runs AFTER the dracut build — so it only shrank the rootfs erofs,
not the initramfs). Move the strip to step 2, before the dracut build: with
--no-hostonly, plymouth's drm dep instmods a broad DRM set and dracut bundles
each module's firmware into the initramfs (amdgpu being the classic UKI
bloater). Trimming first means dracut never sees it, so both the rootfs erofs
AND the UKI initramfs shrink. Keep-list is unchanged (i915 + intel/ + mellanox
+ intel-ucode), so the fleet splash GPU (i915) and ice DDP firmware the
initramfs actually needs are retained; dracut just warns 'firmware not found'
for the removed drm modules and continues.
Add a post-build diagnostic (initramfs .img size + bundled firmware, expected
i915 only) so the initrd win is measured in the build log, not estimated.
The 2b rpm-size manifest revealed the biggest packages aren't the kiosk but
base bloat: linux-firmware 678 MB and glibc-all-langpacks 228 MB (vs chromium
370 + llvm 129). The uncompressed erofs is read over NVMe/TCP on every boot,
so this is fleet-wide dead weight + attack surface.
glibc: swap glibc-all-langpacks (every language) for glibc-minimal-langpack +
glibc-langpack-en. ~220 MB; image is English-only.
linux-firmware: a live fleet audit (modinfo -F firmware over loaded modules +
sysfs/lspci driver bindings on the real 820-class hosts and the VMs) proved the
only firmware any host loads is Intel i915 (GPU). Every NIC is firmware-free
(e1000e reports on-chip NVM, not a fw file; virtio_net loads nothing), so the
catastrophic missing-NIC-firmware case is off the table. Keep i915 + intel/
(ice DDP insurance) + mellanox + intel-ucode; strip the ~600 MB of foreign
vendor blobs (amdgpu/nvidia/radeon/ath/qcom/...). Remove iwl*-firmware (Wi-Fi,
~83 MB) on a wired-only fleet. Strip-in-place since the package can't be
removed without losing i915; du reports the result in the build log.
BENCH-VERIFY: kiosk still renders on the 820 (i915 DMC/GuC load) — the one real
risk; widen the keep-list if any host has a non-Intel GPU or an ice/mlx5 NIC.
Test-path enrolment (rd.dot1x=0) succeeded end-to-end on the Dell 820 — the
hard case that previously found the un-clearable OEM platform NV at 0x01800001
and silently bypassed enrolment every boot. Confirms BOTH the NV-index move
(0x01800001 -> 0x01800100) and the signing-key handle move (EK slot -> 0x81000100)
on a physical TPM, not just the Proxmox swtpm: ensure ECDSA key -> CSR -> broker
fingerprint -> approve -> leaf cert -> NV write all run on the metal.
Mark #11 in-progress ([~]). Still to verify: the enrolled cert authenticating
EAP-TLS on the secure 802.1X/MACsec path (this run was test-mode), and
tpm2_nvreadpublic on other fleet hardware models to confirm 0x01800100 is free
fleet-wide.
Two fixes from the booted-host overlay RAM audit (2026-07-03). The writable
root overlay (systemd.volatile=overlay, tmpfs) held 124.5 MiB of runtime
writes; almost all of it is runtime-regenerated state, not real fs deltas.
1. fapolicyd db_max_size 100 -> 50 (TODO #17). /var/lib/fapolicyd/data.mdb was
100 MiB fully allocated (~80% of the overlay). fapolicyd's LMDB trust DB is
rebuilt at boot onto the overlay, so db_max_size is a fixed RAM reservation.
Sed it to 50 before the fapolicyd-cli --update rebuild. 50 was fapolicyd's
long-time upstream default (covered a full workstation rpmdb), so it has
headroom for this smaller image. Reclaims ~50 MiB.
BENCH-VERIFY (STIG-critical exec guard): boot + kiosk + admin SSH + sudo +
custom-binary exec must work (undersize -> MDB_MAP_FULL bricks exec).
2. .pyc invalidation fix (TODO #16 item 2). erofs -T 0 zeroes mtimes, so
Python thinks every baked .pyc is stale and rewrites it into the overlay at
runtime (wasting RAM + throwing lib_t write/unlink AVCs that flood the SIEM).
Observed trigger: dnf-makecache.timer (~10 min post-boot) re-importing the
dnf Python stack. Fix both ends: (a) compileall --invalidation-mode
unchecked-hash over /usr/lib*/python3.* at the end of 2d (mtime-independent
.pyc, no rewrite, also more reproducible); (b) mask dnf-makecache.timer next
to the existing rhsmcertd mask (removes the trigger). Firewalld enablement
(#16 items 1 & 3) still open.
The kiosk dnf transaction was measured at 229 pkgs / 1.0 GB. Nearly all
of the non-core packages are Recommends: a locked-down single-URL kiosk
never uses flatpak + its ostree/bubblewrap sandbox runtime, tracker-miners
(a filesystem indexer — dead weight and a threat surface), the pipewire/
pulseaudio audio stack, or libproxy-webkitgtk4 -> webkit2gtk3-jsc (an 8.9M
second browser engine dragged into a Chromium kiosk for libproxy's PAC
runner).
Add --setopt=install_weak_deps=False to cut them, and an rpm -qa size
manifest to the build log so the next CI build reports the exact new
footprint (turning 'seems unnecessary' into data for the next trim round:
gstreamer/media codecs, or mesa-dri-drivers if the GPU path is abandoned).
The immovable floor is chromium + chromium-common + llvm-libs (mesa sw-GL)
+ gtk3/adwaita + libicu, all hard deps. BENCH-VERIFY the kiosk still
renders with weak deps off (watch for a missing font or libva).
The Dell 820 kept "finding a cert" and bypassing enrolment despite repeated BIOS
TPM clears. tpm2_nvreadpublic revealed 0x01800001 on that hardware is a FIRMWARE
platform index (platformcreate|policydelete|writelocked, 70 bytes — not our
cert; siblings at 0x01800003/4 + 0x01810000). platformcreate survives TPM2_Clear
and policydelete needs the platform's secret policy, so it's permanent and
unusable by us — the blind check false-positived on it, and enrolment could
never write there anyway. Same class of bug as the EK-handle collision.
Move the NV cert index default 0x01800001 -> 0x01800100 (free on the 820, owner
range, overridable via rd.dot1x.nvindex): cmdline (extract-artifacts.sh),
fleet-enrol -nvindex, the dracut hooks, bench scripts, CLAUDE.md invariant #1,
and design/SSP docs. The signing-key handle (0x81000100) is unchanged.
Harden WriteNVCert: refuse a platformcreate index with a clear "pick a free
owner index via rd.dot1x.nvindex" error instead of a confusing undefine
failure. Regression simtest/TestWriteNVCertRefusesPlatformIndex; NV roundtrip +
ReadNVCert tests moved to 0x01800100. All go tests + packer validate pass.
Captures the decision to re-enable a CONFIGURED firewalld (currently removed),
the required .pyc/-T0 root-cause fix (compileall --unchecked-hash), the 4 STIG
waivers it converts, and the load-bearing NVMe/TCP-survives-firewalld bench gate.
The NM dispatcher 10-chrony-dhcp-ntp runs `chronyc reload sources` to load DHCP
option-42 NTP servers. That chronyc runs in the confined custom-dispatcher
domain (NetworkManager_dispatcher_custom_t), so its reply socket carries that
label and the shipped policy has no rule for chronyd_t to sendto it. Under
enforcing, chronyd does the reload but its ack is denied — chronyc reports
failure and the AVC floods the SIEM (seen 2026-07-03; permissive=1 only because
the box was booted enforcing=0).
Ship a minimal local allow module (files/stig/chrony-dhcp-dispatcher.te):
allow chronyd_t NetworkManager_dispatcher_custom_t:unix_dgram_socket sendto;
Built + semodule -i'd in the 2c-STIG provisioner (checkpolicy installed if
absent), with an install assertion. `allow` not `dontaudit` — it also lets the
reload ack succeed so chronyc returns cleanly. Documented as
docs/stig-deviations.md #14 (not a rule waiver — a local functional allow).
BENCH-VERIFY: enforcing boot shows no AVC, chronyc reload returns 0, a
DHCP-handed NTP server appears in `chronyc sources`; and re-check verity-hash
reproducibility (semodule rebuilds the policy store, like the semanage step).
The nvmeof (spdk-target) service carried a versioned bootstrap mount
(type: image, source ${OCI_IMAGE}:${IMAGE_VERSION} -> /artifacts) plus an
IMAGE_VERSION env. Bumping IMAGE_VERSION therefore changed the service config,
so `docker compose up -d` recreated the container -> nvmf_tgt restarted ->
every connected host's NVMe/TCP root dropped at once. That is precisely the
fleet-wide outage the long-lived-target design exists to prevent.
Remove the /artifacts image-volume and IMAGE_VERSION env from nvmeof so its
config no longer depends on the version — compose leaves it running across a
bump. Bootstrap and rolling updates are already fully covered by spdk-loader
(stages the current release into ./spdk/releases and adds it online via
JSON-RPC); spdk-target.sh's seed block is guarded, so it just no-ops when
/artifacts isn't mounted. Verified nvmeof no longer references IMAGE_VERSION
(only spdk-loader does); compose YAML validates.
BENCH-VERIFY: bump IMAGE_VERSION + up -d with a host connected -> nvmeof
container not recreated, host root survives, loader adds the new version.
Audiobuzz