Merge pull request 'optimise-ram' (#11) from optimise-ram into main

Reviewed-on: https://git.bitasylum.org/audiobuzz/diskless-workstation/pulls/11

diskless.pkr.hcl

@@ -393,6 +393,17 @@ build {
       # AVC in Wazuh — that would also hide future rhsmcertd_t denials.)
       "systemctl mask rhsmcertd.service 2>/dev/null || true",
       "systemctl disable rhsmcertd.service 2>/dev/null || true",
+      # Same class of noise, different trigger: dnf-makecache.timer fires ~10 min
+      # after boot and imports the whole dnf Python stack (dnf/libdnf/librepo/rhsm).
+      # Because the reproducible erofs `-T 0` reset makes every source .py look
+      # newer than its baked .pyc, that import REWRITES bytecode into the tmpfs
+      # overlay (measured 2026-07-03: ~MBs of dnf/libdnf/rhsm __pycache__) and
+      # throws lib_t write/unlink AVCs. An immutable diskless host never refreshes
+      # dnf caches (no persistent store, no runtime installs), so mask the timer to
+      # remove the trigger. Belt-and-braces with the compileall unchecked-hash step
+      # in 2d (which stops the rewrite at the source). See docs/TODO.md #16.
+      "systemctl mask dnf-makecache.timer 2>/dev/null || true",
+      "systemctl disable dnf-makecache.timer 2>/dev/null || true",
       # Read-only erofs/verity root; runtime state on a tmpfs overlay provided
       # by systemd.volatile=overlay (set in the cmdline by extract-artifacts.sh).
       # RHEL/Rocky dracut has no 'overlayfs'/'rootovl' module.
@@ -886,6 +897,21 @@ build {
       # custom kiosk files: add them to fapolicyd's trust DB and rebuild it. weston
       # /chromium are RPM-installed so already trusted. (See docs/stig-deviations.md.)
       "if command -v fapolicyd-cli >/dev/null 2>&1; then",
+      # RAM: fapolicyd's LMDB trust DB is rebuilt at BOOT onto the tmpfs overlay,
+      # so its map size (db_max_size, in MB) is a fixed RAM RESERVATION, not a
+      # ceiling it grows into. The EL9 default (100) was measured consuming a real
+      # 100 MiB of overlay RAM on a booted host (2026-07-03) — ~80% of the entire
+      # overlay. The minimal+kiosk trust set is a fraction of that (fapolicyd
+      # upstream shipped db_max_size=50 for years as the default that covers a
+      # FULL workstation rpmdb, which is larger than this image). Halve it to
+      # reclaim ~50 MiB. MUST run before the --update below so the rebuilt db uses
+      # the smaller map. BENCH-VERIFY the exec-guard still works end-to-end (boot +
+      # kiosk render + admin SSH + sudo): undersize this and fapolicyd hits
+      # MDB_MAP_FULL, allowlisting breaks, and login/kiosk exec is bricked (keep
+      # fapolicyd working — it's the writable-overlay exec guard dm-verity can't
+      # provide). Can drop lower after measuring `fapolicyd-cli --dump | wc -l` on
+      # the bench; see docs/TODO.md.
+      "  if [ -f /etc/fapolicyd/fapolicyd.conf ]; then sed -ri 's/^[[:space:]]*#?[[:space:]]*db_max_size[[:space:]]*=.*/db_max_size = 50/' /etc/fapolicyd/fapolicyd.conf; grep -q '^db_max_size' /etc/fapolicyd/fapolicyd.conf || printf 'db_max_size = 50\\n' >> /etc/fapolicyd/fapolicyd.conf; echo '==> fapolicyd: db_max_size set to 50 MiB (was EL9 default ~100)'; fi",
       "  for f in /usr/local/bin/kiosk-session /usr/local/bin/fleet-sb-enroll /usr/local/bin/fleet-hostapi /usr/sbin/fleet-enrol; do { [ -f \"$f\" ] && fapolicyd-cli --file add \"$f\"; } || true; done",
       "  fapolicyd-cli --update 2>/dev/null || true",
       "  echo '==> fapolicyd: trusted custom binaries (kiosk-session, fleet-sb-enroll, fleet-hostapi) + rebuilt trust db'",
@@ -1030,6 +1056,23 @@ build {
       # UKI initramfs shrinks too — see TODO #18. Nothing re-adds firmware after
       # that point, so the rootfs erofs is already trimmed by the time we get here.)
       #
+      # .pyc invalidation vs reproducible mtimes: the erofs is built `-T 0`, which
+      # zeroes every file's mtime. Python's DEFAULT .pyc invalidation is
+      # timestamp-based (the .pyc header stores the source's mtime), so at runtime
+      # every source .py looks newer than its baked .pyc and Python recompiles into
+      # the tmpfs overlay — wasting RAM and, under enforcing SELinux, throwing
+      # lib_t write/unlink AVCs (seen 2026-07-03: dnf/libdnf/rhsm __pycache__
+      # rewrites 10 min after boot). Recompile ALL bytecode with hash-based,
+      # UNCHECKED invalidation: the .pyc then carries a source hash flagged
+      # 'do not check', so Python loads it WITHOUT stat-ing the source, and never
+      # rewrites regardless of the `-T 0` mtimes. Bonus: hash .pyc embed no mtime,
+      # so they're more reproducible than timestamp .pyc (reinforces the -T 0
+      # invariant). MUST run LAST (after every dnf install/remove) so nothing lays
+      # down a fresh timestamp .pyc afterward; -f forces existing .pyc to this mode.
+      # Version-agnostic loop so an EL point-release python bump doesn't silently
+      # skip it. See docs/TODO.md #16.
+      "for d in /usr/lib/python3.* /usr/lib64/python3.*; do [ -d \"$d\" ] && python3 -m compileall -f -q --invalidation-mode unchecked-hash \"$d\" 2>/dev/null || true; done",
+      #
       # ---- FINAL GUARD: assert the kiosk stack survived the whole pipeline ----
       # 2026-07-03 incident: chromium installed cleanly in the 2b kiosk step (the
       # footprint audit showed chromium-149 + chromium-common, 523 pkgs), but by
@@ -1043,7 +1086,9 @@ build {
       # kiosk-critical package is still installed at the very end of the pipeline
       # (after remediation AND all 2d removes). This catches ANY future silent drop
       # — a new hard-Requires cascade, an oscap package_*_removed rule, autoremove.
-      # enable_kiosk=false legitimately ships no browser, so gate on it.
+      # Run AFTER the .pyc recompile (read-only rpm -q, lays down no bytecode, so it
+      # doesn't violate the "recompile last" rule). enable_kiosk=false legitimately
+      # ships no browser, so gate on it.
       "if [ \"${var.enable_kiosk}\" = \"true\" ] && [ -n \"${var.kiosk_url}\" ]; then for p in chromium chromium-common weston mesa-dri-drivers mesa-libEGL mesa-libgbm; do rpm -q \"$p\" >/dev/null 2>&1 || { echo \"ERROR: kiosk-critical package '$p' missing from the FINAL image — a later step removed it after the 2b kiosk install (e.g. a dnf remove whose dependency cascade took chromium, cf. the 2026-07-03 nss-mdns incident). Refusing to ship a bricked kiosk.\" >&2; exit 1; }; done; echo '==> guard: kiosk stack intact (chromium/weston/mesa present after remediation + hygiene)'; fi",
     ]
   }

docs/TODO.md

@@ -1037,7 +1037,7 @@ reasonable for a controlled fleet; the highest-assurance path is attended/firmwa
 or manufacturer pre-provisioning. Default recommendation: **separate provisioning
 UKI + isolated VLAN + read-back verify.**
 
-## [ ] 11. (DESIGN) TPM 802.1X identity enrolment (TOFU + OOB fingerprint)
+## [~] 11. (DESIGN) TPM 802.1X identity enrolment (TOFU + OOB fingerprint)
 
 **Authoritative design consolidated 2026-06-28:
 [tpm-enrolment-design.md](tpm-enrolment-design.md)** — unifies this item with the
@@ -1062,7 +1062,9 @@ approve → cert). **BENCH-VERIFY:** run against real step-ca (`bootstrap.sh` ro
 `enrol` provisioner password readable by the broker) and confirm an issued cert
 authenticates EAP-TLS.
 
-**HOST SIDE BUILT 2026-07-02 — Go binary (`fleet-enrol`), awaiting bench.**
+**HOST SIDE BUILT 2026-07-02 — Go binary (`fleet-enrol`); TEST-PATH ENROLMENT
+BENCH-VERIFIED ON REAL HARDWARE (Dell 820, `rd.dot1x=0`, 2026-07-03).** Secure-path
+(802.1X/MACsec) enrolment still to verify — see the NV-index bench finding below.
 Pivoted from a shell hook to a static **Go + go-tpm** binary (`files/enrol/`):
 it talks to `/dev/tpmrm0` DIRECTLY (via `transport.FromReadWriter` on an
 `os.File` — no `linuxtpm`, so go-tpm is the only vendored dep), so the enrolment
@@ -1157,7 +1159,7 @@ have their EK there), so enrolment could never create our key.
   (+ `tpm2_evictcontrol -C o -c 0x81000100` for a stale *app* key — never the EK), or
   BIOS "Clear TPM". Document in the bench README.
 
-**BENCH FINDING 2026-07-03 (FIXED) — NV index `0x01800001` collided with OEM
+**BENCH FINDING 2026-07-03 (FIXED + BENCH-VERIFIED) — NV index `0x01800001` collided with OEM
 platform NV; moved to `0x01800100`.** The Dell 820 kept "finding a cert" and
 bypassing enrolment even after repeated BIOS TPM clears. `tpm2_nvreadpublic`
 showed the truth: `0x01800001` on that hardware is a **firmware/platform** index
@@ -1173,9 +1175,18 @@ through the cmdline (`extract-artifacts.sh`), `fleet-enrol -nvindex`, the dracut
 hooks, bench scripts, invariant #1, and docs. Also **hardened `WriteNVCert`** to
 refuse a `platformcreate` index with a clear "pick a free owner index" error
 instead of a confusing undefine failure (regression `simtest/
-TestWriteNVCertRefusesPlatformIndex`). **BENCH-VERIFY:** 820 now enrols at
+TestWriteNVCertRefusesPlatformIndex`). **BENCH-VERIFIED 2026-07-03 — the Dell 820
-`0x01800100`; and check `tpm2_nvreadpublic` on any OTHER hardware model in the
+enrols end-to-end at `0x01800100`.** First success on REAL TPM hardware (past the
-fleet — if a vendor also squats `0x018001xx`, pick a value free on all of them.
+un-clearable OEM platform-NV at `0x01800001`): validates BOTH the NV-index move
+AND the `0x81000100` key-handle move on a physical TPM, not just the Proxmox
+swtpm — the full host ceremony (ensure ECDSA key → CSR → broker fingerprint →
+approve → leaf cert → NV write) runs on the metal. This was the hard case (the
+820 previously "found a cert" and silently bypassed enrolment every boot). **STILL
+BENCH-VERIFY:** (a) `tpm2_nvreadpublic` on any OTHER hardware model in the fleet —
+if a vendor also squats `0x018001xx`, pick a value free on all of them; (b) the
+enrolled cert actually **authenticates EAP-TLS on the secure path** (this run was
+test-mode `rd.dot1x=0`; 802.1X/MACsec enrolment still unverified — see the
+test-mode note above).
 
 **BENCH-VERIFY (host side, before trusting):** (a) go-tpm against a REAL TPM +
 swtpm on the target (the simulator proves the ceremony, not the device) — **the
@@ -1492,6 +1503,15 @@ tailoring; retires deviation-register #4).
    unchecked-hash` over the Python trees BEFORE the erofs build → mtime-independent
    `.pyc`, no rewrites, and MORE reproducible. (Worth doing regardless of firewalld —
    it's the true root cause of the firewalld AVCs seen 2026-07-03.)
+   **[x] DONE (2026-07-03):** implemented in `diskless.pkr.hcl` — (a) the compileall
+   `--invalidation-mode unchecked-hash` pass over `/usr/lib*/python3.*` at the end of the
+   2d hygiene block (last step before erofs extraction), and (b) masked
+   `dnf-makecache.timer` next to the existing `rhsmcertd` mask (it was the observed ~10-min
+   post-boot trigger that re-imported the dnf Python stack). Confirmed live on a booted host
+   via the overlay audit: `dnf/libdnf/librepo/rhsm` `__pycache__` rewritten at boot+10min.
+   **BENCH-VERIFY:** after a rebuild, no more `.pyc` files appear in the overlay
+   (`find / -xdev -newermt '1980-01-01' -path '*__pycache__*'` empty) and the `lib_t`
+   `.pyc` AVCs stop in the SIEM. Items 1 & 3 (firewalld enablement + tailoring) still OPEN.
 3. **Tailoring:** re-select the 4 firewalld rules.
 
 **THE load-bearing bench gate (why #4 deferred it):** the NVMe/TCP root is an OUTBOUND
@@ -1501,6 +1521,47 @@ default → should, but UNVERIFIED); a reconnect within `ctrl_loss_tmo` re-estab
 *through* the firewall; DHCP renew + MKA keepalives unaffected; SSH in; no AVCs; verity
 hash reproducible. If it drops the root, that is the fleet-wide outage risk — gate hard.
 
+## [~] 17. Cap fapolicyd trust-DB RAM (`db_max_size`) — biggest overlay consumer
+
+**Finding (overlay RAM audit, 2026-07-03, host `cygnus-cruiser-99`):** the writable root
+is `systemd.volatile=overlay` (a tmpfs). Enumerating runtime-written files via the `-T 0`
+trick — every baked file has mtime = epoch, so anything newer is an overlay write:
+
+```bash
+sudo find / -xdev -type f -newermt '1980-01-01' -printf '%b\n' | awk '{s+=$1*512} END{print s/1048576" MiB"}'
+```
+
+showed **124.5 MiB** total, of which **`/var/lib/fapolicyd/data.mdb` = 100 MiB, FULLY
+allocated** (`du` == `du --apparent-size`, not sparse) — **~80% of the entire overlay.**
+fapolicyd's LMDB trust DB is rebuilt at boot onto the tmpfs overlay, so its `db_max_size`
+(MB, in `/etc/fapolicyd/fapolicyd.conf`) is a **fixed RAM reservation**, not a ceiling it
+grows into. The EL9 default is ~100; the minimal+kiosk trust set is a fraction of that.
+
+**[x] DONE (2026-07-03):** `diskless.pkr.hcl` 2c-STIG-postfix now seds
+`db_max_size = 50` into `fapolicyd.conf` BEFORE the `fapolicyd-cli --update` rebuild (so
+the rebuilt db uses the smaller map). Value 50 chosen conservatively — it was fapolicyd's
+upstream default for years and covered a FULL workstation rpmdb (larger than this image),
+so it has real headroom. Reclaims ~50 MiB of overlay RAM.
+
+**[ ] BENCH-VERIFY (STIG-critical — fapolicyd is the writable-overlay exec guard):**
+1. Boot the rebuilt image; confirm kiosk renders, admin SSH (ECDSA) logs in, `sudo`
+   works, and custom binaries (`kiosk-session`, `fleet-hostapi`, `fleet-enrol`) still
+   execute — i.e. the trust DB rebuilt cleanly at 50 MiB with no `MDB_MAP_FULL`.
+   Undersizing bricks exec (login/kiosk) — that's the hard gate.
+2. Re-run the overlay audit: `data.mdb` should now be ~50 MiB and total overlay ~75 MiB.
+3. **Can go lower:** measure the real trust set with `fapolicyd-cli --dump | wc -l` on the
+   booted host; if the content is only ~15–25 MiB, 40 (or less) is safe and reclaims more.
+
+**Secondary lever (free, already in flight):** the trust DB scales with trusted files, so
+the kiosk weak-deps trim (commit `a193dc3`, ~100 fewer packages) shrinks the *actual*
+`data.mdb` content — widening the headroom under `db_max_size`.
+
+**Other overlay consumers noted (not fapolicyd, for context):** Chromium `/dev/shm`
+(~38 MiB, its own tmpfs) + kiosk `/tmp/kiosk-profile` (on the root overlay here — `/tmp`
+is NOT a separate tmpfs, ties to #15's mount separation); `/var/log/{messages,audit}`
+grow unbounded in RAM (audit is local-only) — logrotate caps or SIEM offload on
+long-lived hosts.
+
 ## [~] 18. Slim the image — locale + firmware (biggest erofs-size wins)
 
 **Finding (image-size audit via the 2b `rpm -qa --qf '%{SIZE}'` manifest, 2026-07-03):**