From 796dfe9f4e10c4eb53316e51de26c07da3bcb805 Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 04:03:01 +0000 Subject: [PATCH] reduce overlay RAM + kill .pyc AVC churn (TODO #16 item2, #17) Two fixes from the booted-host overlay RAM audit (2026-07-03). The writable root overlay (systemd.volatile=overlay, tmpfs) held 124.5 MiB of runtime writes; almost all of it is runtime-regenerated state, not real fs deltas. 1. fapolicyd db_max_size 100 -> 50 (TODO #17). /var/lib/fapolicyd/data.mdb was 100 MiB fully allocated (~80% of the overlay). fapolicyd's LMDB trust DB is rebuilt at boot onto the overlay, so db_max_size is a fixed RAM reservation. Sed it to 50 before the fapolicyd-cli --update rebuild. 50 was fapolicyd's long-time upstream default (covered a full workstation rpmdb), so it has headroom for this smaller image. Reclaims ~50 MiB. BENCH-VERIFY (STIG-critical exec guard): boot + kiosk + admin SSH + sudo + custom-binary exec must work (undersize -> MDB_MAP_FULL bricks exec). 2. .pyc invalidation fix (TODO #16 item 2). erofs -T 0 zeroes mtimes, so Python thinks every baked .pyc is stale and rewrites it into the overlay at runtime (wasting RAM + throwing lib_t write/unlink AVCs that flood the SIEM). Observed trigger: dnf-makecache.timer (~10 min post-boot) re-importing the dnf Python stack. Fix both ends: (a) compileall --invalidation-mode unchecked-hash over /usr/lib*/python3.* at the end of 2d (mtime-independent .pyc, no rewrite, also more reproducible); (b) mask dnf-makecache.timer next to the existing rhsmcertd mask (removes the trigger). Firewalld enablement (#16 items 1 & 3) still open. --- diskless.pkr.hcl | 42 ++++++++++++++++++++++++++++++++++++++++ docs/TODO.md | 50 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+) diff --git a/diskless.pkr.hcl b/diskless.pkr.hcl index b7a85fd..a35d0fe 100644 --- a/diskless.pkr.hcl +++ b/diskless.pkr.hcl @@ -383,6 +383,17 @@ build { # AVC in Wazuh — that would also hide future rhsmcertd_t denials.) "systemctl mask rhsmcertd.service 2>/dev/null || true", "systemctl disable rhsmcertd.service 2>/dev/null || true", + # Same class of noise, different trigger: dnf-makecache.timer fires ~10 min + # after boot and imports the whole dnf Python stack (dnf/libdnf/librepo/rhsm). + # Because the reproducible erofs `-T 0` reset makes every source .py look + # newer than its baked .pyc, that import REWRITES bytecode into the tmpfs + # overlay (measured 2026-07-03: ~MBs of dnf/libdnf/rhsm __pycache__) and + # throws lib_t write/unlink AVCs. An immutable diskless host never refreshes + # dnf caches (no persistent store, no runtime installs), so mask the timer to + # remove the trigger. Belt-and-braces with the compileall unchecked-hash step + # in 2d (which stops the rewrite at the source). See docs/TODO.md #16. + "systemctl mask dnf-makecache.timer 2>/dev/null || true", + "systemctl disable dnf-makecache.timer 2>/dev/null || true", # Read-only erofs/verity root; runtime state on a tmpfs overlay provided # by systemd.volatile=overlay (set in the cmdline by extract-artifacts.sh). # RHEL/Rocky dracut has no 'overlayfs'/'rootovl' module. @@ -812,6 +823,21 @@ build { # custom kiosk files: add them to fapolicyd's trust DB and rebuild it. weston # /chromium are RPM-installed so already trusted. (See docs/stig-deviations.md.) "if command -v fapolicyd-cli >/dev/null 2>&1; then", + # RAM: fapolicyd's LMDB trust DB is rebuilt at BOOT onto the tmpfs overlay, + # so its map size (db_max_size, in MB) is a fixed RAM RESERVATION, not a + # ceiling it grows into. The EL9 default (100) was measured consuming a real + # 100 MiB of overlay RAM on a booted host (2026-07-03) — ~80% of the entire + # overlay. The minimal+kiosk trust set is a fraction of that (fapolicyd + # upstream shipped db_max_size=50 for years as the default that covers a + # FULL workstation rpmdb, which is larger than this image). Halve it to + # reclaim ~50 MiB. MUST run before the --update below so the rebuilt db uses + # the smaller map. BENCH-VERIFY the exec-guard still works end-to-end (boot + + # kiosk render + admin SSH + sudo): undersize this and fapolicyd hits + # MDB_MAP_FULL, allowlisting breaks, and login/kiosk exec is bricked (keep + # fapolicyd working — it's the writable-overlay exec guard dm-verity can't + # provide). Can drop lower after measuring `fapolicyd-cli --dump | wc -l` on + # the bench; see docs/TODO.md. + " if [ -f /etc/fapolicyd/fapolicyd.conf ]; then sed -ri 's/^[[:space:]]*#?[[:space:]]*db_max_size[[:space:]]*=.*/db_max_size = 50/' /etc/fapolicyd/fapolicyd.conf; grep -q '^db_max_size' /etc/fapolicyd/fapolicyd.conf || printf 'db_max_size = 50\\n' >> /etc/fapolicyd/fapolicyd.conf; echo '==> fapolicyd: db_max_size set to 50 MiB (was EL9 default ~100)'; fi", " for f in /usr/local/bin/kiosk-session /usr/local/bin/fleet-sb-enroll /usr/local/bin/fleet-hostapi /usr/sbin/fleet-enrol; do { [ -f \"$f\" ] && fapolicyd-cli --file add \"$f\"; } || true; done", " fapolicyd-cli --update 2>/dev/null || true", " echo '==> fapolicyd: trusted custom binaries (kiosk-session, fleet-sb-enroll, fleet-hostapi) + rebuilt trust db'", @@ -914,6 +940,22 @@ build { "if [ -d /usr/lib64/chromium-browser/locales ]; then find /usr/lib64/chromium-browser/locales -type f -name '*.pak' ! -name 'en-US.pak' -delete; fi", # App translation catalogs (.mo) for locales this fleet never uses. "[ -d /usr/share/locale ] && find /usr/share/locale -mindepth 1 -maxdepth 1 -type d ! -name 'en*' ! -name 'C*' -exec rm -rf {} + || true", + # .pyc invalidation vs reproducible mtimes: the erofs is built `-T 0`, which + # zeroes every file's mtime. Python's DEFAULT .pyc invalidation is + # timestamp-based (the .pyc header stores the source's mtime), so at runtime + # every source .py looks newer than its baked .pyc and Python recompiles into + # the tmpfs overlay — wasting RAM and, under enforcing SELinux, throwing + # lib_t write/unlink AVCs (seen 2026-07-03: dnf/libdnf/rhsm __pycache__ + # rewrites 10 min after boot). Recompile ALL bytecode with hash-based, + # UNCHECKED invalidation: the .pyc then carries a source hash flagged + # 'do not check', so Python loads it WITHOUT stat-ing the source, and never + # rewrites regardless of the `-T 0` mtimes. Bonus: hash .pyc embed no mtime, + # so they're more reproducible than timestamp .pyc (reinforces the -T 0 + # invariant). MUST run LAST (after every dnf install/remove) so nothing lays + # down a fresh timestamp .pyc afterward; -f forces existing .pyc to this mode. + # Version-agnostic loop so an EL point-release python bump doesn't silently + # skip it. See docs/TODO.md #16. + "for d in /usr/lib/python3.* /usr/lib64/python3.*; do [ -d \"$d\" ] && python3 -m compileall -f -q --invalidation-mode unchecked-hash \"$d\" 2>/dev/null || true; done", ] } diff --git a/docs/TODO.md b/docs/TODO.md index 66ba6d4..c6c01ba 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1461,6 +1461,15 @@ tailoring; retires deviation-register #4). unchecked-hash` over the Python trees BEFORE the erofs build → mtime-independent `.pyc`, no rewrites, and MORE reproducible. (Worth doing regardless of firewalld — it's the true root cause of the firewalld AVCs seen 2026-07-03.) + **[x] DONE (2026-07-03):** implemented in `diskless.pkr.hcl` — (a) the compileall + `--invalidation-mode unchecked-hash` pass over `/usr/lib*/python3.*` at the end of the + 2d hygiene block (last step before erofs extraction), and (b) masked + `dnf-makecache.timer` next to the existing `rhsmcertd` mask (it was the observed ~10-min + post-boot trigger that re-imported the dnf Python stack). Confirmed live on a booted host + via the overlay audit: `dnf/libdnf/librepo/rhsm` `__pycache__` rewritten at boot+10min. + **BENCH-VERIFY:** after a rebuild, no more `.pyc` files appear in the overlay + (`find / -xdev -newermt '1980-01-01' -path '*__pycache__*'` empty) and the `lib_t` + `.pyc` AVCs stop in the SIEM. Items 1 & 3 (firewalld enablement + tailoring) still OPEN. 3. **Tailoring:** re-select the 4 firewalld rules. **THE load-bearing bench gate (why #4 deferred it):** the NVMe/TCP root is an OUTBOUND @@ -1469,3 +1478,44 @@ connection to `:4420` established in the initramfs BEFORE firewalld starts. Veri default → should, but UNVERIFIED); a reconnect within `ctrl_loss_tmo` re-establishes *through* the firewall; DHCP renew + MKA keepalives unaffected; SSH in; no AVCs; verity hash reproducible. If it drops the root, that is the fleet-wide outage risk — gate hard. + +## [~] 17. Cap fapolicyd trust-DB RAM (`db_max_size`) — biggest overlay consumer + +**Finding (overlay RAM audit, 2026-07-03, host `cygnus-cruiser-99`):** the writable root +is `systemd.volatile=overlay` (a tmpfs). Enumerating runtime-written files via the `-T 0` +trick — every baked file has mtime = epoch, so anything newer is an overlay write: + +```bash +sudo find / -xdev -type f -newermt '1980-01-01' -printf '%b\n' | awk '{s+=$1*512} END{print s/1048576" MiB"}' +``` + +showed **124.5 MiB** total, of which **`/var/lib/fapolicyd/data.mdb` = 100 MiB, FULLY +allocated** (`du` == `du --apparent-size`, not sparse) — **~80% of the entire overlay.** +fapolicyd's LMDB trust DB is rebuilt at boot onto the tmpfs overlay, so its `db_max_size` +(MB, in `/etc/fapolicyd/fapolicyd.conf`) is a **fixed RAM reservation**, not a ceiling it +grows into. The EL9 default is ~100; the minimal+kiosk trust set is a fraction of that. + +**[x] DONE (2026-07-03):** `diskless.pkr.hcl` 2c-STIG-postfix now seds +`db_max_size = 50` into `fapolicyd.conf` BEFORE the `fapolicyd-cli --update` rebuild (so +the rebuilt db uses the smaller map). Value 50 chosen conservatively — it was fapolicyd's +upstream default for years and covered a FULL workstation rpmdb (larger than this image), +so it has real headroom. Reclaims ~50 MiB of overlay RAM. + +**[ ] BENCH-VERIFY (STIG-critical — fapolicyd is the writable-overlay exec guard):** +1. Boot the rebuilt image; confirm kiosk renders, admin SSH (ECDSA) logs in, `sudo` + works, and custom binaries (`kiosk-session`, `fleet-hostapi`, `fleet-enrol`) still + execute — i.e. the trust DB rebuilt cleanly at 50 MiB with no `MDB_MAP_FULL`. + Undersizing bricks exec (login/kiosk) — that's the hard gate. +2. Re-run the overlay audit: `data.mdb` should now be ~50 MiB and total overlay ~75 MiB. +3. **Can go lower:** measure the real trust set with `fapolicyd-cli --dump | wc -l` on the + booted host; if the content is only ~15–25 MiB, 40 (or less) is safe and reclaims more. + +**Secondary lever (free, already in flight):** the trust DB scales with trusted files, so +the kiosk weak-deps trim (commit `a193dc3`, ~100 fewer packages) shrinks the *actual* +`data.mdb` content — widening the headroom under `db_max_size`. + +**Other overlay consumers noted (not fapolicyd, for context):** Chromium `/dev/shm` +(~38 MiB, its own tmpfs) + kiosk `/tmp/kiosk-profile` (on the root overlay here — `/tmp` +is NOT a separate tmpfs, ties to #15's mount separation); `/var/log/{messages,audit}` +grow unbounded in RAM (audit is local-only) — logrotate caps or SIEM offload on +long-lived hosts.