From c0a08438b2ac8a39606eb23d9536fe2091028744 Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 03:27:58 +0000 Subject: [PATCH 01/13] kiosk: disable weak deps + add package-footprint audit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The kiosk dnf transaction was measured at 229 pkgs / 1.0 GB. Nearly all of the non-core packages are Recommends: a locked-down single-URL kiosk never uses flatpak + its ostree/bubblewrap sandbox runtime, tracker-miners (a filesystem indexer — dead weight and a threat surface), the pipewire/ pulseaudio audio stack, or libproxy-webkitgtk4 -> webkit2gtk3-jsc (an 8.9M second browser engine dragged into a Chromium kiosk for libproxy's PAC runner). Add --setopt=install_weak_deps=False to cut them, and an rpm -qa size manifest to the build log so the next CI build reports the exact new footprint (turning 'seems unnecessary' into data for the next trim round: gstreamer/media codecs, or mesa-dri-drivers if the GPU path is abandoned). The immovable floor is chromium + chromium-common + llvm-libs (mesa sw-GL) + gtk3/adwaita + libicu, all hard deps. BENCH-VERIFY the kiosk still renders with weak deps off (watch for a missing font or libva). --- diskless.pkr.hcl | 22 +++++++++++++++++++++- docs/TODO.md | 13 +++++++++++++ 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/diskless.pkr.hcl b/diskless.pkr.hcl index a0855ad..b7a85fd 100644 --- a/diskless.pkr.hcl +++ b/diskless.pkr.hcl @@ -459,7 +459,27 @@ build { # the heaviest items here and are likely unnecessary given weston # --use-pixman + chromium --disable-gpu — candidate to drop once the kiosk # is display-verified; see TODO #4.) - "dnf install -y --setopt=tsflags=nodocs weston chromium mesa-dri-drivers mesa-libEGL mesa-libgbm dejavu-sans-fonts", + # + # install_weak_deps=False: a locked-down single-URL kiosk needs none of the + # Recommends: this transaction otherwise drags in — flatpak + its ostree/ + # bubblewrap sandbox runtime, tracker-miners (a filesystem indexer, both + # dead weight AND a threat surface), the pipewire/pulseaudio audio stack, + # and libproxy-webkitgtk4 → webkit2gtk3-jsc (a whole SECOND browser engine + # pulled into a Chromium kiosk just for libproxy's PAC runner). Measured: + # the default (weak-deps=on) transaction was 229 pkgs / 1.0 GB. The floor + # that survives is chromium + chromium-common + llvm-libs (mesa sw-GL) + + # gtk3/adwaita + libicu — all HARD deps. If a genuine runtime dep turns out + # to be a weak dep (e.g. a font, or libva for VA-API on the GPU path), add + # it to the explicit list below rather than re-enabling weak deps wholesale. + "dnf install -y --setopt=tsflags=nodocs --setopt=install_weak_deps=False weston chromium mesa-dri-drivers mesa-libEGL mesa-libgbm dejavu-sans-fonts", + # Footprint audit (surfaces in the CI build log): the installed set, largest + # first, + a total count. Turns 'seems unnecessary' into data for the next + # trim round (e.g. dropping gstreamer/media codecs if the SPA plays no HTML5 + # media, or mesa-dri-drivers if the GPU path is abandoned). %%{...} is escaped + # for Packer's HCL2 template engine (a bare %%{ is parsed as a directive). + "echo '==> kiosk package footprint (installed size, largest 40 first):'", + "rpm -qa --qf '%%{SIZE}\\t%%{NAME}-%%{VERSION}-%%{RELEASE}\\n' | sort -rn | head -40", + "echo \"==> total packages installed after kiosk: $(rpm -qa | wc -l)\"", # Unprivileged, passwordless kiosk user. Groups: 'video' (KMS /dev/dri/card0 # for weston) + 'input' (libinput) as a seat fallback; 'render' # (/dev/dri/renderD128) so Chromium's GPU process can do hardware raster on diff --git a/docs/TODO.md b/docs/TODO.md index a0c251d..66ba6d4 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -306,6 +306,19 @@ step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source trim, ship via CI `enable_kiosk=true`) and the phone-home/NTP lockdown (done) — all captured in the sub-notes below. Secure-path render check is folded into #8. +**Package-footprint trim (2026-07-03):** the kiosk dnf transaction was measured at +**229 pkgs / 1.0 GB** (weak deps ON). Added `--setopt=install_weak_deps=False` to +the install in `diskless.pkr.hcl` — drops the flatpak/ostree sandbox runtime, +tracker-miners (fs indexer = dead weight + threat surface), the pipewire/pulseaudio +audio stack, and `libproxy-webkitgtk4 → webkit2gtk3-jsc` (an 8.9M *second* browser +engine). Also added an `rpm -qa --qf %%{SIZE}` size-manifest diagnostic to the build +log to quantify the result. The immovable floor is chromium + chromium-common + +llvm-libs (mesa sw-GL) + gtk3/adwaita + libicu — all HARD deps. **BENCH-VERIFY:** +with weak deps off, confirm the kiosk still renders (no genuine runtime dep was a +weak dep — watch for a missing font or `libva`). NEXT trim candidates (measure via +the new manifest): gstreamer/media codecs if the SPA plays no HTML5 media; +mesa-dri-drivers + `--use-pixman`-only if the GPU path is abandoned. + **WORKING / display-verified 2026-06-17** (kiosk renders Chromium fullscreen on the configured URL). Build-flagged OFF by default for now (`enable_kiosk=false`) From 9464e7fc2aca22368a7b418eaf53922b243d0bed Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 05:23:49 +0000 Subject: [PATCH 02/13] =?UTF-8?q?trim:=20slim=20image=20~800=20MB=20?= =?UTF-8?q?=E2=80=94=20glibc=20langpacks=20en-only=20+=20linux-firmware=20?= =?UTF-8?q?strip=20(TODO=20#18)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2b rpm-size manifest revealed the biggest packages aren't the kiosk but base bloat: linux-firmware 678 MB and glibc-all-langpacks 228 MB (vs chromium 370 + llvm 129). The uncompressed erofs is read over NVMe/TCP on every boot, so this is fleet-wide dead weight + attack surface. glibc: swap glibc-all-langpacks (every language) for glibc-minimal-langpack + glibc-langpack-en. ~220 MB; image is English-only. linux-firmware: a live fleet audit (modinfo -F firmware over loaded modules + sysfs/lspci driver bindings on the real 820-class hosts and the VMs) proved the only firmware any host loads is Intel i915 (GPU). Every NIC is firmware-free (e1000e reports on-chip NVM, not a fw file; virtio_net loads nothing), so the catastrophic missing-NIC-firmware case is off the table. Keep i915 + intel/ (ice DDP insurance) + mellanox + intel-ucode; strip the ~600 MB of foreign vendor blobs (amdgpu/nvidia/radeon/ath/qcom/...). Remove iwl*-firmware (Wi-Fi, ~83 MB) on a wired-only fleet. Strip-in-place since the package can't be removed without losing i915; du reports the result in the build log. BENCH-VERIFY: kiosk still renders on the 820 (i915 DMC/GuC load) — the one real risk; widen the keep-list if any host has a non-Intel GPU or an ice/mlx5 NIC. --- diskless.pkr.hcl | 34 ++++++++++++++++++++++++++++++++++ docs/TODO.md | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+) diff --git a/diskless.pkr.hcl b/diskless.pkr.hcl index b7a85fd..2a02198 100644 --- a/diskless.pkr.hcl +++ b/diskless.pkr.hcl @@ -296,6 +296,16 @@ build { # talks to the TPM directly via go-tpm. So this list stays at the baseline.) "dnf -y install wpa_supplicant tpm2-tools tpm2-tss nvme-cli cryptsetup dracut-network NetworkManager zstd openssl", + # Locale trim: @^minimal-environment pulls glibc-all-langpacks — ~228 MB of + # locale data for EVERY language (measured 2026-07-03, 3rd-largest package in + # the whole image). This fleet is English-only (we also strip + # /usr/share/locale in 2d). Swap it for the English langpack + the C/POSIX + # minimal set → reclaims ~220 MB of the read-only erofs. Install the small + # replacements first, then drop the monolith (nothing Requires it; strict so + # a surprise dep surfaces in CI rather than silently bloating the image). + "dnf -y install glibc-minimal-langpack glibc-langpack-en", + "dnf -y remove glibc-all-langpacks", + # EFI / Secure Boot management tools (efivar, mokutil, efibootmgr — all base # repos). Used over SSH on a workstation: `mokutil --sb-state/--pk/--kek/--db` # reads Secure Boot state + the enrolled PK/KEK/db; efibootmgr manages boot @@ -914,6 +924,30 @@ build { "if [ -d /usr/lib64/chromium-browser/locales ]; then find /usr/lib64/chromium-browser/locales -type f -name '*.pak' ! -name 'en-US.pak' -delete; fi", # App translation catalogs (.mo) for locales this fleet never uses. "[ -d /usr/share/locale ] && find /usr/share/locale -mindepth 1 -maxdepth 1 -type d ! -name 'en*' ! -name 'C*' -exec rm -rf {} + || true", + # Firmware slimming (biggest single win — see docs/TODO #18). linux-firmware + # ships ~678 MB of blobs for thousands of devices this fixed-hardware fleet + # will never have (amdgpu/nvidia/radeon/ath/qcom/rtl/…). A live fleet audit + # (2026-07-03, `modinfo -F firmware` across all loaded modules + `lspci`/sysfs + # driver bindings on the real 820-class hosts and the VMs) proved the ONLY + # loadable firmware ANY host requests is Intel **i915** (GPU). Every other + # driver is firmware-free: NICs are e1000e / virtio_net (e1000e's ethtool + # "firmware-version" is on-chip NVM, not a /usr/lib/firmware file), storage + # ahci, USB xhci, audio snd_hda_intel — none call request_firmware(). So NO + # NIC needs firmware, which removes the catastrophic case. KEEP: i915 (GPU, + # all Intel gens for mixed models) + intel/ (ice DDP insurance for the + # ice/mlx5 server-NIC build variant + small Intel blobs) + mellanox + + # intel-ucode; STRIP the rest (~600 MB off the uncompressed erofs). Risk + # asymmetry: missing NIC fw would brick the fleet (none needed) but missing + # GPU fw only degrades a display. BENCH-VERIFY the kiosk still renders on the + # 820 (i915 DMC/GuC load) after this; widen the keep-list if a future host + # has a non-Intel GPU or an ice/mlx5 NIC. Strip-in-place (rm, not rpm -e): + # linux-firmware can't be removed as a package without also losing i915. + "if [ -d /usr/lib/firmware ]; then find /usr/lib/firmware -mindepth 1 -maxdepth 1 ! -name i915 ! -name intel ! -name mellanox ! -name intel-ucode -exec rm -rf {} + ; fi", + # Wi-Fi firmware: wired-only fleet (no iwlwifi). Remove the packages cleanly + # so the rpmdb stays honest (the find above already dropped their top-level + # iwlwifi-*.ucode files). Glob quoted so the shell doesn't expand it. + "dnf remove -y 'iwl*-firmware' 2>/dev/null || true", + "echo \"==> firmware after strip: $(du -sh /usr/lib/firmware 2>/dev/null | cut -f1) (was ~678 MB)\"", ] } diff --git a/docs/TODO.md b/docs/TODO.md index 66ba6d4..a59ce99 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1469,3 +1469,42 @@ connection to `:4420` established in the initramfs BEFORE firewalld starts. Veri default → should, but UNVERIFIED); a reconnect within `ctrl_loss_tmo` re-establishes *through* the firewall; DHCP renew + MKA keepalives unaffected; SSH in; no AVCs; verity hash reproducible. If it drops the root, that is the fleet-wide outage risk — gate hard. + +## [~] 18. Slim the image — locale + firmware (biggest erofs-size wins) + +**Finding (image-size audit via the 2b `rpm -qa --qf '%{SIZE}'` manifest, 2026-07-03):** +the largest packages in the 534-package image are NOT the kiosk — they're base bloat: +`linux-firmware` **678 MB**, `glibc-all-langpacks` **228 MB**, then chromium (370) + +llvm-libs (129). The uncompressed erofs is read over NVMe/TCP on every boot of every +host, so every dead byte is fleet-wide waste + attack surface. + +**[x] DONE — glibc langpacks → en-only** (`diskless.pkr.hcl` step 2). `@^minimal-environment` +pulls `glibc-all-langpacks` (all languages); swapped for `glibc-minimal-langpack` + +`glibc-langpack-en`. ~220 MB reclaimed; image is English-only (we also strip +`/usr/share/locale` in 2d). **BENCH-VERIFY:** locale still `en_US.UTF-8`; kiosk + login OK. + +**[x] DONE — linux-firmware strip to a hardware-audited keep-list** (`diskless.pkr.hcl` 2d). +Live fleet audit (`modinfo -F firmware` over all loaded modules + sysfs/`lspci` driver +bindings on the real 820-class hosts AND the VMs) proved the **only** loadable firmware +any host requests is Intel **i915** (GPU). Every other driver is firmware-free — +crucially **e1000e** (on-chip NVM, not a fw file) and **virtio_net**, so NO NIC needs +firmware and the fleet-brick case is off the table. Keep-list: `i915` + `intel/` (ice +DDP insurance for the `ice`/`mlx5` server-NIC build variant + small Intel blobs) + +`mellanox` + `intel-ucode`; strip the rest (amdgpu/nvidia/radeon/ath/qcom/rtl/iwlwifi… +~600 MB). Wi-Fi (`iwl*-firmware`, ~83 MB) removed as packages (wired-only fleet). +Strip-in-place (`find … -exec rm -rf`, not `rpm -e` — can't drop the package without +losing i915); a `du -sh /usr/lib/firmware` line reports the result in the build log. +**BENCH-VERIFY:** (a) kiosk still RENDERS on the 820 after the strip (i915 DMC/GuC still +load — the one real risk); (b) if any host has a non-Intel GPU or an `ice`/`mlx5` NIC, +widen the keep-list before trusting fleet-wide. Audited models so far: HP-820-class +(i915 + e1000e + iwlwifi-dropped) and virtio VMs. + +**Combined:** ~**800 MB** off the uncompressed erofs (langpacks + firmware), dwarfing the +kiosk weak-deps trim (#4). NEXT candidates from the manifest (measure, don't guess): +`iso-codes` (19 MB), `samba-client-libs`/`intel-mediasdk`/`gstreamer` (chromium-common +HARD deps — only removable by dropping HTML5 media), `binutils` (25 MB, build-only?), +`cracklib-dicts` (9.8 MB, needed for pam pwquality — keep). + +**Stack note:** committed on `trim-rpm`. When `optimise-ram` rebases onto the new +`trim-rpm`, expect a trivial `docs/TODO.md` tail conflict (both branches appended after +#16 — `optimise-ram` added #17 fapolicyd, this added #18); resolve by keeping both. From f59eb7942a087ab0818450925ecc9620264a3613 Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 05:36:51 +0000 Subject: [PATCH 03/13] trim: run firmware strip pre-dracut so the UKI initramfs shrinks too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the linux-firmware strip (prev commit put it in the 2d hygiene block, which runs AFTER the dracut build — so it only shrank the rootfs erofs, not the initramfs). Move the strip to step 2, before the dracut build: with --no-hostonly, plymouth's drm dep instmods a broad DRM set and dracut bundles each module's firmware into the initramfs (amdgpu being the classic UKI bloater). Trimming first means dracut never sees it, so both the rootfs erofs AND the UKI initramfs shrink. Keep-list is unchanged (i915 + intel/ + mellanox + intel-ucode), so the fleet splash GPU (i915) and ice DDP firmware the initramfs actually needs are retained; dracut just warns 'firmware not found' for the removed drm modules and continues. Add a post-build diagnostic (initramfs .img size + bundled firmware, expected i915 only) so the initrd win is measured in the build log, not estimated. --- diskless.pkr.hcl | 60 +++++++++++++++++++++++++++++------------------- docs/TODO.md | 7 +++++- 2 files changed, 42 insertions(+), 25 deletions(-) diff --git a/diskless.pkr.hcl b/diskless.pkr.hcl index 2a02198..46d55ff 100644 --- a/diskless.pkr.hcl +++ b/diskless.pkr.hcl @@ -406,6 +406,32 @@ build { "install -D -m0755 /tmp/fleet-enrol /usr/sbin/fleet-enrol", "restorecon /usr/sbin/fleet-enrol 2>/dev/null || true", + # ---- Firmware slimming (BEFORE dracut so the initramfs shrinks too) --- + # linux-firmware ships ~678 MB of blobs for hardware this fixed fleet will + # never have. A live fleet audit (`modinfo -F firmware` over loaded modules + + # sysfs/`lspci` driver bindings on the real 820-class hosts AND the VMs, + # 2026-07-03) proved the ONLY loadable firmware any host requests is Intel + # i915 (GPU); every NIC is firmware-free (e1000e's ethtool "firmware-version" + # is on-chip NVM, not a /usr/lib/firmware file; virtio_net loads nothing), so + # NO NIC needs firmware. This runs PRE-dracut deliberately: --no-hostonly + + # plymouth's `drm` dep instmods a broad DRM set (i915/amdgpu/radeon/…) and + # dracut bundles each module's firmware into the initramfs (amdgpu fw is the + # classic UKI bloater), so trimming FIRST shrinks BOTH the rootfs erofs AND + # the UKI's initramfs. Keep i915 (all Intel gens) + intel/ (ice DDP for the + # ice/mlx5 server-NIC build variant + small Intel blobs) + mellanox + + # intel-ucode; strip the rest (~600 MB). Safe for the initramfs: the fleet + # splash GPU (i915) fw is kept and the build-VM splash uses virtio-gpu/bochs + # (no fw) — dracut just warns "firmware not found" for the removed drm modules + # and continues. Risk asymmetry: missing NIC fw would brick the fleet (none + # needed), missing GPU fw only degrades a display. BENCH-VERIFY the kiosk + # renders on the 820. Strip-in-place (rm, not rpm -e): linux-firmware can't be + # dropped as a package without also losing i915. See docs/TODO #18. + "if [ -d /usr/lib/firmware ]; then find /usr/lib/firmware -mindepth 1 -maxdepth 1 ! -name i915 ! -name intel ! -name mellanox ! -name intel-ucode -exec rm -rf {} + ; fi", + # Wi-Fi firmware: wired-only fleet (no iwlwifi). Remove the packages cleanly + # so the rpmdb stays honest (the find above dropped their iwlwifi-*.ucode). + "dnf remove -y 'iwl*-firmware' 2>/dev/null || true", + "echo \"==> firmware after strip: $(du -sh /usr/lib/firmware 2>/dev/null | cut -f1) (was ~678 MB)\"", + # ---- Build the fleet-generic initramfs ------------------------------ # %%{ escapes the rpm query tags so Packer's HCL template engine treats # them as literals (a bare %{...} is parsed as a template directive). @@ -416,6 +442,13 @@ build { # under pipefail the pipeline reports failure even when the file is # present (a false negative that wrongly failed the build for many runs). "lsinitrd /boot/initramfs-generic.img > /tmp/lsinitrd.txt", + # Initramfs footprint (this .img is embedded in the signed UKI and pulled + # over PXE/HTTP on every boot). Report the built size + any firmware that + # still made it in — post-strip this should be i915 only (pulled by + # plymouth's drm dep); if amdgpu/radeon/etc. appear, the strip didn't run + # before dracut. See docs/TODO #18. + "echo \"==> initramfs size: $(du -h /boot/initramfs-generic.img | cut -f1)\"", + "echo '==> firmware bundled in initramfs (expect i915 only):'; grep -i '/firmware/' /tmp/lsinitrd.txt || echo ' (none)'", "grep -E 'macsec|wpa_supplicant|tpm2_nvread|nvme.tcp|erofs|veritysetup' /tmp/lsinitrd.txt", # Assert NetworkManager is in the initramfs — the network-manager dracut # module is what renews the DHCP lease post-switch-root. A missing/renamed @@ -924,30 +957,9 @@ build { "if [ -d /usr/lib64/chromium-browser/locales ]; then find /usr/lib64/chromium-browser/locales -type f -name '*.pak' ! -name 'en-US.pak' -delete; fi", # App translation catalogs (.mo) for locales this fleet never uses. "[ -d /usr/share/locale ] && find /usr/share/locale -mindepth 1 -maxdepth 1 -type d ! -name 'en*' ! -name 'C*' -exec rm -rf {} + || true", - # Firmware slimming (biggest single win — see docs/TODO #18). linux-firmware - # ships ~678 MB of blobs for thousands of devices this fixed-hardware fleet - # will never have (amdgpu/nvidia/radeon/ath/qcom/rtl/…). A live fleet audit - # (2026-07-03, `modinfo -F firmware` across all loaded modules + `lspci`/sysfs - # driver bindings on the real 820-class hosts and the VMs) proved the ONLY - # loadable firmware ANY host requests is Intel **i915** (GPU). Every other - # driver is firmware-free: NICs are e1000e / virtio_net (e1000e's ethtool - # "firmware-version" is on-chip NVM, not a /usr/lib/firmware file), storage - # ahci, USB xhci, audio snd_hda_intel — none call request_firmware(). So NO - # NIC needs firmware, which removes the catastrophic case. KEEP: i915 (GPU, - # all Intel gens for mixed models) + intel/ (ice DDP insurance for the - # ice/mlx5 server-NIC build variant + small Intel blobs) + mellanox + - # intel-ucode; STRIP the rest (~600 MB off the uncompressed erofs). Risk - # asymmetry: missing NIC fw would brick the fleet (none needed) but missing - # GPU fw only degrades a display. BENCH-VERIFY the kiosk still renders on the - # 820 (i915 DMC/GuC load) after this; widen the keep-list if a future host - # has a non-Intel GPU or an ice/mlx5 NIC. Strip-in-place (rm, not rpm -e): - # linux-firmware can't be removed as a package without also losing i915. - "if [ -d /usr/lib/firmware ]; then find /usr/lib/firmware -mindepth 1 -maxdepth 1 ! -name i915 ! -name intel ! -name mellanox ! -name intel-ucode -exec rm -rf {} + ; fi", - # Wi-Fi firmware: wired-only fleet (no iwlwifi). Remove the packages cleanly - # so the rpmdb stays honest (the find above already dropped their top-level - # iwlwifi-*.ucode files). Glob quoted so the shell doesn't expand it. - "dnf remove -y 'iwl*-firmware' 2>/dev/null || true", - "echo \"==> firmware after strip: $(du -sh /usr/lib/firmware 2>/dev/null | cut -f1) (was ~678 MB)\"", + # (linux-firmware slimming moved to step 2, BEFORE the dracut build, so the + # UKI initramfs shrinks too — see TODO #18. Nothing re-adds firmware after + # that point, so the rootfs erofs is already trimmed by the time we get here.) ] } diff --git a/docs/TODO.md b/docs/TODO.md index a59ce99..c9b0e1f 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1483,7 +1483,12 @@ pulls `glibc-all-langpacks` (all languages); swapped for `glibc-minimal-langpack `glibc-langpack-en`. ~220 MB reclaimed; image is English-only (we also strip `/usr/share/locale` in 2d). **BENCH-VERIFY:** locale still `en_US.UTF-8`; kiosk + login OK. -**[x] DONE — linux-firmware strip to a hardware-audited keep-list** (`diskless.pkr.hcl` 2d). +**[x] DONE — linux-firmware strip to a hardware-audited keep-list** (`diskless.pkr.hcl` +step 2, run **BEFORE the dracut build** so the UKI initramfs shrinks too — not just the +rootfs. `--no-hostonly` + plymouth's `drm` dep instmods a broad DRM set and dracut bundles +each module's firmware into the initramfs, amdgpu being the classic UKI bloater; the strip +runs first so dracut never sees it. A post-build diagnostic reports the `.img` size + any +firmware still bundled — expected i915 only.) Live fleet audit (`modinfo -F firmware` over all loaded modules + sysfs/`lspci` driver bindings on the real 820-class hosts AND the VMs) proved the **only** loadable firmware any host requests is Intel **i915** (GPU). Every other driver is firmware-free — From 5cc72ccac31361bd37101352bc4427712250e84e Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 05:55:00 +0000 Subject: [PATCH 04/13] =?UTF-8?q?trim:=20drop=20nss-mdns=20(+=20orphaned?= =?UTF-8?q?=20avahi)=20=E2=80=94=20media-independent=20mDNS=20resolver?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Investigating avahi/samba/rabbitmq in the image: ldd on libavformat.so confirms samba (libsmbclient), librabbitmq, librist, libsrt and libzmq are all DT_NEEDED by FFmpeg's libavformat-free — removing any breaks libavformat and thus future HTML5 media in the kiosk, so they stay. avahi is the exception: it's absent from libavformat's links and 'rpm -e --test avahi' shows its only reverse-dep is nss-mdns (the mDNS/.local NSS resolver). The fleet resolves via DHCP/DNS (boot.fleet.lab), never .local, so mDNS is dead weight and a network-listening service unwanted on a locked-down kiosk. Remove nss-mdns; dnf's clean_requirements_on_remove reaps avahi/avahi-glib/avahi-libs as orphans. Belt-and-braces alongside install_weak_deps=False (which already drops nss-mdns if it came in as a Recommends). --- diskless.pkr.hcl | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/diskless.pkr.hcl b/diskless.pkr.hcl index 46d55ff..6edd4fa 100644 --- a/diskless.pkr.hcl +++ b/diskless.pkr.hcl @@ -941,6 +941,17 @@ build { # hard-needs it) and mask as a backstop against future weak-dep reinstalls. "dnf remove -y firewalld 2>/dev/null || true", "systemctl mask firewalld.service 2>/dev/null || true", + # nss-mdns: the mDNS/.local NSS resolver, and the ONLY reverse-dep of avahi + # (confirmed 2026-07-03: `rpm -e --test avahi` → "avahi is needed by nss-mdns", + # and avahi is absent from `ldd libavformat` so it's unrelated to Chromium's + # media stack — unlike samba/librabbitmq/librist/libsrt/libzmq, which ARE + # libavformat DT_NEEDED and must stay for future HTML5 media). This fleet + # resolves via DHCP/DNS (boot.fleet.lab), never .local, so mDNS is dead weight + # AND a network-listening service unwanted on a locked-down kiosk. Removing + # nss-mdns lets dnf's clean_requirements_on_remove reap avahi/avahi-glib/ + # avahi-libs as orphans. (install_weak_deps=False may already drop nss-mdns if + # it's a Recommends; this is the belt-and-braces for the hard-dep case.) + "dnf remove -y nss-mdns 2>/dev/null || true", # (No Go toolchain to remove — fleet-hostapi + fleet-enrol are built OUT OF # the VM on the CI runner and file-provisioned in as static binaries.) # The build-time STIG reports must not ship inside the read-only rootfs. From 0bfe8728285d8640a8da95f4e5fc125f4b3ac237 Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 06:14:08 +0000 Subject: [PATCH 05/13] nvmf: targeted per-version connect (rd.nvmf.nqn) instead of connect-all MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hosts connected with 'nvme connect-all', which attaches EVERY subsystem in the discovery log. Each host then mounts only its own by-UUID verity namespaces but leaves a live controller on every other served version's subsystem. Observed via spdk-release list: more controllers than workstations, each box on every version (1/3/3/1/3 across 5 versions for ~3 boxes). Beyond wasted connections (O(hosts x versions)), it defeated the rolling-update drain/reap: no version ever reached 0 connections while any host was up, so spdk-loader's auto-reap could never fire and old versions accumulated without bound. Fix — targeted connect to this build's subsystem only: - extract-artifacts.sh bakes the per-version SUBNQN (already in verity.json .subnqn, matched by the serving side) into the signed UKI cmdline as rd.nvmf.nqn=. NQN is version identity, not per-host state, and the cmdline is already per-version + immutable-per-signed-UKI, so invariant #1 holds. - nvmf-connect.sh (test path) uses 'nvme connect -n '; legacy UKIs without rd.nvmf.nqn fall back to connect-all with a warning so they still boot. - dot1x-macsec-up.sh (secure path) gains an explicit step-6 targeted connect over macsec0 (after it's addressed, per invariant #2), replacing the delegation to the dracut nvmf module's discovery connect-all. Now each host = 1 controller on its own subsystem; versions drain to 0 when their hosts migrate, so auto-reap works and connections scale O(hosts). Requires a rebuild + re-sign (cmdline change, invariant #6). BENCH-VERIFY both paths; confirm the dracut nvmf module isn't also connect-all-ing on the secure path. CLAUDE.md invariant #2 + TODO #6 updated. --- CLAUDE.md | 8 +++- docs/TODO.md | 27 ++++++++++++ .../dracut/95dot1x-macsec/dot1x-macsec-up.sh | 44 ++++++++++++++++++- files/dracut/95dot1x-macsec/nvmf-connect.sh | 23 ++++++++-- scripts/extract-artifacts.sh | 15 +++++-- 5 files changed, 107 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 5a1aed0..44d6144 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -115,9 +115,13 @@ instead of working around it. `rd.dot1x.nvindex=` / `fleet-enrol -handle -nvindex`.) 1. **The hook ordering in `dot1x-macsec-up.sh` is load-bearing.** NV-cert extraction → supplicant start → wait for `Authorized` + macsec0 → - DHCP **on macsec0** → hostnqn derivation. Never DHCP before auth + DHCP **on macsec0** → hostnqn derivation → **targeted NVMe/TCP connect** + (`nvme connect -n $rd.nvmf.nqn`, step 6). Never DHCP before auth completes (pre-auth leases are invalid after VLAN assignment); never - connect NVMe before macsec0 is addressed. + connect NVMe before macsec0 is addressed. The connect is **targeted to this + version's subsystem** (`rd.nvmf.nqn`, baked per-version into the signed + cmdline) — never a discovery `connect-all`, which attaches every served + version and defeats the per-version drain/reap gate (see TODO #6). 1. **The supplicant must survive switch-root.** It is launched with `exec -a @wpa_supplicant` (systemd root-storage-daemon convention) and its state lives under `/run`. Do not “clean up” the `@`, move state off diff --git a/docs/TODO.md b/docs/TODO.md index a0c251d..9c8afca 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -675,6 +675,33 @@ must be built around **never restarting the process that holds live connections. the current UKI), so auto-reap is safe; removal is still drain-gated + grace-debounced. The loader + target share the RPC socket on a named volume (`nvmf_tgt -r`); the loader reuses the SPDK image (has python3 + rpc.py). + - **BENCH FINDING 2026-07-03 (FIXED) — hosts did `connect-all`, so NO version + ever drained → auto-reap could never fire.** Observed: `spdk-release list` + showed more controllers than workstations, with every box appearing on *every* + served version's subsystem (e.g. 1/3/3/1/3 across 5 versions for ~3 boxes). + Root cause: the initramfs connected with **`nvme connect-all`** (test path + `nvmf-connect.sh`; secure path delegated to the dracut nvmf module's discovery + autoconnect), which attaches EVERY subsystem in the discovery log — the host + then mounts only its own by-UUID verity namespaces but leaves a live controller + on all the others. So each running host pinned every version above 0 conns, and + `spdk-loader`'s "auto-reap once drained (0 conns)" **could never trigger** while + the fleet was up — the served-version list grew without bound. This defeats the + whole drain/reap model, not just cosmetics. **Fix:** targeted connect. The + per-version subsystem NQN (`SUBNQN`, already in `verity.json .subnqn`) is now + also baked into the signed UKI cmdline as **`rd.nvmf.nqn=`** (`extract-artifacts.sh`), + and both connect paths use **`nvme connect -n `** — `nvmf-connect.sh` (test) + and a new explicit step 6 in `dot1x-macsec-up.sh` (secure, replacing the module + delegation; runs after macsec0 is addressed per invariant #2). Legacy UKIs + without `rd.nvmf.nqn` fall back to `connect-all` (with a warning) so they still + boot. Now each host = 1 controller on its own subsystem → versions drain to 0 + when their hosts move off → auto-reap works; connections scale O(hosts) not + O(hosts × versions). Requires a rebuild + re-sign (cmdline change, invariant #6). + **BENCH-VERIFY:** (a) test path — after rebuild, `spdk-release list` shows each + box on ONLY its version; old versions reap after hosts migrate; (b) secure path + — the new step-6 connect brings up the NVMe root over macsec0 (this path was not + yet bench-confirmed end-to-end); (c) confirm the dracut nvmf module isn't ALSO + connect-all-ing on the secure path (if it is, suppress it — the targeted connect + is meant to be the sole connector). - **BENCH FINDING 2026-07-03 (FIXED) — the target WAS still recreated on an `IMAGE_VERSION` bump.** Despite the above, the `nvmeof` (spdk-target) service still carried a versioned `type: image` bootstrap mount diff --git a/files/dracut/95dot1x-macsec/dot1x-macsec-up.sh b/files/dracut/95dot1x-macsec/dot1x-macsec-up.sh index 5f55cc4..cbc70ed 100755 --- a/files/dracut/95dot1x-macsec/dot1x-macsec-up.sh +++ b/files/dracut/95dot1x-macsec/dot1x-macsec-up.sh @@ -5,8 +5,9 @@ # 2. start wpa_supplicant (argv[0]=@wpa_supplicant => survives switch-root) # 3. wait for EAP success + macsec0 # 4. DHCP on macsec0 (the pre-auth lease, if any, is invalid post-auth/VLAN) -# 5. derive hostnqn/hostid from the cert so dracut's nvmf module can -# connect the NVMe/TCP root namespace over macsec0 +# 5. derive hostnqn/hostid from the cert +# 6. TARGETED NVMe/TCP connect over macsec0 (rd.nvmf.nqn -> this version's +# subsystem only, not a discovery connect-all of every served version) # Idempotent: guarded by a stamp file because initqueue may re-run hooks. type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh @@ -159,5 +160,44 @@ if [ ! -s /etc/nvme/hostnqn ]; then info "dot1x-macsec: hostnqn $(cat /etc/nvme/hostnqn)" fi +# --- 6. connect the NVMe/TCP root over macsec0 (TARGETED to this version) ---- +# Explicit targeted connect. This hook used to only set /etc/nvme/hostnqn and +# leave the connect to the dracut nvmf module's discovery mechanism — a +# `connect-all` that attaches EVERY served version's subsystem to this host, +# inflating target controllers to O(hosts x versions) and keeping old versions +# pinned above zero connections so the per-version drain/reap gate never fires +# (see TODO #6). rd.nvmf.nqn pins THIS build's subsystem (baked per-version into +# the signed cmdline by extract-artifacts.sh, matching verity.json .subnqn on the +# target); nvme uses the hostnqn/hostid written in step 5. Load-bearing order +# (invariant #2): this runs AFTER macsec0 is addressed — never connect NVMe +# before macsec0 has an IP. Keep-alive tuning matches the test path (-k 30 long +# KATO, -l -1 retry forever, -c 10) so the root link survives kiosk CPU pressure +# and blips. BENCH-VERIFY: the secure path is not yet confirmed end-to-end. +_disc=$(getarg rd.nvmf.discover=) +_nqn=$(getarg rd.nvmf.nqn=) +if [ -n "$_disc" ] && [ -n "$_nqn" ] && [ ! -e /run/dot1x/.nvmf-connected ]; then + _trtype=${_disc%%,*}; _rest=${_disc#*,} + _traddr=${_rest%%,*}; _trsvcid=${_rest##*,} + modprobe nvme_tcp 2>/dev/null + ply nvme-active + i=0 + while [ "$i" -lt "$TIMEOUT" ]; do + nvme connect -t "$_trtype" -a "$_traddr" -s "$_trsvcid" -n "$_nqn" \ + -k 30 -l -1 -c 10 2>/dev/null + for _ns in /dev/nvme[0-9]*n[0-9]*; do + if [ -b "$_ns" ]; then + : > /run/dot1x/.nvmf-connected + ply nvme-done + info "dot1x-macsec: NVMe root connected over macsec0 ($_ns)" + break 2 + fi + done + sleep 1; i=$((i + 1)) + done + [ -e /run/dot1x/.nvmf-connected ] || warn "dot1x-macsec: NVMe/TCP connect to $_nqn timed out after ${TIMEOUT}s" +elif [ -z "$_nqn" ]; then + warn "dot1x-macsec: rd.nvmf.nqn not set (legacy UKI) — leaving NVMe connect to the dracut nvmf module (connect-all)" +fi + : > "$STAMP" exit 0 diff --git a/files/dracut/95dot1x-macsec/nvmf-connect.sh b/files/dracut/95dot1x-macsec/nvmf-connect.sh index cb29e60..090f3a6 100644 --- a/files/dracut/95dot1x-macsec/nvmf-connect.sh +++ b/files/dracut/95dot1x-macsec/nvmf-connect.sh @@ -26,6 +26,15 @@ _rest=${_disc#*,} _traddr=${_rest%%,*} _trsvcid=${_rest##*,} +# Targeted connect: rd.nvmf.nqn pins THIS build's subsystem (baked per-version +# into the signed cmdline by extract-artifacts.sh, matching verity.json .subnqn +# on the target). This replaces a discovery `connect-all`, which attached EVERY +# served version's subsystem to every host — inflating target controllers and +# keeping old versions pinned above zero connections so they never drained/reaped +# (see TODO #6). If rd.nvmf.nqn is absent (a legacy UKI), fall back to connect-all +# so the host still boots — with a warning. +_nqn=$(getarg rd.nvmf.nqn=) + # nvme_tcp is usually auto-loaded; ensure it so /dev/nvme-fabrics exists. modprobe nvme_tcp 2>/dev/null @@ -50,11 +59,19 @@ _ctrl_loss_tmo=-1 _reconnect_delay=10 # Networking (ip=dhcp) is brought up in parallel, so poll-and-connect until a -# fabrics namespace appears or we time out. connect-all is idempotent. +# fabrics namespace appears or we time out. Both connect forms are idempotent +# (re-connecting an already-attached subsystem just errors, harmlessly). +[ -n "$_nqn" ] || echo "nvmf-connect: rd.nvmf.nqn not set — falling back to connect-all (legacy UKI; attaches every served version)" >&2 _i=0 while [ "$_i" -lt 60 ]; do - nvme connect-all -t "$_trtype" -a "$_traddr" -s "$_trsvcid" \ - -k "$_kato" -l "$_ctrl_loss_tmo" -c "$_reconnect_delay" 2>/dev/null + if [ -n "$_nqn" ]; then + # TARGETED: attach only this build's subsystem (its data + hash namespaces). + nvme connect -t "$_trtype" -a "$_traddr" -s "$_trsvcid" -n "$_nqn" \ + -k "$_kato" -l "$_ctrl_loss_tmo" -c "$_reconnect_delay" 2>/dev/null + else + nvme connect-all -t "$_trtype" -a "$_traddr" -s "$_trsvcid" \ + -k "$_kato" -l "$_ctrl_loss_tmo" -c "$_reconnect_delay" 2>/dev/null + fi for _ns in /dev/nvme[0-9]*n[0-9]*; do if [ -b "$_ns" ]; then mkdir -p /run/dot1x diff --git a/scripts/extract-artifacts.sh b/scripts/extract-artifacts.sh index c781f8b..c95753b 100755 --- a/scripts/extract-artifacts.sh +++ b/scripts/extract-artifacts.sh @@ -73,9 +73,17 @@ ROOT_DATA_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_DATA_UUID}" ROOT_HASH_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_HASH_UUID}" # Per-version subsystem NQN (distinct NQN per release) — lets a host attach to # exactly one version's subsystem and gives clean per-release drain/allow-listing -# on the long-lived target. Written to verity.json for the serving side; NOT in -# the cmdline (host still discovers + connects; the by-UUID verity refs select -# what mounts). An explicit SUBNQN env wins. +# on the long-lived target. Written to verity.json for the serving side AND baked +# into the UKI cmdline as rd.nvmf.nqn= (below), so the host does a TARGETED +# `nvme connect -n ` instead of a discovery `connect-all`. connect-all +# attached EVERY served version's subsystem to EVERY host, which inflated target +# controllers to O(hosts x versions) and — worse — kept every old version pinned +# above zero connections so the per-version drain/reap gate could never fire +# (spdk-loader never reaps; see TODO #6). The NQN is version identity, not per-host +# state, and the cmdline is already per-version + immutable-per-signed-UKI +# (roothash, by-id UUIDs), so baking it does NOT violate invariant #1. It MUST +# match verity.json .subnqn (same var here) so host and target agree. Explicit +# SUBNQN env wins. SUBNQN_BASE=${SUBNQN_BASE:-nqn.2026-06.org.example.fleet:rootfs} SUBNQN=${SUBNQN:-${SUBNQN_BASE}-${IMAGE_VERSION}} @@ -226,6 +234,7 @@ printf '%s' \ "rd.dot1x.nvindex=0x01800100 rd.dot1x.handle=0x81000100 rd.dot1x.timeout=60 " \ "rd.fleet.enrol_url=${ENROL_URL} " \ "rd.nvmf.discover=tcp,${NVMET_TRADDR},4420 " \ + "rd.nvmf.nqn=${SUBNQN} " \ "root=/dev/mapper/root rootfstype=erofs rootflags=ro " \ "systemd.verity=yes roothash=${ROOT_HASH} " \ "systemd.verity_root_data=${ROOT_DATA_DEV} " \ From 853ae20e68c4bec65e54dfcc1713c012a1254884 Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 06:36:32 +0000 Subject: [PATCH 06/13] trim: omit unused GPU drivers + tighten firmware keep-list (TODO #18) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First trim build confirmed firmware 678 MB -> 147 MB, but the initramfs came out 97 MB and is now kernel-MODULE-bound, not firmware: --no-hostonly + plymouth's drm dep instmods every GPU driver (amdgpu.ko etc.) though the fleet is i915-only. - omit_drivers += amdgpu radeon nouveau (90-diskless.conf): drop the discrete-GPU kmods (amdgpu.ko is the biggest single module in the initramfs); keep i915 + virtio-gpu/bochs for the splash. Shrinks the initramfs/UKI pulled over PXE on every boot. - Firmware keep-list tightened: drop mellanox/ (mlxsw_spectrum SWITCH-ASIC blobs, not NIC fw — mlx5 NIC fw is on-card) and narrow intel/ to just ice (drop ish/sof/ibt sensor-hub/audio/bluetooth, none loaded at boot per the audit). - Add an 'initramfs largest files' diagnostic (sort lsinitrd.txt by size) so the next build shows what dominates the .img. BENCH-VERIFY: kiosk still renders on the 820 (i915 kept); watch the new initramfs largest-files log line for the post-omit size. --- diskless.pkr.hcl | 19 ++++++++++++---- docs/TODO.md | 28 ++++++++++++++++++++---- files/etc/dracut.conf.d/90-diskless.conf | 8 +++++++ 3 files changed, 47 insertions(+), 8 deletions(-) diff --git a/diskless.pkr.hcl b/diskless.pkr.hcl index 6edd4fa..e82712d 100644 --- a/diskless.pkr.hcl +++ b/diskless.pkr.hcl @@ -417,16 +417,23 @@ build { # plymouth's `drm` dep instmods a broad DRM set (i915/amdgpu/radeon/…) and # dracut bundles each module's firmware into the initramfs (amdgpu fw is the # classic UKI bloater), so trimming FIRST shrinks BOTH the rootfs erofs AND - # the UKI's initramfs. Keep i915 (all Intel gens) + intel/ (ice DDP for the - # ice/mlx5 server-NIC build variant + small Intel blobs) + mellanox + - # intel-ucode; strip the rest (~600 MB). Safe for the initramfs: the fleet + # the UKI's initramfs. Keep i915 (all Intel gens — the fleet's only + # firmware-loading GPU) + intel/ice (the ice server-NIC DDP, the SOLE intel/ + # blob any host needs — ish/sof/ibt are sensor-hub/audio/bluetooth, none + # loaded at boot) + intel-ucode; strip the rest (~600 MB). NOT mellanox/ — + # those are mlxsw_spectrum SWITCH-ASIC blobs, not NIC firmware (mlx5 NIC fw is + # on-card, never request_firmware'd from /usr/lib/firmware), so they're dead + # weight even for an mlx5 build. Safe for the initramfs: the fleet # splash GPU (i915) fw is kept and the build-VM splash uses virtio-gpu/bochs # (no fw) — dracut just warns "firmware not found" for the removed drm modules # and continues. Risk asymmetry: missing NIC fw would brick the fleet (none # needed), missing GPU fw only degrades a display. BENCH-VERIFY the kiosk # renders on the 820. Strip-in-place (rm, not rpm -e): linux-firmware can't be # dropped as a package without also losing i915. See docs/TODO #18. - "if [ -d /usr/lib/firmware ]; then find /usr/lib/firmware -mindepth 1 -maxdepth 1 ! -name i915 ! -name intel ! -name mellanox ! -name intel-ucode -exec rm -rf {} + ; fi", + "if [ -d /usr/lib/firmware ]; then find /usr/lib/firmware -mindepth 1 -maxdepth 1 ! -name i915 ! -name intel ! -name intel-ucode -exec rm -rf {} + ; fi", + # Narrow intel/ to just ice (NIC DDP); drop ish (sensor hub), sof (audio DSP), + # ibt (bluetooth) etc. — none are loaded on this fleet (audit 2026-07-03). + "[ -d /usr/lib/firmware/intel ] && find /usr/lib/firmware/intel -mindepth 1 -maxdepth 1 ! -name ice -exec rm -rf {} + || true", # Wi-Fi firmware: wired-only fleet (no iwlwifi). Remove the packages cleanly # so the rpmdb stays honest (the find above dropped their iwlwifi-*.ucode). "dnf remove -y 'iwl*-firmware' 2>/dev/null || true", @@ -449,6 +456,10 @@ build { # before dracut. See docs/TODO #18. "echo \"==> initramfs size: $(du -h /boot/initramfs-generic.img | cut -f1)\"", "echo '==> firmware bundled in initramfs (expect i915 only):'; grep -i '/firmware/' /tmp/lsinitrd.txt || echo ' (none)'", + # Largest files IN the initramfs (in-cpio size, before the outer zstd) — after + # the firmware strip the .img is module-bound (the --no-hostonly DRM/net/crypto + # set), so this surfaces the next trim targets (e.g. GPU driver kmods). See #18. + "echo '==> initramfs largest files:'; awk '/^-/{print $5, $NF}' /tmp/lsinitrd.txt | sort -rn | head -25", "grep -E 'macsec|wpa_supplicant|tpm2_nvread|nvme.tcp|erofs|veritysetup' /tmp/lsinitrd.txt", # Assert NetworkManager is in the initramfs — the network-manager dracut # module is what renews the DHCP lease post-switch-root. A missing/renamed diff --git a/docs/TODO.md b/docs/TODO.md index c9b0e1f..328db7a 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1504,11 +1504,31 @@ load — the one real risk); (b) if any host has a non-Intel GPU or an `ice`/`ml widen the keep-list before trusting fleet-wide. Audited models so far: HP-820-class (i915 + e1000e + iwlwifi-dropped) and virtio VMs. +**Build-verified 2026-07-03:** firmware **678 MB → 147 MB**; the initramfs diagnostic +confirms only the keep-list is bundled (i915 all-gens + intel/ice + intel/ish + mellanox) +— proving the pre-dracut ordering works. But the initramfs came out **97 MB**, and it's +now **kernel-MODULE-bound, not firmware** (~10 MB fw): the `--no-hostonly` + plymouth +`drm` set pulls every GPU driver (`amdgpu.ko` etc.) even though the fleet is i915-only. + +**[x] DONE — follow-on trims (2026-07-03):** +- **`omit_drivers+=" amdgpu radeon nouveau "`** in `90-diskless.conf` — drops the + discrete-GPU kmods plymouth's `drm` dep would instmod (amdgpu.ko is the biggest single + module); keeps i915 + virtio-gpu/bochs for the splash. Targets the 97 MB initramfs/UKI. +- **Firmware keep-list tightened**: dropped `mellanox/` (those are `mlxsw_spectrum` + SWITCH-ASIC blobs, NOT NIC fw — mlx5 NIC fw is on-card) and narrowed `intel/` to just + `ice` (dropped ish/sof/ibt = sensor-hub/audio/bluetooth, none loaded at boot). +- **Initramfs top-25-files diagnostic** added after the dracut build (sorts `lsinitrd.txt` + by size) so the next build shows what dominates the `.img` — measure before the next cut. +- **BENCH-VERIFY:** kiosk still renders on the 820 (i915 kept); check the new + "initramfs largest files" log line for the post-omit size + next targets. + **Combined:** ~**800 MB** off the uncompressed erofs (langpacks + firmware), dwarfing the -kiosk weak-deps trim (#4). NEXT candidates from the manifest (measure, don't guess): -`iso-codes` (19 MB), `samba-client-libs`/`intel-mediasdk`/`gstreamer` (chromium-common -HARD deps — only removable by dropping HTML5 media), `binutils` (25 MB, build-only?), -`cracklib-dicts` (9.8 MB, needed for pam pwquality — keep). +kiosk weak-deps trim (#4), plus a smaller UKI/initramfs from the DRM-driver omit. NEXT +candidates from the manifest (measure, don't guess): `iso-codes` (19 MB), +`samba-client-libs`/`intel-mediasdk`/`gstreamer` (chromium-common HARD deps — only +removable by dropping HTML5 media, and samba/rabbitmq/rist/srt/zmq are libavformat +DT_NEEDED so they can't go without breaking future media — confirmed via ldd 2026-07-03), +`binutils` (25 MB, build-only?), `cracklib-dicts` (9.8 MB, needed for pam pwquality — keep). **Stack note:** committed on `trim-rpm`. When `optimise-ram` rebases onto the new `trim-rpm`, expect a trivial `docs/TODO.md` tail conflict (both branches appended after diff --git a/files/etc/dracut.conf.d/90-diskless.conf b/files/etc/dracut.conf.d/90-diskless.conf index a58470e..49184a3 100644 --- a/files/etc/dracut.conf.d/90-diskless.conf +++ b/files/etc/dracut.conf.d/90-diskless.conf @@ -25,5 +25,13 @@ add_dracutmodules+=" dot1x-macsec network-manager nvmf systemd-veritysetup " add_dracutmodules+=" plymouth " filesystems+=" erofs overlay " add_drivers+=" macsec nvme-tcp nvme-fabrics dm-verity @NIC_DRIVERS@ " +# Omit the discrete-GPU DRM drivers plymouth's `drm` dep would otherwise instmod +# in --no-hostonly mode. This fleet is Intel i915 ONLY (the build VM uses +# virtio-gpu/bochs), so amdgpu/radeon/nouveau are dead weight — amdgpu.ko alone is +# the single biggest module in the initramfs. i915 + virtio-gpu/bochs stay for the +# splash. (Their FIRMWARE is already stripped pre-dracut in diskless.pkr.hcl; this +# drops the MODULES too.) If a future host has a non-Intel GPU, remove it here. +# See docs/TODO #18. +omit_drivers+=" amdgpu radeon nouveau " install_items+=" /etc/dot1x/radius-ca.pem /etc/dot1x/wpa_supplicant-dot1x.conf " compress="zstd" From bf0c61a025166e9663ae7ce081436a1c6758abb3 Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 06:55:52 +0000 Subject: [PATCH 07/13] docs/TODO #18: record f59eb79 built + booted successfully MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ~800 MB trim (glibc langpacks en-only + linux-firmware strip to the i915 keep-list, run pre-dracut) builds and boots — the boot/root path survives it (firmware 678->147 MB, initramfs bundles only the keep-list). Notes still-open checks: kiosk RENDER on real i915 hardware ('booted' != 'display up'), and that the two later commits (nss-mdns drop 5cc72cc; omit_drivers + keep-list tightening 853ae20) are NOT in f59eb79 and need separate bench-verify. --- docs/TODO.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/TODO.md b/docs/TODO.md index 328db7a..1cdd7f2 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1499,10 +1499,16 @@ DDP insurance for the `ice`/`mlx5` server-NIC build variant + small Intel blobs) ~600 MB). Wi-Fi (`iwl*-firmware`, ~83 MB) removed as packages (wired-only fleet). Strip-in-place (`find … -exec rm -rf`, not `rpm -e` — can't drop the package without losing i915); a `du -sh /usr/lib/firmware` line reports the result in the build log. -**BENCH-VERIFY:** (a) kiosk still RENDERS on the 820 after the strip (i915 DMC/GuC still -load — the one real risk); (b) if any host has a non-Intel GPU or an `ice`/`mlx5` NIC, -widen the keep-list before trusting fleet-wide. Audited models so far: HP-820-class -(i915 + e1000e + iwlwifi-dropped) and virtio VMs. +**BOOT-VERIFIED 2026-07-03 (commit `f59eb79`): built AND booted successfully** — langpacks +en-only + the pre-dracut firmware strip (keep i915/intel/mellanox/intel-ucode) do NOT +break the boot/root path (firmware 678→147 MB, initramfs bundles only the keep-list). +Still to confirm on that build: (a) the kiosk actually **RENDERS** (i915 DMC/GuC load) — +"booted" ≠ "display up"; only exercised on real i915 hardware (the 820), not a virtio VM; +(b) if any host has a non-Intel GPU or an `ice`/`mlx5` NIC, widen the keep-list before +trusting fleet-wide. **NOT yet in `f59eb79`** (bench-verify separately): the `nss-mdns`/ +avahi drop (`5cc72cc`) and the `omit_drivers` GPU-kmod trim + keep-list tightening — +dropped `mellanox/`, narrowed `intel/` to `ice` only (`853ae20`). Audited models so far: +HP-820-class (i915 + e1000e + iwlwifi-dropped) and virtio VMs. **Build-verified 2026-07-03:** firmware **678 MB → 147 MB**; the initramfs diagnostic confirms only the keep-list is bundled (i915 all-gens + intel/ice + intel/ish + mellanox) From 8e105c8bf1cc47e0b6e977a5c35d5c281a733b54 Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 06:58:52 +0000 Subject: [PATCH 08/13] docs/TODO #18: kiosk render bench-verified on VM + 820 (i915 firmware confirmed) f59eb79 now fully bench-verified: the fleet SPA renders on both the VM (virtio-gpu) and the 820 (i915), so the i915 DMC/GuC firmware loads on real hardware and the firmware keep-list is validated end-to-end. The two later commits (nss-mdns drop, omit_drivers + keep-list tightening) don't touch i915, so render should be safe, but still need their own build+boot. --- docs/TODO.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/TODO.md b/docs/TODO.md index 1cdd7f2..ffbf6fb 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -1499,16 +1499,16 @@ DDP insurance for the `ice`/`mlx5` server-NIC build variant + small Intel blobs) ~600 MB). Wi-Fi (`iwl*-firmware`, ~83 MB) removed as packages (wired-only fleet). Strip-in-place (`find … -exec rm -rf`, not `rpm -e` — can't drop the package without losing i915); a `du -sh /usr/lib/firmware` line reports the result in the build log. -**BOOT-VERIFIED 2026-07-03 (commit `f59eb79`): built AND booted successfully** — langpacks -en-only + the pre-dracut firmware strip (keep i915/intel/mellanox/intel-ucode) do NOT -break the boot/root path (firmware 678→147 MB, initramfs bundles only the keep-list). -Still to confirm on that build: (a) the kiosk actually **RENDERS** (i915 DMC/GuC load) — -"booted" ≠ "display up"; only exercised on real i915 hardware (the 820), not a virtio VM; -(b) if any host has a non-Intel GPU or an `ice`/`mlx5` NIC, widen the keep-list before -trusting fleet-wide. **NOT yet in `f59eb79`** (bench-verify separately): the `nss-mdns`/ -avahi drop (`5cc72cc`) and the `omit_drivers` GPU-kmod trim + keep-list tightening — -dropped `mellanox/`, narrowed `intel/` to `ice` only (`853ae20`). Audited models so far: -HP-820-class (i915 + e1000e + iwlwifi-dropped) and virtio VMs. +**BENCH-VERIFIED 2026-07-03 (commit `f59eb79`): built, booted, AND the kiosk SPA RENDERS +on both the VM (virtio-gpu) and the 820 (i915).** This clears the firmware strip's one +real risk — the i915 DMC/GuC blobs load on real hardware, so the keep-list +(i915/intel/mellanox/intel-ucode, pre-dracut) is validated end-to-end (firmware 678→147 MB, +initramfs bundles only the keep-list). Remaining caveat: if a future host has a non-Intel +GPU or an `ice`/`mlx5` NIC, widen the keep-list before trusting fleet-wide. **NOT yet in +`f59eb79`** (bench-verify separately — none touch i915, so render should be safe): the +`nss-mdns`/avahi drop (`5cc72cc`) and the `omit_drivers` GPU-kmod trim + keep-list +tightening — dropped `mellanox/`, narrowed `intel/` to `ice` only (`853ae20`). Audited +models: HP-820-class (i915 + e1000e + iwlwifi-dropped) and virtio VMs. **Build-verified 2026-07-03:** firmware **678 MB → 147 MB**; the initramfs diagnostic confirms only the keep-list is bundled (i915 all-gens + intel/ice + intel/ish + mellanox) From 0d725af716accc383180d2614db516c6188b7956 Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 07:33:46 +0000 Subject: [PATCH 09/13] =?UTF-8?q?ci:=20add=20registry-gc=20=E2=80=94=20pru?= =?UTF-8?q?ne=20orphan=20container=20images=20for=20deleted=20branches?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reconciliation workflow (schedule + workflow_dispatch): lists live branches and every container version across the four images this repo builds (fleet-boot, fleet-ipxe, fleet-dnsmasq, spdk-nvmf) and deletes dev/branch builds whose branch no longer exists, via the Gitea Packages API. Not event-driven, so it catches branches deleted any way and never depends on a 'delete' webhook. Safety (per request — never touch release tags or latest): - Only tags ending in the dev commit-sha suffix '-<7 hex>' are ever candidates. Release/version tags (1.2.3, v1.2.3, spdk v24.09, dnsmasq main-) and 'latest' never carry that suffix, so they're structurally out of scope; there's also an explicit 'latest' skip. Verified by simulation across representative tags incl. spdk's -- form and pre-release semver. - Keeps any image whose sanitized branch is still live (+ always 'main'); branch sanitization mirrors the build workflows. - Fail-safe: aborts before deleting anything if the live-branch list can't be fetched (0 branches), so an API/auth blip can't trigger mass deletion. - Conservative matching: a suffix collision only ever causes a KEEP. DRY RUN by default: workflow_dispatch previews unless dry_run=false; scheduled runs dry-run until armed via repo variable REGISTRY_GC_ENABLE=true (so a scheduled dry-run can be eyeballed first). Uses the existing PACKAGE_TOKEN (write:package). shell: bash (runner defaults to sh). --- .gitea/workflows/registry-gc.yaml | 162 ++++++++++++++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 .gitea/workflows/registry-gc.yaml diff --git a/.gitea/workflows/registry-gc.yaml b/.gitea/workflows/registry-gc.yaml new file mode 100644 index 0000000..31640e8 --- /dev/null +++ b/.gitea/workflows/registry-gc.yaml @@ -0,0 +1,162 @@ +# .gitea/workflows/registry-gc.yaml +# +# Garbage-collect ORPHANED container images: dev/branch builds whose branch no +# longer exists. Reconciliation model (NOT event-driven) — it lists the live +# branches and every container version, then prunes the mismatch. So it catches +# branches deleted any way (UI, `git push --delete`, API) and never depends on a +# `delete` webhook the runner might not support or might miss. +# +# ┌─ SAFETY — what it will NEVER delete ───────────────────────────────────────┐ +# │ * `latest`. │ +# │ * ANY release / version tag — semver `1.2.3` / `v1.2.3`, spdk `v24.09`, the │ +# │ dnsmasq `main-…` line, etc. The guard: only tags ending in a `-<7 hex>` │ +# │ commit-sha suffix are ever CANDIDATES, and ONLY dev/branch builds get │ +# │ that suffix (`-`, spdk `--`). Release │ +# │ tags never carry it, so they are structurally out of scope. │ +# │ * Any image whose (sanitized) branch is still LIVE, and anything on `main`. │ +# │ * Anything it cannot confidently classify (fail-safe = KEEP). │ +# │ * Everything, if the live-branch list can't be fetched (aborts first). │ +# └────────────────────────────────────────────────────────────────────────────┘ +# +# DRY RUN by default. `workflow_dispatch` previews unless dispatched with +# dry_run=false. SCHEDULED runs also dry-run until you ARM them by setting the +# repo variable REGISTRY_GC_ENABLE=true — so you can eyeball a scheduled dry-run +# first, then flip the variable to let it prune automatically. +# +# Needs the existing PACKAGE_TOKEN secret (write:package scope, which the +# registry-push token already has). Only touches images THIS repo builds; the +# fleet-kiosk image is owned by the kiosk-app submodule's own CI. + +name: registry-gc +run-name: prune orphan container images (dry_run default; ${{ github.event_name }}) + +on: + schedule: + - cron: "17 4 * * *" # nightly ~04:17 UTC + workflow_dispatch: + inputs: + dry_run: + description: "Preview only (true) or actually delete (false)" + type: boolean + default: true + +concurrency: + group: registry-gc + cancel-in-progress: false + +jobs: + prune: + runs-on: ubuntu-latest + env: + # API base — hardcoded to match the registry host baked into the other + # workflows' image names (git.bitasylum.org/audiobuzz/*). Keep in sync. + SERVER: "https://git.bitasylum.org" + OWNER: ${{ github.repository_owner }} + REPO: ${{ github.repository }} + TOKEN: ${{ secrets.PACKAGE_TOKEN }} + EVENT: ${{ github.event_name }} + DISPATCH_DRY: ${{ inputs.dry_run }} + GC_ENABLE: ${{ vars.REGISTRY_GC_ENABLE }} + # Only images built by THIS repo's workflows. + IMAGES: "fleet-boot fleet-ipxe fleet-dnsmasq spdk-nvmf" + steps: + - name: Reconcile registry against live branches + # The runner defaults `run:` to sh (dash); this script uses bashisms + # (process substitution, arrays) — force bash. (Cf. gitea-main-ci.yaml.) + shell: bash + run: | + set -euo pipefail + command -v jq >/dev/null 2>&1 || { sudo apt-get update -qq && sudo apt-get install -y -qq jq; } + + API="${SERVER}/api/v1" + auth=(-H "Authorization: token ${TOKEN}" -H "Accept: application/json") + + # ---- dry-run resolution ------------------------------------------ + if [ "$EVENT" = "workflow_dispatch" ]; then + DRY="${DISPATCH_DRY:-true}" + elif [ "${GC_ENABLE:-}" = "true" ]; then + DRY="false" + else + DRY="true" + echo "::notice::scheduled run is DRY — set repo variable REGISTRY_GC_ENABLE=true to arm real deletion" + fi + echo "event=$EVENT dry_run=$DRY" + + # Sanitize a branch name EXACTLY as the build workflows do, then + # lowercase (registries lowercase tags). MUST stay in sync with the + # `safe_branch` sed in {gitea-main-ci,ipxe-image,dnsmasq-image,spdk-image}.yaml. + sanitize() { printf '%s' "$1" | sed -e 's#[^A-Za-z0-9._-]#-#g' -e 's#^[.-]*##' | cut -c1-80 | tr 'A-Z' 'a-z'; } + + # ---- 1. live branches (fail-safe: abort if none) ----------------- + live=" main " # main is always protected, even if listing is odd + total=0 page=1 + while :; do + resp="$(curl -fsSL --retry 3 "${auth[@]}" "${API}/repos/${REPO}/branches?limit=50&page=${page}")" + cnt="$(printf '%s' "$resp" | jq 'length')" + [ "$cnt" -eq 0 ] && break + while IFS= read -r b; do + [ -n "$b" ] || continue + live="${live}$(sanitize "$b") " + total=$((total + 1)) + done < <(printf '%s' "$resp" | jq -r '.[].name') + page=$((page + 1)) + done + if [ "$total" -eq 0 ]; then + echo "::error::fetched 0 live branches — aborting so we never mass-delete on an API/auth failure" + exit 1 + fi + echo "live branches ($total):${live}" + + # ---- 2. reconcile each image ------------------------------------- + orphans=0 kept=0 deleted=0 failed=0 + for img in $IMAGES; do + page=1 + while :; do + resp="$(curl -fsSL --retry 3 "${auth[@]}" "${API}/packages/${OWNER}?type=container&q=${img}&limit=50&page=${page}")" + cnt="$(printf '%s' "$resp" | jq 'length')" + [ "$cnt" -eq 0 ] && break + while IFS= read -r ver; do + [ -n "$ver" ] || continue + + # PROTECT releases + latest: a tag WITHOUT the dev commit-sha + # suffix (`-<7 hex>`) is a release/version/`latest`/`main-ver` + # tag and is NEVER a candidate. This single rule keeps every real + # version tag and latest, exactly as required. + if [ "$ver" = "latest" ] || ! printf '%s' "$ver" | grep -Eq -- '-[0-9a-f]{7}$'; then + kept=$((kept + 1)); continue + fi + + # It's a dev/commit tag. KEEP iff its branch component is a LIVE + # branch (anchored so `-` and spdk `--` + # both match). Conservative: a suffix collision only ever causes a + # KEEP, never a wrongful delete. + is_live="false" + for lb in $live; do + lre="$(printf '%s' "$lb" | sed 's/\./\\./g')" # escape regex dots + if printf '%s' "$ver" | grep -Eq -- "(^|-)${lre}-[0-9a-f]{7}\$"; then + is_live="true"; break + fi + done + if [ "$is_live" = "true" ]; then kept=$((kept + 1)); continue; fi + + # Orphan: dev tag whose branch is gone. + orphans=$((orphans + 1)) + echo "ORPHAN ${img}:${ver}" + if [ "$DRY" != "true" ]; then + code="$(curl -sS -o /dev/null -w '%{http_code}' -X DELETE "${auth[@]}" "${API}/packages/${OWNER}/container/${img}/${ver}" || echo 000)" + case "$code" in + 2*) deleted=$((deleted + 1)); echo " deleted ${img}:${ver}" ;; + *) failed=$((failed + 1)); echo "::warning::delete ${img}:${ver} returned HTTP ${code}" ;; + esac + fi + done < <(printf '%s' "$resp" | jq -r --arg n "$img" '.[] | select(.name == $n) | .version') + page=$((page + 1)) + done + done + + echo "----------------------------------------------------------------" + echo "kept=${kept} orphans=${orphans} deleted=${deleted} failed=${failed} dry_run=${DRY}" + if [ "$DRY" = "true" ] && [ "$orphans" -gt 0 ]; then + echo "::notice::${orphans} orphan version(s) would be pruned. Dispatch with dry_run=false, or set REGISTRY_GC_ENABLE=true for scheduled runs, to actually delete." + fi + [ "$failed" -eq 0 ] || { echo "::error::${failed} deletion(s) failed"; exit 1; } From 51d9dc08888dc8c049e5cb7412368ed7d72cd498 Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 07:43:24 +0000 Subject: [PATCH 10/13] =?UTF-8?q?ci:=20registry-gc=20=E2=80=94=20read=20br?= =?UTF-8?q?anches=20with=20the=20Actions=20token,=20not=20PACKAGE=5FTOKEN?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dry-run failed with HTTP 403 on the branches API: PACKAGE_TOKEN is a package-scoped token (registry push/delete) and can't read repo branches. Split by capability: branch reads now use the automatic Actions token (github.token, repo read via 'permissions: contents: read'); package list + delete keep PACKAGE_TOKEN. Also replaced the bare 'curl -f' failures with a get() helper that reports the HTTP code + a token-scope hint, so an auth problem can't look like 'no results' (which would otherwise skew the reconcile). --- .gitea/workflows/registry-gc.yaml | 40 ++++++++++++++++++++++++++----- 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/.gitea/workflows/registry-gc.yaml b/.gitea/workflows/registry-gc.yaml index 31640e8..66fadf2 100644 --- a/.gitea/workflows/registry-gc.yaml +++ b/.gitea/workflows/registry-gc.yaml @@ -23,7 +23,9 @@ # repo variable REGISTRY_GC_ENABLE=true — so you can eyeball a scheduled dry-run # first, then flip the variable to let it prune automatically. # -# Needs the existing PACKAGE_TOKEN secret (write:package scope, which the +# TOKENS (two, by capability — a package token can't read branches, hence the +# original 403): branch reads use the automatic Actions token (github.token, +# repo read); package list + delete use PACKAGE_TOKEN (write:package, which the # registry-push token already has). Only touches images THIS repo builds; the # fleet-kiosk image is owned by the kiosk-app submodule's own CI. @@ -40,6 +42,9 @@ on: type: boolean default: true +permissions: + contents: read # branch listing via the auto Actions token + concurrency: group: registry-gc cancel-in-progress: false @@ -53,7 +58,8 @@ jobs: SERVER: "https://git.bitasylum.org" OWNER: ${{ github.repository_owner }} REPO: ${{ github.repository }} - TOKEN: ${{ secrets.PACKAGE_TOKEN }} + REPO_TOKEN: ${{ github.token }} # auto Actions token: repo/branch reads + PKG_TOKEN: ${{ secrets.PACKAGE_TOKEN }} # package list + delete (write:package) EVENT: ${{ github.event_name }} DISPATCH_DRY: ${{ inputs.dry_run }} GC_ENABLE: ${{ vars.REGISTRY_GC_ENABLE }} @@ -69,7 +75,27 @@ jobs: command -v jq >/dev/null 2>&1 || { sudo apt-get update -qq && sudo apt-get install -y -qq jq; } API="${SERVER}/api/v1" - auth=(-H "Authorization: token ${TOKEN}" -H "Accept: application/json") + + # Authenticated GET. $1 = which token (repo|pkg), $2 = url. Echoes the + # body on 2xx; on anything else prints an actionable error and exits + # (so a scope/auth problem can never be mistaken for "no results"). + get() { + local who="$1" url="$2" tok tmp code body + case "$who" in repo) tok="$REPO_TOKEN";; pkg) tok="$PKG_TOKEN";; esac + tmp="$(mktemp)" + code="$(curl -sS --retry 3 -o "$tmp" -w '%{http_code}' \ + -H "Authorization: token ${tok}" -H "Accept: application/json" "$url" || echo 000)" + body="$(cat "$tmp")"; rm -f "$tmp" + case "$code" in + 2*) printf '%s' "$body"; return 0 ;; + 401|403) + echo "::error::GET ${url} -> HTTP ${code} (token '${who}' lacks scope). branches need repo read (github.token); packages need read/write:package (PACKAGE_TOKEN)." >&2 + exit 1 ;; + *) + echo "::error::GET ${url} -> HTTP ${code}" >&2 + exit 1 ;; + esac + } # ---- dry-run resolution ------------------------------------------ if [ "$EVENT" = "workflow_dispatch" ]; then @@ -91,7 +117,7 @@ jobs: live=" main " # main is always protected, even if listing is odd total=0 page=1 while :; do - resp="$(curl -fsSL --retry 3 "${auth[@]}" "${API}/repos/${REPO}/branches?limit=50&page=${page}")" + resp="$(get repo "${API}/repos/${REPO}/branches?limit=50&page=${page}")" cnt="$(printf '%s' "$resp" | jq 'length')" [ "$cnt" -eq 0 ] && break while IFS= read -r b; do @@ -112,7 +138,7 @@ jobs: for img in $IMAGES; do page=1 while :; do - resp="$(curl -fsSL --retry 3 "${auth[@]}" "${API}/packages/${OWNER}?type=container&q=${img}&limit=50&page=${page}")" + resp="$(get pkg "${API}/packages/${OWNER}?type=container&q=${img}&limit=50&page=${page}")" cnt="$(printf '%s' "$resp" | jq 'length')" [ "$cnt" -eq 0 ] && break while IFS= read -r ver; do @@ -143,7 +169,9 @@ jobs: orphans=$((orphans + 1)) echo "ORPHAN ${img}:${ver}" if [ "$DRY" != "true" ]; then - code="$(curl -sS -o /dev/null -w '%{http_code}' -X DELETE "${auth[@]}" "${API}/packages/${OWNER}/container/${img}/${ver}" || echo 000)" + code="$(curl -sS -o /dev/null -w '%{http_code}' -X DELETE \ + -H "Authorization: token ${PKG_TOKEN}" \ + "${API}/packages/${OWNER}/container/${img}/${ver}" || echo 000)" case "$code" in 2*) deleted=$((deleted + 1)); echo " deleted ${img}:${ver}" ;; *) failed=$((failed + 1)); echo "::warning::delete ${img}:${ver} returned HTTP ${code}" ;; From 480d926a98ec3817b095c9c997b2aac7ec6eb1eb Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 07:48:09 +0000 Subject: [PATCH 11/13] trim: fix SIGPIPE (exit 141) in the initramfs-largest-files diagnostic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 853ae20's diagnostic did 'awk … | sort -rn | head -25'. head closes the pipe after 25 lines, sort dies of SIGPIPE (141), and step 2's set -euxo pipefail turns that into a build failure (the exact trap the lsinitrd '| grep -q' comment warns about). The kiosk step's rpm-manifest '| head' is safe only because that step runs plain set -e, no pipefail. Fix with the codebase convention: sort into a file, then head the file — no live pipe for head to break. (Bonus: the crash log confirms omit_drivers worked — amdgpu/radeon/nouveau kmods are gone from the top-25; the new biggest initramfs item is GenuineIntel.bin CPU microcode at 22 MB, a future trim candidate.) --- diskless.pkr.hcl | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/diskless.pkr.hcl b/diskless.pkr.hcl index e82712d..0854930 100644 --- a/diskless.pkr.hcl +++ b/diskless.pkr.hcl @@ -459,7 +459,12 @@ build { # Largest files IN the initramfs (in-cpio size, before the outer zstd) — after # the firmware strip the .img is module-bound (the --no-hostonly DRM/net/crypto # set), so this surfaces the next trim targets (e.g. GPU driver kmods). See #18. - "echo '==> initramfs largest files:'; awk '/^-/{print $5, $NF}' /tmp/lsinitrd.txt | sort -rn | head -25", + # Sort into a FILE then head the file — NOT `sort | head`: head closes the pipe + # after 25 lines, sort dies of SIGPIPE (exit 141), and step 2's + # `set -euxo pipefail` turns that into a build failure (same trap as the + # lsinitrd `| grep -q` note above). (The kiosk step's rpm-manifest `| head` is + # safe only because that step runs plain `set -e`, no pipefail.) + "echo '==> initramfs largest files:'; awk '/^-/{print $5, $NF}' /tmp/lsinitrd.txt | sort -rn > /tmp/initrd-sizes.txt; head -25 /tmp/initrd-sizes.txt; rm -f /tmp/initrd-sizes.txt", "grep -E 'macsec|wpa_supplicant|tpm2_nvread|nvme.tcp|erofs|veritysetup' /tmp/lsinitrd.txt", # Assert NetworkManager is in the initramfs — the network-manager dracut # module is what renews the DHCP lease post-switch-root. A missing/renamed From f882ddae1f31babcf899b1f445a01d71b502a9eb Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 10:43:24 +0000 Subject: [PATCH 12/13] docs/TODO #6: targeted-connect test-path bench-verified (0bfe872 mounts one image) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 0bfe872 builds, deploys, and the host now mounts ONLY its own version's image instead of connect-all'ing every served version — the targeted 'nvme connect -n rd.nvmf.nqn' works. Each host = 1 controller on its own subsystem, which is what lets old versions drain to 0 and spdk-loader auto-reap. Still to confirm: the reap payoff (migrate a host, watch its old version drain + reap), and the secure path's new step-6 connect over macsec0. --- docs/TODO.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/TODO.md b/docs/TODO.md index 9c8afca..5351344 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -696,12 +696,16 @@ must be built around **never restarting the process that holds live connections. boot. Now each host = 1 controller on its own subsystem → versions drain to 0 when their hosts move off → auto-reap works; connections scale O(hosts) not O(hosts × versions). Requires a rebuild + re-sign (cmdline change, invariant #6). - **BENCH-VERIFY:** (a) test path — after rebuild, `spdk-release list` shows each - box on ONLY its version; old versions reap after hosts migrate; (b) secure path - — the new step-6 connect brings up the NVMe root over macsec0 (this path was not - yet bench-confirmed end-to-end); (c) confirm the dracut nvmf module isn't ALSO - connect-all-ing on the secure path (if it is, suppress it — the targeted connect - is meant to be the sole connector). + **(a) TEST PATH BENCH-VERIFIED 2026-07-03 (commit `0bfe872`): builds, deploys, and + the host mounts ONLY ONE image** (its own version's subsystem) instead of connect- + all'ing every served version — the targeted `nvme connect -n rd.nvmf.nqn` works. + So each host now = 1 controller on its own subsystem, which is what lets old + versions drain to 0 and `spdk-loader` auto-reap. **Still to confirm:** the reap + payoff itself (migrate a host, watch its old version drain to 0 and get reaped); + (b) SECURE PATH — the new step-6 connect brings up the NVMe root over macsec0 + (this path is not yet bench-confirmed end-to-end); (c) that the dracut nvmf module + isn't ALSO connect-all-ing on the secure path (if it is, suppress it — the targeted + connect is meant to be the sole connector). - **BENCH FINDING 2026-07-03 (FIXED) — the target WAS still recreated on an `IMAGE_VERSION` bump.** Despite the above, the `nvmeof` (spdk-target) service still carried a versioned `type: image` bootstrap mount From 2bc2dafb1cdf5a7a0985af1dba68c1f1a99ca65c Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 12:21:22 +0000 Subject: [PATCH 13/13] kiosk: stop nss-mdns removal from cascading chromium out of the image MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause of the trim-rpm-48dc3e1 'flashing cursor, no kiosk' + mysteriously small (~500 MB) image: the 2d hygiene step ran 'dnf remove -y nss-mdns', but the EPEL chromium package HARD-Requires nss-mdns (Chromium uses .local mDNS hostnames for WebRTC ICE candidates). So dnf removed chromium as a dependent package and then reaped its orphaned GUI stack (avahi, at-spi2, atk, adwaita, chromium-common ...) — a ~300 MB drop. The shipped image had no browser, so kiosk-session died at the ${BROWSER:?} guard before ever starting weston: a bare KMS cursor. The earlier nss-mdns commit checked what nss-mdns NEEDS (avahi) but not what NEEDS nss-mdns (chromium). Fix: - Keep nss-mdns + avahi (satisfy chromium's Requires); instead MASK the avahi mDNS daemon/socket so nothing listens on :5353 — that was the real attack-surface concern, and nss-mdns is only an NSS module (no daemon needed). - Add a FINAL GUARD at the end of 2d: assert chromium/chromium-common/weston/mesa are still installed in the final image, else fail the BUILD. A missing browser must brick the build, never the fleet. Catches any future silent removal cascade, oscap package_*_removed rule, or autoremove. --- diskless.pkr.hcl | 46 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/diskless.pkr.hcl b/diskless.pkr.hcl index 0854930..720a79d 100644 --- a/diskless.pkr.hcl +++ b/diskless.pkr.hcl @@ -957,17 +957,25 @@ build { # hard-needs it) and mask as a backstop against future weak-dep reinstalls. "dnf remove -y firewalld 2>/dev/null || true", "systemctl mask firewalld.service 2>/dev/null || true", - # nss-mdns: the mDNS/.local NSS resolver, and the ONLY reverse-dep of avahi - # (confirmed 2026-07-03: `rpm -e --test avahi` → "avahi is needed by nss-mdns", - # and avahi is absent from `ldd libavformat` so it's unrelated to Chromium's - # media stack — unlike samba/librabbitmq/librist/libsrt/libzmq, which ARE - # libavformat DT_NEEDED and must stay for future HTML5 media). This fleet - # resolves via DHCP/DNS (boot.fleet.lab), never .local, so mDNS is dead weight - # AND a network-listening service unwanted on a locked-down kiosk. Removing - # nss-mdns lets dnf's clean_requirements_on_remove reap avahi/avahi-glib/ - # avahi-libs as orphans. (install_weak_deps=False may already drop nss-mdns if - # it's a Recommends; this is the belt-and-braces for the hard-dep case.) - "dnf remove -y nss-mdns 2>/dev/null || true", + # nss-mdns / avahi: DO NOT REMOVE — the EPEL `chromium` package hard-Requires + # nss-mdns (Chromium resolves `.local` mDNS hostnames for WebRTC ICE + # candidates, to avoid leaking local IPs in peer connections). `dnf remove + # nss-mdns` therefore takes CHROMIUM with it as a dependent package, then + # reaps its whole orphaned GUI stack (avahi, at-spi2, atk, adwaita, + # chromium-common, ...) — a ~300 MB drop that ships a browserless, bricked + # kiosk (flashing cursor; kiosk-session dies at `${BROWSER:?}`). This bit us + # 2026-07-03: the earlier commit checked what nss-mdns NEEDS (avahi) but not + # what NEEDS nss-mdns (chromium). The FINAL GUARD below now asserts the kiosk + # stack survived so this can never ship silently again. + # + # We still don't want the avahi mDNS DAEMON listening on :5353 on a + # locked-down kiosk (that was the real attack-surface concern). nss-mdns is + # only an NSS module (libnss_mdns.so) — it needs no daemon; if avahi-daemon + # isn't running, `.local` lookups just fall through. So keep the PACKAGES + # (satisfy chromium's Requires) but mask the listener: no socket, no daemon, + # dep satisfied, browser intact. ~1.5 MB kept — negligible vs a working kiosk. + "systemctl mask avahi-daemon.service avahi-daemon.socket 2>/dev/null || true", + "systemctl disable avahi-daemon.service avahi-daemon.socket 2>/dev/null || true", # (No Go toolchain to remove — fleet-hostapi + fleet-enrol are built OUT OF # the VM on the CI runner and file-provisioned in as static binaries.) # The build-time STIG reports must not ship inside the read-only rootfs. @@ -987,6 +995,22 @@ build { # (linux-firmware slimming moved to step 2, BEFORE the dracut build, so the # UKI initramfs shrinks too — see TODO #18. Nothing re-adds firmware after # that point, so the rootfs erofs is already trimmed by the time we get here.) + # + # ---- FINAL GUARD: assert the kiosk stack survived the whole pipeline ---- + # 2026-07-03 incident: chromium installed cleanly in the 2b kiosk step (the + # footprint audit showed chromium-149 + chromium-common, 523 pkgs), but by + # the final image it was GONE (472 pkgs) — `dnf remove -y nss-mdns` in 2d + # removed chromium AS A DEPENDENT PACKAGE (EPEL chromium hard-Requires + # nss-mdns for WebRTC mDNS ICE candidates), then reaped its orphaned GUI + # stack. Result: a ~500 MB image whose kiosk died at the `${BROWSER:?}` guard + # in kiosk-session — a flashing-cursor boot with no weston, shipped green. + # The nss-mdns removal is fixed above (kept + avahi listener masked), but a + # missing browser must fail the BUILD, never brick the fleet: assert every + # kiosk-critical package is still installed at the very end of the pipeline + # (after remediation AND all 2d removes). This catches ANY future silent drop + # — a new hard-Requires cascade, an oscap package_*_removed rule, autoremove. + # enable_kiosk=false legitimately ships no browser, so gate on it. + "if [ \"${var.enable_kiosk}\" = \"true\" ] && [ -n \"${var.kiosk_url}\" ]; then for p in chromium chromium-common weston mesa-dri-drivers mesa-libEGL mesa-libgbm; do rpm -q \"$p\" >/dev/null 2>&1 || { echo \"ERROR: kiosk-critical package '$p' missing from the FINAL image — a later step removed it after the 2b kiosk install (e.g. a dnf remove whose dependency cascade took chromium, cf. the 2026-07-03 nss-mdns incident). Refusing to ship a bricked kiosk.\" >&2; exit 1; }; done; echo '==> guard: kiosk stack intact (chromium/weston/mesa present after remediation + hygiene)'; fi", ] }