Audiobuzz
f59eb7942a
Follow-up to the linux-firmware strip (prev commit put it in the 2d hygiene block, which runs AFTER the dracut build — so it only shrank the rootfs erofs, not the initramfs). Move the strip to step 2, before the dracut build: with --no-hostonly, plymouth's drm dep instmods a broad DRM set and dracut bundles each module's firmware into the initramfs (amdgpu being the classic UKI bloater). Trimming first means dracut never sees it, so both the rootfs erofs AND the UKI initramfs shrink. Keep-list is unchanged (i915 + intel/ + mellanox + intel-ucode), so the fleet splash GPU (i915) and ice DDP firmware the initramfs actually needs are retained; dracut just warns 'firmware not found' for the removed drm modules and continues. Add a post-build diagnostic (initramfs .img size + bundled firmware, expected i915 only) so the initrd win is measured in the build log, not estimated.
1516 lines
102 KiB
Markdown
1516 lines
102 KiB
Markdown
# TODO — diskless fleet boot system
|
||
|
||
Persistent backlog, kept in sync with the working session task tracker. Status:
|
||
`[ ]` pending · `[~]` in progress · `[x]` done.
|
||
|
||
---
|
||
|
||
## Current build posture (LAB / test) — as of 2026-06-19
|
||
|
||
The image currently builds in **open-network test mode**. These toggles must be
|
||
flipped back for a secure/production build (consolidated as item #8):
|
||
- `disable_dot1x=true` / CI `DISABLE_DOT1X="true"` → `rd.dot1x=0` (NO 802.1X /
|
||
MACsec; open NVMe-oF). **TEMPORARY.**
|
||
- `nvmet_traddr` / CI `NVMET_TRADDR="192.168.50.10"` → hardcoded IP (should be a
|
||
per-env DNS name like `nvme.fleet.lab` once DNS-at-boot is trusted).
|
||
- Unsigned UKI (`sb_key`/`sb_cert` not passed) → Secure Boot must be OFF on the
|
||
test target. Production needs CI/HSM signing.
|
||
- `enable_kiosk=true`; `boot_splash=true`; NTP via DHCP→serving-host chrony.
|
||
- **`enable_stig` / CI `ENABLE_STIG="false"` RIGHT NOW** — temporarily disabled
|
||
to get a working box while the STIG regressions were diagnosed (now fixed,
|
||
below). **Flip back to `true`** and bench-verify the hardened build.
|
||
- **`debug_shell=true` (DEBUG HOLE — passwordless root on tty2). REMOVE before
|
||
prod** (`-var debug_shell=false`). Note: it didn't actually come up on tty2 in
|
||
testing; diagnosis ended up done via offline `rootfs.erofs` mount + SMBIOS
|
||
`enforcing=0`. SMBIOS cmdline-extra works on Proxmox only after a full VM
|
||
**stop/start** (not `reset`).
|
||
- **Admin SSH key MUST be ECDSA or RSA, NOT ed25519**, when STIG is on — the STIG
|
||
crypto policy drops ed25519 from sshd's accepted pubkey algorithms. Set the
|
||
`SSH_AUTHORIZED_KEY` CI secret to an ECDSA/RSA key (see #2).
|
||
- **Hostname:** image is hostname-neutral (empty `/etc/hostname`,
|
||
`hostname-mode=default`); the serving dnsmasq's `fleet-namer` (`--dhcp-script`)
|
||
assigns a stable creative starship name per MAC (e.g. `nebula-vanguard-07`) via
|
||
DHCP option 12 + DNS A/PTR. Host picks it up on next DHCP renewal/reboot.
|
||
- **Serving `.env`:** must set `AUDIT_HOST=audit.fleet.lab` (+ `docker compose up
|
||
-d dnsmasq` to reload) for remote log offload to resolve/flow.
|
||
|
||
What's working/verified vs awaiting bench is per-item below. Cross-cutting
|
||
bench-verify checklist lives in `CLAUDE.md` → "Known bench-verify items".
|
||
|
||
---
|
||
|
||
## [x] 1. Port the dracut module off `network-legacy` → `network-manager`
|
||
|
||
**DONE — test path VERIFIED ON THE BENCH 2026-06-17** (host stays up past the
|
||
DHCP lease time; no more lease-expiry outage). Fixed the confirmed outage:
|
||
network-legacy's one-shot DHCP lease was never renewed post-switch-root, so the
|
||
address dropped at lease expiry → NVMe/TCP root died ("keep alive timeout" /
|
||
"no route to host"). NM hands its initramfs connections to the real-root NM
|
||
(`/run/NetworkManager` state), which renews.
|
||
|
||
REMAINING: the **secure path** (macsec0 DHCP via `nmcli`, physical NIC unmanaged
|
||
via `nm-unmanaged.sh`) is only exercised once `disable_dot1x=false` — verify on
|
||
the swtpm + hostapd + FreeRADIUS bench when re-enabling security (see #2/cleanup).
|
||
|
||
Done:
|
||
- `90-diskless.conf` `add_dracutmodules`: `network-legacy` → `network-manager`.
|
||
- `module-setup.sh` `depends()` → `network-manager`; added `nmcli`; installs the
|
||
new cmdline hook.
|
||
- NEW `nm-unmanaged.sh` (cmdline hook): secure mode writes
|
||
`/run/NetworkManager/conf.d/10-dot1x-unmanaged.conf`
|
||
(`unmanaged-devices=interface-name:en*`) **before NM starts**; no-op in test
|
||
mode. Replaces the deleted static `99-dot1x-unmanaged.conf` (couldn't be
|
||
mode-aware — it wrongly unmanaged `en*` in test mode, blocking renewal).
|
||
- `dot1x-macsec-up.sh` secure path: `ifup macsec0` → `nmcli` (managed + DHCP,
|
||
poll for IPv4). macsec0 MANAGED so NM renews; physical NIC UNMANAGED.
|
||
- `diskless.pkr.hcl`: dropped `dhcp-client`, added explicit `NetworkManager`;
|
||
removed the static-config install; lsinitrd now asserts `NetworkManager` +
|
||
`nm-unmanaged.sh` present.
|
||
- Static lint clean (`bash -n`, `shellcheck`, `packer validate`).
|
||
|
||
**BENCH-VERIFY before trusting (the gate):**
|
||
- Test path (`disable_dot1x=true`): boot, `nmcli device` shows boot NIC MANAGED;
|
||
run past the DHCP lease time (drop the lab dnsmasq lease low) — address
|
||
persists, lease renews (`journalctl -u NetworkManager`), NVMe + SSH stay up.
|
||
- Secure path (swtpm + hostapd + FreeRADIUS, `disable_dot1x=false`): `en*`
|
||
UNMANAGED, `macsec0` MANAGED + renewing; MKA survives switch-root; verity root
|
||
mapped; NVMe stays connected past lease time; no "Address already assigned".
|
||
- **Risk:** ordering of the unmanaged drop-in vs NM start in secure mode. If NM
|
||
transiently grabs `en*`, fall back to also baking a static `en*`-unmanaged file
|
||
for secure-only builds.
|
||
|
||
## [~] 2. Apply RHEL9 STIG hardening to the built image
|
||
|
||
**IMPLEMENTED 2026-06-18 — awaiting build/bench verification.** OpenSCAP + SSG
|
||
remediation in a provisioner (`enable_stig` var, default on), tailored for the
|
||
diskless/UKI/kiosk architecture via `files/stig/diskless-tailoring.xml`. STIG
|
||
kernel params baked into the UKI cmdline (`extract-artifacts.sh`). Decisions:
|
||
**FIPS off**, **audit local-only**, **CI report-only**. Deviations are documented
|
||
in **[stig-deviations.md](stig-deviations.md)** (curated register) and reported
|
||
per-build by **[../scripts/stig-deviations.sh](../scripts/stig-deviations.sh)**
|
||
(non-pass rules tagged intentional vs REVIEW); CI publishes
|
||
`stig-report.html` + `stig-arf.xml` + `stig-deviations.md`. Scanner tooling is
|
||
removed from the shipped image. **BENCH-VERIFY:** boot/login **VERIFIED ON THE
|
||
BENCH 2026-06-19** — hardened build with `enable_stig=true`, host
|
||
`cygnus-cruiser-99`: kiosk auto-starts + renders, admin SSH (ECDSA) logs in,
|
||
audit offload reaches the collector, DHCP creative hostname applied, NVMe/TCP
|
||
verity root mounts and the host stays up. **STILL OPEN:** reproducible verity
|
||
hash across two builds; triage the REVIEW rows (below); the secure path (#8,
|
||
802.1X/MACsec — this was the `disable_dot1x=true` test path).
|
||
|
||
**FINDING 2026-06-18 (FIXED) — false-clean scan.** The first hardened build
|
||
(branch `hardening`, `ada4aa4`) produced a report reading "0 REVIEW / clean",
|
||
but the ARF had **0 pass, 0 fail** — only 462 `notapplicable` + 1068
|
||
`notselected`. Root cause: the provisioner selected the datastream with
|
||
`ls ssg-rhel9-ds.xml ssg-rl9-ds.xml | head -1`, which picks **rhel9**
|
||
alphabetically; the RHEL9 datastream's CPE dictionary has **no Rocky entry**, so
|
||
on Rocky 9.8 every selected rule went `notapplicable` and **nothing was actually
|
||
evaluated or remediated** — the image was unhardened despite a clean-looking
|
||
report. Fixes: (a) provisioner now maps `/etc/os-release $ID` → datastream
|
||
(Rocky→`ssg-rl9-ds.xml`); (b) provisioner **aborts the build** if the scan
|
||
evaluates 0 rules; (c) `stig-deviations.sh` flags an **INVALID SCAN** and exits
|
||
non-zero on 0-evaluated. **Re-run the hardened build** and confirm pass/fail
|
||
counts are non-zero before trusting any deviations report.
|
||
|
||
**FINDING 2026-06-18 (FIXED) — hardening locked the fleet out.** The first
|
||
genuinely-evaluated hardened image (app `215b0eb`) broke **both** login paths:
|
||
the Chromium kiosk dropped to a getty login prompt, and admin SSH was rejected
|
||
with `Permission denied (publickey)`. Root cause: STIG enables `pam_faillock`
|
||
(account phase) + password aging/inactivity, and the **kiosk + fleetadmin
|
||
accounts are passwordless / key-only**, so the hardened PAM *account* phase
|
||
treats them as expired/lockable (sshd key auth succeeds, then `pam_acct_mgmt`
|
||
fails). Fix (provisioner `2c-STIG-postfix`, runs after remediation): exempt the
|
||
two appliance accounts — `chage -M -1 -m 0 -I -1 -E -1` + `faillock --reset`;
|
||
kiosk uses a dedicated `/etc/pam.d/kiosk` (`PAMName=kiosk`, permissive account /
|
||
full session for seat); sshd gets an `account sufficient pam_succeed_if.so user
|
||
= fleetadmin` first-rule bypass. Scoped to those two accounts; everyone else
|
||
keeps full STIG. Justified as register deviation #10. **VERIFIED ON THE BENCH 2026-06-19**
|
||
(`enable_stig=true`, host `cygnus-cruiser-99`): kiosk auto-starts + renders and
|
||
admin SSH (ECDSA key) logs in with hardening on.**
|
||
|
||
**UPDATE 2026-06-19 — offline erofs inspection of the hardened image (mounted
|
||
`rootfs.erofs` read-only) pinned the real causes; the PAM-account theory above
|
||
was incomplete:**
|
||
- **Kiosk had TWO causes, both FIXED:**
|
||
1. **fapolicyd** is `enabled` and `kiosk-session` wasn't in its trust → it
|
||
blocked `/usr/local/bin/kiosk-session`. Fix: KEEP fapolicyd (it's the
|
||
exec-guard for the writable overlay dm-verity can't cover) and add the kiosk
|
||
binary to its trust DB at build (2c-STIG-postfix). Not waived.
|
||
2. **Auto-start:** kiosk was `WantedBy=graphical.target` but the host boots to
|
||
`multi-user.target` (graphical never reached) → unit `inactive (dead)` at
|
||
boot, fine on manual `restart`. Fix: `WantedBy=multi-user.target` + dropped
|
||
`set-default graphical.target` (kiosk launches weston directly, needs no DM
|
||
target).
|
||
- **SSH = STIG crypto policy excludes ed25519 (CONFIRMED via `ssh -vvv`).** The
|
||
key fingerprint MATCHED (not a mismatch); the server's `server-sig-algs`
|
||
advertised only ecdsa + rsa-sha2 — **no `ssh-ed25519`** — so the ed25519 admin
|
||
key is rejected at the signature stage (before PAM/SELinux, which is why
|
||
`enforcing=0` and the account-bypass didn't help). **Fix (your action): set the
|
||
`SSH_AUTHORIZED_KEY` CI secret to an ECDSA (`ssh-keygen -t ecdsa -b 384`) or RSA
|
||
key**, then rebuild. No code change. (Alt: re-allow ed25519 via
|
||
`PubkeyAcceptedAlgorithms +ssh-ed25519` in the sshd drop-in — but that's a
|
||
non-approved algo and would re-fail a STIG rule; prefer the ECDSA key.)
|
||
- **rsyslog TLS-offload rules waived (lab):** `rsyslog_encrypt_offload_*` force
|
||
gtls+x509+CA and broke the plaintext `2f` forward → unselected in the
|
||
tailoring; production TLS offload tracked under #8.
|
||
- Diagnosis tooling notes: SMBIOS cmdline-extra works only after a Proxmox VM
|
||
**stop/start** (not `reset`); the debug-shell-on-tty2 never came up — the
|
||
decisive diagnosis was **offline `rootfs.erofs` mount** + `ssh -vvv` +
|
||
`enforcing=0`.
|
||
|
||
**REVIEW-ROW TRIAGE — DONE 2026-06-19 (scan `app-9f7d8d1`, 21 REVIEW → 1 open).**
|
||
The post-fix hardened scan was valid (398 pass / 12 fail / 9 notchecked). All 21
|
||
REVIEW rows dispositioned (full detail: [stig-deviations.md](stig-deviations.md)
|
||
#12). Mechanics: a rule only stops being flagged REVIEW if it's `selected="false"`
|
||
in `diskless-tailoring.xml` (the register prose alone doesn't clear it) or made to
|
||
pass / N/A.
|
||
- **Bucket A (waived, register-justified):** `encrypt_partitions`,
|
||
`auditd_audispd_configure_sufficiently_large_partition` (#1); `grub2_password`
|
||
(#2); `sysctl_crypto_fips_enabled`, `harden_sshd_macs_openssh_conf_crypto_policy`,
|
||
`harden_sshd_macs_opensshserver_conf_crypto_policy` (#5);
|
||
`configured_firewalld_default_deny` (#4); `sssd_enable_certmap`,
|
||
`sssd_has_trust_anchor` (#7).
|
||
- **Bucket B (waived N/A):** `account_temp_expire_date`,
|
||
`accounts_user_home_paths_only`, `libreswan_approved_tunnels`,
|
||
`security_patches_up_to_date`, `ssh_keys_passphrase_protected`.
|
||
- **Bucket C (decisions applied):** `installed_OS_is_vendor_supported` (Rocky IS
|
||
supported; RHEL-datastream CPE artifact — waived), `network_configure_name_resolution`
|
||
(single DHCP DNS by design — waived), `sysctl_user_max_user_namespaces_no_remediation`
|
||
(KEEP userns for Chromium sandbox — waived), `accounts_authorized_local_users`
|
||
(appliance accounts authorized, #10 — waived; prod: set the authorized-users
|
||
regex var instead), `package_postfix_installed` + `postfix_prevent_unrestricted_relay`
|
||
(**postfix removed** in 2c-STIG — no MTA needed; audit goes off-box via 2f),
|
||
**USBGuard** `configure_usbguard_auditbackend` + `usbguard_allow_hid`/`_hub`/
|
||
`_hid_and_hub` (**ENABLED + enforcing**: LinuxAudit backend + hardware-independent
|
||
HID/hub allowlist written in 2c-STIG; ⚠ BENCH-VERIFY physical console keyboard/
|
||
mouse on real hardware — the build-VM-generated policy + default block target
|
||
would otherwise lock out console input).
|
||
- **`selinux_context_elevation_for_sudo` — REMEDIATED WITH CONFINEMENT
|
||
(decision 2026-06-20).** 2c-STIG now rewrites the admin sudoers to
|
||
`TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL` and maps the admin to **`staff_u`**
|
||
(`semanage login`) so the sudo role transition is authorized. ⚠ **TWO BENCH
|
||
GATES:** (a) admin SSH login + sudo + privileged ops must work under `staff_u`
|
||
confinement (escape hatch: `enable_stig=false` → reverts to a justified
|
||
deviation); (b) **reproducibility** — `semanage` rebuilds the policy store;
|
||
confirm the verity hash is stable across two builds (normalize in 2d if it
|
||
drifts). Needs `policycoreutils-python-utils` (installed in 2c-STIG if absent).
|
||
|
||
**Expected next-build report: 0–1 REVIEW** (0 if both postfix→N/A and
|
||
usbguard→pass land and sudo confinement holds; the triage was validated by
|
||
re-running `scripts/stig-deviations.sh` against the app-9f7d8d1 ARF with the new
|
||
tailoring — 21 REVIEW → 3, where the 3 are exactly postfix/usbguard/sudo, all
|
||
addressed for the next build).
|
||
|
||
Remaining (future hardening, currently waived — see the register): FIPS,
|
||
firewall, per-host/rotatable admin access. **Remote audit offload — WIRED both
|
||
ends** (2026-06-19): receiver `serving/audit-collector/` (rsyslog TCP/514,
|
||
per-source-host files) + sender in the image (provisioner 2f, `audit_host` var:
|
||
audisp-syslog→LOCAL6 + rsyslog `omfwd` to `audit.fleet.lab`). Lab is plaintext;
|
||
TLS/RELP is the remaining hardening. **VERIFIED END-TO-END ON THE BENCH
|
||
2026-06-19** — booted host `cygnus-cruiser-99` (192.168.50.148) forwards to the
|
||
collector, which writes `audit-collector/logs/192.168.50.148/{audit,messages}.log`.
|
||
**Serving-host firewall gotcha hit + fixed:** the collector is host-networked so
|
||
the serving host's **ufw** governs :514 (Docker doesn't manage iptables for
|
||
host-net containers); 514/tcp was bound to the wrong NIC (`ens18` instead of the
|
||
boot-LAN `ens19`), so SYNs were silently dropped (`SYN-SENT` / "connection timed
|
||
out", not refused). Fix: `ufw allow in on ens19 to any port 514 proto tcp`. The
|
||
sender's `action.resumeRetryCount="-1"` buffered the backlog and drained on
|
||
reconnect — no records lost. Documented in
|
||
[../serving/audit-collector/README.md](../serving/audit-collector/README.md).
|
||
When the **workstation** firewall is re-enabled (#4 / register #4) it needs the
|
||
reciprocal **egress** allow for 514/tcp to the collector.
|
||
|
||
**SIEM (Wazuh) added 2026-06-27 — [serving/siem/](../serving/siem/).** Upgrades the
|
||
"Detect" pillar from file-collection-only to parse/search/correlate/alert.
|
||
**Runs on a DEDICATED node — `wazuh1.bitasylum.org`** (managed independently),
|
||
keeping the memory-hungry indexer JVM **off** the serving host so it can't OOM SPDK
|
||
(the fleet's root disk / SPOF, #6/#7). **Agentless** (diskless hosts stay stateless,
|
||
invariant #1): the serving-host audit-collector **relays a copy** to wazuh1's
|
||
manager syslog listener (`omfwd` → `wazuh1.bitasylum.org:514` in
|
||
`audit-collector/rsyslog.conf`; rebuild the collector to apply) while still writing
|
||
durable per-host files. `serving/siem/` is the INTEGRATION (the `<remote>` syslog
|
||
block to add on wazuh1 + reference deploy overlay), not a local deployment.
|
||
**DECODING BENCH-VERIFIED 2026-06-27** (`wazuh-logtest`): the audisp-syslog records
|
||
carry a `node=<host>` prefix (STIG `name_format=hostname`, SSG `auditd_name_format`
|
||
— kept, it's compliance) that breaks Wazuh's built-in auditd decoder, so a custom
|
||
**`fleet-auditd`** decoder + rules (`serving/siem/config/local_decoder.xml` +
|
||
`local_rules.xml`, mounted declaratively via `/wazuh-config-mount`) decode them
|
||
into the **`fleet_audit`** group. **Decode + alert confirmed:** `wazuh-logtest`
|
||
shows `decoder.name=fleet-auditd` + `audit.type` extracted (base rule `100100`);
|
||
and a live `sudo` on a booted host (`cygnus-cruiser-99`) surfaced `USER_CMD` →
|
||
rule `100130` and `CRED_*`/`USER_ACCT` → rule `100110` (level 5) in the dashboard
|
||
Threat Hunting view. Wazuh runs multi-node on `wazuh1.bitasylum.org`.
|
||
**REMAINING (prod):** the relay→wazuh1 hop is **plaintext** — **TLS it** (gtls/6514,
|
||
fleet CA; ISM protect-logs-in-transit) since it's now on the wire; rotate all Wazuh
|
||
creds; restrict dashboard/API to admin hosts; set `allowed-ips`→/32 of the serving
|
||
host; ISM retention/archival; then update the SSP "Detect" + ISM event-logging
|
||
controls.
|
||
|
||
### Original spec (retained for reference)
|
||
|
||
OpenSCAP + SCAP Security Guide (`xccdf_org.ssgproject.content_profile_stig`), run
|
||
as a Packer provisioner after `dnf install` and before the dracut/initramfs build
|
||
and cleanup.
|
||
|
||
- **Tailoring** (`autotailor`): unselect rules this architecture can't satisfy —
|
||
separate-partition rules (single erofs root + tmpfs overlay; satisfy intent
|
||
with `nodev,nosuid,noexec` tmpfs mounts), GRUB/kernel-cmdline rules (no GRUB),
|
||
and AIDE/file-integrity rules (dm-verity is the documented compensating
|
||
control).
|
||
- **Kernel params → UKI cmdline** in
|
||
[../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh) before
|
||
signing: `audit=1 audit_backlog_limit=8192 slub_debug=P page_poison=1
|
||
vsyscall=none` (+ `fips=1` if FIPS).
|
||
- **Reproducibility guard:** strip non-deterministic remediation output (SSH host
|
||
keys → regen at boot into overlay, AIDE db, timestamps, fresh UUIDs) so the
|
||
verity hash stays stable.
|
||
- **Audit logging:** `audisp-remote`/rsyslog to a central collector addressed by
|
||
DNS name (host-generic).
|
||
- **Optional FIPS:** `fips=1` cmdline + `01fips` dracut module + FIPS crypto
|
||
policy.
|
||
- **CI:** run `oscap eval` (no `--remediate`) to publish an ARF+HTML STIG report
|
||
as a build artifact; optionally hard-fail below a score threshold.
|
||
- **Decisions needed before implementing:** FIPS on/off · remote-audit collector
|
||
DNS name (or ephemeral/none for lab) · CI fail-on-score threshold vs
|
||
report-only.
|
||
- Note: orthogonal to the `rd.dot1x=0` test toggle, but a full STIG production
|
||
posture wants 802.1X re-enabled.
|
||
- **Credentials (partly done):** root is now **locked** and access is via a
|
||
baked sudo-enabled key-only admin user (`var.ssh_authorized_key` /
|
||
`var.admin_user`, `wheel` + NOPASSWD sudo), with sshd drop-in
|
||
`PermitRootLogin no` / `PasswordAuthentication no`. Two items for STIG to
|
||
reconcile: **NOPASSWD sudo** is a finding (acceptable for a key-only account,
|
||
but document/deviate), and the admin key/user are **fleet-wide baked** — for
|
||
per-host/rotatable access without rebuilds, layer the signed HTTP/SMBIOS
|
||
config-agent or an SSH CA (`TrustedUserCAKeys`) discussed in session.
|
||
- **SSH host keys (done):** `extract-artifacts.sh` strips `/etc/ssh/ssh_host_*`
|
||
so they're not baked (baked keys = shared fleet identity + the build's
|
||
perms/group don't survive the tar→erofs round-trip, so sshd rejected them
|
||
"Permissions 640 too open" and refused to start). RHEL's `sshd-keygen@.service`
|
||
regenerates them into the overlay before sshd starts. **⚠ Caveat:** the
|
||
ephemeral overlay means they regenerate **every boot** → the host key changes
|
||
per reboot (clients warn). Follow-up for a stable host identity: persist the
|
||
host keys or derive them from the TPM (ties to the hostnqn/identity model).
|
||
|
||
## [ ] 3. Roll the errata into design doc v3
|
||
|
||
Once the boot path is validated and item 1 lands, fold
|
||
[design-v2-errata.md](design-v2-errata.md) into the source design document and
|
||
re-export a **v3** PDF (key change: `rootovl` → `systemd.volatile=overlay` in §3
|
||
step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source
|
||
(not in this repo).
|
||
|
||
## [x] 4. Kiosk mode — autologin + fullscreen Chromium loading a URL
|
||
|
||
**WORKING (display-verified 2026-06-17).** Remaining bits are polish (GPU/mesa
|
||
trim, ship via CI `enable_kiosk=true`) and the phone-home/NTP lockdown (done) —
|
||
all captured in the sub-notes below. Secure-path render check is folded into #8.
|
||
|
||
**Package-footprint trim (2026-07-03):** the kiosk dnf transaction was measured at
|
||
**229 pkgs / 1.0 GB** (weak deps ON). Added `--setopt=install_weak_deps=False` to
|
||
the install in `diskless.pkr.hcl` — drops the flatpak/ostree sandbox runtime,
|
||
tracker-miners (fs indexer = dead weight + threat surface), the pipewire/pulseaudio
|
||
audio stack, and `libproxy-webkitgtk4 → webkit2gtk3-jsc` (an 8.9M *second* browser
|
||
engine). Also added an `rpm -qa --qf %%{SIZE}` size-manifest diagnostic to the build
|
||
log to quantify the result. The immovable floor is chromium + chromium-common +
|
||
llvm-libs (mesa sw-GL) + gtk3/adwaita + libicu — all HARD deps. **BENCH-VERIFY:**
|
||
with weak deps off, confirm the kiosk still renders (no genuine runtime dep was a
|
||
weak dep — watch for a missing font or `libva`). NEXT trim candidates (measure via
|
||
the new manifest): gstreamer/media codecs if the SPA plays no HTML5 media;
|
||
mesa-dri-drivers + `--use-pixman`-only if the GPU path is abandoned.
|
||
|
||
|
||
**WORKING / display-verified 2026-06-17** (kiosk renders Chromium fullscreen on
|
||
the configured URL). Build-flagged OFF by default for now (`enable_kiosk=false`)
|
||
while iterating on boot/network — set `-var enable_kiosk=true` to ship it.
|
||
|
||
Path to getting here (don't re-discover):
|
||
- **cage is NOT packaged for EL9** (`dnf install cage` → "Unable to find a
|
||
match"). Tried weston instead.
|
||
- **EPEL 9 ships weston 8.0**, which has NEITHER `kiosk-shell.so` NOR the
|
||
`[autolaunch]` weston.ini section — both arrived in **weston 9.0**. The first
|
||
cut wrongly assumed 8.0 had them; `shell=kiosk-shell.so` failed to load.
|
||
`rpm -ql weston` shows only desktop-shell / fullscreen-shell / ivi-shell.
|
||
- **Resolution: `desktop-shell.so`** (the only packaged shell Chromium's
|
||
xdg-shell works with — fullscreen-shell needs a client protocol Chromium
|
||
doesn't speak), stripped bare (`panel-position=none`, black bg, no
|
||
animation/lock), and Chromium launched by a **wrapper** (kiosk-session) since
|
||
there's no `[autolaunch]`. Chromium `--kiosk` fullscreens over the shell.
|
||
- **Seat fix (load-bearing):** weston died with `logind: failed to get session
|
||
seat` → `failed to create compositor backend`. logind's auto-derivation
|
||
didn't attach a seat; `Environment=XDG_SEAT=seat0` + `XDG_VTNR=1` in the unit
|
||
(put into the PAM env before pam_open_session) registers the session on
|
||
seat0/vt1. `/dev/dri/card0` was present all along (not the issue).
|
||
- Tested live without a rebuild: the whole root is `systemd.volatile=overlay`,
|
||
so /etc AND /usr are writable tmpfs at runtime — drop-in the files + restart.
|
||
- **`enable_kiosk` Packer var — currently `false` by DEFAULT (set 2026-06-17).**
|
||
The whole kiosk provisioner is feature-flagged off while iterating on the
|
||
boot/network stack (DHCP-lease/keep-alive work), for faster builds and a
|
||
smaller image. **Re-enable with `-var enable_kiosk=true` when returning to
|
||
kiosk work** — CI does not pass the var, so CI currently builds lean (no
|
||
kiosk) too; add `-var enable_kiosk=true` to the Gitea workflow to ship it.
|
||
- `kiosk_url` Packer var (default `http://boot.fleet.lab/kiosk/` — the kiosk
|
||
**SPA** (`serving/kiosk-app/`, Svelte 5 + Tailwind v4, a cyber sci-fi
|
||
spaceship monitoring/control terminal) served by the boot HTTP server, which
|
||
**reverse-proxies** `/kiosk/` to the `kiosk` container; self-contained so a
|
||
boot validates without external connectivity. Empty disables the whole
|
||
install). Wired in CI as `KIOSK_URL` env.
|
||
- **Kiosk SPA (`serving/kiosk-app/`)** — replaced the old single static
|
||
`serving/http/kiosk/index.html` (2026-06-18). Top navbar + 5 pages
|
||
(Bridge/Reactor/Nav/Life Support/Comms) with dummy telemetry; hash routing;
|
||
builds in a 2-stage Docker image (Vite → nginx), reverse-proxied at `/kiosk/`
|
||
by the main serving nginx (loopback `127.0.0.1:${KIOSK_APP_PORT:-8088}`).
|
||
`npm run build` verified locally; container build untested (no docker here).
|
||
Dummy data lives in `src/lib/data.js` (swap for a fleet API later).
|
||
- Provisioner enables EPEL, installs `weston chromium mesa-dri-drivers
|
||
mesa-libEGL mesa-libgbm dejavu-sans-fonts`, creates a passwordless
|
||
unprivileged `kiosk` user (video/input groups).
|
||
- `files/kiosk/kiosk.service` — runs `/usr/local/bin/kiosk-session` as `kiosk`
|
||
on tty1 via `PAMName=login`+`TTYPath`+`Environment=XDG_SEAT=seat0`/`XDG_VTNR=1`
|
||
(gets seat0 for DRM/input), `Conflicts=getty@tty1`, `WantedBy=graphical.target`;
|
||
default target set to graphical.
|
||
- `files/kiosk/weston.ini` — `shell=desktop-shell.so`, `idle-time=0`,
|
||
`require-input=false`, `xwayland=false`; `[shell] panel-position=none`,
|
||
black `background-color`, `animation=none`, `locking=false`.
|
||
- `files/kiosk/kiosk-session` — `exec`s `weston --use-pixman --socket=wayland-kiosk`
|
||
in the foreground (owns the seat), and launches Chromium (`--kiosk
|
||
--ozone-platform=wayland --disable-gpu`, tmpfs profile/cache, URL from
|
||
`/etc/kiosk/kiosk.env`) in a background crash-restart loop.
|
||
- **Remaining polish (now that it renders):** dropping `--disable-gpu` on real
|
||
GPUs, `/dev/dri` present in the VM (weston needs a DRM device — set Display to
|
||
VirtIO-GPU/std), and **SELinux AVCs** for the GUI stack (check `ausearch -m
|
||
avc`; the kiosk user is unconfined so likely none — but do NOT set permissive
|
||
if they appear; add a policy module). Admin SSH (the key user) still works for
|
||
debug even if the kiosk fails (it lands at graphical.target either way).
|
||
- **Phone-home lockdown (done 2026-06-17/18):** Chromium was dialing Google
|
||
(accounts/variations/optimization-guide/component+auto update/safe browsing —
|
||
seen in dnsmasq logs). Two layers:
|
||
- **Flags** in `kiosk-session` (`--disable-background-networking`,
|
||
`--disable-component-update`, `--disable-features=OptimizationHints,...`) —
|
||
confirmed killing variations/optimization-guide/update queries.
|
||
- **Managed policy** `files/kiosk/chromium-kiosk-policy.json` for the
|
||
policy-governed bits (sign-in `accounts.google.com`, sync). FIX 2026-06-18:
|
||
the managed-policy dir differs by build, so it's installed to BOTH
|
||
`/etc/chromium/policies/managed/` AND `/etc/chromium-browser/policies/managed/`
|
||
(a policy in the wrong dir is silently ignored — the reason
|
||
`accounts.google.com`/`android.clients.google.com` persisted). Verify via
|
||
`chrome://policy` (Status: OK per entry).
|
||
- **Serving-side sinkhole** (defense-in-depth, `serving/docker-compose.yml`
|
||
dnsmasq `--address=/google.com|googleapis.com|gstatic.com/0.0.0.0`): the
|
||
kiosk is local-only, so block egress at the resolver. Stops the *leak*; the
|
||
query still logs (answered 0.0.0.0, not forwarded) — eliminating the query
|
||
is the browser policy's job. Production: do this at the boot-LAN firewall.
|
||
- **NTP egress closed (done 2026-06-18):** the `*.pool.ntp.org` lookups were
|
||
the OS (chrony), not Chromium. Now: a `chrony` NTP server in the serving
|
||
stack (`serving/chrony/`), dnsmasq hands it via **DHCP option 42**, and the
|
||
image's NM dispatcher (`files/etc/NetworkManager/dispatcher.d/10-chrony-dhcp-ntp`)
|
||
feeds it into chrony's `sourcedir /run/chrony-dhcp`; the baked public pool is
|
||
stripped from `/etc/chrony.conf`. Result: the workstation syncs time only
|
||
from the lab server — no external NTP egress. BENCH-VERIFY: on a booted host
|
||
`chronyc sources` shows the lab server (the DHCP one), not a public pool.
|
||
- **Image size:** Chromium+Mesa add hundreds of MB to the *uncompressed* erofs —
|
||
feeds the "shrink the rootfs" concern (and #7); reconsider whether a lighter
|
||
browser/render path is viable.
|
||
|
||
### Original task
|
||
|
||
|
||
|
||
Make the diskless workstation boot straight into a locked-down kiosk: a
|
||
fullscreen Chromium showing a configured site (default: the boot HTTP server's
|
||
`/kiosk/` test page). All
|
||
baked at **build time** into the root tree (immutable erofs/verity root; runtime
|
||
is the tmpfs overlay), fleet-generic (no per-host state).
|
||
|
||
- **Graphical stack (keep it minimal):** add a Packer provisioner installing a
|
||
lightweight kiosk compositor + browser. Preferred: `cage` (Wayland
|
||
single-app kiosk compositor) + Mesa (software GL fallback for VMs/no-GPU);
|
||
X11 + a minimal WM (matchbox/openbox) as the alternative. Chromium on Rocky 9
|
||
is in **EPEL** — enabling EPEL adds a repo/supply-chain surface (decision
|
||
below).
|
||
- **Autologin + session:** dedicated unprivileged `kiosk` user; launch the
|
||
session via a `kiosk.service` running `cage -s -- chromium --kiosk …`
|
||
(or `greetd` + cage). Stage units/configs under `files/` like the other
|
||
provisioned config.
|
||
- **Chromium flags:** `--kiosk --noerrdialogs --disable-infobars --incognito
|
||
--no-first-run --disable-session-crashed-bubble <URL>`. Writable profile/cache
|
||
land on the `systemd.volatile=overlay` tmpfs (RAM-backed, ephemeral) — fine,
|
||
but watch RAM sizing; pin `--user-data-dir`/`--disk-cache-dir` to a tmpfs path
|
||
if needed.
|
||
- **Kiosk URL = Packer var** (e.g. `kiosk_url`, default the target site), baked
|
||
into the launch command; keep it host-generic (per-env via DNS if ever needed).
|
||
- **Verify in boot test:** graphical session comes up on real/VM display,
|
||
Chromium fullscreen loads the URL over the booted network (works with the
|
||
current `rd.dot1x=0` test path / open network).
|
||
- **Decisions:** Wayland(cage) vs X11; accept EPEL for Chromium (vs another
|
||
source); confirm the target URL.
|
||
- **⚠ Conflict with item 2 (STIG):** passwordless autologin and a desktop
|
||
browser are STIG findings and enlarge the attack surface/image. Reconcile —
|
||
e.g. a tightly-constrained kiosk session + documented deviation — when
|
||
hardening lands.
|
||
|
||
## [x] 5. Deploy & validate the end-to-end boot path (open-network test)
|
||
|
||
✅ **DONE (2026-06-15).** Full chain validated on QEMU/Proxmox: UEFI HTTP boot →
|
||
initramfs → DHCP → `nvmf-connect.service` auto-connects the NVMe/TCP subsystem →
|
||
two namespaces (data + verity hash) appear by pinned UUID → `systemd-veritysetup`
|
||
maps `/dev/mapper/root` → uncompressed erofs mounts → `systemd.volatile=overlay`
|
||
→ multi-user. Booted with `rd.dot1x=0` (open network) and the IP discovery
|
||
address. See bench findings below for the full list of fixes. Remaining cleanup
|
||
to leave the lab-test posture (now unblocked): revert the hardcoded IP to the
|
||
`nvme.fleet.lab` DNS name, and re-enable security (`DISABLE_DOT1X=false`) — the
|
||
secure path needs the same connect trigger over macsec0 (see ⚠ note below).
|
||
|
||
Original task: stand up `serving/` (dnsmasq + nginx + SPDK) per
|
||
[serving/README.md](../serving/README.md), rebuild the UKI with
|
||
`nvmet_traddr=nvme.fleet.lab` (now a Packer var — older UKIs have the literal
|
||
`NVMET_TRADDR` placeholder and won't connect), then boot a UEFI HTTP-boot client
|
||
(**Secure Boot OFF**, unsigned UKI; no TPM needed in `rd.dot1x=0` mode) on the
|
||
boot LAN.
|
||
|
||
- **Pass criteria:** UKI fetched over HTTP → initramfs → DHCP on NIC → NVMe/TCP
|
||
namespace connected → dm-verity root mapped → `systemd.volatile=overlay`
|
||
writable → multi-user.
|
||
- This validation **gates** item #1 (network-manager port) and item #3
|
||
(errata → design v3).
|
||
- **Bench findings (boot debug, 2026-06-14):**
|
||
- Initramfs has no resolver (dracut writes `resolv.conf` only via its own
|
||
DHCP path, not a manual `dhclient`), so the NVMe discovery address must be
|
||
an **IP**, not a DNS name — build with `-var nvmet_traddr=192.168.50.10`.
|
||
Old UKIs carrying `nvme.fleet.lab` can't connect early in boot.
|
||
- NVMe controller indices are **not stable**: a transient discovery
|
||
controller can take `nvme0`, pushing the I/O controller to `nvme1`. The
|
||
hardcoded `/dev/nvme0n1` in the cmdline was a latent brick. **Fixed:**
|
||
namespace UUID pinned on the target (`spdk-setup.sh` `add_ns -u
|
||
f1ee7000-0000-4000-8000-000000000001`) and the cmdline now references
|
||
`/dev/disk/by-id/nvme-uuid.<uuid>` (`extract-artifacts.sh`). The two
|
||
constants MUST stay in sync; make them version-specific for #6.
|
||
- `nvme_tcp`/`nvme_fabrics` must be loaded before `/dev/nvme-fabrics` exists
|
||
(dracut's `nvmf` module does this in a normal boot; at `rd.break` modprobe
|
||
manually).
|
||
- **Don't double-configure the NIC in test mode.** With `ip=dhcp` on the
|
||
cmdline, dracut's network module brings the NIC up AND emits the "online"
|
||
event that the `nvmf` module's `initqueue/online` hook waits on to run
|
||
`nvme connect-all`. The `rd.dot1x=0` bypass used to *also* `ifup $NIC`,
|
||
which (a) raced dracut → `Error: ipv4: Address already assigned`, and (b)
|
||
didn't fire the nvmf online hook, so the namespace never auto-connected and
|
||
boot spun on `/dev/mapper/root` forever (manual `connect-all` worked, which
|
||
is what masked it). **Fixed:** the bypass branch no longer touches the NIC;
|
||
`ip=dhcp` owns networking. (The secure path still addresses macsec0 itself —
|
||
dracut doesn't know about that interface.)
|
||
- **dracut's `nvmf` module never auto-connects on this distro.** Its cmdline
|
||
hook (`92-parse-nvmf-boot-connections.sh`) parses `rd.nvmf.discover` into
|
||
`/etc/nvme/discovery.conf` correctly, but `initqueue/online/` is **empty** —
|
||
no autoconnect hook is registered, so `nvme connect-all` is never run (no
|
||
attempt in dmesg) and the namespace never appears. Drivers load and
|
||
`/dev/nvme-fabrics` exists; only the connect trigger is missing. A job
|
||
placed on the *main* initqueue from a settled hook gets registered but is
|
||
**never re-executed** once the loop blocks on udev events (verified: job
|
||
present, manual `connect-all` works, namespace stays absent). An
|
||
`initqueue/online` hook also does NOT work: this initramfs is **systemd-
|
||
driven** ("A start job is running for /dev/mapper/root") and never sources
|
||
dracut online hooks on link-up (verified: hook installs as
|
||
`90-nvmf-connect.sh` but never fires — no log, no stamp). **Fixed:** the
|
||
connect is now a **systemd unit** (`nvmf-connect.service`,
|
||
`ConditionKernelCommandLine=rd.dot1x=0`, `Before=initrd-root-device.target`)
|
||
whose ExecStart (`/usr/sbin/nvmf-connect`) polls for the network then runs
|
||
`nvme connect-all` until the namespace appears. The dracut module installs
|
||
the unit + script and enables it via an `initrd-root-device.target.wants`
|
||
symlink; `diskless.pkr.hcl` hard-asserts the unit is in the initramfs.
|
||
Gotcha: `$systemdsystemunitdir` is **empty** in this dracut's module-setup
|
||
context — `inst_simple` then silently drops the unit (empty dest dir). Use a
|
||
`${systemdsystemunitdir:-/usr/lib/systemd/system}` fallback and `return 1`
|
||
if the unit didn't land, so the build fails loudly instead of shipping a UKI
|
||
that can't connect its root.
|
||
Second gotcha: the build assertion `lsinitrd … | grep -q 'nvmf-connect.service'`
|
||
was a **false negative** — `grep -q` exits on first match, closes the pipe,
|
||
`lsinitrd` dies of SIGPIPE, and under `pipefail` the pipeline reports failure
|
||
even though the unit IS present. This masked the fact that the systemd unit
|
||
install was working all along. Fix: `lsinitrd > file; grep -q … file` (no
|
||
pipe). Never `lsinitrd | grep -q` under pipefail.
|
||
Bonus: the survivor listing confirmed nvme-cli's native autoconnect bits ARE
|
||
in the initramfs (`/usr/sbin/nvmf-autoconnect.sh`,
|
||
`/etc/udev/rules.d/95-nvmf-initqueue.rules`) — so the fully-native
|
||
`rd.nvmf.discover` autoconnect path is available as a fallback if the unit
|
||
doesn't fire on boot.
|
||
- **dm-verity: `hash-offset` is unsupported → two namespaces.** With NVMe
|
||
connecting and the right by-id device present, verity still failed:
|
||
`systemd-veritysetup` logged `unknown option 'hash-offset=…'` then `Failed
|
||
to load verity superblock` — exactly the CLAUDE.md bench-verify item. The
|
||
in-file (data+hash in one namespace, split by offset) layout is not
|
||
supported by Rocky 9.8 systemd. **Fixed** via the documented fallback:
|
||
`extract-artifacts.sh` now emits `rootfs.erofs` (data) + `rootfs.hash` (hash
|
||
tree) as separate files, served as **two NVMe namespaces** with distinct
|
||
pinned UUIDs (`f1ee7000-…-01` data, `…-02` hash); the cmdline uses
|
||
`systemd.verity_root_data=`/`systemd.verity_root_hash=` by-id devices with no
|
||
offset. Touches `extract-artifacts.sh`, `build-oci.sh` (artifact list +
|
||
SHA256SUMS), `serving/spdk-setup.sh` (2nd bdev/namespace), and invariant #9.
|
||
- **SELinux labels must survive into the erofs.** It booted to multi-user but
|
||
`root`/`packer-build-only` login was accepted then the session was killed
|
||
instantly. Cause: kickstart sets `selinux --enforcing`, but the build dropped
|
||
`security.selinux` xattrs (`virt-tar-out` + plain `tar -xpf` don't carry
|
||
them), so the enforcing target denied the login session on an unlabeled root
|
||
— and a read-only erofs can't autorelabel. Confirmed by booting with
|
||
`enforcing=0` (SMBIOS-injected) → login worked. **Fixed:** export via
|
||
`guestfish tar-out … selinux:true xattrs:true acls:true`, restore with
|
||
`tar --xattrs --xattrs-include='*' --acls --selinux`, and a `getfattr`
|
||
build-time assertion that `/usr/bin/bash` is labeled before `mkfs.erofs`
|
||
(mkfs.erofs preserves xattrs). Needs `attr` on the build host. The image now
|
||
boots **enforcing**.
|
||
- **`mkfs.erofs --all-root` was flattening all ownership to root:root.** Mode
|
||
bits + (now) xattrs survive the export, but `--all-root` forced every inode
|
||
to uid/gid 0 — discarding group ownership (`ssh_keys` on host keys →
|
||
sshd "permissions 640 too open" → refused; `tty`/`crontab` setgid groups)
|
||
and user ownership (home dirs). **Fixed:** dropped `--all-root` so real
|
||
ownership is preserved from the tree. That now requires `extract-artifacts.sh`
|
||
to run as **root** (added an `id -u` guard; `tar -x` needs CAP_CHOWN to
|
||
restore ownership) — the post-processor runs it via `sudo` and `chown`s the
|
||
output files back to the build user so `build-oci.sh` (rootless buildah +
|
||
its own login) still works. Combined with the host-key strip above, sshd
|
||
starts cleanly.
|
||
**⚠ The secure path has the same dependency on the
|
||
(absent) nvmf autoconnect** — when 802.1X is re-enabled, the macsec0 path
|
||
must likewise drive `connect-all` (or the network-manager port #1 must wire
|
||
a working autoconnect). Currently only the test path is fixed.
|
||
- The target Rocky 9 kernel's erofs is built **without
|
||
`CONFIG_EROFS_FS_ZIP`** — mounting a compressed image fails `erofs: lz4
|
||
algorithm isn't enabled`, which silently blocks the root mount. **Fixed:**
|
||
`extract-artifacts.sh` now builds the erofs **uncompressed** (`mkfs.erofs`
|
||
without `-z`). Tradeoff: larger rootfs namespace + more NVMe/TCP bandwidth
|
||
per read. Re-enable compression only against a kernel verified to have
|
||
`EROFS_FS_ZIP` (`grep CONFIG_EROFS /boot/config-$(uname -r)`). This is the
|
||
CLAUDE.md bench-verify item ("erofs module availability on the target RHEL
|
||
kernel"). Larger image size also feeds the "shrink the rootfs" angle of #7.
|
||
- SPDK: add the listener to the discovery subsystem explicitly
|
||
(`nqn.2014-08.org.nvmexpress.discovery`) to silence the deprecation warning.
|
||
|
||
## [ ] 6. Rolling fleet updates (multi-version serve → cohort reboots)
|
||
|
||
Image-based, reboot-to-update across many live workstations. Hosts are immutable
|
||
(read-only erofs/verity root over NVMe/TCP, ephemeral overlay), so updates are
|
||
**atomic A/B reboots**, never in-place. The released OCI artifact + digest-pinned
|
||
manifest + roothash coupling already give the atomic unit and instant rollback;
|
||
this item builds the rollout machinery around it.
|
||
|
||
- **Serve versions side-by-side** (must not overwrite a live rootfs): extend
|
||
`serving/` so SPDK/nvmet exposes each release as its **own subsystem/NQN**
|
||
(`nqn…:rootfs-<ver>`) and nginx serves multiple `fleet-<ver>.efi` (keep
|
||
current + previous). Make the UKI's NQN version-specific so each signed UKI
|
||
connects only its matching namespace (constant discovery DNS name).
|
||
- **Version selection at boot**: dnsmasq **cohort tagging** (MAC / vendor class /
|
||
assigned tag) hands each cohort a different option-67 UKI URL. Moving a
|
||
cohort's URL = assigning the release.
|
||
- **Control plane + in-image reboot agent**: a baked-in systemd timer polls a
|
||
"desired version" endpoint and reboots **when idle / in a maintenance window**
|
||
(or out-of-band via Redfish/BMC). Stagger + jitter reboots to avoid UKI-fetch
|
||
/ NVMe-connect storms.
|
||
- **Staged rollout**: canary → waves (10/50/100%) with health gates
|
||
(multi-user reached, NVMe connected, dm-verity mapped, kiosk URL rendered,
|
||
prod: MACsec/802.1X up); auto-halt on threshold.
|
||
- **Rollback**: repoint cohort URL to previous UKI + reboot (current+previous
|
||
stay served); security rollback via dbx-revoke / TPM monotonic counter
|
||
(design §6).
|
||
- **Drain & retire** an old version only when SPDK/nvmet reports zero connected
|
||
controllers on it. **Drain accuracy:** account for hosts powered off mid-rollout
|
||
— an assigned-but-offline host will reconnect to a version that may be gone, so
|
||
don't remove a version while any host that could return is still assigned to it.
|
||
- **Availability**: run NVMe/TCP target replicas behind distinct discovery
|
||
entries (design §7) — never serve the whole fleet from one unreplicated node,
|
||
especially mid-rollout (N versions + reboot bursts).
|
||
- Note: kiosk *content* changes are server-side (the website) and need no fleet
|
||
update — only OS/UKI/rootfs changes require a reboot.
|
||
|
||
### SPDK restart / availability is the load-bearing constraint (analysis 2026-06-25)
|
||
|
||
SPDK sits in the **live data path of every running host's root fs**, so a process
|
||
restart isn't like restarting a stateless service — it tears down **every**
|
||
NVMe/TCP association at once (fleet-wide, NOT version-scoped). The rollout design
|
||
must be built around **never restarting the process that holds live connections.**
|
||
|
||
- **Core principle:** a host changes root version *only by rebooting* (no live
|
||
swap). So: **version transitions = reboots; SPDK changes = additive/online; the
|
||
process stays up.** Routine rollouts never restart SPDK.
|
||
- **Online add/remove via JSON-RPC, no restart.** `nvmf_tgt` is fully
|
||
runtime-configurable (`bdev_aio_create`, `nvmf_create_subsystem`,
|
||
`nvmf_subsystem_add_ns`, `..._add_listener`). Add vY's namespaces live (existing
|
||
vX connections untouched); remove vX live once drained. **Per-version subsystem
|
||
(distinct NQN)** matters here: adding vY as a *new subsystem* never disturbs the
|
||
subsystem vX hosts are attached to — adding a *namespace* to a shared subsystem
|
||
fires a namespace-change AER + rescan at running hosts. Per-version subsystem
|
||
also gives clean per-release drain + per-release hostNQN allow-listing.
|
||
- **What a connection loss does to a running host:** root is **read-only erofs +
|
||
tmpfs overlay**. tmpfs writes (RAM) + page-cached reads keep working; **cold
|
||
reads block**. The nvme-tcp host driver reconnects (`reconnect_delay`) until
|
||
`ctrl_loss_tmo`. **Fast SPDK restart** (same NQN/UUID/addr within the window) →
|
||
host **survives with a stall**; **long/changed/expired** → root I/O errors, host
|
||
needs a reboot ("keep alive timeout / no route to host"). Read-only +
|
||
content-addressed makes reconnect unusually clean (no write/reservation state).
|
||
- **Cheap host-side mitigation:** tune `ctrl_loss_tmo` (even `-1` = retry forever)
|
||
+ `reconnect_delay` at `nvme connect` time (dracut nvmf step). For a diskless box
|
||
"hang and recover" beats "die" — there's nothing better to fail to.
|
||
- **Structural answer — HA + NVMe-oF multipath (ANA):** run ≥2 targets serving
|
||
**byte-identical, same-UUID** namespaces; hosts multipath across them, so one
|
||
target can be restarted/upgraded while the other serves (no root loss). The
|
||
**read-only, content-addressed root makes replication trivially coherent** (no
|
||
write-coherence problem) — this architecture is ideally suited to it. Wrinkle:
|
||
the **immutable UKI cmdline** points at one discovery address, so multipath needs
|
||
discovery to return multiple paths (or a DNS name resolving to several targets).
|
||
- **Planned SPDK restarts only via drain/failover** (multipath fail-over, or a
|
||
maintenance window with generous `ctrl_loss_tmo` so hosts recover not die). There
|
||
is **no zero-impact single-node SPDK restart.**
|
||
- **[x] Long-lived target + online release control — IMPLEMENTED 2026-06-27.**
|
||
The old gap (SPDK ran `spdk-setup.sh` as entrypoint, single-version,
|
||
configured-at-start → changing versions = container restart = drop ALL
|
||
connections) is closed. Now in `serving/spdk/`:
|
||
- **`spdk-target.sh`** — persistent entrypoint: starts `nvmf_tgt` ONCE
|
||
(release-agnostic), creates the TCP transport + discovery listener, then
|
||
reconciles every staged release. Never restarted to change versions; on
|
||
(re)start re-adds everything under `./releases/` (a serving-host reboot
|
||
restores all served versions).
|
||
- **`spdk-release.sh`** — JSON-RPC control: `add`/`remove`/`list`/
|
||
`drain-status`/`reconcile`, **one subsystem per version** (distinct NQN, so
|
||
adding/removing one never disturbs others' connections), **drain-gated
|
||
removal** (refuses while a version has connected controllers unless
|
||
`--force`).
|
||
- **`spdk-ctl.sh`** — host ops: `load`/`unload`/`list`; stages a release's
|
||
artifacts from its OCI image into `./releases/<ver>/` and drives the live
|
||
target. Rolling update = `load` new → reboot hosts → `unload` old once
|
||
drained.
|
||
- **`spdk-loader.sh`** — DECLARATIVE rolling updates (the "init container that
|
||
stays up", compose service `spdk-loader`). On `compose up` after an
|
||
`IMAGE_VERSION` bump it stages + adds the new rootfs to the running target
|
||
ONLINE, then polls and **auto-reaps** any non-current version once it has
|
||
drained (0 conns) for a sustained grace window (`REAP_GRACE_SECONDS`, keep >
|
||
host `ctrl_loss_tmo`). So migration = bump `IMAGE_VERSION` in `.env` +
|
||
`docker compose up -d` (compose recreates the loader + the http/dnsmasq UKI
|
||
servers, but NOT the nvmeof target → no dropped connections). Single served
|
||
UKI ⇒ a 0-conn old version is genuinely orphaned (a returning host fetches
|
||
the current UKI), so auto-reap is safe; removal is still drain-gated +
|
||
grace-debounced. The loader + target share the RPC socket on a named volume
|
||
(`nvmf_tgt -r`); the loader reuses the SPDK image (has python3 + rpc.py).
|
||
- **BENCH FINDING 2026-07-03 (FIXED) — the target WAS still recreated on an
|
||
`IMAGE_VERSION` bump.** Despite the above, the `nvmeof` (spdk-target) service
|
||
still carried a versioned `type: image` bootstrap mount
|
||
(`source: ${OCI_IMAGE}:${IMAGE_VERSION}` → `/artifacts`) **and** an
|
||
`IMAGE_VERSION` env, so bumping the version changed THIS service's config and
|
||
`compose up -d` recreated it → `nvmf_tgt` restarted → **every connected host's
|
||
NVMe/TCP root dropped** (the exact fleet-wide outage the long-lived-target
|
||
design exists to prevent). Fix: **made the target version-agnostic** — removed
|
||
the `/artifacts` image-volume + `IMAGE_VERSION` env from `nvmeof` in
|
||
`serving/docker-compose.yml`. Bootstrap AND rolling updates are now solely the
|
||
`spdk-loader`'s job (stages the current release into `./spdk/releases`, adds it
|
||
online); `spdk-target.sh`'s seed block is guarded
|
||
(`[ -s /artifacts/verity.json ] && [ -n IMAGE_VERSION ]`) so it no-ops when
|
||
unmounted. Verified `nvmeof` no longer references `IMAGE_VERSION` (only
|
||
`spdk-loader` does). **BENCH-VERIFY:** bump `IMAGE_VERSION` + `up -d` with a
|
||
host connected → `nvmeof` container NOT recreated (`docker ps` ID/uptime
|
||
unchanged), host root survives, loader adds the new version online.
|
||
- **No jq dependency:** `spdk-release.sh` parses RPC JSON with **python3** (the
|
||
SPDK image ships no jq); legacy artifacts (no per-version UUIDs in
|
||
verity.json) fall back to the fixed UUID pair with a warning.
|
||
- **Per-version namespace UUIDs** (`scripts/extract-artifacts.sh`): derived
|
||
deterministically (UUIDv5 of `image_version`), baked into the UKI cmdline +
|
||
`verity.json` (now also carries `subnqn`); the serving side reads them back
|
||
from each release's `verity.json`. Required so two versions don't collide on
|
||
the same `/dev/disk/by-id/nvme-uuid.<uuid>` host symlink.
|
||
- **Bootstrap unchanged:** `docker compose up` with one version still works —
|
||
the `/artifacts` image volume is seeded into `./releases/${IMAGE_VERSION}`.
|
||
- **PARTIALLY BENCH-VERIFIED 2026-06-28:** long-lived target boots a host end
|
||
to end — UKI served over HTTP → host boots → NVMe/TCP root connects to the
|
||
persistent `nvmf_tgt` (single version `spdk-permanence-02ffd7d`). Surfaced +
|
||
fixed several real bugs along the way: bootstrap seed must COPY not symlink
|
||
(a symlink into the version-pinned `/artifacts` re-pointed a stale release at
|
||
new data); `spdk-release` parses RPC JSON via **python3** (SPDK image has no
|
||
jq); legacy `verity.json` (no per-version UUIDs) falls back to the fixed pair;
|
||
mislabeled-dir guard (dir name vs verity.json `image_version`, case-insensitive);
|
||
dropped the redundant discovery-listener re-add (daemon logged "Listener
|
||
already exists"). **CI lowercases `image_version`** (gitea-main-ci vars step) —
|
||
a mixed-case branch (`SPDK-permanence`) had baked a capital
|
||
`fleet-SPDK-….efi` the lowercase boot URL couldn't fetch (iPXE "no such
|
||
file"); booted via a local retag workaround pending a lowercase rebuild.
|
||
- **STILL BENCH-VERIFY:** (a) `load` a 2nd version while a host is connected to
|
||
the 1st → 1st's connection survives (no AER/reset on the running host); (b)
|
||
`unload`/auto-reap drain-gate blocks while CONNS>0 and reaps after grace on a
|
||
LIVE host; (c) serving-host reboot reconciles all staged versions; (d)
|
||
per-version UUID build reproducible (two builds, same version → identical
|
||
`data_ns_uuid`/`hash_ns_uuid` + `root_hash`); (e) the full fix on a clean
|
||
LOWERCASE CI rebuild (no retag). See `serving/spdk/README.md`.
|
||
- **Capacity:** each concurrently-served version costs target memory/hugepages +
|
||
file-backed bdev overhead — plan for N live versions (current + previous +
|
||
canary) during long rollouts (raise `HUGEPAGES`).
|
||
- **Attach gating + discovery:** with several subsystems advertised, the dracut
|
||
connect + per-subsystem hostNQN allow-list must ensure a vX host attaches **only**
|
||
vX's namespace (the by-UUID verity ref protects *what mounts*, but gate the
|
||
*attach* too); `rd.nvmf.discover` selection must be deterministic, not connect-all.
|
||
|
||
## [ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience)
|
||
|
||
There is **no production-grade memory-safe NVMe-oF target** — the Rust NVMe work
|
||
is all driver/initiator-side (vroom, RedLeaf, Rust-for-Linux); the only
|
||
production targets are C (Linux kernel `nvmet`, SPDK). So this is **serving-node
|
||
hardening, not a language swap**. dm-verity already removes the target from the
|
||
clients' integrity TCB (verify-on-read rejects tampered blocks), so the risk is
|
||
RCE/DoS on the node that holds the fleet's disk, not client integrity. The
|
||
export is read-only + single-namespace = inherently small attack surface.
|
||
|
||
- **Evaluate kernel `nvmet` vs SPDK** for the target: kernel `nvmet` keeps the
|
||
data path in the hardened/fuzzed kernel with a tiny userspace footprint
|
||
(arguably a better attack-surface story); SPDK is a userspace C daemon
|
||
(current choice, needs hugepages + caps). Decide per environment.
|
||
- **Confine the target** (whichever): unprivileged, drop caps, seccomp +
|
||
SELinux/AppArmor profile, read-only container, minimal mounts; minimal
|
||
immutable serving image, digest-pinned.
|
||
- **Slim the SPDK image (multi-stage build)**: the current single-stage
|
||
from-source image is ~600 MB (it keeps the whole toolchain — gcc, meson/ninja,
|
||
kernel headers, the python venv — plus the SPDK/DPDK source and build trees).
|
||
Convert `serving/spdk/Dockerfile` to a builder stage + a minimal runtime stage
|
||
that copies only `nvmf_tgt` + its `ldd` shared libs (libnuma, libssl/crypto,
|
||
libuuid, libaio…) + `rpc.py`/`python3`. Expect ~150–250 MB (debian-slim base
|
||
or a static SPDK link trims further). Smaller attack surface + faster pulls,
|
||
and it pairs with the confinement above.
|
||
- **Network-isolate :4420** so only the boot VLAN reaches the target (compose
|
||
with 802.1X/MACsec segmentation in prod).
|
||
- **Replicas** behind distinct discovery entries (availability is the real risk
|
||
— it's the fleet's disk). The target is a **fleet-wide SPOF in every running
|
||
host's root data path**; a process restart drops *all* NVMe/TCP associations at
|
||
once. Structural fix = **HA + NVMe-oF multipath (ANA)** over byte-identical,
|
||
same-UUID, read-only namespaces (replication is coherent precisely because the
|
||
root is read-only + content-addressed); cheap mitigation = generous host
|
||
`ctrl_loss_tmo`/`reconnect_delay`. See the **"SPDK restart / availability"**
|
||
analysis under #6 (rolling updates) — same constraint, fuller treatment.
|
||
- Revisit a memory-safe target only if/when an off-the-shelf one matures
|
||
(e.g. a future Rust-for-Linux `nvmet`).
|
||
|
||
## [ ] 8. Return to secure / production posture (un-toggle the lab flags)
|
||
|
||
Consolidates the cleanup of the temporary lab toggles (see "Current build
|
||
posture" at the top) and the secure-path bench-verifies deferred while testing
|
||
the open-network path. Do these together, then run the full secure boot-chain
|
||
smoke test.
|
||
|
||
- **Re-enable 802.1X/MACsec:** `disable_dot1x=false` (CI `DISABLE_DOT1X="false"`).
|
||
Removes `rd.dot1x=0`/`ip=dhcp`; the hook owns the link (macsec0).
|
||
- **Bench harness scaffolded 2026-06-22: [serving/dot1x-bench/](../serving/dot1x-bench/).**
|
||
step-ca (fleet CA, on the serving host) signs each host's **TPM-generated CSR**
|
||
+ issues the RADIUS server cert; FreeRADIUS does EAP-TLS (trusts step-ca root,
|
||
returns MSK); a `hostapd` wired authenticator does 802.1X + MACsec on an
|
||
**isolated bridge per workstation** (flat VLAN can't auth — EAPOL dropped +
|
||
MACsec is P2P). `enrol/tpm-provision.sh` creates the key in the TPM at
|
||
`0x81000100` + writes the cert to NV `0x01800100`. Starting solution **signs a
|
||
CSR** (key stays in TPM); **TPM attestation** (ACME `device-attest-01`) is the
|
||
documented upgrade on the same CA. **BENCH-VERIFY** items flagged in the files:
|
||
hostapd MACsec build, openssl tpm2 provider vs pkcs11 key ref, EAP-TLS version.
|
||
PKI product decision (greenfield): start step-ca; OpenBao/EJBCA/Dogtag later.
|
||
- **First step (DONE 2026-06-22): `enrol/tpm-selftest.sh`** — non-destructive TPM
|
||
capability check (run on the booted open-port image, no `disable_dot1x` needed)
|
||
to de-risk the TPM half before the full path. Likely first finding: no openssl
|
||
TPM2 **provider** in the image (`tpm2-tss` ≠ the provider) — add `tpm2-openssl`.
|
||
- **Enrolment of the per-host identity** (key@`0x81000100` + cert@NV) is its own
|
||
design — see **#11** (TOFU + OOB fingerprint broker, pre-auth permit-set,
|
||
provisioning UKI). `tpm-provision.sh` is the manual bench stand-in for it.
|
||
- **NVMe discovery by DNS:** `nvmet_traddr=nvme.fleet.lab` (CI), resolved at boot
|
||
by the serving dnsmasq — drop the hardcoded `192.168.50.10`.
|
||
- **Signing — SIGNED-UKI + SECURE-BOOT-ON PATH BENCH-VERIFIED on the Proxmox VM
|
||
(HTTP boot, 2026-06-20).** CI signs the UKI when `SB_KEY`/`SB_CERT` secrets are
|
||
set (`db` key); `fleet-sb-enroll` enrols db→KEK→PK from the baked public `.auth`
|
||
files; OVMF then boots the signed UKI under enforcing Secure Boot. Confirmed
|
||
end-to-end: CI-signed UKI verifies (`sbverify`), enrolment succeeds, VM boots
|
||
with `mokutil --sb-state` = enabled.
|
||
**Key generation + enrolment procedure: [secure-boot-keys.md](secure-boot-keys.md)**
|
||
(the PK/KEK/db trio; `db` = `sb_key`/`sb_cert`, PK/KEK stay offline).
|
||
- **Bug fixed during bring-up (2026-06-20):** `fleet-sb-enroll` built its 4-byte
|
||
EFI attribute prefix in a bash variable (`ATTR=$'\x27\x00\x00\x00'`). Bash
|
||
strings can't hold NULs → only `0x27` survived → efivarfs read a corrupt
|
||
attribute u32 → every write failed with `EINVAL` ("dd: Invalid argument").
|
||
Now emitted directly with `printf` at write time. **Baked fix VERIFIED on the
|
||
820 (2026-06-22): `sudo fleet-sb-enroll` enrolled db→KEK→PK cleanly, no inline
|
||
workaround.** (Red herring on the way:
|
||
Proxmox EFI disks default to `pre-enrolled-keys=1`, so `mokutil --db` shows
|
||
the MS cert even in Setup Mode — looked like a non-empty-db rejection but was
|
||
the NUL bug all along; it failed identically on a clean `pre-enrolled-keys=0`
|
||
disk.)
|
||
- **iPXE signing WIRED (2026-06-21):** `snponly.efi` is now built + signed
|
||
(`db`) + pushed by CI as a separate FROM-scratch **`fleet-ipxe`** image
|
||
(`scripts/build-ipxe-oci.sh`, `.gitea/workflows/ipxe-image.yaml`), mounted into
|
||
dnsmasq at `/ipxe`. dnsmasq itself moved to a generic CI-built/pulled image
|
||
(`.gitea/workflows/dnsmasq-image.yaml`) — no longer built at deploy time, and
|
||
carries no keys/binary. (Decision: separate `fleet-ipxe` image, not baked into
|
||
dnsmasq, so the daemon image stays key-agnostic across SB environments.)
|
||
- **820 / PXE signed chain — BENCH-VERIFIED 2026-06-22.** Real hardware (HP
|
||
EliteBook 820 G4), Secure Boot **enforcing** (`mokutil --sb-state` = enabled,
|
||
Setup Mode no, db/KEK/PK enrolled): firmware verifies the signed `snponly.efi`
|
||
(fleet-ipxe image) → signed UKI and boots. Confirms the whole `fleet-ipxe`
|
||
signing pipeline + the PXE→iPXE→UKI chain under SB on real silicon (the VM only
|
||
covered the HTTP path, which has no iPXE stage).
|
||
- **REMAINING for production:** (a) **secure path** — re-verify signed boot with
|
||
`disable_dot1x=false` (802.1X/MACsec), not just the test path; (b) confirm
|
||
signing is reproducible (verity hash stable across two signed builds).
|
||
- **PXE-only iPXE fallback — BENCH-VERIFIED on the EliteBook 820 G4 (2026-06-20).**
|
||
Firmware without UEFI HTTP boot chainloads `serving/tftp/snponly.efi` → renders
|
||
`autoexec.ipxe` (ifopen + DHCP-retry + `imgargs <real cmdline>`) → UKI over HTTP
|
||
→ NVMe/verity root. dnsmasq tags `pxe`/`ipxe`/`httpboot` give graceful per-client
|
||
fallback; HTTP-boot hosts are unaffected (firmware → UKI direct). Gotchas that
|
||
cost real time (all captured in [../serving/tftp/README.md](../serving/tftp/README.md)):
|
||
(i) the I219-LM needs **`snponly.efi`** (native iPXE driver won't bind);
|
||
(ii) **PortFast/edge on the switch port is REQUIRED** — iPXE's PHY bounce
|
||
triggers STP forwarding delay that drops DHCP (this was the real blocker);
|
||
(iii) iPXE aborts the script on any unguarded command failure (use `||`);
|
||
(iv) Secure Boot OFF makes systemd-stub take iPXE's load options as the cmdline,
|
||
so the entrypoint extracts the UKI's real `.cmdline` and feeds it via `imgargs`.
|
||
**Tidied 2026-06-20** once PortFast was identified: dropped the `autoexec.ipxe` +
|
||
`ifopen` + DHCP-retry STP workaround; `boot.ipxe` is now just
|
||
`dhcp`+`imgfetch`+`imgargs`+`boot` reached via normal iPXE autoboot. Re-verified
|
||
on the 820 G4. **SB-on simplification:** when Secure Boot is enabled the cmdline
|
||
injection is unnecessary (the stub ignores the loader cmdline) — collapse
|
||
`boot.ipxe` to a plain `chain` of the UKI and drop the `.cmdline` extraction +
|
||
`binutils` + the artifact mount from the dnsmasq service.
|
||
- **Sign iPXE for the PXE-only fallback (Secure Boot, production).** The lab
|
||
`snponly.efi` is **UNSIGNED** → that path needs **Secure Boot OFF**. For
|
||
production: `sbsign` the iPXE binary with the fleet **db** key (only `db` needed —
|
||
no MS CA / shim), so SB stays enforced firmware → iPXE → UKI; then the cmdline
|
||
contraption becomes a no-op (the stub ignores loader cmdline under SB). See
|
||
[secure-boot-keys.md](secure-boot-keys.md) "PXE-only hosts".
|
||
- **Bench-verify on the secure path** (swtpm + hostapd + FreeRADIUS,
|
||
`disable_dot1x=false`) — the items deferred this session:
|
||
- NM port secure path: `en*` UNMANAGED + `macsec0` MANAGED/renewing, MKA
|
||
survives switch-root, NVMe stays up past lease (item #1 risk).
|
||
- NTP via DHCP option 42 reaches chrony over macsec0 (`chronyc sources`).
|
||
- Plymouth `fleet` theme: test-path render VERIFIED 2026-06-18; still confirm
|
||
the auth→macsec rows reach `done` live on the secure path.
|
||
- STIG remediation didn't break the secure path; reproducible verity hash;
|
||
triage the generated `stig-deviations.md` REVIEW rows (item #2).
|
||
- **Secure-path NVMe connect trigger:** the test path uses
|
||
`nvmf-connect.service` (gated `rd.dot1x=0`); confirm the secure path's
|
||
connect (dracut nvmf over macsec0) actually fires, or give it an equivalent
|
||
trigger. Apply the same `-k 30 -l -1 -c 10` keep-alive/reconnect flags to the
|
||
secure `nvme connect` as the test path has.
|
||
- **Future hardening waived for now** (from the STIG register): FIPS, firewall,
|
||
per-host/rotatable admin (SSH CA / config-agent).
|
||
- **TLS-encrypted audit/log offload (production).** The lab forwards logs to
|
||
`serving/audit-collector` in **plaintext**, and the STIG TLS-offload rules
|
||
(`rsyslog_encrypt_offload_defaultnetstreamdriver`,
|
||
`..._actionsendstreamdrivermode`, `..._actionsendstreamdriverauthmode`) are
|
||
**waived** in the tailoring because applying them forces `gtls` + x509 peer
|
||
auth + a CA file and breaks the plaintext `2f` forward. For production, do it
|
||
properly and DROP the waivers:
|
||
1. Stand up a CA (or reuse the boot-infra CA); issue the collector a server
|
||
cert (CN/SAN = `audit.fleet.lab`).
|
||
2. Configure the collector's rsyslog for TLS: `imtcp` with
|
||
`StreamDriver.Name="gtls"`, `StreamDriver.Mode="1"`, server cert/key + CA;
|
||
listen on the syslog-TLS port (6514) — add a TLS receiver to
|
||
`serving/audit-collector/rsyslog.conf` + compose.
|
||
3. Bundle the CA into the image (host-generic, no per-host state — same model
|
||
as the RADIUS CA) and point `$DefaultNetstreamDriverCAFile` at it.
|
||
4. Set the `2f` `omfwd` action to `StreamDriver="gtls" StreamDriverMode="1"
|
||
StreamDriverAuthMode="x509/name"` + the CA file, target port 6514.
|
||
5. Re-select the three `rsyslog_encrypt_offload_*` rules (remove from
|
||
`files/stig/diskless-tailoring.xml`) — they should then PASS, and update
|
||
`docs/stig-deviations.md` #6 to reflect encrypted offload.
|
||
(RELP+TLS via `omrelp`/`imrelp` is the alternative if guaranteed delivery is
|
||
wanted alongside encryption.)
|
||
- **fleet-hostapi: scope CORS to the kiosk origin.** The host-info agent ships
|
||
with `FLEET_HOSTAPI_ALLOW_ORIGIN=*` (lab). For prod set it to the exact kiosk
|
||
origin (default `http://boot.fleet.lab`) via the unit's commented
|
||
`Environment=` line (or a drop-in). See #12.
|
||
|
||
## [ ] 9. (OPTIONAL / nice-to-have) Convert provisioning to Ansible where it pays off
|
||
|
||
Low priority — only the *config-management* half of the build is a candidate, and
|
||
the ROI is modest because the build VM is single-shot (idempotency buys little)
|
||
and reproducibility/auditability favour minimal shell. **Not** a wholesale
|
||
rewrite: the load-bearing logic (dracut module hook ordering, `extract-artifacts.sh`,
|
||
`build-oci.sh`) is initramfs-runtime / host-side build tooling, not provisioning,
|
||
and stays shell.
|
||
|
||
The high-ROI, low-risk slice if/when we pick this up:
|
||
- **STIG via the SSG Ansible playbook** (`ssg-rl9-playbook-stig.yml`, shipped
|
||
with scap-security-guide) instead of `oscap --remediate` — the upstream path
|
||
and far more *reviewable* (you can diff exactly what each rule changes). Keep
|
||
`oscap eval` (no remediate) for the report + the 0-eval guard (item #2).
|
||
- Restyle the brittle imperative bits as declarative tasks: chrony.conf edit
|
||
(`lineinfile`/`blockinfile` vs `sed`), sshd drop-in / NM dispatcher / kiosk
|
||
file installs (`template`/`copy`), datastream-by-`os-release` as `assert`.
|
||
Bonus: sidesteps the `%{...}`→`%%{` HCL-escaping footgun.
|
||
|
||
Tips the balance toward "do it" only if: we start managing the **serving hosts**
|
||
(dnsmasq/nginx/SPDK/chrony/audit-collector) with config-management too (shared
|
||
roles to amortise — currently they're docker-compose), provisioning grows much
|
||
more branchy, or we want molecule-tested hardening roles validated without a
|
||
full ~17-min Packer build.
|
||
|
||
## [ ] 10. (OPTIONAL) Secure Boot cert auto-enrollment for provisioning
|
||
|
||
Speed up bringing up new workstations: bake the **public** PK/KEK/db certs into an
|
||
image, detect Secure Boot **Setup Mode** at boot, and auto-enroll them so a
|
||
fresh/reset box adopts the fleet trust root without a manual firmware-setup visit.
|
||
Pairs with #8 (signing) + [secure-boot-keys.md](secure-boot-keys.md). Shares the
|
||
provisioning-UKI + isolated-environment vehicle with **#11** (which enrols the
|
||
*network* identity); build them together.
|
||
|
||
**Tooling gap (found 2026-06-20): `efitools` is NOT packaged for EL9** — no
|
||
`efi-updatevar`/`efi-readvar`/`KeyTool`. Resolved for the OS-side path by writing
|
||
the signed `.auth` variables to **efivarfs directly** (de-immutable + single
|
||
write), which needs no extra package. **Done 2026-06-20:** baked
|
||
`/usr/local/bin/fleet-sb-enroll` (efivarfs method, Setup-Mode-gated, fapolicyd-
|
||
trusted) + the public `PK/KEK/db.auth` at `/usr/share/fleet-secureboot/` (supplied
|
||
via `files/secureboot/keys/` or `SB_*_AUTH_B64` CI secrets); `mokutil` covers
|
||
reading. So **manual** OS-side enrollment over SSH now works (`sudo
|
||
fleet-sb-enroll`). What's left for *auto*-enroll: drive `fleet-sb-enroll`
|
||
unattended on a **Setup-Mode detection** at boot, gated to a provisioning mode/UKI
|
||
(below), with read-back verify + audit — NOT armed in the production image.
|
||
|
||
**What's safe:** the certs are public (no key material — only the **db** private
|
||
key signs, and it stays in CI/HSM, invariant #7); Setup Mode accepts
|
||
*unauthenticated* writes to PK/KEK/db, so public certs (ESL) are sufficient. It's
|
||
fleet-generic (no per-host state, invariant #1) and deterministic (reproducible
|
||
verity hash).
|
||
|
||
**The real risk — provisioning-time TOFU (trust-on-first-use):** in Setup Mode
|
||
Secure Boot isn't enforcing, so the enroller runs as *unverified* code, and the
|
||
fleet's UEFI **HTTP boot is unauthenticated** in that state. An on-path attacker
|
||
during provisioning can serve a malicious image that enrolls **their** PK/KEK/db —
|
||
the box then "has Secure Boot" but trusts the attacker. Auto-enroll is only as
|
||
trustworthy as the provisioning channel, and it removes the physical-presence
|
||
approval gate. So treat it as a controlled-environment provisioning step, not an
|
||
always-on behaviour.
|
||
|
||
**Optional separate provisioning UKI (preferred shape):** build a dedicated
|
||
**provisioning UKI** that carries the certs + the enroll logic, instead of arming
|
||
the enroller in the production image. Keeps the standing footgun out of the normal
|
||
image (a factory reset + tampered image can't silently re-root trust), and lets
|
||
the enroller be a one-shot, attended/isolated step. The production UKI never
|
||
enrolls.
|
||
- Build it as a separate Packer target / flag (e.g. `provisioning=true`) →
|
||
its own small UKI; signed or unsigned (it runs in Setup Mode anyway, but sign
|
||
it so it *also* boots once keys are enrolled / for re-runs).
|
||
- Or a tiny standalone EFI tool (KeyTool-style) served alongside, not a full OS.
|
||
|
||
**Mitigations (do these regardless of UKI vs separate UKI):**
|
||
- Run provisioning on a **trusted, isolated VLAN** (or a physically controlled
|
||
bench) — the integrity Setup Mode can't give must come from the environment.
|
||
- **Gate** enrollment behind an explicit provisioning mode / one-shot token;
|
||
never arm it on every boot of the production image.
|
||
- **Read-back verify before declaring success:** `efi-readvar` the enrolled
|
||
PK/KEK/db and compare fingerprints to the values baked in; mismatch = fail
|
||
loudly (turns silent TOFU into tamper-evidence).
|
||
- **Audit** the enrollment event + read-back fingerprints to the collector.
|
||
- **Keep the Microsoft UEFI CA in `db`** (or enroll your db alongside it) unless
|
||
every NIC/GPU option ROM is covered by your keys — clearing MS db can break
|
||
signed **option ROMs** (NIC PXE/HTTP-boot, GPU init), bricking the boot path.
|
||
- **PCR 7 check:** enrollment changes the Secure Boot measurement in PCR 7;
|
||
confirm nothing the fleet seals to the TPM (the EAP-TLS key at `0x81000100`,
|
||
NV cert) is PCR-7-policy-bound before relying on this.
|
||
|
||
**Decision before building:** auto-enroll trades assurance for convenience and is
|
||
reasonable for a controlled fleet; the highest-assurance path is attended/firmware
|
||
or manufacturer pre-provisioning. Default recommendation: **separate provisioning
|
||
UKI + isolated VLAN + read-back verify.**
|
||
|
||
## [ ] 11. (DESIGN) TPM 802.1X identity enrolment (TOFU + OOB fingerprint)
|
||
|
||
**Authoritative design consolidated 2026-06-28:
|
||
[tpm-enrolment-design.md](tpm-enrolment-design.md)** — unifies this item with the
|
||
fleet-manager (the broker = the fleet-manager's one unauthenticated bootstrap
|
||
endpoint; approval births the inventory record) and the full identity lifecycle
|
||
(enrol / renew / alert). **v1 decisions (2026-06-28):** fingerprint-only OOB
|
||
assurance (TPM attestation via ACME `device-attest-01` is the documented later
|
||
upgrade); **inline single-UKI** vehicle for now (empty-NV early-boot branch — keeps
|
||
provisioning network-boot-only, no USB; a separate provisioning UKI behind an
|
||
`enrol_mode` build flag is the planned later upgrade); enrolment transport is plain
|
||
**stdlib HTTP/JSON** (Go, the one unauthenticated edge — gRPC reserved for the
|
||
steady-state agent path). The notes below are retained as the originating decisions;
|
||
the design note is canonical.
|
||
|
||
**Broker BUILT (server side, 2026-06-28): [serving/enrol-broker/](../serving/enrol-broker/)**
|
||
— Go/stdlib; public unauth `POST /request`+`GET /status/{id}` (behind the serving
|
||
nginx `/enrol/`) + admin **mTLS web UI** (`/`, `/pending`, `/approve|deny/{id}`)
|
||
with a **dedicated admin CA** (`admin-pki.sh`); wordlist fingerprint, time-boxed
|
||
store, per-IP rate limit; signs via `step` (dev `mock` signer). Ships in
|
||
`serving/dot1x-bench/docker-compose.yml`; end-to-end smoke-tested (CSR → mTLS
|
||
approve → cert). **BENCH-VERIFY:** run against real step-ca (`bootstrap.sh` root +
|
||
`enrol` provisioner password readable by the broker) and confirm an issued cert
|
||
authenticates EAP-TLS.
|
||
|
||
**HOST SIDE BUILT 2026-07-02 — Go binary (`fleet-enrol`), awaiting bench.**
|
||
Pivoted from a shell hook to a static **Go + go-tpm** binary (`files/enrol/`):
|
||
it talks to `/dev/tpmrm0` DIRECTLY (via `transport.FromReadWriter` on an
|
||
`os.File` — no `linuxtpm`, so go-tpm is the only vendored dep), so the enrolment
|
||
path needs **no tpm2-tools key-gen / no openssl tpm2 provider / no curl** in the
|
||
image (removes the biggest bench risk — the provider `handle:` key-ref). It
|
||
ensures the ECDSA key at `0x81000100`, builds the CSR, POSTs CSR+context to
|
||
`rd.fleet.enrol_url`, shows the broker fingerprint on Plymouth, polls `/status`,
|
||
and writes the leaf cert to NV `0x01800100`. The dracut hook `tpm-enrol.sh`
|
||
(initqueue/settled prio 25) is now a thin GATE: empty-NV check (`tpm2_nvread`,
|
||
still needed for the secure-path read) → pre-auth DHCP on `en*` (nmcli override of
|
||
the secure-mode unmanaged drop-in) → `exec /usr/sbin/fleet-enrol` → reboot to
|
||
converge on exit 0. No-op when a cert is present or under `rd.dot1x=0`.
|
||
|
||
**Built OUT OF THE VM (2026-07-02, per decision):** `fleet-enrol` AND
|
||
`fleet-hostapi` now compile on the CI runner (`.gitea/workflows/gitea-main-ci.yaml`
|
||
"Build host Go binaries" step, Go 1.26.4, `CGO_ENABLED=0`, vendored/stdlib,
|
||
`GOPROXY=off`), and Packer **file-provisions** the static binaries in — the Rocky
|
||
build image no longer carries the Go toolchain (dropped the in-VM `dnf install
|
||
golang` + the 2d removal). `fleet-enrol` lands at `/usr/sbin` before the initramfs
|
||
build (dracut `inst_simple`s it; asserted in `lsinitrd`), fapolicyd-trusted.
|
||
New `enrol_url` Packer var → `rd.fleet.enrol_url` cmdline.
|
||
|
||
**TEST-MODE ENROLMENT ENABLED (2026-07-02, temporary):** the hook no longer
|
||
no-ops under `rd.dot1x=0` — enrolment runs on the open/test path too, so it can be
|
||
exercised on a host with a TPM + the broker WITHOUT standing up
|
||
802.1X/MACsec/RADIUS. In test mode `ip=dhcp` owns the NIC (the hook skips its own
|
||
DHCP to avoid the dracut "address already assigned" race) and DNS resolves the
|
||
broker via the lease. `rd.fleet.enrol=0` disables it. Production intent is still
|
||
secure-path-only; revisit when the secure path is bench-verified.
|
||
|
||
**TESTED (2026-07-02):** transaction logic (CSR → mock broker → approve → cert
|
||
extract) as stdlib unit tests, AND the **full TPM ceremony (key-gen → persist →
|
||
ECDSA sign → CSR verify → chunked NV write → re-enrol) against the go-tpm software
|
||
simulator** (separate `files/enrol/simtest/` module so the CGO simulator never
|
||
enters the shipped build). Both run in CI.
|
||
|
||
**BENCH FINDING 2026-07-02 (FIXED) — the signing-key handle `0x81000100` was the
|
||
EK slot; moved to `0x81000100`.** First real-TPM boot (host `cygnus-cruiser-99`,
|
||
Proxmox vTPM, `rd.dot1x=0`) failed: `fleet-enrol: ensure key at 0x81010001: key at
|
||
handle is not ECC: did not contain eccDetail (selector value was 1)`. `EnsureKey`
|
||
died reading an existing RSA object as ECC → enrolment died before the CSR/POST, so
|
||
the splash flashed "FIRST-BOOT TPM ENROLMENT" then the test system booted with **no
|
||
fingerprint and nothing at the broker**. Root cause (confirmed via `tpm2_readpublic`
|
||
under `enforcing=0`): **`0x81000100` is the TCG-standard RSA Endorsement Key handle**
|
||
— the object there was the swtpm-provisioned **EK** (`restricted|decrypt|adminwithpolicy`,
|
||
RSA-2048/AES128CFB, canonical EK authPolicy `837197674484b3f8…`), with its EK cert at
|
||
NV `0x1c00002`. Our design squatted the EK slot fleet-wide (most firmware/swtpm TPMs
|
||
have their EK there), so enrolment could never create our key.
|
||
- **Fix (commit c574fa4 + follow-up):** the fleet signing key moved to
|
||
**`0x81000100`** (non-reserved owner handle; threaded through `fleet-enrol
|
||
-handle` default, `rd.dot1x.handle=` cmdline knob baked by `extract-artifacts.sh`,
|
||
`wpa_supplicant-dot1x.conf`, the bench scripts, invariant #1, and the design docs).
|
||
NV index was `0x01800001` here (owner range); it MOVED to `0x01800100` in a later finding (OEM platform-NV collision on the 820 — see below).
|
||
- **EnsureKey hardening:** (i) evicts a genuine *stale signing key* at the handle
|
||
and recreates (self-heal for old provisioning leftovers); (ii) **refuses to evict
|
||
any restricted-decrypt object (EK/SRK)** as a safety net — never clobber the TPM's
|
||
endorsement/storage identity (also protects the future EK/AK attestation).
|
||
Regressions: `simtest/TestEnsureKeyEvictsStaleNonECC` + `TestEnsureKeyRefusesToEvictProtectedKey`.
|
||
- **IDevID note (future):** if EK/AK attestation (`device-attest-01`, §"Stronger
|
||
assurance") lands, the natural home for a TPM device-identity key is the TCG
|
||
**IDevID** handle `0x81020001` (LDevID `0x81030001`) — see tpm-enrolment-design.md.
|
||
Not adopted for v1 (implies the IDevID template/cert conventions).
|
||
- **Also seen — leftover NV suppresses enrolment:** a *different* host (the 820)
|
||
booted `tpm-enrol: client cert present at NV 0x01800001 — normal boot` with nothing
|
||
enrolled this session — leftover NV state (from an earlier `tpm-provision.sh` /
|
||
`tpm-selftest.sh --use-real-handles`) **persists in the hardware TPM** across
|
||
reinstalls. Our presence check is content-blind (index defined ⇒ "enrolled"), so
|
||
foreign/stale NV silently skips enrolment (→ secure-path auth would then fail). See
|
||
the **content-aware NV check** item below.
|
||
- **Needs a rebuild + re-boot to confirm** (the sim proves the ceremony, not the
|
||
device). ⚠ **Don't hand-run `sudo fleet-enrol` on the booted STIG host** — the
|
||
confined admin domain (`sysadm_t`) is SELinux-**denied** write to `/dev/tpmrm0`
|
||
(`avc: denied … tpm_device_t`); the faithful path is the **initramfs** run
|
||
(reboot, unconfined), or `enforcing=0` for hand-run TPM inspection.
|
||
|
||
**FOLLOW-UP HARDENING:**
|
||
- **Content-aware NV presence check — BUILT 2026-07-03.** `fleet-enrol -check-only`
|
||
(no network / no key-gen) reads NV `0x01800100` via `tpmkey.ReadNVCert`, parses it as
|
||
X.509, checks the validity window, and — when a real fleet CA is bundled (`-cacert`,
|
||
default `radius-ca.pem`) — verifies it **chains** to it. Only a valid fleet cert counts
|
||
as enrolled (rc 0); empty/garbage/foreign/expired → rc 20 (enrol + overwrite;
|
||
`WriteNVCert` undefine+redefines). The dracut hook now calls it as the single gate,
|
||
replacing the blind `tpm2_nvread` (removed from `tpm-enrol.sh`; the binary's `NVPresent`
|
||
stays for cheap existence checks). **Lab fallback:** the shipped `radius-ca.pem` is a
|
||
*placeholder* (comments, not a cert) → `loadCAPool` returns nil → chain-less "parseable
|
||
+ unexpired" acceptance, so the reboot-converge doesn't loop on a bench-CA cert;
|
||
**production must replace `radius-ca.pem` with the real fleet CA** for strict chaining.
|
||
Tests: `classifyNVCert` unit table (valid/expired/garbage/foreign/no-CA) + `simtest`
|
||
ReadNVCert roundtrip across the 512B chunk. **BENCH-VERIFY** on a real boot: a garbage/
|
||
foreign NV now enrols instead of "normal boot".
|
||
- **De-enrol / TPM-reset procedure** for re-testing: `tpm2_nvundefine 0x01800100 -C o`
|
||
(+ `tpm2_evictcontrol -C o -c 0x81000100` for a stale *app* key — never the EK), or
|
||
BIOS "Clear TPM". Document in the bench README.
|
||
|
||
**BENCH FINDING 2026-07-03 (FIXED) — NV index `0x01800001` collided with OEM
|
||
platform NV; moved to `0x01800100`.** The Dell 820 kept "finding a cert" and
|
||
bypassing enrolment even after repeated BIOS TPM clears. `tpm2_nvreadpublic`
|
||
showed the truth: `0x01800001` on that hardware is a **firmware/platform** index
|
||
(`platformcreate|policydelete|writelocked`, 70 bytes — NOT our cert; siblings at
|
||
`0x01800003/4` + `0x01810000` share it), which **no owner/BIOS clear can remove**
|
||
(`TPM2_Clear` spares `platformcreate`; `policydelete` needs the platform's secret
|
||
policy). So our chosen index was permanently occupied by Dell — the blind check
|
||
false-positived on it, and even the content-aware check would have failed at the
|
||
NV **write** (can't undefine a platform index). Same class as the EK-handle
|
||
collision. Fix: **moved the NV cert index default `0x01800001` → `0x01800100`**
|
||
(free on the 820; owner range; overridable via `rd.dot1x.nvindex=`), threaded
|
||
through the cmdline (`extract-artifacts.sh`), `fleet-enrol -nvindex`, the dracut
|
||
hooks, bench scripts, invariant #1, and docs. Also **hardened `WriteNVCert`** to
|
||
refuse a `platformcreate` index with a clear "pick a free owner index" error
|
||
instead of a confusing undefine failure (regression `simtest/
|
||
TestWriteNVCertRefusesPlatformIndex`). **BENCH-VERIFY:** 820 now enrols at
|
||
`0x01800100`; and check `tpm2_nvreadpublic` on any OTHER hardware model in the
|
||
fleet — if a vendor also squats `0x018001xx`, pick a value free on all of them.
|
||
|
||
**BENCH-VERIFY (host side, before trusting):** (a) go-tpm against a REAL TPM +
|
||
swtpm on the target (the simulator proves the ceremony, not the device) — **the
|
||
stale-key fix (above) is the first real-TPM result; re-boot to confirm it now
|
||
reaches the broker**; (b) empty-NV create/persist on a fresh TPM with **empty owner
|
||
auth** (the code assumes it — a set owner auth needs handling); (c) pre-auth DHCP on `en*` in secure mode
|
||
(the `nmcli set managed yes` override of nm-unmanaged.sh actually leases); (d)
|
||
fingerprint renders on the `fleet` Plymouth theme + admin can match it; (e) NV
|
||
write + **reboot converges** (2nd boot runs the secure path); (f) prod: broker over
|
||
**HTTPS** (`fleet-enrol -cacert radius-ca.pem`), currently lab-HTTP.
|
||
|
||
**TEMPORARY (2026-06-28): broker also added to the MAIN serving stack**
|
||
(`serving/docker-compose.yml`, `enrol-broker` service + `ENROL_*` in
|
||
`serving/.env.example`) so a single `compose up` in `serving/` brings the
|
||
`/enrol/` endpoint up while the host side is built. Defaults to the
|
||
zero-dependency lab path — **`mock` signer** (ephemeral CA; certs NOT trusted by
|
||
RADIUS/NVMe) + **plaintext loopback admin** (`ENROL_ADMIN_INSECURE=true`). The
|
||
serving nginx already proxies `/enrol/`→`127.0.0.1:9090`. **SPLIT BACK OUT:** the
|
||
broker's proper home is a CA-co-located stack (it's still defined in
|
||
`dot1x-bench/` next to step-ca); real signing needs the fleet CA. Don't run both
|
||
copies on one host (they collide on 9090/9443). To use real signing from the
|
||
serving stack meanwhile: set `ENROL_SIGNER=step` + `ENROL_STEP_CA_URL` at a
|
||
reachable step-ca, populate `serving/enrol-pki/`, and `ENROL_ADMIN_INSECURE=false`.
|
||
|
||
**STEP SIGNER VERIFIED end-to-end 2026-06-28** (broker → real step-ca → signed
|
||
cert that chains to the fleet root; hostnqn derived from serial; EKU clientAuth).
|
||
**Re-VERIFIED ON THE SERVING HOST 2026-07-02** against step-ca 0.30.2 via
|
||
`verify-serving.sh` through the nginx `/enrol/` proxy: issuer = Fleet Bench CA
|
||
Intermediate, cert validity exactly 720h (confirms the maxTLSCertDuration claim
|
||
raise took effect), EKU clientAuth, `enrol` provisioner extension present,
|
||
hostnqn = cert serial. Bring-up gotchas hit + fixed (all in the commits): the CA
|
||
wouldn't init (compose used `_PASSWORD_FILE` for a nonexistent file → use
|
||
`_PASSWORD`); Docker auto-created `./pki` root-owned (bootstrap self-heals perms +
|
||
pre-create the bind-mount dirs); the CLI couldn't resolve `ca.fleet.lab` on the
|
||
serving host (connect via `localhost`, which is in the CA SAN).
|
||
Verification found + fixed two things the `mock` path had hidden:
|
||
- **BUG FIXED (`signer.go`):** `--not-after` was passed to `step ca token`
|
||
(validity capped ~1h → "token validity too long") instead of `step ca sign`.
|
||
Moved to the sign step. Without it the step signer never issues a cert.
|
||
- **CA CLAIM REQUIRED:** a stock step-ca caps certs at 24h, so 720h is rejected
|
||
("authorized maximum certificate duration of 24h"). `dot1x-bench/step-ca/bootstrap.sh`
|
||
now raises `authority.claims.maxTLSCertDuration` (+restarts the CA) — this also
|
||
unblocks bootstrap's own 720h RADIUS cert. Any other CA needs the same claim.
|
||
- **BENCH-VERIFY still open:** issued cert actually authenticates EAP-TLS through
|
||
FreeRADIUS (next, before host enrolment).
|
||
|
||
How a host gets its **802.1X EAP-TLS client identity** the first time — the key at
|
||
TPM handle `0x81000100` + cert at NV `0x01800100` that invariant #1 says must come
|
||
from enrolment, never the image. Distinct from #10 (that enrols *Secure Boot*
|
||
PK/KEK/db); this enrols the *network identity*. They share the provisioning UKI +
|
||
isolated-environment vehicle. Pairs with #8 + the CA decision (start **step-ca**,
|
||
[serving/dot1x-bench/](../serving/dot1x-bench/); OpenBao/EJBCA/Dogtag later).
|
||
|
||
**Chosen pattern — trust-on-first-use with out-of-band fingerprint approval:**
|
||
on an empty TPM the host generates a key *in* the TPM, builds a CSR, submits it to
|
||
an enrolment **broker**, and shows a **fingerprint** on the Plymouth splash; an
|
||
admin visually cross-checks that fingerprint against the broker's pending-list and
|
||
approves; the broker has step-ca sign the CSR and returns the cert; the host writes
|
||
it to NV `0x01800100` and proceeds to authenticate. Sound, recognised pattern
|
||
(SSH/WireGuard/Signal-style OOB verification); good fit for invariant #1.
|
||
|
||
**Key design decisions (settled 2026-06-22):**
|
||
|
||
- **The pre-auth permit-set already exists — enrolment rides in it.** The firmware
|
||
can't do 802.1X, so the UKI fetch (DHCP/DNS/TFTP/HTTP to the serving host) is
|
||
**always pre-auth**; the secure model therefore *requires* a pre-auth permit-set
|
||
(low-impact 802.1X / pre-auth dACL / boot→secure dynamic VLAN; RADIUS returns the
|
||
secure VLAN on Access-Accept). The enrolment endpoint is just **another entry in
|
||
that set** — expose it as an `/enrol/` location on the **existing serving nginx**,
|
||
reverse-proxied to the broker. No new network mechanism.
|
||
- **The broker is the only new component.** step-ca signs but has no human-approval
|
||
queue: broker = `POST CSR + context → PENDING + fingerprint → admin approves →
|
||
step-ca signs → host polls → cert`. Exposed to *unauthenticated* devices by
|
||
design → harden: OOB-fingerprint gate (no auto-sign), rate-limit, time-boxed
|
||
pending, HTTPS (image trusts the serving CA), pre-auth permit scoped to only
|
||
these services + the serving host.
|
||
- **Trigger on EMPTY-NV only — split the three events:** empty NV → enrol (human);
|
||
valid key + expired/near-expiry cert → **renew** automatically (no human, it
|
||
already holds a trusted key); auth fails with a present cert → **alert + retry,
|
||
do NOT re-enrol** (auto-reenrol on failure would mask revocation and churn keys
|
||
on a transient RADIUS outage). Re-enrol after the first time = explicit admin
|
||
action (clear/flag the host).
|
||
- **Fingerprint: what it proves + entropy.** It proves "this CSR came from *this
|
||
physical device*" (approver reads the screen) — it does NOT prove TPM residency.
|
||
Pair it with **TPM attestation** (step-ca ACME `device-attest-01`, EK/AK) for the
|
||
residency proof: fingerprint = human authorises the device, attestation = the key
|
||
is really in a TPM (together → strong SSP R-01 story). Size for entropy — a short
|
||
hex fp is grindable; use a word-list/BIP39-style encoding (reads short, more
|
||
bits) + rate-limiting + a server nonce. Show device context (asset/port/location)
|
||
so the admin knows *what* they're approving.
|
||
- **Pre-auth traffic has no MACsec (plaintext at L2).** Fine for enrol (CSR/cert
|
||
are public; OOB fingerprint gives authenticity) — and it's exactly **why the UKI
|
||
must be Secure-Boot-signed** (#8): integrity of the boot comes from the signature,
|
||
not the network. Enrolment design + Secure Boot are two halves of the same
|
||
"pre-auth path is hostile" assumption.
|
||
- **Run it in a provisioning UKI/mode, not the production initramfs** — the boot
|
||
hook ordering is load-bearing (invariant #2); don't park it on a multi-minute
|
||
human-approval wait. Shares the #10 provisioning-UKI vehicle.
|
||
|
||
**Bench caveat:** `hostapd` wired is strictly port-based (no pre-auth per-flow
|
||
ACL) — before EAP success it passes nothing, so the bench can't reproduce
|
||
"boot + enrol pre-auth on the same port." Model the pre-auth state with a separate
|
||
provisioning bridge/VLAN on the rig; the in-place low-impact pre-auth permit is a
|
||
real-switch feature to validate on the 820's switch.
|
||
|
||
**First step DONE (2026-06-22):** `serving/dot1x-bench/enrol/tpm-selftest.sh` — a
|
||
non-destructive TPM capability check (device, key persist, NV roundtrip, openssl
|
||
TPM2 provider, CSR signed by the TPM key) to de-risk the TPM half before any of
|
||
this. Expected gap: the image installs `tpm2-tss` but no openssl TPM2 **provider**
|
||
(`tpm2-openssl`) — add it to the build once the self-test confirms.
|
||
|
||
## [~] 12. Workstation host-info API (`fleet-hostapi`) for the kiosk
|
||
|
||
**IMPLEMENTED 2026-06-28.** A small **Go, read-only, loopback-only** HTTP agent on
|
||
the workstation that reports host/OS/hardware facts to the kiosk SPA, per the
|
||
[HOST-API.md](../HOST-API.md) contract (`GET http://localhost:9099/v1/host`).
|
||
Source: `files/hostapi/` (stdlib-only, reproducible static build). Built + enabled
|
||
by provisioner **2b-hostapi**; toolchain removed in 2d; binary fapolicyd-trusted
|
||
in 2c-STIG-postfix (register #10). Chromium policy grants Private Network Access
|
||
scoped to the kiosk origin (`files/kiosk/chromium-kiosk-policy.json`).
|
||
|
||
- **Per-host safety:** host-specific fields (kernel, hostname, IP, MAC, board,
|
||
uptime) are read at RUNTIME; only image-generic facts (image version +
|
||
weston/chromium versions) are baked, to `/etc/fleet-hostinfo.json` (invariant
|
||
#1 preserved). Deliberately does NOT expose DMI serial (root-only/per-host) or
|
||
any TPM/cert/identity material.
|
||
- **Hardening shipped:** loopback bind enforced in code; `DynamicUser`, no caps,
|
||
`ProtectSystem=strict`, `MemoryDenyWriteExecute`, `SystemCallFilter=@system-service`,
|
||
`IPAddressDeny=any` (loopback only). Runs unprivileged, reads world-readable
|
||
files only.
|
||
|
||
**Lab posture LEFT AS-IS (decided 2026-06-28) — revisit for prod:**
|
||
- **CORS `Allow-Origin=*`** → scope to the kiosk origin (`http://boot.fleet.lab`).
|
||
Consolidated into #8.
|
||
- **SELinux:** runs in the default service domain (no custom policy module).
|
||
**BENCH-VERIFY** no AVC denials for `fleet-hostapi` on an enforcing booted host
|
||
(Wazuh/`fleet_audit` rule 100120 will surface any). If denials appear, decide:
|
||
write a confined SELinux policy module vs adjust the sandbox.
|
||
- **No auth** (justified: loopback + read-only + single-user appliance). Option if
|
||
the threat model tightens: shared token in `kiosk.env` + header check.
|
||
- **Info sensitivity:** exposes hostname/IP/MAC/board/versions to the kiosk. Drop
|
||
MAC/IP if that's later deemed too much.
|
||
- **Reproducibility:** the `golang` toolchain version is a reproducibility-sensitive
|
||
input (a package bump changes the binary → verity hash); covered by the
|
||
build-twice gate. Pin the go version only if drift becomes a problem.
|
||
|
||
**BENCH-VERIFY (functional):** (a) service starts under enforcing SELinux +
|
||
fapolicyd (not blocked); (b) the kiosk SPA's System page fetches `/v1/host`
|
||
successfully through the PNA allowance (preflight + GET); (c) facts render
|
||
correctly on real hardware (board/CPU/GPU from DMI/DRM, not the virtio build VM).
|
||
|
||
## [~] 13. Kiosk GPU compositing + Chromium flag ownership
|
||
|
||
**GPU FIX APPLIED 2026-06-28.** The 820 (Intel HD620/i915) ran Chromium at ~150%
|
||
CPU with periodic crashes: `--disable-gpu` in our `kiosk-session` forced software
|
||
raster/compositing. Root cause confirmed from the live cmdline. Fixes in
|
||
`files/kiosk/kiosk-session` + `diskless.pkr.hcl`:
|
||
- **Runtime GPU probe** (fleet-generic image boots real HW *and* virtio VMs): a
|
||
real KMS driver (`i915`/`amdgpu`/…) + `/dev/dri/renderD128` → GPU path
|
||
(`--ignore-gpu-blocklist --enable-gpu-rasterization --enable-zero-copy`, weston
|
||
GL renderer); else software (`--disable-gpu` + `weston --use-pixman`) so the VMs
|
||
still render. No per-host state baked.
|
||
- **kiosk user added to `render`** (renderD128 isn't logind-seat-ACL'd on EL9;
|
||
`video` only covers card0/KMS for weston).
|
||
- **Merged the distro wrapper's `--disable-features` into ours** — Chromium takes
|
||
the LAST `--disable-features` (not the union), so ours was silently dropping the
|
||
wrapper's Lens/Gemini/Compose disables.
|
||
|
||
**BENCH-VERIFY (on the 820):** GPU process alive (`chrome://gpu`), Compositor/
|
||
ThreadPool threads near-idle + CPU drop, `gt_act_freq_mhz` rises under load, no
|
||
SELinux AVC for `dri_device_t` from the kiosk domain (Wazuh `fleet_audit` 100120
|
||
will surface any — if present, add a scoped SELinux allow or confirm the kiosk
|
||
user's SELinux mapping is unconfined). Confirm VMs still render (software path).
|
||
|
||
**DECIDED 2026-06-28 — Option 1 (policy lock), APPLIED.** The distro chromium
|
||
wrapper (`/usr/bin/chromium-browser`) layers flags as `$CHROMIUM_FLAGS`
|
||
(`/etc/chromium/chromium.conf` → VA-API enables + AI disables; overridable via
|
||
`CHROMIUM_USER_FLAGS`) + **hardcoded** `$CHROMIUM_DISTRO_FLAGS`
|
||
(`--enable-plugins --enable-extensions --enable-user-scripts --enable-printing
|
||
--enable-sync --auto-ssl-client-auth`) + our `"$@"` (last). The hardcoded layer
|
||
isn't removable via env/conf — only by bypassing the wrapper. Rather than bypass
|
||
(Option 2, which would force us to replicate the wrapper's stdio sanitisation +
|
||
own the VA-API feature list across updates), we **neutralise in managed policy**
|
||
(policy beats cmdline flags, survives updates): `ExtensionInstallBlocklist=["*"]`
|
||
added to `files/kiosk/chromium-kiosk-policy.json` (blocks all extensions → also
|
||
kills the user-scripts surface). `--enable-sync` already locked by `SyncDisabled`;
|
||
`--auto-ssl-client-auth` is moot (no browser-store client cert; the TPM 802.1X
|
||
cert isn't one) — `AutoSelectCertificateForUrls` left unset. VA-API enables
|
||
inherited from `chromium.conf` unchanged. Option 2 (full flag ownership, bypass
|
||
the wrapper, own the VA-API `--enable-features`:
|
||
`AllowQt,WaylandLinuxDrmSyncobj,WaylandPerSurfaceScale,WaylandUiScale,AcceleratedVideoDecodeLinuxGL,AcceleratedVideoDecodeLinuxZeroCopyGL,AcceleratedVideoEncoder`)
|
||
remains documented here if a minimal/auditable cmdline is ever wanted.
|
||
**BENCH-VERIFY:** `chrome://policy` shows `ExtensionInstallBlocklist=["*"]` active;
|
||
extensions can't install.
|
||
|
||
## [ ] 14. (DESIGN) Centralised fleet management plane (health/version/usage/maintenance)
|
||
|
||
**DESIGN CAPTURED 2026-06-28 — [docs/fleet-management-design.md](fleet-management-design.md).**
|
||
Server-authoritative, pull-based, TPM-identified management plane reusing existing
|
||
pieces: `fleet-hostapi` as the on-host data source, the audit-collector→Wazuh
|
||
pipeline as telemetry transport, the digest-pinned deployment manifest as version
|
||
truth, and the SPDK `spdk-loader`/`spdk-release` rolling-update flow (#6) as the
|
||
redeploy/drain mechanics. Hosts stay stateless (identity from the TPM cert, no
|
||
baked enrollment); control is pull/reconcile (no inbound surface); convergence to
|
||
a new version = reboot. **Phased build:** (1) push hostapi snapshots as structured
|
||
events into Wazuh + fleet dashboard (health/version-drift/offline/usage/alerting);
|
||
(2) metrics push (OTLP/remote-write) → VictoriaMetrics + Grafana; (3) mTLS
|
||
desired-state pull (version pin, reboot/drain, kiosk URL) wired to the deployment
|
||
manifest + SPDK flow. Build-time decisions (telemetry encoding, metrics/desired-
|
||
state store, per-host vs cohort, osquery/FleetDM eval) listed in the design note.
|
||
Depends on the step-ca fleet CA (#8) for the mTLS identity.
|
||
|
||
## [ ] 15. Scope the root overlay → per-subtree tmpfs/overlay (un-shadowable `/usr` + STIG mount options)
|
||
|
||
**DESIGN CAPTURED 2026-07-02.** Today the whole root is one `systemd.volatile=overlay`
|
||
tmpfs over `/` (see #5, invariant on the volatile overlay). Replace it with a
|
||
**read-only verity root + a handful of scoped writable mounts**, for two payoffs
|
||
that land from the same change:
|
||
|
||
1. **Security — un-shadowable `/usr`.** dm-verity seals the backing store, but the
|
||
whole-root overlay lets any root process write a tmpfs *copy* over a system
|
||
binary (e.g. `/usr/sbin/fleet-enrol`) that then executes in preference to the
|
||
sealed one until reboot. fapolicyd is the current compensating control (CLAUDE.md:
|
||
"the writable-overlay exec guard dm-verity can't provide"). Keeping `/usr` (and
|
||
`/`, `/opt`, `/boot`) strictly **read-only** makes the executable set both
|
||
verity-sealed *and* un-shadowable — defence in depth, shrinks fapolicyd reliance.
|
||
Also: writes outside the sanctioned set fail loudly (`EROFS`) instead of silently
|
||
landing in RAM and vanishing at reboot — enforces invariant #1 (no per-host state)
|
||
architecturally.
|
||
|
||
2. **STIG — converts the partition/mount family from `notapplicable`/waived to real
|
||
`pass`.** Quantified against the scan in `reports/stig-arf.xml` (NOT estimates):
|
||
the `mount_option_*` rules currently score **`notapplicable`** (no separate mount
|
||
to check — the "false-clean via notapplicable" pattern #2 warns about), and
|
||
`partition_for_*` are **`notselected`** (waived in `diskless-tailoring.xml`
|
||
lines 61-66). Scoped tmpfs mounts with `noexec,nosuid,nodev` + re-selecting the
|
||
partition rules convert **~21 rules to genuine `pass`**:
|
||
- 15 `mount_option_*` N/A→pass: `/tmp` (nodev,nosuid[,noexec*]), `/var/tmp`
|
||
(all 3), `/var/log` (all 3), `/var/log/audit` (all 3), `/var` (nodev), `/home`
|
||
(nodev,nosuid).
|
||
- 6 `partition_for_*` re-selects→pass: tmp, var, var_log, var_log_audit, var_tmp,
|
||
home (+1 easy: `partition_for_dev_shm` — already a separate tmpfs).
|
||
- Retires **deviation-register entry #1** (the narrative hand-wave over the whole
|
||
partition/mount family) — one fewer waiver to defend at ATO.
|
||
- `/dev/shm`'s three option rules **already pass** — no work there.
|
||
|
||
**Writable-set + flag matrix (the design):**
|
||
- `/` `/usr` `/opt` `/boot` → **read-only** (verity as-is, no overlay). `/usr` keeps
|
||
suid for legit setuid bins (sudo/su/passwd).
|
||
- `/etc` → overlay (lower=verity `/etc`, upper=tmpfs) — needs defaults + writability
|
||
(machine-id, regenerated ssh host keys, NM connections, resolv.conf, chrony).
|
||
- `/var` → overlay (NOT bare tmpfs — bare tmpfs hides `/var/lib/rpm`, selinux store);
|
||
`nodev`. Nest the noexec leaf mounts on top (order: `/var` first, then children).
|
||
- `/var/tmp` `/var/log` `/var/log/audit` → tmpfs `noexec,nosuid,nodev`.
|
||
- `/tmp` → tmpfs `nosuid,nodev` (+`noexec` only after the kiosk repoint, below).
|
||
- `/home` `/root` `/srv` → tmpfs `nodev,nosuid`.
|
||
- Each tmpfs needs the **correct SELinux context** (`tmp_t`, `var_log_t`,
|
||
`auditd_log_t`, …) via `context=` or restorecon — this is the login-deny hazard
|
||
class (#5); a mislabeled tree denies logins / stops auditd.
|
||
|
||
**Prerequisites / caveats that gate the count:**
|
||
- `mount_option_tmp_noexec` (1 of the 21) is **conditional**: register entry #1 notes
|
||
the Chromium kiosk uses `/tmp` for profile/cache and breaks under `noexec`. Must
|
||
**repoint the kiosk scratch** (Chromium `--user-data-dir`/`--disk-cache-dir`) to a
|
||
dedicated exec-capable tmpfs (e.g. `/run/kiosk`) first → ties to #13/#4. So ~20
|
||
land immediately, +1 after the repoint.
|
||
- **fstab-vs-runtime OVAL — mostly de-risked:** the rules return `notapplicable`
|
||
(evaluating the live mountpoint) not `fail` (missing fstab line), evidence they read
|
||
the runtime mount table — so systemd mount units / initramfs mounts should score
|
||
without fstab entries. Confirm per-rule on the bench; add fstab lines if any rule is
|
||
fstab-based (compatible with the empty-fstab invariant — root still comes from
|
||
netroot+verity).
|
||
- **Does NOT fix red findings** (this family is ~0 fail today) — the value is
|
||
evidenced compliance + one fewer deviation + kernel-enforced noexec, not clearing
|
||
fails. Framed honestly so it isn't oversold.
|
||
- **`partition_for_usr` stays waived** — `/usr` should be verity-sealed, not a
|
||
separate writable partition (a *stronger* control than the rule asks). Same for
|
||
`encrypt_partitions`, `auditd_..._large_partition`, boot/boot_efi (genuinely N/A).
|
||
|
||
**Mechanism:** `systemd.volatile=` has no per-dir knob (it's all-or-var-only), so this
|
||
means dropping the global volatile overlay for explicit early mount units (ordered
|
||
before `local-fs.target`) or initramfs mounts. Reproducibility unaffected (runtime
|
||
mounts, not image content → no verity-hash drift).
|
||
|
||
**Load-bearing / bench-gated:** touches the boot path (#5 territory) — full boot-chain
|
||
bench (`serving/dot1x-bench/`) + careful SELinux-label verification + a reconciliation
|
||
note in `design-v2-errata.md`. **Do AFTER TPM enrolment (#11) is bench-verified.**
|
||
Tailoring edits: re-select the 6 `partition_for_*` (+dev_shm), delete/gut register
|
||
entry #1, re-run `scripts/stig-deviations.sh` to confirm the new passes.
|
||
|
||
## [ ] 16. Enable a locked-down host firewall (firewalld) — defense-in-depth + STIG passes
|
||
|
||
**DECISION 2026-07-03: enable a *configured* firewalld.** Currently REMOVED + masked
|
||
(`diskless.pkr.hcl` 2d, commit 4346ce7) because an UNCONFIGURED weak-dep firewalld was
|
||
running and flooding the SIEM with `lib_t` `.pyc` AVCs. The design deferred the firewall
|
||
as "future hardening, currently waived" ([stig-deviations.md](stig-deviations.md) #4) —
|
||
NOT incompatible, just unverified during boot-chain bring-up. A diskless kiosk is an
|
||
ideal firewall candidate: near-zero legit inbound (admin SSH only; `fleet-hostapi` is
|
||
loopback), and the load-bearing paths are outbound (NVMe/TCP, DHCP) or L2 (EAPOL/MACsec).
|
||
|
||
**Payoff:** defense-in-depth (limits a compromised same-segment peer) + converts **4
|
||
STIG waivers → passes** (`service_firewalld_enabled`, `package_firewalld_installed`,
|
||
`set_firewalld_default_zone`, `configured_firewalld_default_deny` — re-select in the
|
||
tailoring; retires deviation-register #4).
|
||
|
||
**Must do together:**
|
||
1. **Config:** default-DENY inbound zone; allow **SSH** (key-only admin) +
|
||
established/related; assign `macsec0` (secure) / `en*` (test) to it. hostapi
|
||
(:9099 loopback) + kiosk→hostapi PNA need no rule.
|
||
2. **Fix the `.pyc` AVC root cause (REQUIRED — an enabled firewalld re-hits it):** the
|
||
reproducible erofs `-T 0` zeroes mtimes, so Python thinks every site-package `.pyc`
|
||
is stale and rewrites it → `firewalld_t` (and ANY Python daemon, e.g. `tuned`)
|
||
write/unlink AVCs on `lib_t`. Fix once with `python3 -m compileall --invalidation-mode
|
||
unchecked-hash` over the Python trees BEFORE the erofs build → mtime-independent
|
||
`.pyc`, no rewrites, and MORE reproducible. (Worth doing regardless of firewalld —
|
||
it's the true root cause of the firewalld AVCs seen 2026-07-03.)
|
||
3. **Tailoring:** re-select the 4 firewalld rules.
|
||
|
||
**THE load-bearing bench gate (why #4 deferred it):** the NVMe/TCP root is an OUTBOUND
|
||
connection to `:4420` established in the initramfs BEFORE firewalld starts. Verify it
|
||
**SURVIVES** firewalld start + `firewall-cmd --reload` (established/related allowed by
|
||
default → should, but UNVERIFIED); a reconnect within `ctrl_loss_tmo` re-establishes
|
||
*through* the firewall; DHCP renew + MKA keepalives unaffected; SSH in; no AVCs; verity
|
||
hash reproducible. If it drops the root, that is the fleet-wide outage risk — gate hard.
|
||
|
||
## [~] 18. Slim the image — locale + firmware (biggest erofs-size wins)
|
||
|
||
**Finding (image-size audit via the 2b `rpm -qa --qf '%{SIZE}'` manifest, 2026-07-03):**
|
||
the largest packages in the 534-package image are NOT the kiosk — they're base bloat:
|
||
`linux-firmware` **678 MB**, `glibc-all-langpacks` **228 MB**, then chromium (370) +
|
||
llvm-libs (129). The uncompressed erofs is read over NVMe/TCP on every boot of every
|
||
host, so every dead byte is fleet-wide waste + attack surface.
|
||
|
||
**[x] DONE — glibc langpacks → en-only** (`diskless.pkr.hcl` step 2). `@^minimal-environment`
|
||
pulls `glibc-all-langpacks` (all languages); swapped for `glibc-minimal-langpack` +
|
||
`glibc-langpack-en`. ~220 MB reclaimed; image is English-only (we also strip
|
||
`/usr/share/locale` in 2d). **BENCH-VERIFY:** locale still `en_US.UTF-8`; kiosk + login OK.
|
||
|
||
**[x] DONE — linux-firmware strip to a hardware-audited keep-list** (`diskless.pkr.hcl`
|
||
step 2, run **BEFORE the dracut build** so the UKI initramfs shrinks too — not just the
|
||
rootfs. `--no-hostonly` + plymouth's `drm` dep instmods a broad DRM set and dracut bundles
|
||
each module's firmware into the initramfs, amdgpu being the classic UKI bloater; the strip
|
||
runs first so dracut never sees it. A post-build diagnostic reports the `.img` size + any
|
||
firmware still bundled — expected i915 only.)
|
||
Live fleet audit (`modinfo -F firmware` over all loaded modules + sysfs/`lspci` driver
|
||
bindings on the real 820-class hosts AND the VMs) proved the **only** loadable firmware
|
||
any host requests is Intel **i915** (GPU). Every other driver is firmware-free —
|
||
crucially **e1000e** (on-chip NVM, not a fw file) and **virtio_net**, so NO NIC needs
|
||
firmware and the fleet-brick case is off the table. Keep-list: `i915` + `intel/` (ice
|
||
DDP insurance for the `ice`/`mlx5` server-NIC build variant + small Intel blobs) +
|
||
`mellanox` + `intel-ucode`; strip the rest (amdgpu/nvidia/radeon/ath/qcom/rtl/iwlwifi…
|
||
~600 MB). Wi-Fi (`iwl*-firmware`, ~83 MB) removed as packages (wired-only fleet).
|
||
Strip-in-place (`find … -exec rm -rf`, not `rpm -e` — can't drop the package without
|
||
losing i915); a `du -sh /usr/lib/firmware` line reports the result in the build log.
|
||
**BENCH-VERIFY:** (a) kiosk still RENDERS on the 820 after the strip (i915 DMC/GuC still
|
||
load — the one real risk); (b) if any host has a non-Intel GPU or an `ice`/`mlx5` NIC,
|
||
widen the keep-list before trusting fleet-wide. Audited models so far: HP-820-class
|
||
(i915 + e1000e + iwlwifi-dropped) and virtio VMs.
|
||
|
||
**Combined:** ~**800 MB** off the uncompressed erofs (langpacks + firmware), dwarfing the
|
||
kiosk weak-deps trim (#4). NEXT candidates from the manifest (measure, don't guess):
|
||
`iso-codes` (19 MB), `samba-client-libs`/`intel-mediasdk`/`gstreamer` (chromium-common
|
||
HARD deps — only removable by dropping HTML5 media), `binutils` (25 MB, build-only?),
|
||
`cracklib-dicts` (9.8 MB, needed for pam pwquality — keep).
|
||
|
||
**Stack note:** committed on `trim-rpm`. When `optimise-ram` rebases onto the new
|
||
`trim-rpm`, expect a trivial `docs/TODO.md` tail conflict (both branches appended after
|
||
#16 — `optimise-ram` added #17 fapolicyd, this added #18); resolve by keeping both.
|