Audiobuzz
2ef0c1747d
Complete the deployment story for the server-side control plane: - docker-compose.yml: private InfluxDB (published to loopback only) + a host-networked fleet-manager (reaches the loopback enrol-broker on 127.0.0.1:9443 and binds a routable gRPC agent listener, like the broker). - .env.example: RP origin (diskless-server.bitasylum.org), Influx token/DB password, cert/PKI paths; the Influx token wires influxdb + manager with one value. - server-pki.sh: issue the agent-plane server cert from the fleet CA (step) and stage the fleet root as the client-verify CA. - DEPLOY.md: end-to-end runbook — three-cert PKI (fleet-CA agent cert, broker admin-client cert, Let's Encrypt DNS-01 UI cert), register-admin passkey bootstrap, compose up, hardware-free smoke test, backups. - .dockerignore: keep host node_modules/dist/pki/secrets out of the build context so the embedded SPA is the one built in-image. - .gitignore: never commit .env / issued PKI / data. Design docs marked as-built: fleet-management-design.md as-built note (resolved forks — gRPC bidi transport, InfluxDB metrics, SPA graphs, passkey admin auth, broker fold-in "alongside"); tpm-enrolment-design.md (broker fold-in + agent transport fork resolved); TODO.md #14 → partly built, with the on-host agent + per-host boot-version assignment (#6) called out as the remaining work.
1626 lines
110 KiB
Markdown
1626 lines
110 KiB
Markdown
# TODO — diskless fleet boot system
|
||
|
||
Persistent backlog, kept in sync with the working session task tracker. Status:
|
||
`[ ]` pending · `[~]` in progress · `[x]` done.
|
||
|
||
---
|
||
|
||
## Current build posture (LAB / test) — as of 2026-06-19
|
||
|
||
The image currently builds in **open-network test mode**. These toggles must be
|
||
flipped back for a secure/production build (consolidated as item #8):
|
||
- `disable_dot1x=true` / CI `DISABLE_DOT1X="true"` → `rd.dot1x=0` (NO 802.1X /
|
||
MACsec; open NVMe-oF). **TEMPORARY.**
|
||
- `nvmet_traddr` / CI `NVMET_TRADDR="192.168.50.10"` → hardcoded IP (should be a
|
||
per-env DNS name like `nvme.fleet.lab` once DNS-at-boot is trusted).
|
||
- Unsigned UKI (`sb_key`/`sb_cert` not passed) → Secure Boot must be OFF on the
|
||
test target. Production needs CI/HSM signing.
|
||
- `enable_kiosk=true`; `boot_splash=true`; NTP via DHCP→serving-host chrony.
|
||
- **`enable_stig` / CI `ENABLE_STIG="false"` RIGHT NOW** — temporarily disabled
|
||
to get a working box while the STIG regressions were diagnosed (now fixed,
|
||
below). **Flip back to `true`** and bench-verify the hardened build.
|
||
- **`debug_shell=true` (DEBUG HOLE — passwordless root on tty2). REMOVE before
|
||
prod** (`-var debug_shell=false`). Note: it didn't actually come up on tty2 in
|
||
testing; diagnosis ended up done via offline `rootfs.erofs` mount + SMBIOS
|
||
`enforcing=0`. SMBIOS cmdline-extra works on Proxmox only after a full VM
|
||
**stop/start** (not `reset`).
|
||
- **Admin SSH key MUST be ECDSA or RSA, NOT ed25519**, when STIG is on — the STIG
|
||
crypto policy drops ed25519 from sshd's accepted pubkey algorithms. Set the
|
||
`SSH_AUTHORIZED_KEY` CI secret to an ECDSA/RSA key (see #2).
|
||
- **Hostname:** image is hostname-neutral (empty `/etc/hostname`,
|
||
`hostname-mode=default`); the serving dnsmasq's `fleet-namer` (`--dhcp-script`)
|
||
assigns a stable creative starship name per MAC (e.g. `nebula-vanguard-07`) via
|
||
DHCP option 12 + DNS A/PTR. Host picks it up on next DHCP renewal/reboot.
|
||
- **Serving `.env`:** must set `AUDIT_HOST=audit.fleet.lab` (+ `docker compose up
|
||
-d dnsmasq` to reload) for remote log offload to resolve/flow.
|
||
|
||
What's working/verified vs awaiting bench is per-item below. Cross-cutting
|
||
bench-verify checklist lives in `CLAUDE.md` → "Known bench-verify items".
|
||
|
||
---
|
||
|
||
## [x] 1. Port the dracut module off `network-legacy` → `network-manager`
|
||
|
||
**DONE — test path VERIFIED ON THE BENCH 2026-06-17** (host stays up past the
|
||
DHCP lease time; no more lease-expiry outage). Fixed the confirmed outage:
|
||
network-legacy's one-shot DHCP lease was never renewed post-switch-root, so the
|
||
address dropped at lease expiry → NVMe/TCP root died ("keep alive timeout" /
|
||
"no route to host"). NM hands its initramfs connections to the real-root NM
|
||
(`/run/NetworkManager` state), which renews.
|
||
|
||
REMAINING: the **secure path** (macsec0 DHCP via `nmcli`, physical NIC unmanaged
|
||
via `nm-unmanaged.sh`) is only exercised once `disable_dot1x=false` — verify on
|
||
the swtpm + hostapd + FreeRADIUS bench when re-enabling security (see #2/cleanup).
|
||
|
||
Done:
|
||
- `90-diskless.conf` `add_dracutmodules`: `network-legacy` → `network-manager`.
|
||
- `module-setup.sh` `depends()` → `network-manager`; added `nmcli`; installs the
|
||
new cmdline hook.
|
||
- NEW `nm-unmanaged.sh` (cmdline hook): secure mode writes
|
||
`/run/NetworkManager/conf.d/10-dot1x-unmanaged.conf`
|
||
(`unmanaged-devices=interface-name:en*`) **before NM starts**; no-op in test
|
||
mode. Replaces the deleted static `99-dot1x-unmanaged.conf` (couldn't be
|
||
mode-aware — it wrongly unmanaged `en*` in test mode, blocking renewal).
|
||
- `dot1x-macsec-up.sh` secure path: `ifup macsec0` → `nmcli` (managed + DHCP,
|
||
poll for IPv4). macsec0 MANAGED so NM renews; physical NIC UNMANAGED.
|
||
- `diskless.pkr.hcl`: dropped `dhcp-client`, added explicit `NetworkManager`;
|
||
removed the static-config install; lsinitrd now asserts `NetworkManager` +
|
||
`nm-unmanaged.sh` present.
|
||
- Static lint clean (`bash -n`, `shellcheck`, `packer validate`).
|
||
|
||
**BENCH-VERIFY before trusting (the gate):**
|
||
- Test path (`disable_dot1x=true`): boot, `nmcli device` shows boot NIC MANAGED;
|
||
run past the DHCP lease time (drop the lab dnsmasq lease low) — address
|
||
persists, lease renews (`journalctl -u NetworkManager`), NVMe + SSH stay up.
|
||
- Secure path (swtpm + hostapd + FreeRADIUS, `disable_dot1x=false`): `en*`
|
||
UNMANAGED, `macsec0` MANAGED + renewing; MKA survives switch-root; verity root
|
||
mapped; NVMe stays connected past lease time; no "Address already assigned".
|
||
- **Risk:** ordering of the unmanaged drop-in vs NM start in secure mode. If NM
|
||
transiently grabs `en*`, fall back to also baking a static `en*`-unmanaged file
|
||
for secure-only builds.
|
||
|
||
## [~] 2. Apply RHEL9 STIG hardening to the built image
|
||
|
||
**IMPLEMENTED 2026-06-18 — awaiting build/bench verification.** OpenSCAP + SSG
|
||
remediation in a provisioner (`enable_stig` var, default on), tailored for the
|
||
diskless/UKI/kiosk architecture via `files/stig/diskless-tailoring.xml`. STIG
|
||
kernel params baked into the UKI cmdline (`extract-artifacts.sh`). Decisions:
|
||
**FIPS off**, **audit local-only**, **CI report-only**. Deviations are documented
|
||
in **[stig-deviations.md](stig-deviations.md)** (curated register) and reported
|
||
per-build by **[../scripts/stig-deviations.sh](../scripts/stig-deviations.sh)**
|
||
(non-pass rules tagged intentional vs REVIEW); CI publishes
|
||
`stig-report.html` + `stig-arf.xml` + `stig-deviations.md`. Scanner tooling is
|
||
removed from the shipped image. **BENCH-VERIFY:** boot/login **VERIFIED ON THE
|
||
BENCH 2026-06-19** — hardened build with `enable_stig=true`, host
|
||
`cygnus-cruiser-99`: kiosk auto-starts + renders, admin SSH (ECDSA) logs in,
|
||
audit offload reaches the collector, DHCP creative hostname applied, NVMe/TCP
|
||
verity root mounts and the host stays up. **STILL OPEN:** reproducible verity
|
||
hash across two builds; triage the REVIEW rows (below); the secure path (#8,
|
||
802.1X/MACsec — this was the `disable_dot1x=true` test path).
|
||
|
||
**FINDING 2026-06-18 (FIXED) — false-clean scan.** The first hardened build
|
||
(branch `hardening`, `ada4aa4`) produced a report reading "0 REVIEW / clean",
|
||
but the ARF had **0 pass, 0 fail** — only 462 `notapplicable` + 1068
|
||
`notselected`. Root cause: the provisioner selected the datastream with
|
||
`ls ssg-rhel9-ds.xml ssg-rl9-ds.xml | head -1`, which picks **rhel9**
|
||
alphabetically; the RHEL9 datastream's CPE dictionary has **no Rocky entry**, so
|
||
on Rocky 9.8 every selected rule went `notapplicable` and **nothing was actually
|
||
evaluated or remediated** — the image was unhardened despite a clean-looking
|
||
report. Fixes: (a) provisioner now maps `/etc/os-release $ID` → datastream
|
||
(Rocky→`ssg-rl9-ds.xml`); (b) provisioner **aborts the build** if the scan
|
||
evaluates 0 rules; (c) `stig-deviations.sh` flags an **INVALID SCAN** and exits
|
||
non-zero on 0-evaluated. **Re-run the hardened build** and confirm pass/fail
|
||
counts are non-zero before trusting any deviations report.
|
||
|
||
**FINDING 2026-06-18 (FIXED) — hardening locked the fleet out.** The first
|
||
genuinely-evaluated hardened image (app `215b0eb`) broke **both** login paths:
|
||
the Chromium kiosk dropped to a getty login prompt, and admin SSH was rejected
|
||
with `Permission denied (publickey)`. Root cause: STIG enables `pam_faillock`
|
||
(account phase) + password aging/inactivity, and the **kiosk + fleetadmin
|
||
accounts are passwordless / key-only**, so the hardened PAM *account* phase
|
||
treats them as expired/lockable (sshd key auth succeeds, then `pam_acct_mgmt`
|
||
fails). Fix (provisioner `2c-STIG-postfix`, runs after remediation): exempt the
|
||
two appliance accounts — `chage -M -1 -m 0 -I -1 -E -1` + `faillock --reset`;
|
||
kiosk uses a dedicated `/etc/pam.d/kiosk` (`PAMName=kiosk`, permissive account /
|
||
full session for seat); sshd gets an `account sufficient pam_succeed_if.so user
|
||
= fleetadmin` first-rule bypass. Scoped to those two accounts; everyone else
|
||
keeps full STIG. Justified as register deviation #10. **VERIFIED ON THE BENCH 2026-06-19**
|
||
(`enable_stig=true`, host `cygnus-cruiser-99`): kiosk auto-starts + renders and
|
||
admin SSH (ECDSA key) logs in with hardening on.**
|
||
|
||
**UPDATE 2026-06-19 — offline erofs inspection of the hardened image (mounted
|
||
`rootfs.erofs` read-only) pinned the real causes; the PAM-account theory above
|
||
was incomplete:**
|
||
- **Kiosk had TWO causes, both FIXED:**
|
||
1. **fapolicyd** is `enabled` and `kiosk-session` wasn't in its trust → it
|
||
blocked `/usr/local/bin/kiosk-session`. Fix: KEEP fapolicyd (it's the
|
||
exec-guard for the writable overlay dm-verity can't cover) and add the kiosk
|
||
binary to its trust DB at build (2c-STIG-postfix). Not waived.
|
||
2. **Auto-start:** kiosk was `WantedBy=graphical.target` but the host boots to
|
||
`multi-user.target` (graphical never reached) → unit `inactive (dead)` at
|
||
boot, fine on manual `restart`. Fix: `WantedBy=multi-user.target` + dropped
|
||
`set-default graphical.target` (kiosk launches weston directly, needs no DM
|
||
target).
|
||
- **SSH = STIG crypto policy excludes ed25519 (CONFIRMED via `ssh -vvv`).** The
|
||
key fingerprint MATCHED (not a mismatch); the server's `server-sig-algs`
|
||
advertised only ecdsa + rsa-sha2 — **no `ssh-ed25519`** — so the ed25519 admin
|
||
key is rejected at the signature stage (before PAM/SELinux, which is why
|
||
`enforcing=0` and the account-bypass didn't help). **Fix (your action): set the
|
||
`SSH_AUTHORIZED_KEY` CI secret to an ECDSA (`ssh-keygen -t ecdsa -b 384`) or RSA
|
||
key**, then rebuild. No code change. (Alt: re-allow ed25519 via
|
||
`PubkeyAcceptedAlgorithms +ssh-ed25519` in the sshd drop-in — but that's a
|
||
non-approved algo and would re-fail a STIG rule; prefer the ECDSA key.)
|
||
- **rsyslog TLS-offload rules waived (lab):** `rsyslog_encrypt_offload_*` force
|
||
gtls+x509+CA and broke the plaintext `2f` forward → unselected in the
|
||
tailoring; production TLS offload tracked under #8.
|
||
- Diagnosis tooling notes: SMBIOS cmdline-extra works only after a Proxmox VM
|
||
**stop/start** (not `reset`); the debug-shell-on-tty2 never came up — the
|
||
decisive diagnosis was **offline `rootfs.erofs` mount** + `ssh -vvv` +
|
||
`enforcing=0`.
|
||
|
||
**REVIEW-ROW TRIAGE — DONE 2026-06-19 (scan `app-9f7d8d1`, 21 REVIEW → 1 open).**
|
||
The post-fix hardened scan was valid (398 pass / 12 fail / 9 notchecked). All 21
|
||
REVIEW rows dispositioned (full detail: [stig-deviations.md](stig-deviations.md)
|
||
#12). Mechanics: a rule only stops being flagged REVIEW if it's `selected="false"`
|
||
in `diskless-tailoring.xml` (the register prose alone doesn't clear it) or made to
|
||
pass / N/A.
|
||
- **Bucket A (waived, register-justified):** `encrypt_partitions`,
|
||
`auditd_audispd_configure_sufficiently_large_partition` (#1); `grub2_password`
|
||
(#2); `sysctl_crypto_fips_enabled`, `harden_sshd_macs_openssh_conf_crypto_policy`,
|
||
`harden_sshd_macs_opensshserver_conf_crypto_policy` (#5);
|
||
`configured_firewalld_default_deny` (#4); `sssd_enable_certmap`,
|
||
`sssd_has_trust_anchor` (#7).
|
||
- **Bucket B (waived N/A):** `account_temp_expire_date`,
|
||
`accounts_user_home_paths_only`, `libreswan_approved_tunnels`,
|
||
`security_patches_up_to_date`, `ssh_keys_passphrase_protected`.
|
||
- **Bucket C (decisions applied):** `installed_OS_is_vendor_supported` (Rocky IS
|
||
supported; RHEL-datastream CPE artifact — waived), `network_configure_name_resolution`
|
||
(single DHCP DNS by design — waived), `sysctl_user_max_user_namespaces_no_remediation`
|
||
(KEEP userns for Chromium sandbox — waived), `accounts_authorized_local_users`
|
||
(appliance accounts authorized, #10 — waived; prod: set the authorized-users
|
||
regex var instead), `package_postfix_installed` + `postfix_prevent_unrestricted_relay`
|
||
(**postfix removed** in 2c-STIG — no MTA needed; audit goes off-box via 2f),
|
||
**USBGuard** `configure_usbguard_auditbackend` + `usbguard_allow_hid`/`_hub`/
|
||
`_hid_and_hub` (**ENABLED + enforcing**: LinuxAudit backend + hardware-independent
|
||
HID/hub allowlist written in 2c-STIG; ⚠ BENCH-VERIFY physical console keyboard/
|
||
mouse on real hardware — the build-VM-generated policy + default block target
|
||
would otherwise lock out console input).
|
||
- **`selinux_context_elevation_for_sudo` — REMEDIATED WITH CONFINEMENT
|
||
(decision 2026-06-20).** 2c-STIG now rewrites the admin sudoers to
|
||
`TYPE=sysadm_t ROLE=sysadm_r NOPASSWD:ALL` and maps the admin to **`staff_u`**
|
||
(`semanage login`) so the sudo role transition is authorized. ⚠ **TWO BENCH
|
||
GATES:** (a) admin SSH login + sudo + privileged ops must work under `staff_u`
|
||
confinement (escape hatch: `enable_stig=false` → reverts to a justified
|
||
deviation); (b) **reproducibility** — `semanage` rebuilds the policy store;
|
||
confirm the verity hash is stable across two builds (normalize in 2d if it
|
||
drifts). Needs `policycoreutils-python-utils` (installed in 2c-STIG if absent).
|
||
|
||
**Expected next-build report: 0–1 REVIEW** (0 if both postfix→N/A and
|
||
usbguard→pass land and sudo confinement holds; the triage was validated by
|
||
re-running `scripts/stig-deviations.sh` against the app-9f7d8d1 ARF with the new
|
||
tailoring — 21 REVIEW → 3, where the 3 are exactly postfix/usbguard/sudo, all
|
||
addressed for the next build).
|
||
|
||
Remaining (future hardening, currently waived — see the register): FIPS,
|
||
firewall, per-host/rotatable admin access. **Remote audit offload — WIRED both
|
||
ends** (2026-06-19): receiver `serving/audit-collector/` (rsyslog TCP/514,
|
||
per-source-host files) + sender in the image (provisioner 2f, `audit_host` var:
|
||
audisp-syslog→LOCAL6 + rsyslog `omfwd` to `audit.fleet.lab`). Lab is plaintext;
|
||
TLS/RELP is the remaining hardening. **VERIFIED END-TO-END ON THE BENCH
|
||
2026-06-19** — booted host `cygnus-cruiser-99` (192.168.50.148) forwards to the
|
||
collector, which writes `audit-collector/logs/192.168.50.148/{audit,messages}.log`.
|
||
**Serving-host firewall gotcha hit + fixed:** the collector is host-networked so
|
||
the serving host's **ufw** governs :514 (Docker doesn't manage iptables for
|
||
host-net containers); 514/tcp was bound to the wrong NIC (`ens18` instead of the
|
||
boot-LAN `ens19`), so SYNs were silently dropped (`SYN-SENT` / "connection timed
|
||
out", not refused). Fix: `ufw allow in on ens19 to any port 514 proto tcp`. The
|
||
sender's `action.resumeRetryCount="-1"` buffered the backlog and drained on
|
||
reconnect — no records lost. Documented in
|
||
[../serving/audit-collector/README.md](../serving/audit-collector/README.md).
|
||
When the **workstation** firewall is re-enabled (#4 / register #4) it needs the
|
||
reciprocal **egress** allow for 514/tcp to the collector.
|
||
|
||
**SIEM (Wazuh) added 2026-06-27 — [serving/siem/](../serving/siem/).** Upgrades the
|
||
"Detect" pillar from file-collection-only to parse/search/correlate/alert.
|
||
**Runs on a DEDICATED node — `wazuh1.bitasylum.org`** (managed independently),
|
||
keeping the memory-hungry indexer JVM **off** the serving host so it can't OOM SPDK
|
||
(the fleet's root disk / SPOF, #6/#7). **Agentless** (diskless hosts stay stateless,
|
||
invariant #1): the serving-host audit-collector **relays a copy** to wazuh1's
|
||
manager syslog listener (`omfwd` → `wazuh1.bitasylum.org:514` in
|
||
`audit-collector/rsyslog.conf`; rebuild the collector to apply) while still writing
|
||
durable per-host files. `serving/siem/` is the INTEGRATION (the `<remote>` syslog
|
||
block to add on wazuh1 + reference deploy overlay), not a local deployment.
|
||
**DECODING BENCH-VERIFIED 2026-06-27** (`wazuh-logtest`): the audisp-syslog records
|
||
carry a `node=<host>` prefix (STIG `name_format=hostname`, SSG `auditd_name_format`
|
||
— kept, it's compliance) that breaks Wazuh's built-in auditd decoder, so a custom
|
||
**`fleet-auditd`** decoder + rules (`serving/siem/config/local_decoder.xml` +
|
||
`local_rules.xml`, mounted declaratively via `/wazuh-config-mount`) decode them
|
||
into the **`fleet_audit`** group. **Decode + alert confirmed:** `wazuh-logtest`
|
||
shows `decoder.name=fleet-auditd` + `audit.type` extracted (base rule `100100`);
|
||
and a live `sudo` on a booted host (`cygnus-cruiser-99`) surfaced `USER_CMD` →
|
||
rule `100130` and `CRED_*`/`USER_ACCT` → rule `100110` (level 5) in the dashboard
|
||
Threat Hunting view. Wazuh runs multi-node on `wazuh1.bitasylum.org`.
|
||
**REMAINING (prod):** the relay→wazuh1 hop is **plaintext** — **TLS it** (gtls/6514,
|
||
fleet CA; ISM protect-logs-in-transit) since it's now on the wire; rotate all Wazuh
|
||
creds; restrict dashboard/API to admin hosts; set `allowed-ips`→/32 of the serving
|
||
host; ISM retention/archival; then update the SSP "Detect" + ISM event-logging
|
||
controls.
|
||
|
||
### Original spec (retained for reference)
|
||
|
||
OpenSCAP + SCAP Security Guide (`xccdf_org.ssgproject.content_profile_stig`), run
|
||
as a Packer provisioner after `dnf install` and before the dracut/initramfs build
|
||
and cleanup.
|
||
|
||
- **Tailoring** (`autotailor`): unselect rules this architecture can't satisfy —
|
||
separate-partition rules (single erofs root + tmpfs overlay; satisfy intent
|
||
with `nodev,nosuid,noexec` tmpfs mounts), GRUB/kernel-cmdline rules (no GRUB),
|
||
and AIDE/file-integrity rules (dm-verity is the documented compensating
|
||
control).
|
||
- **Kernel params → UKI cmdline** in
|
||
[../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh) before
|
||
signing: `audit=1 audit_backlog_limit=8192 slub_debug=P page_poison=1
|
||
vsyscall=none` (+ `fips=1` if FIPS).
|
||
- **Reproducibility guard:** strip non-deterministic remediation output (SSH host
|
||
keys → regen at boot into overlay, AIDE db, timestamps, fresh UUIDs) so the
|
||
verity hash stays stable.
|
||
- **Audit logging:** `audisp-remote`/rsyslog to a central collector addressed by
|
||
DNS name (host-generic).
|
||
- **Optional FIPS:** `fips=1` cmdline + `01fips` dracut module + FIPS crypto
|
||
policy.
|
||
- **CI:** run `oscap eval` (no `--remediate`) to publish an ARF+HTML STIG report
|
||
as a build artifact; optionally hard-fail below a score threshold.
|
||
- **Decisions needed before implementing:** FIPS on/off · remote-audit collector
|
||
DNS name (or ephemeral/none for lab) · CI fail-on-score threshold vs
|
||
report-only.
|
||
- Note: orthogonal to the `rd.dot1x=0` test toggle, but a full STIG production
|
||
posture wants 802.1X re-enabled.
|
||
- **Credentials (partly done):** root is now **locked** and access is via a
|
||
baked sudo-enabled key-only admin user (`var.ssh_authorized_key` /
|
||
`var.admin_user`, `wheel` + NOPASSWD sudo), with sshd drop-in
|
||
`PermitRootLogin no` / `PasswordAuthentication no`. Two items for STIG to
|
||
reconcile: **NOPASSWD sudo** is a finding (acceptable for a key-only account,
|
||
but document/deviate), and the admin key/user are **fleet-wide baked** — for
|
||
per-host/rotatable access without rebuilds, layer the signed HTTP/SMBIOS
|
||
config-agent or an SSH CA (`TrustedUserCAKeys`) discussed in session.
|
||
- **SSH host keys (done):** `extract-artifacts.sh` strips `/etc/ssh/ssh_host_*`
|
||
so they're not baked (baked keys = shared fleet identity + the build's
|
||
perms/group don't survive the tar→erofs round-trip, so sshd rejected them
|
||
"Permissions 640 too open" and refused to start). RHEL's `sshd-keygen@.service`
|
||
regenerates them into the overlay before sshd starts. **⚠ Caveat:** the
|
||
ephemeral overlay means they regenerate **every boot** → the host key changes
|
||
per reboot (clients warn). Follow-up for a stable host identity: persist the
|
||
host keys or derive them from the TPM (ties to the hostnqn/identity model).
|
||
|
||
## [ ] 3. Roll the errata into design doc v3
|
||
|
||
Once the boot path is validated and item 1 lands, fold
|
||
[design-v2-errata.md](design-v2-errata.md) into the source design document and
|
||
re-export a **v3** PDF (key change: `rootovl` → `systemd.volatile=overlay` in §3
|
||
step 8 and §5). Retire/reset the errata afterward. Needs the design-doc source
|
||
(not in this repo).
|
||
|
||
## [x] 4. Kiosk mode — autologin + fullscreen Chromium loading a URL
|
||
|
||
**WORKING (display-verified 2026-06-17).** Remaining bits are polish (GPU/mesa
|
||
trim, ship via CI `enable_kiosk=true`) and the phone-home/NTP lockdown (done) —
|
||
all captured in the sub-notes below. Secure-path render check is folded into #8.
|
||
|
||
**Package-footprint trim (2026-07-03):** the kiosk dnf transaction was measured at
|
||
**229 pkgs / 1.0 GB** (weak deps ON). Added `--setopt=install_weak_deps=False` to
|
||
the install in `diskless.pkr.hcl` — drops the flatpak/ostree sandbox runtime,
|
||
tracker-miners (fs indexer = dead weight + threat surface), the pipewire/pulseaudio
|
||
audio stack, and `libproxy-webkitgtk4 → webkit2gtk3-jsc` (an 8.9M *second* browser
|
||
engine). Also added an `rpm -qa --qf %%{SIZE}` size-manifest diagnostic to the build
|
||
log to quantify the result. The immovable floor is chromium + chromium-common +
|
||
llvm-libs (mesa sw-GL) + gtk3/adwaita + libicu — all HARD deps. **BENCH-VERIFY:**
|
||
with weak deps off, confirm the kiosk still renders (no genuine runtime dep was a
|
||
weak dep — watch for a missing font or `libva`). NEXT trim candidates (measure via
|
||
the new manifest): gstreamer/media codecs if the SPA plays no HTML5 media;
|
||
mesa-dri-drivers + `--use-pixman`-only if the GPU path is abandoned.
|
||
|
||
|
||
**WORKING / display-verified 2026-06-17** (kiosk renders Chromium fullscreen on
|
||
the configured URL). Build-flagged OFF by default for now (`enable_kiosk=false`)
|
||
while iterating on boot/network — set `-var enable_kiosk=true` to ship it.
|
||
|
||
Path to getting here (don't re-discover):
|
||
- **cage is NOT packaged for EL9** (`dnf install cage` → "Unable to find a
|
||
match"). Tried weston instead.
|
||
- **EPEL 9 ships weston 8.0**, which has NEITHER `kiosk-shell.so` NOR the
|
||
`[autolaunch]` weston.ini section — both arrived in **weston 9.0**. The first
|
||
cut wrongly assumed 8.0 had them; `shell=kiosk-shell.so` failed to load.
|
||
`rpm -ql weston` shows only desktop-shell / fullscreen-shell / ivi-shell.
|
||
- **Resolution: `desktop-shell.so`** (the only packaged shell Chromium's
|
||
xdg-shell works with — fullscreen-shell needs a client protocol Chromium
|
||
doesn't speak), stripped bare (`panel-position=none`, black bg, no
|
||
animation/lock), and Chromium launched by a **wrapper** (kiosk-session) since
|
||
there's no `[autolaunch]`. Chromium `--kiosk` fullscreens over the shell.
|
||
- **Seat fix (load-bearing):** weston died with `logind: failed to get session
|
||
seat` → `failed to create compositor backend`. logind's auto-derivation
|
||
didn't attach a seat; `Environment=XDG_SEAT=seat0` + `XDG_VTNR=1` in the unit
|
||
(put into the PAM env before pam_open_session) registers the session on
|
||
seat0/vt1. `/dev/dri/card0` was present all along (not the issue).
|
||
- Tested live without a rebuild: the whole root is `systemd.volatile=overlay`,
|
||
so /etc AND /usr are writable tmpfs at runtime — drop-in the files + restart.
|
||
- **`enable_kiosk` Packer var — currently `false` by DEFAULT (set 2026-06-17).**
|
||
The whole kiosk provisioner is feature-flagged off while iterating on the
|
||
boot/network stack (DHCP-lease/keep-alive work), for faster builds and a
|
||
smaller image. **Re-enable with `-var enable_kiosk=true` when returning to
|
||
kiosk work** — CI does not pass the var, so CI currently builds lean (no
|
||
kiosk) too; add `-var enable_kiosk=true` to the Gitea workflow to ship it.
|
||
- `kiosk_url` Packer var (default `http://boot.fleet.lab/kiosk/` — the kiosk
|
||
**SPA** (`serving/kiosk-app/`, Svelte 5 + Tailwind v4, a cyber sci-fi
|
||
spaceship monitoring/control terminal) served by the boot HTTP server, which
|
||
**reverse-proxies** `/kiosk/` to the `kiosk` container; self-contained so a
|
||
boot validates without external connectivity. Empty disables the whole
|
||
install). Wired in CI as `KIOSK_URL` env.
|
||
- **Kiosk SPA (`serving/kiosk-app/`)** — replaced the old single static
|
||
`serving/http/kiosk/index.html` (2026-06-18). Top navbar + 5 pages
|
||
(Bridge/Reactor/Nav/Life Support/Comms) with dummy telemetry; hash routing;
|
||
builds in a 2-stage Docker image (Vite → nginx), reverse-proxied at `/kiosk/`
|
||
by the main serving nginx (loopback `127.0.0.1:${KIOSK_APP_PORT:-8088}`).
|
||
`npm run build` verified locally; container build untested (no docker here).
|
||
Dummy data lives in `src/lib/data.js` (swap for a fleet API later).
|
||
- Provisioner enables EPEL, installs `weston chromium mesa-dri-drivers
|
||
mesa-libEGL mesa-libgbm dejavu-sans-fonts`, creates a passwordless
|
||
unprivileged `kiosk` user (video/input groups).
|
||
- `files/kiosk/kiosk.service` — runs `/usr/local/bin/kiosk-session` as `kiosk`
|
||
on tty1 via `PAMName=login`+`TTYPath`+`Environment=XDG_SEAT=seat0`/`XDG_VTNR=1`
|
||
(gets seat0 for DRM/input), `Conflicts=getty@tty1`, `WantedBy=graphical.target`;
|
||
default target set to graphical.
|
||
- `files/kiosk/weston.ini` — `shell=desktop-shell.so`, `idle-time=0`,
|
||
`require-input=false`, `xwayland=false`; `[shell] panel-position=none`,
|
||
black `background-color`, `animation=none`, `locking=false`.
|
||
- `files/kiosk/kiosk-session` — `exec`s `weston --use-pixman --socket=wayland-kiosk`
|
||
in the foreground (owns the seat), and launches Chromium (`--kiosk
|
||
--ozone-platform=wayland --disable-gpu`, tmpfs profile/cache, URL from
|
||
`/etc/kiosk/kiosk.env`) in a background crash-restart loop.
|
||
- **Remaining polish (now that it renders):** dropping `--disable-gpu` on real
|
||
GPUs, `/dev/dri` present in the VM (weston needs a DRM device — set Display to
|
||
VirtIO-GPU/std), and **SELinux AVCs** for the GUI stack (check `ausearch -m
|
||
avc`; the kiosk user is unconfined so likely none — but do NOT set permissive
|
||
if they appear; add a policy module). Admin SSH (the key user) still works for
|
||
debug even if the kiosk fails (it lands at graphical.target either way).
|
||
- **Phone-home lockdown (done 2026-06-17/18):** Chromium was dialing Google
|
||
(accounts/variations/optimization-guide/component+auto update/safe browsing —
|
||
seen in dnsmasq logs). Two layers:
|
||
- **Flags** in `kiosk-session` (`--disable-background-networking`,
|
||
`--disable-component-update`, `--disable-features=OptimizationHints,...`) —
|
||
confirmed killing variations/optimization-guide/update queries.
|
||
- **Managed policy** `files/kiosk/chromium-kiosk-policy.json` for the
|
||
policy-governed bits (sign-in `accounts.google.com`, sync). FIX 2026-06-18:
|
||
the managed-policy dir differs by build, so it's installed to BOTH
|
||
`/etc/chromium/policies/managed/` AND `/etc/chromium-browser/policies/managed/`
|
||
(a policy in the wrong dir is silently ignored — the reason
|
||
`accounts.google.com`/`android.clients.google.com` persisted). Verify via
|
||
`chrome://policy` (Status: OK per entry).
|
||
- **Serving-side sinkhole** (defense-in-depth, `serving/docker-compose.yml`
|
||
dnsmasq `--address=/google.com|googleapis.com|gstatic.com/0.0.0.0`): the
|
||
kiosk is local-only, so block egress at the resolver. Stops the *leak*; the
|
||
query still logs (answered 0.0.0.0, not forwarded) — eliminating the query
|
||
is the browser policy's job. Production: do this at the boot-LAN firewall.
|
||
- **NTP egress closed (done 2026-06-18):** the `*.pool.ntp.org` lookups were
|
||
the OS (chrony), not Chromium. Now: a `chrony` NTP server in the serving
|
||
stack (`serving/chrony/`), dnsmasq hands it via **DHCP option 42**, and the
|
||
image's NM dispatcher (`files/etc/NetworkManager/dispatcher.d/10-chrony-dhcp-ntp`)
|
||
feeds it into chrony's `sourcedir /run/chrony-dhcp`; the baked public pool is
|
||
stripped from `/etc/chrony.conf`. Result: the workstation syncs time only
|
||
from the lab server — no external NTP egress. BENCH-VERIFY: on a booted host
|
||
`chronyc sources` shows the lab server (the DHCP one), not a public pool.
|
||
- **Image size:** Chromium+Mesa add hundreds of MB to the *uncompressed* erofs —
|
||
feeds the "shrink the rootfs" concern (and #7); reconsider whether a lighter
|
||
browser/render path is viable.
|
||
|
||
### Original task
|
||
|
||
|
||
|
||
Make the diskless workstation boot straight into a locked-down kiosk: a
|
||
fullscreen Chromium showing a configured site (default: the boot HTTP server's
|
||
`/kiosk/` test page). All
|
||
baked at **build time** into the root tree (immutable erofs/verity root; runtime
|
||
is the tmpfs overlay), fleet-generic (no per-host state).
|
||
|
||
- **Graphical stack (keep it minimal):** add a Packer provisioner installing a
|
||
lightweight kiosk compositor + browser. Preferred: `cage` (Wayland
|
||
single-app kiosk compositor) + Mesa (software GL fallback for VMs/no-GPU);
|
||
X11 + a minimal WM (matchbox/openbox) as the alternative. Chromium on Rocky 9
|
||
is in **EPEL** — enabling EPEL adds a repo/supply-chain surface (decision
|
||
below).
|
||
- **Autologin + session:** dedicated unprivileged `kiosk` user; launch the
|
||
session via a `kiosk.service` running `cage -s -- chromium --kiosk …`
|
||
(or `greetd` + cage). Stage units/configs under `files/` like the other
|
||
provisioned config.
|
||
- **Chromium flags:** `--kiosk --noerrdialogs --disable-infobars --incognito
|
||
--no-first-run --disable-session-crashed-bubble <URL>`. Writable profile/cache
|
||
land on the `systemd.volatile=overlay` tmpfs (RAM-backed, ephemeral) — fine,
|
||
but watch RAM sizing; pin `--user-data-dir`/`--disk-cache-dir` to a tmpfs path
|
||
if needed.
|
||
- **Kiosk URL = Packer var** (e.g. `kiosk_url`, default the target site), baked
|
||
into the launch command; keep it host-generic (per-env via DNS if ever needed).
|
||
- **Verify in boot test:** graphical session comes up on real/VM display,
|
||
Chromium fullscreen loads the URL over the booted network (works with the
|
||
current `rd.dot1x=0` test path / open network).
|
||
- **Decisions:** Wayland(cage) vs X11; accept EPEL for Chromium (vs another
|
||
source); confirm the target URL.
|
||
- **⚠ Conflict with item 2 (STIG):** passwordless autologin and a desktop
|
||
browser are STIG findings and enlarge the attack surface/image. Reconcile —
|
||
e.g. a tightly-constrained kiosk session + documented deviation — when
|
||
hardening lands.
|
||
|
||
## [x] 5. Deploy & validate the end-to-end boot path (open-network test)
|
||
|
||
✅ **DONE (2026-06-15).** Full chain validated on QEMU/Proxmox: UEFI HTTP boot →
|
||
initramfs → DHCP → `nvmf-connect.service` auto-connects the NVMe/TCP subsystem →
|
||
two namespaces (data + verity hash) appear by pinned UUID → `systemd-veritysetup`
|
||
maps `/dev/mapper/root` → uncompressed erofs mounts → `systemd.volatile=overlay`
|
||
→ multi-user. Booted with `rd.dot1x=0` (open network) and the IP discovery
|
||
address. See bench findings below for the full list of fixes. Remaining cleanup
|
||
to leave the lab-test posture (now unblocked): revert the hardcoded IP to the
|
||
`nvme.fleet.lab` DNS name, and re-enable security (`DISABLE_DOT1X=false`) — the
|
||
secure path needs the same connect trigger over macsec0 (see ⚠ note below).
|
||
|
||
Original task: stand up `serving/` (dnsmasq + nginx + SPDK) per
|
||
[serving/README.md](../serving/README.md), rebuild the UKI with
|
||
`nvmet_traddr=nvme.fleet.lab` (now a Packer var — older UKIs have the literal
|
||
`NVMET_TRADDR` placeholder and won't connect), then boot a UEFI HTTP-boot client
|
||
(**Secure Boot OFF**, unsigned UKI; no TPM needed in `rd.dot1x=0` mode) on the
|
||
boot LAN.
|
||
|
||
- **Pass criteria:** UKI fetched over HTTP → initramfs → DHCP on NIC → NVMe/TCP
|
||
namespace connected → dm-verity root mapped → `systemd.volatile=overlay`
|
||
writable → multi-user.
|
||
- This validation **gates** item #1 (network-manager port) and item #3
|
||
(errata → design v3).
|
||
- **Bench findings (boot debug, 2026-06-14):**
|
||
- Initramfs has no resolver (dracut writes `resolv.conf` only via its own
|
||
DHCP path, not a manual `dhclient`), so the NVMe discovery address must be
|
||
an **IP**, not a DNS name — build with `-var nvmet_traddr=192.168.50.10`.
|
||
Old UKIs carrying `nvme.fleet.lab` can't connect early in boot.
|
||
- NVMe controller indices are **not stable**: a transient discovery
|
||
controller can take `nvme0`, pushing the I/O controller to `nvme1`. The
|
||
hardcoded `/dev/nvme0n1` in the cmdline was a latent brick. **Fixed:**
|
||
namespace UUID pinned on the target (`spdk-setup.sh` `add_ns -u
|
||
f1ee7000-0000-4000-8000-000000000001`) and the cmdline now references
|
||
`/dev/disk/by-id/nvme-uuid.<uuid>` (`extract-artifacts.sh`). The two
|
||
constants MUST stay in sync; make them version-specific for #6.
|
||
- `nvme_tcp`/`nvme_fabrics` must be loaded before `/dev/nvme-fabrics` exists
|
||
(dracut's `nvmf` module does this in a normal boot; at `rd.break` modprobe
|
||
manually).
|
||
- **Don't double-configure the NIC in test mode.** With `ip=dhcp` on the
|
||
cmdline, dracut's network module brings the NIC up AND emits the "online"
|
||
event that the `nvmf` module's `initqueue/online` hook waits on to run
|
||
`nvme connect-all`. The `rd.dot1x=0` bypass used to *also* `ifup $NIC`,
|
||
which (a) raced dracut → `Error: ipv4: Address already assigned`, and (b)
|
||
didn't fire the nvmf online hook, so the namespace never auto-connected and
|
||
boot spun on `/dev/mapper/root` forever (manual `connect-all` worked, which
|
||
is what masked it). **Fixed:** the bypass branch no longer touches the NIC;
|
||
`ip=dhcp` owns networking. (The secure path still addresses macsec0 itself —
|
||
dracut doesn't know about that interface.)
|
||
- **dracut's `nvmf` module never auto-connects on this distro.** Its cmdline
|
||
hook (`92-parse-nvmf-boot-connections.sh`) parses `rd.nvmf.discover` into
|
||
`/etc/nvme/discovery.conf` correctly, but `initqueue/online/` is **empty** —
|
||
no autoconnect hook is registered, so `nvme connect-all` is never run (no
|
||
attempt in dmesg) and the namespace never appears. Drivers load and
|
||
`/dev/nvme-fabrics` exists; only the connect trigger is missing. A job
|
||
placed on the *main* initqueue from a settled hook gets registered but is
|
||
**never re-executed** once the loop blocks on udev events (verified: job
|
||
present, manual `connect-all` works, namespace stays absent). An
|
||
`initqueue/online` hook also does NOT work: this initramfs is **systemd-
|
||
driven** ("A start job is running for /dev/mapper/root") and never sources
|
||
dracut online hooks on link-up (verified: hook installs as
|
||
`90-nvmf-connect.sh` but never fires — no log, no stamp). **Fixed:** the
|
||
connect is now a **systemd unit** (`nvmf-connect.service`,
|
||
`ConditionKernelCommandLine=rd.dot1x=0`, `Before=initrd-root-device.target`)
|
||
whose ExecStart (`/usr/sbin/nvmf-connect`) polls for the network then runs
|
||
`nvme connect-all` until the namespace appears. The dracut module installs
|
||
the unit + script and enables it via an `initrd-root-device.target.wants`
|
||
symlink; `diskless.pkr.hcl` hard-asserts the unit is in the initramfs.
|
||
Gotcha: `$systemdsystemunitdir` is **empty** in this dracut's module-setup
|
||
context — `inst_simple` then silently drops the unit (empty dest dir). Use a
|
||
`${systemdsystemunitdir:-/usr/lib/systemd/system}` fallback and `return 1`
|
||
if the unit didn't land, so the build fails loudly instead of shipping a UKI
|
||
that can't connect its root.
|
||
Second gotcha: the build assertion `lsinitrd … | grep -q 'nvmf-connect.service'`
|
||
was a **false negative** — `grep -q` exits on first match, closes the pipe,
|
||
`lsinitrd` dies of SIGPIPE, and under `pipefail` the pipeline reports failure
|
||
even though the unit IS present. This masked the fact that the systemd unit
|
||
install was working all along. Fix: `lsinitrd > file; grep -q … file` (no
|
||
pipe). Never `lsinitrd | grep -q` under pipefail.
|
||
Bonus: the survivor listing confirmed nvme-cli's native autoconnect bits ARE
|
||
in the initramfs (`/usr/sbin/nvmf-autoconnect.sh`,
|
||
`/etc/udev/rules.d/95-nvmf-initqueue.rules`) — so the fully-native
|
||
`rd.nvmf.discover` autoconnect path is available as a fallback if the unit
|
||
doesn't fire on boot.
|
||
- **dm-verity: `hash-offset` is unsupported → two namespaces.** With NVMe
|
||
connecting and the right by-id device present, verity still failed:
|
||
`systemd-veritysetup` logged `unknown option 'hash-offset=…'` then `Failed
|
||
to load verity superblock` — exactly the CLAUDE.md bench-verify item. The
|
||
in-file (data+hash in one namespace, split by offset) layout is not
|
||
supported by Rocky 9.8 systemd. **Fixed** via the documented fallback:
|
||
`extract-artifacts.sh` now emits `rootfs.erofs` (data) + `rootfs.hash` (hash
|
||
tree) as separate files, served as **two NVMe namespaces** with distinct
|
||
pinned UUIDs (`f1ee7000-…-01` data, `…-02` hash); the cmdline uses
|
||
`systemd.verity_root_data=`/`systemd.verity_root_hash=` by-id devices with no
|
||
offset. Touches `extract-artifacts.sh`, `build-oci.sh` (artifact list +
|
||
SHA256SUMS), `serving/spdk-setup.sh` (2nd bdev/namespace), and invariant #9.
|
||
- **SELinux labels must survive into the erofs.** It booted to multi-user but
|
||
`root`/`packer-build-only` login was accepted then the session was killed
|
||
instantly. Cause: kickstart sets `selinux --enforcing`, but the build dropped
|
||
`security.selinux` xattrs (`virt-tar-out` + plain `tar -xpf` don't carry
|
||
them), so the enforcing target denied the login session on an unlabeled root
|
||
— and a read-only erofs can't autorelabel. Confirmed by booting with
|
||
`enforcing=0` (SMBIOS-injected) → login worked. **Fixed:** export via
|
||
`guestfish tar-out … selinux:true xattrs:true acls:true`, restore with
|
||
`tar --xattrs --xattrs-include='*' --acls --selinux`, and a `getfattr`
|
||
build-time assertion that `/usr/bin/bash` is labeled before `mkfs.erofs`
|
||
(mkfs.erofs preserves xattrs). Needs `attr` on the build host. The image now
|
||
boots **enforcing**.
|
||
- **`mkfs.erofs --all-root` was flattening all ownership to root:root.** Mode
|
||
bits + (now) xattrs survive the export, but `--all-root` forced every inode
|
||
to uid/gid 0 — discarding group ownership (`ssh_keys` on host keys →
|
||
sshd "permissions 640 too open" → refused; `tty`/`crontab` setgid groups)
|
||
and user ownership (home dirs). **Fixed:** dropped `--all-root` so real
|
||
ownership is preserved from the tree. That now requires `extract-artifacts.sh`
|
||
to run as **root** (added an `id -u` guard; `tar -x` needs CAP_CHOWN to
|
||
restore ownership) — the post-processor runs it via `sudo` and `chown`s the
|
||
output files back to the build user so `build-oci.sh` (rootless buildah +
|
||
its own login) still works. Combined with the host-key strip above, sshd
|
||
starts cleanly.
|
||
**⚠ The secure path has the same dependency on the
|
||
(absent) nvmf autoconnect** — when 802.1X is re-enabled, the macsec0 path
|
||
must likewise drive `connect-all` (or the network-manager port #1 must wire
|
||
a working autoconnect). Currently only the test path is fixed.
|
||
- The target Rocky 9 kernel's erofs is built **without
|
||
`CONFIG_EROFS_FS_ZIP`** — mounting a compressed image fails `erofs: lz4
|
||
algorithm isn't enabled`, which silently blocks the root mount. **Fixed:**
|
||
`extract-artifacts.sh` now builds the erofs **uncompressed** (`mkfs.erofs`
|
||
without `-z`). Tradeoff: larger rootfs namespace + more NVMe/TCP bandwidth
|
||
per read. Re-enable compression only against a kernel verified to have
|
||
`EROFS_FS_ZIP` (`grep CONFIG_EROFS /boot/config-$(uname -r)`). This is the
|
||
CLAUDE.md bench-verify item ("erofs module availability on the target RHEL
|
||
kernel"). Larger image size also feeds the "shrink the rootfs" angle of #7.
|
||
- SPDK: add the listener to the discovery subsystem explicitly
|
||
(`nqn.2014-08.org.nvmexpress.discovery`) to silence the deprecation warning.
|
||
|
||
## [ ] 6. Rolling fleet updates (multi-version serve → cohort reboots)
|
||
|
||
Image-based, reboot-to-update across many live workstations. Hosts are immutable
|
||
(read-only erofs/verity root over NVMe/TCP, ephemeral overlay), so updates are
|
||
**atomic A/B reboots**, never in-place. The released OCI artifact + digest-pinned
|
||
manifest + roothash coupling already give the atomic unit and instant rollback;
|
||
this item builds the rollout machinery around it.
|
||
|
||
- **Serve versions side-by-side** (must not overwrite a live rootfs): extend
|
||
`serving/` so SPDK/nvmet exposes each release as its **own subsystem/NQN**
|
||
(`nqn…:rootfs-<ver>`) and nginx serves multiple `fleet-<ver>.efi` (keep
|
||
current + previous). Make the UKI's NQN version-specific so each signed UKI
|
||
connects only its matching namespace (constant discovery DNS name).
|
||
- **Version selection at boot**: dnsmasq **cohort tagging** (MAC / vendor class /
|
||
assigned tag) hands each cohort a different option-67 UKI URL. Moving a
|
||
cohort's URL = assigning the release.
|
||
- **Control plane + in-image reboot agent**: a baked-in systemd timer polls a
|
||
"desired version" endpoint and reboots **when idle / in a maintenance window**
|
||
(or out-of-band via Redfish/BMC). Stagger + jitter reboots to avoid UKI-fetch
|
||
/ NVMe-connect storms.
|
||
- **Staged rollout**: canary → waves (10/50/100%) with health gates
|
||
(multi-user reached, NVMe connected, dm-verity mapped, kiosk URL rendered,
|
||
prod: MACsec/802.1X up); auto-halt on threshold.
|
||
- **Rollback**: repoint cohort URL to previous UKI + reboot (current+previous
|
||
stay served); security rollback via dbx-revoke / TPM monotonic counter
|
||
(design §6).
|
||
- **Drain & retire** an old version only when SPDK/nvmet reports zero connected
|
||
controllers on it. **Drain accuracy:** account for hosts powered off mid-rollout
|
||
— an assigned-but-offline host will reconnect to a version that may be gone, so
|
||
don't remove a version while any host that could return is still assigned to it.
|
||
- **Availability**: run NVMe/TCP target replicas behind distinct discovery
|
||
entries (design §7) — never serve the whole fleet from one unreplicated node,
|
||
especially mid-rollout (N versions + reboot bursts).
|
||
- Note: kiosk *content* changes are server-side (the website) and need no fleet
|
||
update — only OS/UKI/rootfs changes require a reboot.
|
||
|
||
### SPDK restart / availability is the load-bearing constraint (analysis 2026-06-25)
|
||
|
||
SPDK sits in the **live data path of every running host's root fs**, so a process
|
||
restart isn't like restarting a stateless service — it tears down **every**
|
||
NVMe/TCP association at once (fleet-wide, NOT version-scoped). The rollout design
|
||
must be built around **never restarting the process that holds live connections.**
|
||
|
||
- **Core principle:** a host changes root version *only by rebooting* (no live
|
||
swap). So: **version transitions = reboots; SPDK changes = additive/online; the
|
||
process stays up.** Routine rollouts never restart SPDK.
|
||
- **Online add/remove via JSON-RPC, no restart.** `nvmf_tgt` is fully
|
||
runtime-configurable (`bdev_aio_create`, `nvmf_create_subsystem`,
|
||
`nvmf_subsystem_add_ns`, `..._add_listener`). Add vY's namespaces live (existing
|
||
vX connections untouched); remove vX live once drained. **Per-version subsystem
|
||
(distinct NQN)** matters here: adding vY as a *new subsystem* never disturbs the
|
||
subsystem vX hosts are attached to — adding a *namespace* to a shared subsystem
|
||
fires a namespace-change AER + rescan at running hosts. Per-version subsystem
|
||
also gives clean per-release drain + per-release hostNQN allow-listing.
|
||
- **What a connection loss does to a running host:** root is **read-only erofs +
|
||
tmpfs overlay**. tmpfs writes (RAM) + page-cached reads keep working; **cold
|
||
reads block**. The nvme-tcp host driver reconnects (`reconnect_delay`) until
|
||
`ctrl_loss_tmo`. **Fast SPDK restart** (same NQN/UUID/addr within the window) →
|
||
host **survives with a stall**; **long/changed/expired** → root I/O errors, host
|
||
needs a reboot ("keep alive timeout / no route to host"). Read-only +
|
||
content-addressed makes reconnect unusually clean (no write/reservation state).
|
||
- **Cheap host-side mitigation:** tune `ctrl_loss_tmo` (even `-1` = retry forever)
|
||
+ `reconnect_delay` at `nvme connect` time (dracut nvmf step). For a diskless box
|
||
"hang and recover" beats "die" — there's nothing better to fail to.
|
||
- **Structural answer — HA + NVMe-oF multipath (ANA):** run ≥2 targets serving
|
||
**byte-identical, same-UUID** namespaces; hosts multipath across them, so one
|
||
target can be restarted/upgraded while the other serves (no root loss). The
|
||
**read-only, content-addressed root makes replication trivially coherent** (no
|
||
write-coherence problem) — this architecture is ideally suited to it. Wrinkle:
|
||
the **immutable UKI cmdline** points at one discovery address, so multipath needs
|
||
discovery to return multiple paths (or a DNS name resolving to several targets).
|
||
- **Planned SPDK restarts only via drain/failover** (multipath fail-over, or a
|
||
maintenance window with generous `ctrl_loss_tmo` so hosts recover not die). There
|
||
is **no zero-impact single-node SPDK restart.**
|
||
- **[x] Long-lived target + online release control — IMPLEMENTED 2026-06-27.**
|
||
The old gap (SPDK ran `spdk-setup.sh` as entrypoint, single-version,
|
||
configured-at-start → changing versions = container restart = drop ALL
|
||
connections) is closed. Now in `serving/spdk/`:
|
||
- **`spdk-target.sh`** — persistent entrypoint: starts `nvmf_tgt` ONCE
|
||
(release-agnostic), creates the TCP transport + discovery listener, then
|
||
reconciles every staged release. Never restarted to change versions; on
|
||
(re)start re-adds everything under `./releases/` (a serving-host reboot
|
||
restores all served versions).
|
||
- **`spdk-release.sh`** — JSON-RPC control: `add`/`remove`/`list`/
|
||
`drain-status`/`reconcile`, **one subsystem per version** (distinct NQN, so
|
||
adding/removing one never disturbs others' connections), **drain-gated
|
||
removal** (refuses while a version has connected controllers unless
|
||
`--force`).
|
||
- **`spdk-ctl.sh`** — host ops: `load`/`unload`/`list`; stages a release's
|
||
artifacts from its OCI image into `./releases/<ver>/` and drives the live
|
||
target. Rolling update = `load` new → reboot hosts → `unload` old once
|
||
drained.
|
||
- **`spdk-loader.sh`** — DECLARATIVE rolling updates (the "init container that
|
||
stays up", compose service `spdk-loader`). On `compose up` after an
|
||
`IMAGE_VERSION` bump it stages + adds the new rootfs to the running target
|
||
ONLINE, then polls and **auto-reaps** any non-current version once it has
|
||
drained (0 conns) for a sustained grace window (`REAP_GRACE_SECONDS`, keep >
|
||
host `ctrl_loss_tmo`). So migration = bump `IMAGE_VERSION` in `.env` +
|
||
`docker compose up -d` (compose recreates the loader + the http/dnsmasq UKI
|
||
servers, but NOT the nvmeof target → no dropped connections). Single served
|
||
UKI ⇒ a 0-conn old version is genuinely orphaned (a returning host fetches
|
||
the current UKI), so auto-reap is safe; removal is still drain-gated +
|
||
grace-debounced. The loader + target share the RPC socket on a named volume
|
||
(`nvmf_tgt -r`); the loader reuses the SPDK image (has python3 + rpc.py).
|
||
- **BENCH FINDING 2026-07-03 (FIXED) — hosts did `connect-all`, so NO version
|
||
ever drained → auto-reap could never fire.** Observed: `spdk-release list`
|
||
showed more controllers than workstations, with every box appearing on *every*
|
||
served version's subsystem (e.g. 1/3/3/1/3 across 5 versions for ~3 boxes).
|
||
Root cause: the initramfs connected with **`nvme connect-all`** (test path
|
||
`nvmf-connect.sh`; secure path delegated to the dracut nvmf module's discovery
|
||
autoconnect), which attaches EVERY subsystem in the discovery log — the host
|
||
then mounts only its own by-UUID verity namespaces but leaves a live controller
|
||
on all the others. So each running host pinned every version above 0 conns, and
|
||
`spdk-loader`'s "auto-reap once drained (0 conns)" **could never trigger** while
|
||
the fleet was up — the served-version list grew without bound. This defeats the
|
||
whole drain/reap model, not just cosmetics. **Fix:** targeted connect. The
|
||
per-version subsystem NQN (`SUBNQN`, already in `verity.json .subnqn`) is now
|
||
also baked into the signed UKI cmdline as **`rd.nvmf.nqn=`** (`extract-artifacts.sh`),
|
||
and both connect paths use **`nvme connect -n <nqn>`** — `nvmf-connect.sh` (test)
|
||
and a new explicit step 6 in `dot1x-macsec-up.sh` (secure, replacing the module
|
||
delegation; runs after macsec0 is addressed per invariant #2). Legacy UKIs
|
||
without `rd.nvmf.nqn` fall back to `connect-all` (with a warning) so they still
|
||
boot. Now each host = 1 controller on its own subsystem → versions drain to 0
|
||
when their hosts move off → auto-reap works; connections scale O(hosts) not
|
||
O(hosts × versions). Requires a rebuild + re-sign (cmdline change, invariant #6).
|
||
**(a) TEST PATH BENCH-VERIFIED 2026-07-03 (commit `0bfe872`): builds, deploys, and
|
||
the host mounts ONLY ONE image** (its own version's subsystem) instead of connect-
|
||
all'ing every served version — the targeted `nvme connect -n rd.nvmf.nqn` works.
|
||
So each host now = 1 controller on its own subsystem, which is what lets old
|
||
versions drain to 0 and `spdk-loader` auto-reap. **Still to confirm:** the reap
|
||
payoff itself (migrate a host, watch its old version drain to 0 and get reaped);
|
||
(b) SECURE PATH — the new step-6 connect brings up the NVMe root over macsec0
|
||
(this path is not yet bench-confirmed end-to-end); (c) that the dracut nvmf module
|
||
isn't ALSO connect-all-ing on the secure path (if it is, suppress it — the targeted
|
||
connect is meant to be the sole connector).
|
||
- **BENCH FINDING 2026-07-03 (FIXED) — the target WAS still recreated on an
|
||
`IMAGE_VERSION` bump.** Despite the above, the `nvmeof` (spdk-target) service
|
||
still carried a versioned `type: image` bootstrap mount
|
||
(`source: ${OCI_IMAGE}:${IMAGE_VERSION}` → `/artifacts`) **and** an
|
||
`IMAGE_VERSION` env, so bumping the version changed THIS service's config and
|
||
`compose up -d` recreated it → `nvmf_tgt` restarted → **every connected host's
|
||
NVMe/TCP root dropped** (the exact fleet-wide outage the long-lived-target
|
||
design exists to prevent). Fix: **made the target version-agnostic** — removed
|
||
the `/artifacts` image-volume + `IMAGE_VERSION` env from `nvmeof` in
|
||
`serving/docker-compose.yml`. Bootstrap AND rolling updates are now solely the
|
||
`spdk-loader`'s job (stages the current release into `./spdk/releases`, adds it
|
||
online); `spdk-target.sh`'s seed block is guarded
|
||
(`[ -s /artifacts/verity.json ] && [ -n IMAGE_VERSION ]`) so it no-ops when
|
||
unmounted. Verified `nvmeof` no longer references `IMAGE_VERSION` (only
|
||
`spdk-loader` does). **BENCH-VERIFY:** bump `IMAGE_VERSION` + `up -d` with a
|
||
host connected → `nvmeof` container NOT recreated (`docker ps` ID/uptime
|
||
unchanged), host root survives, loader adds the new version online.
|
||
- **No jq dependency:** `spdk-release.sh` parses RPC JSON with **python3** (the
|
||
SPDK image ships no jq); legacy artifacts (no per-version UUIDs in
|
||
verity.json) fall back to the fixed UUID pair with a warning.
|
||
- **Per-version namespace UUIDs** (`scripts/extract-artifacts.sh`): derived
|
||
deterministically (UUIDv5 of `image_version`), baked into the UKI cmdline +
|
||
`verity.json` (now also carries `subnqn`); the serving side reads them back
|
||
from each release's `verity.json`. Required so two versions don't collide on
|
||
the same `/dev/disk/by-id/nvme-uuid.<uuid>` host symlink.
|
||
- **Bootstrap unchanged:** `docker compose up` with one version still works —
|
||
the `/artifacts` image volume is seeded into `./releases/${IMAGE_VERSION}`.
|
||
- **PARTIALLY BENCH-VERIFIED 2026-06-28:** long-lived target boots a host end
|
||
to end — UKI served over HTTP → host boots → NVMe/TCP root connects to the
|
||
persistent `nvmf_tgt` (single version `spdk-permanence-02ffd7d`). Surfaced +
|
||
fixed several real bugs along the way: bootstrap seed must COPY not symlink
|
||
(a symlink into the version-pinned `/artifacts` re-pointed a stale release at
|
||
new data); `spdk-release` parses RPC JSON via **python3** (SPDK image has no
|
||
jq); legacy `verity.json` (no per-version UUIDs) falls back to the fixed pair;
|
||
mislabeled-dir guard (dir name vs verity.json `image_version`, case-insensitive);
|
||
dropped the redundant discovery-listener re-add (daemon logged "Listener
|
||
already exists"). **CI lowercases `image_version`** (gitea-main-ci vars step) —
|
||
a mixed-case branch (`SPDK-permanence`) had baked a capital
|
||
`fleet-SPDK-….efi` the lowercase boot URL couldn't fetch (iPXE "no such
|
||
file"); booted via a local retag workaround pending a lowercase rebuild.
|
||
- **STILL BENCH-VERIFY:** (a) `load` a 2nd version while a host is connected to
|
||
the 1st → 1st's connection survives (no AER/reset on the running host); (b)
|
||
`unload`/auto-reap drain-gate blocks while CONNS>0 and reaps after grace on a
|
||
LIVE host; (c) serving-host reboot reconciles all staged versions; (d)
|
||
per-version UUID build reproducible (two builds, same version → identical
|
||
`data_ns_uuid`/`hash_ns_uuid` + `root_hash`); (e) the full fix on a clean
|
||
LOWERCASE CI rebuild (no retag). See `serving/spdk/README.md`.
|
||
- **Capacity:** each concurrently-served version costs target memory/hugepages +
|
||
file-backed bdev overhead — plan for N live versions (current + previous +
|
||
canary) during long rollouts (raise `HUGEPAGES`).
|
||
- **Attach gating + discovery:** with several subsystems advertised, the dracut
|
||
connect + per-subsystem hostNQN allow-list must ensure a vX host attaches **only**
|
||
vX's namespace (the by-UUID verity ref protects *what mounts*, but gate the
|
||
*attach* too); `rd.nvmf.discover` selection must be deterministic, not connect-all.
|
||
|
||
## [ ] 7. Harden / evaluate the NVMe-oF target (serving-node resilience)
|
||
|
||
There is **no production-grade memory-safe NVMe-oF target** — the Rust NVMe work
|
||
is all driver/initiator-side (vroom, RedLeaf, Rust-for-Linux); the only
|
||
production targets are C (Linux kernel `nvmet`, SPDK). So this is **serving-node
|
||
hardening, not a language swap**. dm-verity already removes the target from the
|
||
clients' integrity TCB (verify-on-read rejects tampered blocks), so the risk is
|
||
RCE/DoS on the node that holds the fleet's disk, not client integrity. The
|
||
export is read-only + single-namespace = inherently small attack surface.
|
||
|
||
- **Evaluate kernel `nvmet` vs SPDK** for the target: kernel `nvmet` keeps the
|
||
data path in the hardened/fuzzed kernel with a tiny userspace footprint
|
||
(arguably a better attack-surface story); SPDK is a userspace C daemon
|
||
(current choice, needs hugepages + caps). Decide per environment.
|
||
- **Confine the target** (whichever): unprivileged, drop caps, seccomp +
|
||
SELinux/AppArmor profile, read-only container, minimal mounts; minimal
|
||
immutable serving image, digest-pinned.
|
||
- **Slim the SPDK image (multi-stage build)**: the current single-stage
|
||
from-source image is ~600 MB (it keeps the whole toolchain — gcc, meson/ninja,
|
||
kernel headers, the python venv — plus the SPDK/DPDK source and build trees).
|
||
Convert `serving/spdk/Dockerfile` to a builder stage + a minimal runtime stage
|
||
that copies only `nvmf_tgt` + its `ldd` shared libs (libnuma, libssl/crypto,
|
||
libuuid, libaio…) + `rpc.py`/`python3`. Expect ~150–250 MB (debian-slim base
|
||
or a static SPDK link trims further). Smaller attack surface + faster pulls,
|
||
and it pairs with the confinement above.
|
||
- **Network-isolate :4420** so only the boot VLAN reaches the target (compose
|
||
with 802.1X/MACsec segmentation in prod).
|
||
- **Replicas** behind distinct discovery entries (availability is the real risk
|
||
— it's the fleet's disk). The target is a **fleet-wide SPOF in every running
|
||
host's root data path**; a process restart drops *all* NVMe/TCP associations at
|
||
once. Structural fix = **HA + NVMe-oF multipath (ANA)** over byte-identical,
|
||
same-UUID, read-only namespaces (replication is coherent precisely because the
|
||
root is read-only + content-addressed); cheap mitigation = generous host
|
||
`ctrl_loss_tmo`/`reconnect_delay`. See the **"SPDK restart / availability"**
|
||
analysis under #6 (rolling updates) — same constraint, fuller treatment.
|
||
- Revisit a memory-safe target only if/when an off-the-shelf one matures
|
||
(e.g. a future Rust-for-Linux `nvmet`).
|
||
|
||
## [ ] 8. Return to secure / production posture (un-toggle the lab flags)
|
||
|
||
Consolidates the cleanup of the temporary lab toggles (see "Current build
|
||
posture" at the top) and the secure-path bench-verifies deferred while testing
|
||
the open-network path. Do these together, then run the full secure boot-chain
|
||
smoke test.
|
||
|
||
- **Re-enable 802.1X/MACsec:** `disable_dot1x=false` (CI `DISABLE_DOT1X="false"`).
|
||
Removes `rd.dot1x=0`/`ip=dhcp`; the hook owns the link (macsec0).
|
||
- **Bench harness scaffolded 2026-06-22: [serving/dot1x-bench/](../serving/dot1x-bench/).**
|
||
step-ca (fleet CA, on the serving host) signs each host's **TPM-generated CSR**
|
||
+ issues the RADIUS server cert; FreeRADIUS does EAP-TLS (trusts step-ca root,
|
||
returns MSK); a `hostapd` wired authenticator does 802.1X + MACsec on an
|
||
**isolated bridge per workstation** (flat VLAN can't auth — EAPOL dropped +
|
||
MACsec is P2P). `enrol/tpm-provision.sh` creates the key in the TPM at
|
||
`0x81000100` + writes the cert to NV `0x01800100`. Starting solution **signs a
|
||
CSR** (key stays in TPM); **TPM attestation** (ACME `device-attest-01`) is the
|
||
documented upgrade on the same CA. **BENCH-VERIFY** items flagged in the files:
|
||
hostapd MACsec build, openssl tpm2 provider vs pkcs11 key ref, EAP-TLS version.
|
||
PKI product decision (greenfield): start step-ca; OpenBao/EJBCA/Dogtag later.
|
||
- **First step (DONE 2026-06-22): `enrol/tpm-selftest.sh`** — non-destructive TPM
|
||
capability check (run on the booted open-port image, no `disable_dot1x` needed)
|
||
to de-risk the TPM half before the full path. Likely first finding: no openssl
|
||
TPM2 **provider** in the image (`tpm2-tss` ≠ the provider) — add `tpm2-openssl`.
|
||
- **Enrolment of the per-host identity** (key@`0x81000100` + cert@NV) is its own
|
||
design — see **#11** (TOFU + OOB fingerprint broker, pre-auth permit-set,
|
||
provisioning UKI). `tpm-provision.sh` is the manual bench stand-in for it.
|
||
- **NVMe discovery by DNS:** `nvmet_traddr=nvme.fleet.lab` (CI), resolved at boot
|
||
by the serving dnsmasq — drop the hardcoded `192.168.50.10`.
|
||
- **Signing — SIGNED-UKI + SECURE-BOOT-ON PATH BENCH-VERIFIED on the Proxmox VM
|
||
(HTTP boot, 2026-06-20).** CI signs the UKI when `SB_KEY`/`SB_CERT` secrets are
|
||
set (`db` key); `fleet-sb-enroll` enrols db→KEK→PK from the baked public `.auth`
|
||
files; OVMF then boots the signed UKI under enforcing Secure Boot. Confirmed
|
||
end-to-end: CI-signed UKI verifies (`sbverify`), enrolment succeeds, VM boots
|
||
with `mokutil --sb-state` = enabled.
|
||
**Key generation + enrolment procedure: [secure-boot-keys.md](secure-boot-keys.md)**
|
||
(the PK/KEK/db trio; `db` = `sb_key`/`sb_cert`, PK/KEK stay offline).
|
||
- **Bug fixed during bring-up (2026-06-20):** `fleet-sb-enroll` built its 4-byte
|
||
EFI attribute prefix in a bash variable (`ATTR=$'\x27\x00\x00\x00'`). Bash
|
||
strings can't hold NULs → only `0x27` survived → efivarfs read a corrupt
|
||
attribute u32 → every write failed with `EINVAL` ("dd: Invalid argument").
|
||
Now emitted directly with `printf` at write time. **Baked fix VERIFIED on the
|
||
820 (2026-06-22): `sudo fleet-sb-enroll` enrolled db→KEK→PK cleanly, no inline
|
||
workaround.** (Red herring on the way:
|
||
Proxmox EFI disks default to `pre-enrolled-keys=1`, so `mokutil --db` shows
|
||
the MS cert even in Setup Mode — looked like a non-empty-db rejection but was
|
||
the NUL bug all along; it failed identically on a clean `pre-enrolled-keys=0`
|
||
disk.)
|
||
- **iPXE signing WIRED (2026-06-21):** `snponly.efi` is now built + signed
|
||
(`db`) + pushed by CI as a separate FROM-scratch **`fleet-ipxe`** image
|
||
(`scripts/build-ipxe-oci.sh`, `.gitea/workflows/ipxe-image.yaml`), mounted into
|
||
dnsmasq at `/ipxe`. dnsmasq itself moved to a generic CI-built/pulled image
|
||
(`.gitea/workflows/dnsmasq-image.yaml`) — no longer built at deploy time, and
|
||
carries no keys/binary. (Decision: separate `fleet-ipxe` image, not baked into
|
||
dnsmasq, so the daemon image stays key-agnostic across SB environments.)
|
||
- **820 / PXE signed chain — BENCH-VERIFIED 2026-06-22.** Real hardware (HP
|
||
EliteBook 820 G4), Secure Boot **enforcing** (`mokutil --sb-state` = enabled,
|
||
Setup Mode no, db/KEK/PK enrolled): firmware verifies the signed `snponly.efi`
|
||
(fleet-ipxe image) → signed UKI and boots. Confirms the whole `fleet-ipxe`
|
||
signing pipeline + the PXE→iPXE→UKI chain under SB on real silicon (the VM only
|
||
covered the HTTP path, which has no iPXE stage).
|
||
- **REMAINING for production:** (a) **secure path** — re-verify signed boot with
|
||
`disable_dot1x=false` (802.1X/MACsec), not just the test path; (b) confirm
|
||
signing is reproducible (verity hash stable across two signed builds).
|
||
- **PXE-only iPXE fallback — BENCH-VERIFIED on the EliteBook 820 G4 (2026-06-20).**
|
||
Firmware without UEFI HTTP boot chainloads `serving/tftp/snponly.efi` → renders
|
||
`autoexec.ipxe` (ifopen + DHCP-retry + `imgargs <real cmdline>`) → UKI over HTTP
|
||
→ NVMe/verity root. dnsmasq tags `pxe`/`ipxe`/`httpboot` give graceful per-client
|
||
fallback; HTTP-boot hosts are unaffected (firmware → UKI direct). Gotchas that
|
||
cost real time (all captured in [../serving/tftp/README.md](../serving/tftp/README.md)):
|
||
(i) the I219-LM needs **`snponly.efi`** (native iPXE driver won't bind);
|
||
(ii) **PortFast/edge on the switch port is REQUIRED** — iPXE's PHY bounce
|
||
triggers STP forwarding delay that drops DHCP (this was the real blocker);
|
||
(iii) iPXE aborts the script on any unguarded command failure (use `||`);
|
||
(iv) Secure Boot OFF makes systemd-stub take iPXE's load options as the cmdline,
|
||
so the entrypoint extracts the UKI's real `.cmdline` and feeds it via `imgargs`.
|
||
**Tidied 2026-06-20** once PortFast was identified: dropped the `autoexec.ipxe` +
|
||
`ifopen` + DHCP-retry STP workaround; `boot.ipxe` is now just
|
||
`dhcp`+`imgfetch`+`imgargs`+`boot` reached via normal iPXE autoboot. Re-verified
|
||
on the 820 G4. **SB-on simplification:** when Secure Boot is enabled the cmdline
|
||
injection is unnecessary (the stub ignores the loader cmdline) — collapse
|
||
`boot.ipxe` to a plain `chain` of the UKI and drop the `.cmdline` extraction +
|
||
`binutils` + the artifact mount from the dnsmasq service.
|
||
- **Sign iPXE for the PXE-only fallback (Secure Boot, production).** The lab
|
||
`snponly.efi` is **UNSIGNED** → that path needs **Secure Boot OFF**. For
|
||
production: `sbsign` the iPXE binary with the fleet **db** key (only `db` needed —
|
||
no MS CA / shim), so SB stays enforced firmware → iPXE → UKI; then the cmdline
|
||
contraption becomes a no-op (the stub ignores loader cmdline under SB). See
|
||
[secure-boot-keys.md](secure-boot-keys.md) "PXE-only hosts".
|
||
- **Bench-verify on the secure path** (swtpm + hostapd + FreeRADIUS,
|
||
`disable_dot1x=false`) — the items deferred this session:
|
||
- NM port secure path: `en*` UNMANAGED + `macsec0` MANAGED/renewing, MKA
|
||
survives switch-root, NVMe stays up past lease (item #1 risk).
|
||
- NTP via DHCP option 42 reaches chrony over macsec0 (`chronyc sources`).
|
||
- Plymouth `fleet` theme: test-path render VERIFIED 2026-06-18; still confirm
|
||
the auth→macsec rows reach `done` live on the secure path.
|
||
- STIG remediation didn't break the secure path; reproducible verity hash;
|
||
triage the generated `stig-deviations.md` REVIEW rows (item #2).
|
||
- **Secure-path NVMe connect trigger:** the test path uses
|
||
`nvmf-connect.service` (gated `rd.dot1x=0`); confirm the secure path's
|
||
connect (dracut nvmf over macsec0) actually fires, or give it an equivalent
|
||
trigger. Apply the same `-k 30 -l -1 -c 10` keep-alive/reconnect flags to the
|
||
secure `nvme connect` as the test path has.
|
||
- **Future hardening waived for now** (from the STIG register): FIPS, firewall,
|
||
per-host/rotatable admin (SSH CA / config-agent).
|
||
- **TLS-encrypted audit/log offload (production).** The lab forwards logs to
|
||
`serving/audit-collector` in **plaintext**, and the STIG TLS-offload rules
|
||
(`rsyslog_encrypt_offload_defaultnetstreamdriver`,
|
||
`..._actionsendstreamdrivermode`, `..._actionsendstreamdriverauthmode`) are
|
||
**waived** in the tailoring because applying them forces `gtls` + x509 peer
|
||
auth + a CA file and breaks the plaintext `2f` forward. For production, do it
|
||
properly and DROP the waivers:
|
||
1. Stand up a CA (or reuse the boot-infra CA); issue the collector a server
|
||
cert (CN/SAN = `audit.fleet.lab`).
|
||
2. Configure the collector's rsyslog for TLS: `imtcp` with
|
||
`StreamDriver.Name="gtls"`, `StreamDriver.Mode="1"`, server cert/key + CA;
|
||
listen on the syslog-TLS port (6514) — add a TLS receiver to
|
||
`serving/audit-collector/rsyslog.conf` + compose.
|
||
3. Bundle the CA into the image (host-generic, no per-host state — same model
|
||
as the RADIUS CA) and point `$DefaultNetstreamDriverCAFile` at it.
|
||
4. Set the `2f` `omfwd` action to `StreamDriver="gtls" StreamDriverMode="1"
|
||
StreamDriverAuthMode="x509/name"` + the CA file, target port 6514.
|
||
5. Re-select the three `rsyslog_encrypt_offload_*` rules (remove from
|
||
`files/stig/diskless-tailoring.xml`) — they should then PASS, and update
|
||
`docs/stig-deviations.md` #6 to reflect encrypted offload.
|
||
(RELP+TLS via `omrelp`/`imrelp` is the alternative if guaranteed delivery is
|
||
wanted alongside encryption.)
|
||
- **fleet-hostapi: scope CORS to the kiosk origin.** The host-info agent ships
|
||
with `FLEET_HOSTAPI_ALLOW_ORIGIN=*` (lab). For prod set it to the exact kiosk
|
||
origin (default `http://boot.fleet.lab`) via the unit's commented
|
||
`Environment=` line (or a drop-in). See #12.
|
||
|
||
## [ ] 9. (OPTIONAL / nice-to-have) Convert provisioning to Ansible where it pays off
|
||
|
||
Low priority — only the *config-management* half of the build is a candidate, and
|
||
the ROI is modest because the build VM is single-shot (idempotency buys little)
|
||
and reproducibility/auditability favour minimal shell. **Not** a wholesale
|
||
rewrite: the load-bearing logic (dracut module hook ordering, `extract-artifacts.sh`,
|
||
`build-oci.sh`) is initramfs-runtime / host-side build tooling, not provisioning,
|
||
and stays shell.
|
||
|
||
The high-ROI, low-risk slice if/when we pick this up:
|
||
- **STIG via the SSG Ansible playbook** (`ssg-rl9-playbook-stig.yml`, shipped
|
||
with scap-security-guide) instead of `oscap --remediate` — the upstream path
|
||
and far more *reviewable* (you can diff exactly what each rule changes). Keep
|
||
`oscap eval` (no remediate) for the report + the 0-eval guard (item #2).
|
||
- Restyle the brittle imperative bits as declarative tasks: chrony.conf edit
|
||
(`lineinfile`/`blockinfile` vs `sed`), sshd drop-in / NM dispatcher / kiosk
|
||
file installs (`template`/`copy`), datastream-by-`os-release` as `assert`.
|
||
Bonus: sidesteps the `%{...}`→`%%{` HCL-escaping footgun.
|
||
|
||
Tips the balance toward "do it" only if: we start managing the **serving hosts**
|
||
(dnsmasq/nginx/SPDK/chrony/audit-collector) with config-management too (shared
|
||
roles to amortise — currently they're docker-compose), provisioning grows much
|
||
more branchy, or we want molecule-tested hardening roles validated without a
|
||
full ~17-min Packer build.
|
||
|
||
## [ ] 10. (OPTIONAL) Secure Boot cert auto-enrollment for provisioning
|
||
|
||
Speed up bringing up new workstations: bake the **public** PK/KEK/db certs into an
|
||
image, detect Secure Boot **Setup Mode** at boot, and auto-enroll them so a
|
||
fresh/reset box adopts the fleet trust root without a manual firmware-setup visit.
|
||
Pairs with #8 (signing) + [secure-boot-keys.md](secure-boot-keys.md). Shares the
|
||
provisioning-UKI + isolated-environment vehicle with **#11** (which enrols the
|
||
*network* identity); build them together.
|
||
|
||
**Tooling gap (found 2026-06-20): `efitools` is NOT packaged for EL9** — no
|
||
`efi-updatevar`/`efi-readvar`/`KeyTool`. Resolved for the OS-side path by writing
|
||
the signed `.auth` variables to **efivarfs directly** (de-immutable + single
|
||
write), which needs no extra package. **Done 2026-06-20:** baked
|
||
`/usr/local/bin/fleet-sb-enroll` (efivarfs method, Setup-Mode-gated, fapolicyd-
|
||
trusted) + the public `PK/KEK/db.auth` at `/usr/share/fleet-secureboot/` (supplied
|
||
via `files/secureboot/keys/` or `SB_*_AUTH_B64` CI secrets); `mokutil` covers
|
||
reading. So **manual** OS-side enrollment over SSH now works (`sudo
|
||
fleet-sb-enroll`). What's left for *auto*-enroll: drive `fleet-sb-enroll`
|
||
unattended on a **Setup-Mode detection** at boot, gated to a provisioning mode/UKI
|
||
(below), with read-back verify + audit — NOT armed in the production image.
|
||
|
||
**What's safe:** the certs are public (no key material — only the **db** private
|
||
key signs, and it stays in CI/HSM, invariant #7); Setup Mode accepts
|
||
*unauthenticated* writes to PK/KEK/db, so public certs (ESL) are sufficient. It's
|
||
fleet-generic (no per-host state, invariant #1) and deterministic (reproducible
|
||
verity hash).
|
||
|
||
**The real risk — provisioning-time TOFU (trust-on-first-use):** in Setup Mode
|
||
Secure Boot isn't enforcing, so the enroller runs as *unverified* code, and the
|
||
fleet's UEFI **HTTP boot is unauthenticated** in that state. An on-path attacker
|
||
during provisioning can serve a malicious image that enrolls **their** PK/KEK/db —
|
||
the box then "has Secure Boot" but trusts the attacker. Auto-enroll is only as
|
||
trustworthy as the provisioning channel, and it removes the physical-presence
|
||
approval gate. So treat it as a controlled-environment provisioning step, not an
|
||
always-on behaviour.
|
||
|
||
**Optional separate provisioning UKI (preferred shape):** build a dedicated
|
||
**provisioning UKI** that carries the certs + the enroll logic, instead of arming
|
||
the enroller in the production image. Keeps the standing footgun out of the normal
|
||
image (a factory reset + tampered image can't silently re-root trust), and lets
|
||
the enroller be a one-shot, attended/isolated step. The production UKI never
|
||
enrolls.
|
||
- Build it as a separate Packer target / flag (e.g. `provisioning=true`) →
|
||
its own small UKI; signed or unsigned (it runs in Setup Mode anyway, but sign
|
||
it so it *also* boots once keys are enrolled / for re-runs).
|
||
- Or a tiny standalone EFI tool (KeyTool-style) served alongside, not a full OS.
|
||
|
||
**Mitigations (do these regardless of UKI vs separate UKI):**
|
||
- Run provisioning on a **trusted, isolated VLAN** (or a physically controlled
|
||
bench) — the integrity Setup Mode can't give must come from the environment.
|
||
- **Gate** enrollment behind an explicit provisioning mode / one-shot token;
|
||
never arm it on every boot of the production image.
|
||
- **Read-back verify before declaring success:** `efi-readvar` the enrolled
|
||
PK/KEK/db and compare fingerprints to the values baked in; mismatch = fail
|
||
loudly (turns silent TOFU into tamper-evidence).
|
||
- **Audit** the enrollment event + read-back fingerprints to the collector.
|
||
- **Keep the Microsoft UEFI CA in `db`** (or enroll your db alongside it) unless
|
||
every NIC/GPU option ROM is covered by your keys — clearing MS db can break
|
||
signed **option ROMs** (NIC PXE/HTTP-boot, GPU init), bricking the boot path.
|
||
- **PCR 7 check:** enrollment changes the Secure Boot measurement in PCR 7;
|
||
confirm nothing the fleet seals to the TPM (the EAP-TLS key at `0x81000100`,
|
||
NV cert) is PCR-7-policy-bound before relying on this.
|
||
|
||
**Decision before building:** auto-enroll trades assurance for convenience and is
|
||
reasonable for a controlled fleet; the highest-assurance path is attended/firmware
|
||
or manufacturer pre-provisioning. Default recommendation: **separate provisioning
|
||
UKI + isolated VLAN + read-back verify.**
|
||
|
||
## [ ] 11. (DESIGN) TPM 802.1X identity enrolment (TOFU + OOB fingerprint)
|
||
|
||
**Authoritative design consolidated 2026-06-28:
|
||
[tpm-enrolment-design.md](tpm-enrolment-design.md)** — unifies this item with the
|
||
fleet-manager (the broker = the fleet-manager's one unauthenticated bootstrap
|
||
endpoint; approval births the inventory record) and the full identity lifecycle
|
||
(enrol / renew / alert). **v1 decisions (2026-06-28):** fingerprint-only OOB
|
||
assurance (TPM attestation via ACME `device-attest-01` is the documented later
|
||
upgrade); **inline single-UKI** vehicle for now (empty-NV early-boot branch — keeps
|
||
provisioning network-boot-only, no USB; a separate provisioning UKI behind an
|
||
`enrol_mode` build flag is the planned later upgrade); enrolment transport is plain
|
||
**stdlib HTTP/JSON** (Go, the one unauthenticated edge — gRPC reserved for the
|
||
steady-state agent path). The notes below are retained as the originating decisions;
|
||
the design note is canonical.
|
||
|
||
**Broker BUILT (server side, 2026-06-28): [serving/enrol-broker/](../serving/enrol-broker/)**
|
||
— Go/stdlib; public unauth `POST /request`+`GET /status/{id}` (behind the serving
|
||
nginx `/enrol/`) + admin **mTLS web UI** (`/`, `/pending`, `/approve|deny/{id}`)
|
||
with a **dedicated admin CA** (`admin-pki.sh`); wordlist fingerprint, time-boxed
|
||
store, per-IP rate limit; signs via `step` (dev `mock` signer). Ships in
|
||
`serving/dot1x-bench/docker-compose.yml`; end-to-end smoke-tested (CSR → mTLS
|
||
approve → cert). **BENCH-VERIFY:** run against real step-ca (`bootstrap.sh` root +
|
||
`enrol` provisioner password readable by the broker) and confirm an issued cert
|
||
authenticates EAP-TLS.
|
||
|
||
**HOST SIDE BUILT 2026-07-02 — Go binary (`fleet-enrol`), awaiting bench.**
|
||
Pivoted from a shell hook to a static **Go + go-tpm** binary (`files/enrol/`):
|
||
it talks to `/dev/tpmrm0` DIRECTLY (via `transport.FromReadWriter` on an
|
||
`os.File` — no `linuxtpm`, so go-tpm is the only vendored dep), so the enrolment
|
||
path needs **no tpm2-tools key-gen / no openssl tpm2 provider / no curl** in the
|
||
image (removes the biggest bench risk — the provider `handle:` key-ref). It
|
||
ensures the ECDSA key at `0x81000100`, builds the CSR, POSTs CSR+context to
|
||
`rd.fleet.enrol_url`, shows the broker fingerprint on Plymouth, polls `/status`,
|
||
and writes the leaf cert to NV `0x01800100`. The dracut hook `tpm-enrol.sh`
|
||
(initqueue/settled prio 25) is now a thin GATE: empty-NV check (`tpm2_nvread`,
|
||
still needed for the secure-path read) → pre-auth DHCP on `en*` (nmcli override of
|
||
the secure-mode unmanaged drop-in) → `exec /usr/sbin/fleet-enrol` → reboot to
|
||
converge on exit 0. No-op when a cert is present or under `rd.dot1x=0`.
|
||
|
||
**Built OUT OF THE VM (2026-07-02, per decision):** `fleet-enrol` AND
|
||
`fleet-hostapi` now compile on the CI runner (`.gitea/workflows/gitea-main-ci.yaml`
|
||
"Build host Go binaries" step, Go 1.26.4, `CGO_ENABLED=0`, vendored/stdlib,
|
||
`GOPROXY=off`), and Packer **file-provisions** the static binaries in — the Rocky
|
||
build image no longer carries the Go toolchain (dropped the in-VM `dnf install
|
||
golang` + the 2d removal). `fleet-enrol` lands at `/usr/sbin` before the initramfs
|
||
build (dracut `inst_simple`s it; asserted in `lsinitrd`), fapolicyd-trusted.
|
||
New `enrol_url` Packer var → `rd.fleet.enrol_url` cmdline.
|
||
|
||
**TEST-MODE ENROLMENT ENABLED (2026-07-02, temporary):** the hook no longer
|
||
no-ops under `rd.dot1x=0` — enrolment runs on the open/test path too, so it can be
|
||
exercised on a host with a TPM + the broker WITHOUT standing up
|
||
802.1X/MACsec/RADIUS. In test mode `ip=dhcp` owns the NIC (the hook skips its own
|
||
DHCP to avoid the dracut "address already assigned" race) and DNS resolves the
|
||
broker via the lease. `rd.fleet.enrol=0` disables it. Production intent is still
|
||
secure-path-only; revisit when the secure path is bench-verified.
|
||
|
||
**TESTED (2026-07-02):** transaction logic (CSR → mock broker → approve → cert
|
||
extract) as stdlib unit tests, AND the **full TPM ceremony (key-gen → persist →
|
||
ECDSA sign → CSR verify → chunked NV write → re-enrol) against the go-tpm software
|
||
simulator** (separate `files/enrol/simtest/` module so the CGO simulator never
|
||
enters the shipped build). Both run in CI.
|
||
|
||
**BENCH FINDING 2026-07-02 (FIXED) — the signing-key handle `0x81000100` was the
|
||
EK slot; moved to `0x81000100`.** First real-TPM boot (host `cygnus-cruiser-99`,
|
||
Proxmox vTPM, `rd.dot1x=0`) failed: `fleet-enrol: ensure key at 0x81010001: key at
|
||
handle is not ECC: did not contain eccDetail (selector value was 1)`. `EnsureKey`
|
||
died reading an existing RSA object as ECC → enrolment died before the CSR/POST, so
|
||
the splash flashed "FIRST-BOOT TPM ENROLMENT" then the test system booted with **no
|
||
fingerprint and nothing at the broker**. Root cause (confirmed via `tpm2_readpublic`
|
||
under `enforcing=0`): **`0x81000100` is the TCG-standard RSA Endorsement Key handle**
|
||
— the object there was the swtpm-provisioned **EK** (`restricted|decrypt|adminwithpolicy`,
|
||
RSA-2048/AES128CFB, canonical EK authPolicy `837197674484b3f8…`), with its EK cert at
|
||
NV `0x1c00002`. Our design squatted the EK slot fleet-wide (most firmware/swtpm TPMs
|
||
have their EK there), so enrolment could never create our key.
|
||
- **Fix (commit c574fa4 + follow-up):** the fleet signing key moved to
|
||
**`0x81000100`** (non-reserved owner handle; threaded through `fleet-enrol
|
||
-handle` default, `rd.dot1x.handle=` cmdline knob baked by `extract-artifacts.sh`,
|
||
`wpa_supplicant-dot1x.conf`, the bench scripts, invariant #1, and the design docs).
|
||
NV index was `0x01800001` here (owner range); it MOVED to `0x01800100` in a later finding (OEM platform-NV collision on the 820 — see below).
|
||
- **EnsureKey hardening:** (i) evicts a genuine *stale signing key* at the handle
|
||
and recreates (self-heal for old provisioning leftovers); (ii) **refuses to evict
|
||
any restricted-decrypt object (EK/SRK)** as a safety net — never clobber the TPM's
|
||
endorsement/storage identity (also protects the future EK/AK attestation).
|
||
Regressions: `simtest/TestEnsureKeyEvictsStaleNonECC` + `TestEnsureKeyRefusesToEvictProtectedKey`.
|
||
- **IDevID note (future):** if EK/AK attestation (`device-attest-01`, §"Stronger
|
||
assurance") lands, the natural home for a TPM device-identity key is the TCG
|
||
**IDevID** handle `0x81020001` (LDevID `0x81030001`) — see tpm-enrolment-design.md.
|
||
Not adopted for v1 (implies the IDevID template/cert conventions).
|
||
- **Also seen — leftover NV suppresses enrolment:** a *different* host (the 820)
|
||
booted `tpm-enrol: client cert present at NV 0x01800001 — normal boot` with nothing
|
||
enrolled this session — leftover NV state (from an earlier `tpm-provision.sh` /
|
||
`tpm-selftest.sh --use-real-handles`) **persists in the hardware TPM** across
|
||
reinstalls. Our presence check is content-blind (index defined ⇒ "enrolled"), so
|
||
foreign/stale NV silently skips enrolment (→ secure-path auth would then fail). See
|
||
the **content-aware NV check** item below.
|
||
- **Needs a rebuild + re-boot to confirm** (the sim proves the ceremony, not the
|
||
device). ⚠ **Don't hand-run `sudo fleet-enrol` on the booted STIG host** — the
|
||
confined admin domain (`sysadm_t`) is SELinux-**denied** write to `/dev/tpmrm0`
|
||
(`avc: denied … tpm_device_t`); the faithful path is the **initramfs** run
|
||
(reboot, unconfined), or `enforcing=0` for hand-run TPM inspection.
|
||
|
||
**FOLLOW-UP HARDENING:**
|
||
- **Content-aware NV presence check — BUILT 2026-07-03.** `fleet-enrol -check-only`
|
||
(no network / no key-gen) reads NV `0x01800100` via `tpmkey.ReadNVCert`, parses it as
|
||
X.509, checks the validity window, and — when a real fleet CA is bundled (`-cacert`,
|
||
default `radius-ca.pem`) — verifies it **chains** to it. Only a valid fleet cert counts
|
||
as enrolled (rc 0); empty/garbage/foreign/expired → rc 20 (enrol + overwrite;
|
||
`WriteNVCert` undefine+redefines). The dracut hook now calls it as the single gate,
|
||
replacing the blind `tpm2_nvread` (removed from `tpm-enrol.sh`; the binary's `NVPresent`
|
||
stays for cheap existence checks). **Lab fallback:** the shipped `radius-ca.pem` is a
|
||
*placeholder* (comments, not a cert) → `loadCAPool` returns nil → chain-less "parseable
|
||
+ unexpired" acceptance, so the reboot-converge doesn't loop on a bench-CA cert;
|
||
**production must replace `radius-ca.pem` with the real fleet CA** for strict chaining.
|
||
Tests: `classifyNVCert` unit table (valid/expired/garbage/foreign/no-CA) + `simtest`
|
||
ReadNVCert roundtrip across the 512B chunk. **BENCH-VERIFY** on a real boot: a garbage/
|
||
foreign NV now enrols instead of "normal boot".
|
||
- **De-enrol / TPM-reset procedure** for re-testing: `tpm2_nvundefine 0x01800100 -C o`
|
||
(+ `tpm2_evictcontrol -C o -c 0x81000100` for a stale *app* key — never the EK), or
|
||
BIOS "Clear TPM". Document in the bench README.
|
||
|
||
**BENCH FINDING 2026-07-03 (FIXED) — NV index `0x01800001` collided with OEM
|
||
platform NV; moved to `0x01800100`.** The Dell 820 kept "finding a cert" and
|
||
bypassing enrolment even after repeated BIOS TPM clears. `tpm2_nvreadpublic`
|
||
showed the truth: `0x01800001` on that hardware is a **firmware/platform** index
|
||
(`platformcreate|policydelete|writelocked`, 70 bytes — NOT our cert; siblings at
|
||
`0x01800003/4` + `0x01810000` share it), which **no owner/BIOS clear can remove**
|
||
(`TPM2_Clear` spares `platformcreate`; `policydelete` needs the platform's secret
|
||
policy). So our chosen index was permanently occupied by Dell — the blind check
|
||
false-positived on it, and even the content-aware check would have failed at the
|
||
NV **write** (can't undefine a platform index). Same class as the EK-handle
|
||
collision. Fix: **moved the NV cert index default `0x01800001` → `0x01800100`**
|
||
(free on the 820; owner range; overridable via `rd.dot1x.nvindex=`), threaded
|
||
through the cmdline (`extract-artifacts.sh`), `fleet-enrol -nvindex`, the dracut
|
||
hooks, bench scripts, invariant #1, and docs. Also **hardened `WriteNVCert`** to
|
||
refuse a `platformcreate` index with a clear "pick a free owner index" error
|
||
instead of a confusing undefine failure (regression `simtest/
|
||
TestWriteNVCertRefusesPlatformIndex`). **BENCH-VERIFY:** 820 now enrols at
|
||
`0x01800100`; and check `tpm2_nvreadpublic` on any OTHER hardware model in the
|
||
fleet — if a vendor also squats `0x018001xx`, pick a value free on all of them.
|
||
|
||
**BENCH-VERIFY (host side, before trusting):** (a) go-tpm against a REAL TPM +
|
||
swtpm on the target (the simulator proves the ceremony, not the device) — **the
|
||
stale-key fix (above) is the first real-TPM result; re-boot to confirm it now
|
||
reaches the broker**; (b) empty-NV create/persist on a fresh TPM with **empty owner
|
||
auth** (the code assumes it — a set owner auth needs handling); (c) pre-auth DHCP on `en*` in secure mode
|
||
(the `nmcli set managed yes` override of nm-unmanaged.sh actually leases); (d)
|
||
fingerprint renders on the `fleet` Plymouth theme + admin can match it; (e) NV
|
||
write + **reboot converges** (2nd boot runs the secure path); (f) prod: broker over
|
||
**HTTPS** (`fleet-enrol -cacert radius-ca.pem`), currently lab-HTTP.
|
||
|
||
**TEMPORARY (2026-06-28): broker also added to the MAIN serving stack**
|
||
(`serving/docker-compose.yml`, `enrol-broker` service + `ENROL_*` in
|
||
`serving/.env.example`) so a single `compose up` in `serving/` brings the
|
||
`/enrol/` endpoint up while the host side is built. Defaults to the
|
||
zero-dependency lab path — **`mock` signer** (ephemeral CA; certs NOT trusted by
|
||
RADIUS/NVMe) + **plaintext loopback admin** (`ENROL_ADMIN_INSECURE=true`). The
|
||
serving nginx already proxies `/enrol/`→`127.0.0.1:9090`. **SPLIT BACK OUT:** the
|
||
broker's proper home is a CA-co-located stack (it's still defined in
|
||
`dot1x-bench/` next to step-ca); real signing needs the fleet CA. Don't run both
|
||
copies on one host (they collide on 9090/9443). To use real signing from the
|
||
serving stack meanwhile: set `ENROL_SIGNER=step` + `ENROL_STEP_CA_URL` at a
|
||
reachable step-ca, populate `serving/enrol-pki/`, and `ENROL_ADMIN_INSECURE=false`.
|
||
|
||
**STEP SIGNER VERIFIED end-to-end 2026-06-28** (broker → real step-ca → signed
|
||
cert that chains to the fleet root; hostnqn derived from serial; EKU clientAuth).
|
||
**Re-VERIFIED ON THE SERVING HOST 2026-07-02** against step-ca 0.30.2 via
|
||
`verify-serving.sh` through the nginx `/enrol/` proxy: issuer = Fleet Bench CA
|
||
Intermediate, cert validity exactly 720h (confirms the maxTLSCertDuration claim
|
||
raise took effect), EKU clientAuth, `enrol` provisioner extension present,
|
||
hostnqn = cert serial. Bring-up gotchas hit + fixed (all in the commits): the CA
|
||
wouldn't init (compose used `_PASSWORD_FILE` for a nonexistent file → use
|
||
`_PASSWORD`); Docker auto-created `./pki` root-owned (bootstrap self-heals perms +
|
||
pre-create the bind-mount dirs); the CLI couldn't resolve `ca.fleet.lab` on the
|
||
serving host (connect via `localhost`, which is in the CA SAN).
|
||
Verification found + fixed two things the `mock` path had hidden:
|
||
- **BUG FIXED (`signer.go`):** `--not-after` was passed to `step ca token`
|
||
(validity capped ~1h → "token validity too long") instead of `step ca sign`.
|
||
Moved to the sign step. Without it the step signer never issues a cert.
|
||
- **CA CLAIM REQUIRED:** a stock step-ca caps certs at 24h, so 720h is rejected
|
||
("authorized maximum certificate duration of 24h"). `dot1x-bench/step-ca/bootstrap.sh`
|
||
now raises `authority.claims.maxTLSCertDuration` (+restarts the CA) — this also
|
||
unblocks bootstrap's own 720h RADIUS cert. Any other CA needs the same claim.
|
||
- **BENCH-VERIFY still open:** issued cert actually authenticates EAP-TLS through
|
||
FreeRADIUS (next, before host enrolment).
|
||
|
||
How a host gets its **802.1X EAP-TLS client identity** the first time — the key at
|
||
TPM handle `0x81000100` + cert at NV `0x01800100` that invariant #1 says must come
|
||
from enrolment, never the image. Distinct from #10 (that enrols *Secure Boot*
|
||
PK/KEK/db); this enrols the *network identity*. They share the provisioning UKI +
|
||
isolated-environment vehicle. Pairs with #8 + the CA decision (start **step-ca**,
|
||
[serving/dot1x-bench/](../serving/dot1x-bench/); OpenBao/EJBCA/Dogtag later).
|
||
|
||
**Chosen pattern — trust-on-first-use with out-of-band fingerprint approval:**
|
||
on an empty TPM the host generates a key *in* the TPM, builds a CSR, submits it to
|
||
an enrolment **broker**, and shows a **fingerprint** on the Plymouth splash; an
|
||
admin visually cross-checks that fingerprint against the broker's pending-list and
|
||
approves; the broker has step-ca sign the CSR and returns the cert; the host writes
|
||
it to NV `0x01800100` and proceeds to authenticate. Sound, recognised pattern
|
||
(SSH/WireGuard/Signal-style OOB verification); good fit for invariant #1.
|
||
|
||
**Key design decisions (settled 2026-06-22):**
|
||
|
||
- **The pre-auth permit-set already exists — enrolment rides in it.** The firmware
|
||
can't do 802.1X, so the UKI fetch (DHCP/DNS/TFTP/HTTP to the serving host) is
|
||
**always pre-auth**; the secure model therefore *requires* a pre-auth permit-set
|
||
(low-impact 802.1X / pre-auth dACL / boot→secure dynamic VLAN; RADIUS returns the
|
||
secure VLAN on Access-Accept). The enrolment endpoint is just **another entry in
|
||
that set** — expose it as an `/enrol/` location on the **existing serving nginx**,
|
||
reverse-proxied to the broker. No new network mechanism.
|
||
- **The broker is the only new component.** step-ca signs but has no human-approval
|
||
queue: broker = `POST CSR + context → PENDING + fingerprint → admin approves →
|
||
step-ca signs → host polls → cert`. Exposed to *unauthenticated* devices by
|
||
design → harden: OOB-fingerprint gate (no auto-sign), rate-limit, time-boxed
|
||
pending, HTTPS (image trusts the serving CA), pre-auth permit scoped to only
|
||
these services + the serving host.
|
||
- **Trigger on EMPTY-NV only — split the three events:** empty NV → enrol (human);
|
||
valid key + expired/near-expiry cert → **renew** automatically (no human, it
|
||
already holds a trusted key); auth fails with a present cert → **alert + retry,
|
||
do NOT re-enrol** (auto-reenrol on failure would mask revocation and churn keys
|
||
on a transient RADIUS outage). Re-enrol after the first time = explicit admin
|
||
action (clear/flag the host).
|
||
- **Fingerprint: what it proves + entropy.** It proves "this CSR came from *this
|
||
physical device*" (approver reads the screen) — it does NOT prove TPM residency.
|
||
Pair it with **TPM attestation** (step-ca ACME `device-attest-01`, EK/AK) for the
|
||
residency proof: fingerprint = human authorises the device, attestation = the key
|
||
is really in a TPM (together → strong SSP R-01 story). Size for entropy — a short
|
||
hex fp is grindable; use a word-list/BIP39-style encoding (reads short, more
|
||
bits) + rate-limiting + a server nonce. Show device context (asset/port/location)
|
||
so the admin knows *what* they're approving.
|
||
- **Pre-auth traffic has no MACsec (plaintext at L2).** Fine for enrol (CSR/cert
|
||
are public; OOB fingerprint gives authenticity) — and it's exactly **why the UKI
|
||
must be Secure-Boot-signed** (#8): integrity of the boot comes from the signature,
|
||
not the network. Enrolment design + Secure Boot are two halves of the same
|
||
"pre-auth path is hostile" assumption.
|
||
- **Run it in a provisioning UKI/mode, not the production initramfs** — the boot
|
||
hook ordering is load-bearing (invariant #2); don't park it on a multi-minute
|
||
human-approval wait. Shares the #10 provisioning-UKI vehicle.
|
||
|
||
**Bench caveat:** `hostapd` wired is strictly port-based (no pre-auth per-flow
|
||
ACL) — before EAP success it passes nothing, so the bench can't reproduce
|
||
"boot + enrol pre-auth on the same port." Model the pre-auth state with a separate
|
||
provisioning bridge/VLAN on the rig; the in-place low-impact pre-auth permit is a
|
||
real-switch feature to validate on the 820's switch.
|
||
|
||
**First step DONE (2026-06-22):** `serving/dot1x-bench/enrol/tpm-selftest.sh` — a
|
||
non-destructive TPM capability check (device, key persist, NV roundtrip, openssl
|
||
TPM2 provider, CSR signed by the TPM key) to de-risk the TPM half before any of
|
||
this. Expected gap: the image installs `tpm2-tss` but no openssl TPM2 **provider**
|
||
(`tpm2-openssl`) — add it to the build once the self-test confirms.
|
||
|
||
## [~] 12. Workstation host-info API (`fleet-hostapi`) for the kiosk
|
||
|
||
**IMPLEMENTED 2026-06-28.** A small **Go, read-only, loopback-only** HTTP agent on
|
||
the workstation that reports host/OS/hardware facts to the kiosk SPA, per the
|
||
[HOST-API.md](../HOST-API.md) contract (`GET http://localhost:9099/v1/host`).
|
||
Source: `files/hostapi/` (stdlib-only, reproducible static build). Built + enabled
|
||
by provisioner **2b-hostapi**; toolchain removed in 2d; binary fapolicyd-trusted
|
||
in 2c-STIG-postfix (register #10). Chromium policy grants Private Network Access
|
||
scoped to the kiosk origin (`files/kiosk/chromium-kiosk-policy.json`).
|
||
|
||
- **Per-host safety:** host-specific fields (kernel, hostname, IP, MAC, board,
|
||
uptime) are read at RUNTIME; only image-generic facts (image version +
|
||
weston/chromium versions) are baked, to `/etc/fleet-hostinfo.json` (invariant
|
||
#1 preserved). Deliberately does NOT expose DMI serial (root-only/per-host) or
|
||
any TPM/cert/identity material.
|
||
- **Hardening shipped:** loopback bind enforced in code; `DynamicUser`, no caps,
|
||
`ProtectSystem=strict`, `MemoryDenyWriteExecute`, `SystemCallFilter=@system-service`,
|
||
`IPAddressDeny=any` (loopback only). Runs unprivileged, reads world-readable
|
||
files only.
|
||
|
||
**Lab posture LEFT AS-IS (decided 2026-06-28) — revisit for prod:**
|
||
- **CORS `Allow-Origin=*`** → scope to the kiosk origin (`http://boot.fleet.lab`).
|
||
Consolidated into #8.
|
||
- **SELinux:** runs in the default service domain (no custom policy module).
|
||
**BENCH-VERIFY** no AVC denials for `fleet-hostapi` on an enforcing booted host
|
||
(Wazuh/`fleet_audit` rule 100120 will surface any). If denials appear, decide:
|
||
write a confined SELinux policy module vs adjust the sandbox.
|
||
- **No auth** (justified: loopback + read-only + single-user appliance). Option if
|
||
the threat model tightens: shared token in `kiosk.env` + header check.
|
||
- **Info sensitivity:** exposes hostname/IP/MAC/board/versions to the kiosk. Drop
|
||
MAC/IP if that's later deemed too much.
|
||
- **Reproducibility:** the `golang` toolchain version is a reproducibility-sensitive
|
||
input (a package bump changes the binary → verity hash); covered by the
|
||
build-twice gate. Pin the go version only if drift becomes a problem.
|
||
|
||
**BENCH-VERIFY (functional):** (a) service starts under enforcing SELinux +
|
||
fapolicyd (not blocked); (b) the kiosk SPA's System page fetches `/v1/host`
|
||
successfully through the PNA allowance (preflight + GET); (c) facts render
|
||
correctly on real hardware (board/CPU/GPU from DMI/DRM, not the virtio build VM).
|
||
|
||
## [~] 13. Kiosk GPU compositing + Chromium flag ownership
|
||
|
||
**GPU FIX APPLIED 2026-06-28.** The 820 (Intel HD620/i915) ran Chromium at ~150%
|
||
CPU with periodic crashes: `--disable-gpu` in our `kiosk-session` forced software
|
||
raster/compositing. Root cause confirmed from the live cmdline. Fixes in
|
||
`files/kiosk/kiosk-session` + `diskless.pkr.hcl`:
|
||
- **Runtime GPU probe** (fleet-generic image boots real HW *and* virtio VMs): a
|
||
real KMS driver (`i915`/`amdgpu`/…) + `/dev/dri/renderD128` → GPU path
|
||
(`--ignore-gpu-blocklist --enable-gpu-rasterization --enable-zero-copy`, weston
|
||
GL renderer); else software (`--disable-gpu` + `weston --use-pixman`) so the VMs
|
||
still render. No per-host state baked.
|
||
- **kiosk user added to `render`** (renderD128 isn't logind-seat-ACL'd on EL9;
|
||
`video` only covers card0/KMS for weston).
|
||
- **Merged the distro wrapper's `--disable-features` into ours** — Chromium takes
|
||
the LAST `--disable-features` (not the union), so ours was silently dropping the
|
||
wrapper's Lens/Gemini/Compose disables.
|
||
|
||
**BENCH-VERIFY (on the 820):** GPU process alive (`chrome://gpu`), Compositor/
|
||
ThreadPool threads near-idle + CPU drop, `gt_act_freq_mhz` rises under load, no
|
||
SELinux AVC for `dri_device_t` from the kiosk domain (Wazuh `fleet_audit` 100120
|
||
will surface any — if present, add a scoped SELinux allow or confirm the kiosk
|
||
user's SELinux mapping is unconfined). Confirm VMs still render (software path).
|
||
|
||
**DECIDED 2026-06-28 — Option 1 (policy lock), APPLIED.** The distro chromium
|
||
wrapper (`/usr/bin/chromium-browser`) layers flags as `$CHROMIUM_FLAGS`
|
||
(`/etc/chromium/chromium.conf` → VA-API enables + AI disables; overridable via
|
||
`CHROMIUM_USER_FLAGS`) + **hardcoded** `$CHROMIUM_DISTRO_FLAGS`
|
||
(`--enable-plugins --enable-extensions --enable-user-scripts --enable-printing
|
||
--enable-sync --auto-ssl-client-auth`) + our `"$@"` (last). The hardcoded layer
|
||
isn't removable via env/conf — only by bypassing the wrapper. Rather than bypass
|
||
(Option 2, which would force us to replicate the wrapper's stdio sanitisation +
|
||
own the VA-API feature list across updates), we **neutralise in managed policy**
|
||
(policy beats cmdline flags, survives updates): `ExtensionInstallBlocklist=["*"]`
|
||
added to `files/kiosk/chromium-kiosk-policy.json` (blocks all extensions → also
|
||
kills the user-scripts surface). `--enable-sync` already locked by `SyncDisabled`;
|
||
`--auto-ssl-client-auth` is moot (no browser-store client cert; the TPM 802.1X
|
||
cert isn't one) — `AutoSelectCertificateForUrls` left unset. VA-API enables
|
||
inherited from `chromium.conf` unchanged. Option 2 (full flag ownership, bypass
|
||
the wrapper, own the VA-API `--enable-features`:
|
||
`AllowQt,WaylandLinuxDrmSyncobj,WaylandPerSurfaceScale,WaylandUiScale,AcceleratedVideoDecodeLinuxGL,AcceleratedVideoDecodeLinuxZeroCopyGL,AcceleratedVideoEncoder`)
|
||
remains documented here if a minimal/auditable cmdline is ever wanted.
|
||
**BENCH-VERIFY:** `chrome://policy` shows `ExtensionInstallBlocklist=["*"]` active;
|
||
extensions can't install.
|
||
|
||
## [~] 14. (PARTLY BUILT) Centralised fleet management plane (health/version/usage/maintenance)
|
||
|
||
**SERVER-SIDE CONTROL PLANE BUILT 2026-07 — `serving/fleet-manager/`** (Go service
|
||
+ embedded Svelte SPA). See the **as-built note** at the top of
|
||
[docs/fleet-management-design.md](fleet-management-design.md). What shipped and how
|
||
the design forks resolved:
|
||
- **Two planes:** admin (HTTPS + **WebAuthn/passkey**, YubiKey; server-side session
|
||
+ CSRF; `register-admin` CLI bootstrap; UI cert via **Let's Encrypt DNS-01**,
|
||
hot-reloaded) and agent (**bespoke gRPC bidi/mTLS**, fleet CA, reused TPM cert —
|
||
health/metrics up, desired-state down; no inbound host port).
|
||
- **Metrics → InfluxDB** (manager is the only reader/writer; host holds no Influx
|
||
credential; identity-tagged writes; Flux reads behind a field whitelist + hex-
|
||
serial guard). Graphs render **in the SPA** (bespoke SVG), not Grafana. This
|
||
replaces the earlier Wazuh-Phase-1 / OTLP-Phase-2 sketch for *this* plane;
|
||
Wazuh stays the audit/SIEM path.
|
||
- **Authorise pillar:** the enrol-broker (#11) folded in "alongside" — manager
|
||
drives its admin plane over mTLS and **births the inventory record** on approve.
|
||
- **Deployment:** `Dockerfile` + `docker-compose.yml` (private InfluxDB +
|
||
host-networked manager) + `server-pki.sh` + `DEPLOY.md`.
|
||
- **STILL TO BUILD:** (a) the **on-host agent** — the gRPC report/reconcile client
|
||
that pairs with `fleet-hostapi` (feeds this plane; nothing on the host talks to
|
||
the manager yet); (b) **per-host boot-version assignment** — `target_version` is
|
||
stored + pushed but does not change what a host *boots*; that needs dnsmasq
|
||
cohort steering + multi-version UKI serving (**#6**). Kiosk URL / reboot / drain
|
||
reconcile via the agent once (a) lands. Version pin is labelled "recorded intent"
|
||
in the Lifecycle UI.
|
||
|
||
Depends on the step-ca fleet CA (#8) for the mTLS identity.
|
||
|
||
<details><summary>Original design capture (2026-06-28) — retained for reference</summary>
|
||
|
||
**DESIGN CAPTURED 2026-06-28 — [docs/fleet-management-design.md](fleet-management-design.md).**
|
||
Server-authoritative, pull-based, TPM-identified management plane reusing existing
|
||
pieces: `fleet-hostapi` as the on-host data source, the audit-collector→Wazuh
|
||
pipeline as telemetry transport, the digest-pinned deployment manifest as version
|
||
truth, and the SPDK `spdk-loader`/`spdk-release` rolling-update flow (#6) as the
|
||
redeploy/drain mechanics. Hosts stay stateless (identity from the TPM cert, no
|
||
baked enrollment); control is pull/reconcile (no inbound surface); convergence to
|
||
a new version = reboot. **Phased build:** (1) push hostapi snapshots as structured
|
||
events into Wazuh + fleet dashboard (health/version-drift/offline/usage/alerting);
|
||
(2) metrics push (OTLP/remote-write) → VictoriaMetrics + Grafana; (3) mTLS
|
||
desired-state pull (version pin, reboot/drain, kiosk URL) wired to the deployment
|
||
manifest + SPDK flow. Build-time decisions (telemetry encoding, metrics/desired-
|
||
state store, per-host vs cohort, osquery/FleetDM eval) listed in the design note.
|
||
Depends on the step-ca fleet CA (#8) for the mTLS identity.
|
||
|
||
</details>
|
||
|
||
## [ ] 15. Scope the root overlay → per-subtree tmpfs/overlay (un-shadowable `/usr` + STIG mount options)
|
||
|
||
**DESIGN CAPTURED 2026-07-02.** Today the whole root is one `systemd.volatile=overlay`
|
||
tmpfs over `/` (see #5, invariant on the volatile overlay). Replace it with a
|
||
**read-only verity root + a handful of scoped writable mounts**, for two payoffs
|
||
that land from the same change:
|
||
|
||
1. **Security — un-shadowable `/usr`.** dm-verity seals the backing store, but the
|
||
whole-root overlay lets any root process write a tmpfs *copy* over a system
|
||
binary (e.g. `/usr/sbin/fleet-enrol`) that then executes in preference to the
|
||
sealed one until reboot. fapolicyd is the current compensating control (CLAUDE.md:
|
||
"the writable-overlay exec guard dm-verity can't provide"). Keeping `/usr` (and
|
||
`/`, `/opt`, `/boot`) strictly **read-only** makes the executable set both
|
||
verity-sealed *and* un-shadowable — defence in depth, shrinks fapolicyd reliance.
|
||
Also: writes outside the sanctioned set fail loudly (`EROFS`) instead of silently
|
||
landing in RAM and vanishing at reboot — enforces invariant #1 (no per-host state)
|
||
architecturally.
|
||
|
||
2. **STIG — converts the partition/mount family from `notapplicable`/waived to real
|
||
`pass`.** Quantified against the scan in `reports/stig-arf.xml` (NOT estimates):
|
||
the `mount_option_*` rules currently score **`notapplicable`** (no separate mount
|
||
to check — the "false-clean via notapplicable" pattern #2 warns about), and
|
||
`partition_for_*` are **`notselected`** (waived in `diskless-tailoring.xml`
|
||
lines 61-66). Scoped tmpfs mounts with `noexec,nosuid,nodev` + re-selecting the
|
||
partition rules convert **~21 rules to genuine `pass`**:
|
||
- 15 `mount_option_*` N/A→pass: `/tmp` (nodev,nosuid[,noexec*]), `/var/tmp`
|
||
(all 3), `/var/log` (all 3), `/var/log/audit` (all 3), `/var` (nodev), `/home`
|
||
(nodev,nosuid).
|
||
- 6 `partition_for_*` re-selects→pass: tmp, var, var_log, var_log_audit, var_tmp,
|
||
home (+1 easy: `partition_for_dev_shm` — already a separate tmpfs).
|
||
- Retires **deviation-register entry #1** (the narrative hand-wave over the whole
|
||
partition/mount family) — one fewer waiver to defend at ATO.
|
||
- `/dev/shm`'s three option rules **already pass** — no work there.
|
||
|
||
**Writable-set + flag matrix (the design):**
|
||
- `/` `/usr` `/opt` `/boot` → **read-only** (verity as-is, no overlay). `/usr` keeps
|
||
suid for legit setuid bins (sudo/su/passwd).
|
||
- `/etc` → overlay (lower=verity `/etc`, upper=tmpfs) — needs defaults + writability
|
||
(machine-id, regenerated ssh host keys, NM connections, resolv.conf, chrony).
|
||
- `/var` → overlay (NOT bare tmpfs — bare tmpfs hides `/var/lib/rpm`, selinux store);
|
||
`nodev`. Nest the noexec leaf mounts on top (order: `/var` first, then children).
|
||
- `/var/tmp` `/var/log` `/var/log/audit` → tmpfs `noexec,nosuid,nodev`.
|
||
- `/tmp` → tmpfs `nosuid,nodev` (+`noexec` only after the kiosk repoint, below).
|
||
- `/home` `/root` `/srv` → tmpfs `nodev,nosuid`.
|
||
- Each tmpfs needs the **correct SELinux context** (`tmp_t`, `var_log_t`,
|
||
`auditd_log_t`, …) via `context=` or restorecon — this is the login-deny hazard
|
||
class (#5); a mislabeled tree denies logins / stops auditd.
|
||
|
||
**Prerequisites / caveats that gate the count:**
|
||
- `mount_option_tmp_noexec` (1 of the 21) is **conditional**: register entry #1 notes
|
||
the Chromium kiosk uses `/tmp` for profile/cache and breaks under `noexec`. Must
|
||
**repoint the kiosk scratch** (Chromium `--user-data-dir`/`--disk-cache-dir`) to a
|
||
dedicated exec-capable tmpfs (e.g. `/run/kiosk`) first → ties to #13/#4. So ~20
|
||
land immediately, +1 after the repoint.
|
||
- **fstab-vs-runtime OVAL — mostly de-risked:** the rules return `notapplicable`
|
||
(evaluating the live mountpoint) not `fail` (missing fstab line), evidence they read
|
||
the runtime mount table — so systemd mount units / initramfs mounts should score
|
||
without fstab entries. Confirm per-rule on the bench; add fstab lines if any rule is
|
||
fstab-based (compatible with the empty-fstab invariant — root still comes from
|
||
netroot+verity).
|
||
- **Does NOT fix red findings** (this family is ~0 fail today) — the value is
|
||
evidenced compliance + one fewer deviation + kernel-enforced noexec, not clearing
|
||
fails. Framed honestly so it isn't oversold.
|
||
- **`partition_for_usr` stays waived** — `/usr` should be verity-sealed, not a
|
||
separate writable partition (a *stronger* control than the rule asks). Same for
|
||
`encrypt_partitions`, `auditd_..._large_partition`, boot/boot_efi (genuinely N/A).
|
||
|
||
**Mechanism:** `systemd.volatile=` has no per-dir knob (it's all-or-var-only), so this
|
||
means dropping the global volatile overlay for explicit early mount units (ordered
|
||
before `local-fs.target`) or initramfs mounts. Reproducibility unaffected (runtime
|
||
mounts, not image content → no verity-hash drift).
|
||
|
||
**Load-bearing / bench-gated:** touches the boot path (#5 territory) — full boot-chain
|
||
bench (`serving/dot1x-bench/`) + careful SELinux-label verification + a reconciliation
|
||
note in `design-v2-errata.md`. **Do AFTER TPM enrolment (#11) is bench-verified.**
|
||
Tailoring edits: re-select the 6 `partition_for_*` (+dev_shm), delete/gut register
|
||
entry #1, re-run `scripts/stig-deviations.sh` to confirm the new passes.
|
||
|
||
## [ ] 16. Enable a locked-down host firewall (firewalld) — defense-in-depth + STIG passes
|
||
|
||
**DECISION 2026-07-03: enable a *configured* firewalld.** Currently REMOVED + masked
|
||
(`diskless.pkr.hcl` 2d, commit 4346ce7) because an UNCONFIGURED weak-dep firewalld was
|
||
running and flooding the SIEM with `lib_t` `.pyc` AVCs. The design deferred the firewall
|
||
as "future hardening, currently waived" ([stig-deviations.md](stig-deviations.md) #4) —
|
||
NOT incompatible, just unverified during boot-chain bring-up. A diskless kiosk is an
|
||
ideal firewall candidate: near-zero legit inbound (admin SSH only; `fleet-hostapi` is
|
||
loopback), and the load-bearing paths are outbound (NVMe/TCP, DHCP) or L2 (EAPOL/MACsec).
|
||
|
||
**Payoff:** defense-in-depth (limits a compromised same-segment peer) + converts **4
|
||
STIG waivers → passes** (`service_firewalld_enabled`, `package_firewalld_installed`,
|
||
`set_firewalld_default_zone`, `configured_firewalld_default_deny` — re-select in the
|
||
tailoring; retires deviation-register #4).
|
||
|
||
**Must do together:**
|
||
1. **Config:** default-DENY inbound zone; allow **SSH** (key-only admin) +
|
||
established/related; assign `macsec0` (secure) / `en*` (test) to it. hostapi
|
||
(:9099 loopback) + kiosk→hostapi PNA need no rule.
|
||
2. **Fix the `.pyc` AVC root cause (REQUIRED — an enabled firewalld re-hits it):** the
|
||
reproducible erofs `-T 0` zeroes mtimes, so Python thinks every site-package `.pyc`
|
||
is stale and rewrites it → `firewalld_t` (and ANY Python daemon, e.g. `tuned`)
|
||
write/unlink AVCs on `lib_t`. Fix once with `python3 -m compileall --invalidation-mode
|
||
unchecked-hash` over the Python trees BEFORE the erofs build → mtime-independent
|
||
`.pyc`, no rewrites, and MORE reproducible. (Worth doing regardless of firewalld —
|
||
it's the true root cause of the firewalld AVCs seen 2026-07-03.)
|
||
3. **Tailoring:** re-select the 4 firewalld rules.
|
||
|
||
**THE load-bearing bench gate (why #4 deferred it):** the NVMe/TCP root is an OUTBOUND
|
||
connection to `:4420` established in the initramfs BEFORE firewalld starts. Verify it
|
||
**SURVIVES** firewalld start + `firewall-cmd --reload` (established/related allowed by
|
||
default → should, but UNVERIFIED); a reconnect within `ctrl_loss_tmo` re-establishes
|
||
*through* the firewall; DHCP renew + MKA keepalives unaffected; SSH in; no AVCs; verity
|
||
hash reproducible. If it drops the root, that is the fleet-wide outage risk — gate hard.
|
||
|
||
## [~] 18. Slim the image — locale + firmware (biggest erofs-size wins)
|
||
|
||
**Finding (image-size audit via the 2b `rpm -qa --qf '%{SIZE}'` manifest, 2026-07-03):**
|
||
the largest packages in the 534-package image are NOT the kiosk — they're base bloat:
|
||
`linux-firmware` **678 MB**, `glibc-all-langpacks` **228 MB**, then chromium (370) +
|
||
llvm-libs (129). The uncompressed erofs is read over NVMe/TCP on every boot of every
|
||
host, so every dead byte is fleet-wide waste + attack surface.
|
||
|
||
**[x] DONE — glibc langpacks → en-only** (`diskless.pkr.hcl` step 2). `@^minimal-environment`
|
||
pulls `glibc-all-langpacks` (all languages); swapped for `glibc-minimal-langpack` +
|
||
`glibc-langpack-en`. ~220 MB reclaimed; image is English-only (we also strip
|
||
`/usr/share/locale` in 2d). **BENCH-VERIFY:** locale still `en_US.UTF-8`; kiosk + login OK.
|
||
|
||
**[x] DONE — linux-firmware strip to a hardware-audited keep-list** (`diskless.pkr.hcl`
|
||
step 2, run **BEFORE the dracut build** so the UKI initramfs shrinks too — not just the
|
||
rootfs. `--no-hostonly` + plymouth's `drm` dep instmods a broad DRM set and dracut bundles
|
||
each module's firmware into the initramfs, amdgpu being the classic UKI bloater; the strip
|
||
runs first so dracut never sees it. A post-build diagnostic reports the `.img` size + any
|
||
firmware still bundled — expected i915 only.)
|
||
Live fleet audit (`modinfo -F firmware` over all loaded modules + sysfs/`lspci` driver
|
||
bindings on the real 820-class hosts AND the VMs) proved the **only** loadable firmware
|
||
any host requests is Intel **i915** (GPU). Every other driver is firmware-free —
|
||
crucially **e1000e** (on-chip NVM, not a fw file) and **virtio_net**, so NO NIC needs
|
||
firmware and the fleet-brick case is off the table. Keep-list: `i915` + `intel/` (ice
|
||
DDP insurance for the `ice`/`mlx5` server-NIC build variant + small Intel blobs) +
|
||
`mellanox` + `intel-ucode`; strip the rest (amdgpu/nvidia/radeon/ath/qcom/rtl/iwlwifi…
|
||
~600 MB). Wi-Fi (`iwl*-firmware`, ~83 MB) removed as packages (wired-only fleet).
|
||
Strip-in-place (`find … -exec rm -rf`, not `rpm -e` — can't drop the package without
|
||
losing i915); a `du -sh /usr/lib/firmware` line reports the result in the build log.
|
||
**BENCH-VERIFIED 2026-07-03 (commit `f59eb79`): built, booted, AND the kiosk SPA RENDERS
|
||
on both the VM (virtio-gpu) and the 820 (i915).** This clears the firmware strip's one
|
||
real risk — the i915 DMC/GuC blobs load on real hardware, so the keep-list
|
||
(i915/intel/mellanox/intel-ucode, pre-dracut) is validated end-to-end (firmware 678→147 MB,
|
||
initramfs bundles only the keep-list). Remaining caveat: if a future host has a non-Intel
|
||
GPU or an `ice`/`mlx5` NIC, widen the keep-list before trusting fleet-wide. **NOT yet in
|
||
`f59eb79`** (bench-verify separately — none touch i915, so render should be safe): the
|
||
`nss-mdns`/avahi drop (`5cc72cc`) and the `omit_drivers` GPU-kmod trim + keep-list
|
||
tightening — dropped `mellanox/`, narrowed `intel/` to `ice` only (`853ae20`). Audited
|
||
models: HP-820-class (i915 + e1000e + iwlwifi-dropped) and virtio VMs.
|
||
|
||
**Build-verified 2026-07-03:** firmware **678 MB → 147 MB**; the initramfs diagnostic
|
||
confirms only the keep-list is bundled (i915 all-gens + intel/ice + intel/ish + mellanox)
|
||
— proving the pre-dracut ordering works. But the initramfs came out **97 MB**, and it's
|
||
now **kernel-MODULE-bound, not firmware** (~10 MB fw): the `--no-hostonly` + plymouth
|
||
`drm` set pulls every GPU driver (`amdgpu.ko` etc.) even though the fleet is i915-only.
|
||
|
||
**[x] DONE — follow-on trims (2026-07-03):**
|
||
- **`omit_drivers+=" amdgpu radeon nouveau "`** in `90-diskless.conf` — drops the
|
||
discrete-GPU kmods plymouth's `drm` dep would instmod (amdgpu.ko is the biggest single
|
||
module); keeps i915 + virtio-gpu/bochs for the splash. Targets the 97 MB initramfs/UKI.
|
||
- **Firmware keep-list tightened**: dropped `mellanox/` (those are `mlxsw_spectrum`
|
||
SWITCH-ASIC blobs, NOT NIC fw — mlx5 NIC fw is on-card) and narrowed `intel/` to just
|
||
`ice` (dropped ish/sof/ibt = sensor-hub/audio/bluetooth, none loaded at boot).
|
||
- **Initramfs top-25-files diagnostic** added after the dracut build (sorts `lsinitrd.txt`
|
||
by size) so the next build shows what dominates the `.img` — measure before the next cut.
|
||
- **BENCH-VERIFY:** kiosk still renders on the 820 (i915 kept); check the new
|
||
"initramfs largest files" log line for the post-omit size + next targets.
|
||
|
||
**[x] FIXED — the `nss-mdns` drop bricked the kiosk (regression found+fixed 2026-07-04):**
|
||
the "render should be safe" call above (5cc72cc) was WRONG — it broke the kiosk, just
|
||
not via i915. `trim-rpm-48dc3e1` booted to a **flashing cursor, no kiosk**, and the image
|
||
was a suspicious **500 MB**. Root cause: the 2d `dnf remove -y nss-mdns` removed
|
||
**chromium** as a *dependent package* — the EPEL `chromium` **hard-Requires `nss-mdns`**
|
||
(Chromium uses `.local` mDNS hostnames for WebRTC ICE candidates) — then reaped chromium's
|
||
orphaned GUI stack (avahi, at-spi2, atk, adwaita, chromium-common; ~49 pkgs, 523→472).
|
||
With no browser, `kiosk-session` died at the `${BROWSER:?}` guard in 4 ms, before weston.
|
||
The original commit checked what `nss-mdns` NEEDS (avahi) but not what NEEDS it (chromium).
|
||
GPU/i915/firmware/plymouth were all healthy — a red herring. Fix (`2bc2daf`):
|
||
- **Keep `nss-mdns` + avahi** (satisfy chromium's Requires); **mask the avahi daemon +
|
||
socket** instead — nss-mdns is only an NSS module, needs no listener, and :5353 was the
|
||
real attack-surface concern. ~1.5 MB kept.
|
||
- **FINAL GUARD in 2d**: `rpm -q chromium chromium-common weston mesa-{dri-drivers,libEGL,libgbm}`
|
||
asserts the kiosk stack survived the whole pipeline, else **fail the build**. A missing
|
||
browser must brick the BUILD, never the fleet — catches any future remove-cascade /
|
||
oscap `package_*_removed` / autoremove. Look for `==> guard: kiosk stack intact` in CI.
|
||
**RE-VERIFIED 2026-07-04 (`trim-rpm-2bc2daf`): boots into the kiosk on the 820; image 755 MiB.**
|
||
|
||
**Measured end-to-end (2026-07-04, `trim-rpm-2bc2daf`): container artifact image
|
||
1.8 GiB → 755 MiB (~58%, >1 GB off every host's per-boot NVMe/TCP read).**
|
||
|
||
**Combined:** ~**800 MB** off the uncompressed erofs (langpacks + firmware), dwarfing the
|
||
kiosk weak-deps trim (#4), plus a smaller UKI/initramfs from the DRM-driver omit. NEXT
|
||
candidates from the manifest (measure, don't guess): `iso-codes` (19 MB),
|
||
`samba-client-libs`/`intel-mediasdk`/`gstreamer` (chromium-common HARD deps — only
|
||
removable by dropping HTML5 media, and samba/rabbitmq/rist/srt/zmq are libavformat
|
||
DT_NEEDED so they can't go without breaking future media — confirmed via ldd 2026-07-03),
|
||
`binutils` (25 MB, build-only?), `cracklib-dicts` (9.8 MB, needed for pam pwquality — keep).
|
||
|
||
**Stack note:** committed on `trim-rpm`. When `optimise-ram` rebases onto the new
|
||
`trim-rpm`, expect a trivial `docs/TODO.md` tail conflict (both branches appended after
|
||
#16 — `optimise-ram` added #17 fapolicyd, this added #18); resolve by keeping both.
|