Reconciliation workflow (schedule + workflow_dispatch): lists live branches and
every container version across the four images this repo builds (fleet-boot,
fleet-ipxe, fleet-dnsmasq, spdk-nvmf) and deletes dev/branch builds whose branch
no longer exists, via the Gitea Packages API. Not event-driven, so it catches
branches deleted any way and never depends on a 'delete' webhook.
Safety (per request — never touch release tags or latest):
- Only tags ending in the dev commit-sha suffix '-<7 hex>' are ever candidates.
Release/version tags (1.2.3, v1.2.3, spdk v24.09, dnsmasq main-<sha>) and
'latest' never carry that suffix, so they're structurally out of scope; there's
also an explicit 'latest' skip. Verified by simulation across representative
tags incl. spdk's <ver>-<branch>-<sha> form and pre-release semver.
- Keeps any image whose sanitized branch is still live (+ always 'main'); branch
sanitization mirrors the build workflows.
- Fail-safe: aborts before deleting anything if the live-branch list can't be
fetched (0 branches), so an API/auth blip can't trigger mass deletion.
- Conservative matching: a suffix collision only ever causes a KEEP.
DRY RUN by default: workflow_dispatch previews unless dry_run=false; scheduled
runs dry-run until armed via repo variable REGISTRY_GC_ENABLE=true (so a
scheduled dry-run can be eyeballed first). Uses the existing PACKAGE_TOKEN
(write:package). shell: bash (runner defaults to sh).
The build failed: checkmodule rejected `policy_module(chrony_dhcp_dispatcher,
1.0.0)` with "Building a policy module, but no module specification found".
policy_module() is a refpolicy m4 macro expanded only by the refpolicy Makefile
(/usr/share/selinux/devel), not by checkmodule. Use the raw module-policy
declaration `module chrony_dhcp_dispatcher 1.0.0;`, which is exactly what
checkmodule -M -m parses. require{} + allow rule unchanged.
The Dell 820 kept "finding a cert" and bypassing enrolment despite repeated BIOS
TPM clears. tpm2_nvreadpublic revealed 0x01800001 on that hardware is a FIRMWARE
platform index (platformcreate|policydelete|writelocked, 70 bytes — not our
cert; siblings at 0x01800003/4 + 0x01810000). platformcreate survives TPM2_Clear
and policydelete needs the platform's secret policy, so it's permanent and
unusable by us — the blind check false-positived on it, and enrolment could
never write there anyway. Same class of bug as the EK-handle collision.
Move the NV cert index default 0x01800001 -> 0x01800100 (free on the 820, owner
range, overridable via rd.dot1x.nvindex): cmdline (extract-artifacts.sh),
fleet-enrol -nvindex, the dracut hooks, bench scripts, CLAUDE.md invariant #1,
and design/SSP docs. The signing-key handle (0x81000100) is unchanged.
Harden WriteNVCert: refuse a platformcreate index with a clear "pick a free
owner index via rd.dot1x.nvindex" error instead of a confusing undefine
failure. Regression simtest/TestWriteNVCertRefusesPlatformIndex; NV roundtrip +
ReadNVCert tests moved to 0x01800100. All go tests + packer validate pass.
Captures the decision to re-enable a CONFIGURED firewalld (currently removed),
the required .pyc/-T0 root-cause fix (compileall --unchecked-hash), the 4 STIG
waivers it converts, and the load-bearing NVMe/TCP-survives-firewalld bench gate.
http/ks.cfg excludes firewalld (-firewalld) and sets firewall --disabled — the
fleet's security boundary is the network (802.1X/MACsec + switch); the host
firewall is a documented OFF item (stig-deviations #4). But a provisioner dnf
pulled firewalld back as a weak dep (Recommends) and it starts by default. Its
Python then tries to rewrite site-package .pyc bytecode, which the reproducible
erofs -T 0 mtime reset makes look stale, so firewalld_t gets write/unlink AVCs
on lib_t that flood the SIEM.
Remove firewalld in 2d hygiene (weak dep, nothing hard-needs it) and mask it as
a backstop against future weak-dep reinstalls. No firewalld => no daemon, no
AVC, matches the kickstart intent.
The NM dispatcher 10-chrony-dhcp-ntp runs `chronyc reload sources` to load DHCP
option-42 NTP servers. That chronyc runs in the confined custom-dispatcher
domain (NetworkManager_dispatcher_custom_t), so its reply socket carries that
label and the shipped policy has no rule for chronyd_t to sendto it. Under
enforcing, chronyd does the reload but its ack is denied — chronyc reports
failure and the AVC floods the SIEM (seen 2026-07-03; permissive=1 only because
the box was booted enforcing=0).
Ship a minimal local allow module (files/stig/chrony-dhcp-dispatcher.te):
allow chronyd_t NetworkManager_dispatcher_custom_t:unix_dgram_socket sendto;
Built + semodule -i'd in the 2c-STIG provisioner (checkpolicy installed if
absent), with an install assertion. `allow` not `dontaudit` — it also lets the
reload ack succeed so chronyc returns cleanly. Documented as
docs/stig-deviations.md #14 (not a rule waiver — a local functional allow).
BENCH-VERIFY: enforcing boot shows no AVC, chronyc reload returns 0, a
DHCP-handed NTP server appears in `chronyc sources`; and re-check verity-hash
reproducibility (semodule rebuilds the policy store, like the semanage step).
The nvmeof (spdk-target) service carried a versioned bootstrap mount
(type: image, source ${OCI_IMAGE}:${IMAGE_VERSION} -> /artifacts) plus an
IMAGE_VERSION env. Bumping IMAGE_VERSION therefore changed the service config,
so `docker compose up -d` recreated the container -> nvmf_tgt restarted ->
every connected host's NVMe/TCP root dropped at once. That is precisely the
fleet-wide outage the long-lived-target design exists to prevent.
Remove the /artifacts image-volume and IMAGE_VERSION env from nvmeof so its
config no longer depends on the version — compose leaves it running across a
bump. Bootstrap and rolling updates are already fully covered by spdk-loader
(stages the current release into ./spdk/releases and adds it online via
JSON-RPC); spdk-target.sh's seed block is guarded, so it just no-ops when
/artifacts isn't mounted. Verified nvmeof no longer references IMAGE_VERSION
(only spdk-loader does); compose YAML validates.
BENCH-VERIFY: bump IMAGE_VERSION + up -d with a host connected -> nvmeof
container not recreated, host root survives, loader adds the new version.
The enrol trigger keyed on "is NV 0x01800001 defined/readable", so leftover
state (an earlier tpm-provision run — bench-seen on the 820: "cert present —
normal boot" with no real identity) or random bytes (tpm-selftest) silently
suppressed enrolment. Now content-aware:
- tpmkey.ReadNVCert reads the full NV contents (chunked).
- classifyNVCert (TPM-agnostic, unit-tested): parses X.509, checks validity
window, and when a real fleet CA bundle is present verifies the cert chains
to it. Only a valid fleet cert = enrolled.
- New `fleet-enrol -check-only` (no network/no key-gen) exits 0=valid cert,
20=enrol needed, 1=error. The full enrol path uses the same classification
instead of NVPresent.
- The dracut hook calls -check-only as the single gate, replacing the blind
tpm2_nvread (and its now-dead TPM2TOOLS_TCTI export). tpm2_nvread stays in
the initramfs for the secure path (dot1x-macsec-up.sh).
Lab fallback: the shipped radius-ca.pem is a placeholder (not a cert) ->
loadCAPool returns nil -> chain-less "parseable + unexpired" acceptance, so
reboot-converge doesn't loop; production replaces radius-ca.pem with the real
fleet CA for strict chaining.
Tests: classifyNVCert table (valid/expired/garbage/foreign/no-CA) + simtest
ReadNVCert roundtrip across the 512B chunk. All pass; packer validate clean.
Bench (Proxmox vTPM, host cygnus-cruiser-99) confirmed via tpm2_readpublic that
0x81010001 — the handle our design used for the fleet signing key — is the
TCG-standard RSA Endorsement Key slot: the object there is the swtpm-provisioned
EK (restricted|decrypt|adminwithpolicy, RSA-2048/AES128CFB, canonical EK
authPolicy 837197…, EK cert at NV 0x1c00002). Most firmware/swtpm TPMs occupy
that slot, so EnsureKey could never create our key there — it errored reading
the EK as ECC and enrolment died before the CSR/POST.
Move the fleet signing key to 0x81000100, a non-reserved owner persistent
handle (clear of EK 0x81010001/2 and SRK 0x81000001/2). Threaded through:
- fleet-enrol -handle default + new rd.dot1x.handle= cmdline knob (baked by
extract-artifacts.sh, overridable via SMBIOS)
- files/etc/dot1x/wpa_supplicant-dot1x.conf private_key handle
- the dracut hook, bench tpm-provision.sh / tpm-selftest.sh
- CLAUDE.md invariant #1 + design/SSP docs + TODO #11
NV index 0x01800001 (owner range) is unchanged.
EnsureKey hardening: refuse to evict any restricted-decrypt object (EK/SRK) as
a safety net — never clobber the TPM endorsement/storage identity (also
protects the future EK/AK attestation path). Still self-heals a genuine stale
signing key. New regression TestEnsureKeyRefusesToEvictProtectedKey; the
stale-key regression and handle constants moved to 0x81000100.
Design note records the IDevID (0x81020001) option for the later EK/AK
attestation work. Follow-up items logged: content-aware NV presence check
(verify the cert chains to the fleet CA, so leftover/foreign NV can't suppress
enrolment) and a de-enrol/TPM-reset procedure.
Bench boot failed with "fleet-enrol: ensure key at 0x81010001: key at handle
is not ECC: did not contain eccDetail (selector value was 1)": a real host had
an RSA key persisted at 0x81010001 (left by an older tpm2-tools/openssl
provisioning path), and EnsureKey fatally errored trying to read it as our ECC
signing key — so enrolment died before the CSR/POST and the host fell through
to a normal boot (no fingerprint, nothing at the broker).
EnsureKey now evicts a wrong-type/unusable key at the handle and creates a
fresh ECC key. Safe on the enrol path: it only runs on an empty-NV host, so no
issued cert references the old key. A valid ECC key is still reused (idempotent
retried enrol).
Regression: simtest TestEnsureKeyEvictsStaleNonECC pre-plants an RSA key and
asserts EnsureKey recreates a usable ECC signer (passes against the go-tpm
simulator).
Captures the design + quantified STIG payoff (~21 mount/partition rules
notapplicable/waived -> pass, per reports/stig-arf.xml), the writable-set +
noexec/nosuid/nodev flag matrix, the kiosk-/tmp-noexec prerequisite, the
fstab-vs-runtime OVAL note, and that it's bench-gated after TPM enrolment (#11).
getfattr --only-values emits the NUL-terminated security.selinux value;
bash's $(...) then warns "ignored null byte in input". Pipe through
tr -d '\0' to silence the cosmetic warning. The label assertion above is
unaffected.
The out-of-VM Go build step used `set -euo pipefail`, but the self-hosted
Gitea/act runner runs `run:` steps under sh (dash), which rejects
`-o pipefail` ("Illegal option"). Add `shell: bash` to the step. Keeping
`set -e` is load-bearing: without it a failed go build/test in the first
subshell group is masked by the trailing `file` command's success, letting
a broken binary reach the Packer build.
Removes the hard no-op on the test path so the enrolment flow can be exercised
on a host with a TPM + a reachable broker WITHOUT standing up 802.1X/MACsec/
RADIUS. Networking is now mode-aware: in test mode ip=dhcp already owns the NIC
(and wrote resolv.conf), so the hook uses the existing lease instead of running
its own nmcli DHCP — avoiding the dracut "address already assigned" race
(CLAUDE.md "don't double-configure the NIC in test mode"). Secure mode still does
the nm-unmanaged override for the pre-auth lease. rd.fleet.enrol=0 disables
enrolment in either mode; production intent stays secure-path-only.
Replaces the shell/openssl-provider host enrolment (a35f2d6) with a static
Go + go-tpm binary, and moves BOTH host Go binaries out of the Packer VM.
fleet-enrol (files/enrol/, Go + go-tpm, no CGO):
- talks to /dev/tpmrm0 DIRECTLY via transport.FromReadWriter(os.File) — no
linuxtpm, so go-tpm is the only dep and there is NO tpm2-tools key-gen /
openssl tpm2 provider / curl in the image (removes the biggest bench risk,
the provider handle: key-ref).
- ensures the ECDSA key @0x81010001, builds the CSR (crypto.Signer over the
TPM key), POSTs CSR+context to the broker, shows the Plymouth fingerprint,
polls /status, writes the leaf cert to NV 0x01800001.
- tpmkey/ is an exported package so the TPM ceremony is tested against the
go-tpm software simulator from a SEPARATE simtest/ module (CGO stays out of
the shipped build). Transaction logic (mock broker) tested in-package.
- deps pinned by go.sum (not vendored): the binary now builds on the CI
runner, which has network — reproducible via pinned Go 1.26.4 + go.sum.
Out-of-VM build (decision): both fleet-enrol AND fleet-hostapi compile in the
CI "Build host Go binaries" step (Go 1.26.4, CGO off, -trimpath) and Packer
FILE-PROVISIONS the static binaries in. Dropped the in-VM `dnf install golang`
+ the 2d removal — the Rocky build image carries no Go toolchain / go-tpm tree.
Wiring:
- tpm-enrol.sh: now a thin GATE (empty-NV check, pre-auth DHCP on en*,
exec /usr/sbin/fleet-enrol, reboot-to-converge). module-setup.sh bundles the
binary (asserted) + reverts the tpm2-openssl/curl installs.
- diskless.pkr.hcl: file provisioners for both binaries; enrol installed to
/usr/sbin before the initramfs build + fapolicyd-trusted; lsinitrd asserts
fleet-enrol; dnf reverted to baseline (no tpm2-openssl/curl).
- gitea-main-ci.yaml: Set up Go + build both binaries + run the simulator
ceremony test, before packer validate.
TESTED here: transaction unit tests + the full TPM ceremony (key-gen -> persist
-> ECDSA sign -> CSR verify -> chunked NV write -> re-enrol) against the go-tpm
simulator. BENCH-VERIFY (real/swtpm, empty owner auth, pre-auth DHCP, Plymouth,
reboot-converge, prod HTTPS) tracked in docs/TODO.md #11.
Builds the host side of TPM identity enrolment — the "inline single-UKI"
vehicle from docs/tpm-enrolment-design.md. A production (secure) UKI now
auto-enrols a fresh host with no client cert, then reboots into the normal
secure path.
New dracut hook files/dracut/95dot1x-macsec/tpm-enrol.sh (initqueue/settled
prio 25, BEFORE dot1x-macsec-up at 30):
- no-op when a cert is already present in NV 0x01800001 (normal boot) or
under rd.dot1x=0 (test path has no trustworthy identity to enrol);
- on EMPTY NV: generate the signing key in the TPM @0x81010001 (openssl tpm2
provider builds the CSR; key never leaves the TPM), pre-auth DHCP on the
PHYSICAL NIC (nmcli re-manages en*, overriding the secure-mode unmanaged
drop-in), POST CSR+context to rd.fleet.enrol_url, show the broker
fingerprint on Plymouth, poll /status, write the issued leaf cert (DER) to
NV, and reboot to converge.
- Empty-NV and the secure path are mutually exclusive, so this never reorders
the load-bearing boot sequence (invariant #2).
Wiring:
- module-setup.sh: require+install the tpm2 key-gen/NV tools, curl, awk, and
the openssl tpm2 PROVIDER (ossl-modules/tpm2.so, asserted present — tpm2-tss
alone can't sign with a handle: key); install the hook.
- diskless.pkr.hcl: install tpm2-openssl + curl; assert hook/tools/provider in
the initramfs (invariant #9). New enrol_url var threaded to extract-artifacts.
- extract-artifacts.sh: bake rd.fleet.enrol_url into the cmdline.
- fleet.script: enrol-active/enrol-wait/enrol-done banner; fingerprint on the
message line (public — states only, invariant #1).
Shell JSON build/parse/cert-extract pipeline (no jq in the initramfs) verified
end-to-end against a live broker. BENCH-VERIFY list in docs/TODO.md #11 (tpm2
provider key-ref on real/swtpm, empty-NV detect, pre-auth DHCP override, Plymouth
render, NV write + reboot convergence, prod HTTPS).
The `step` CLI ran against --ca-url https://ca.fleet.lab:8443, but that name is
only served by the fleet dnsmasq to booting clients — the serving host's own
resolver can't resolve it ("no such host on 127.0.0.53"), so provisioner list +
RADIUS cert issuance failed. Connect over CA_CONNECT (default localhost, which
the compose puts in the CA cert SAN); CA_DNS stays the cert/SAN + RADIUS CN name.
Verifying the step signer against a real step-ca surfaced two real bugs the
mock signer had hidden, plus made the bench CA actually initialise:
- signer.go: move --not-after from `step ca token` (validity caps ~1h ->
"token validity too long") to `step ca sign` (it is the CERT lifetime).
Without this the step signer never issues a cert.
- dot1x-bench compose: DOCKER_STEPCA_INIT_PASSWORD_FILE pointed at a file that
does not exist at first init (it is an INPUT path), so `step ca init` failed
and never wrote secrets/password. Use DOCKER_STEPCA_INIT_PASSWORD (the value)
which the entrypoint writes to secrets/password, where bootstrap + broker read.
- bootstrap.sh: self-heal the root-owned ./pki Docker auto-creates; detect the
CA password at $STEPPATH/password or secrets/password (varies by image
version); raise authority.claims.maxTLSCertDuration so 720h fleet certs are
allowed (stock cap 24h, which also blocked bootstrap's own RADIUS cert); fix
RADIUS cert paths to write via the /export mount (were host paths run inside
the container, so they never resolved).
- serving stack: add the enrol-broker service (temporary home; nginx already
proxies /enrol/) defaulting to the zero-dependency lab path (mock signer +
insecure loopback admin), with ENROL_* knobs to switch to step + mTLS.
- verify-serving.sh: exercise request -> approve -> cert through the deployed
nginx /enrol/ proxy.
- TODO #11: record the step-signer verification + the two fixes.
Step signer VERIFIED end-to-end against a real step-ca: issued cert chains to
the fleet root, EKU clientAuth, hostnqn derived from serial.
audiobuzz