Second bench defect after removing the distro 20-chrony-dhcp clobberer
(secure-ntp-5a3eaed, comet-beacon-54): the dispatcher wrote the correct
`server ntp.fleet.lab iburst nts` line, but chrony loaded ZERO sources --
`chronyc reload sources` returned 200 OK yet authdata/sources stayed empty.
Root cause (AVC): chronyd_t was DENIED read on the .sources files because they and
/run/chrony-dhcp itself were labeled generic var_run_t; chronyd_t may only read its
own chronyd_var_run_t. The dispatcher's `mkdir -p` creates the dir as var_run_t and
files inherit it. The distro 20-chrony-dhcp we removed had been restorecon-ing them
to the readable label, so removing it exposed a latent labelling gap.
Stock RHEL policy already maps /run/chrony-dhcp -> chronyd_var_run_t (a restorecon
relabels it and chrony immediately reaches NTS Cook 8). Fix: ship
/etc/tmpfiles.d/chrony-dhcp.conf so systemd-tmpfiles pre-creates the dir with that
fcontext at sysinit, before any NM event -- the dispatcher's files then inherit
chronyd_var_run_t and chronyd can read them. No SELinux allow rules needed
(bench-verified: after the fix a real NM event wrote the source as chronyd_var_run_t,
no AVC, authdata -> NTS Cook 8). Applies in both secure_ntp modes.
Also: the dispatcher now skips `lo` -- it was writing a spurious lo.sources
duplicate of the same server (the distro script had skipped loopback).
BENCH-VERIFY on the next clean build: a normal boot leaves the source file labeled
chronyd_var_run_t (ls -Z), no lo.sources, and authdata reaches NTS/cookies>0 with no
manual step.
NTS was inactive on the clean-build deploy (secure-ntp-d1621e9, comet-beacon-54):
chronyc authdata showed a plain IP source in Mode '-', Cook 0, with no NTS-KE
attempt at all -- chrony was on plain NTP.
Root cause: NetworkManager runs dispatcher scripts from BOTH
/etc/NetworkManager/dispatcher.d and /usr/lib/NetworkManager/dispatcher.d, merged
and sorted by filename. The chrony RPM ships
/usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp, which writes the DHCP
option-42 server to the SAME /run/chrony-dhcp/<iface>.sources file our
10-chrony-dhcp-ntp uses. Sorting after 10-, it ran last and overwrote our
`server ntp.fleet.lab iburst nts` line with `server 192.168.50.10 iburst`, then
reloaded chrony -- last writer wins on the shared filename, silently defeating
authenticated time.
Isolation proof: removing the distro script + hand-writing the NTS source brought
NTS up immediately (ntp.fleet.lab NTS, Cook 8, zero NAKs), confirming the host
image, the fleet-signed NTS-KE cert, DNS, and tcp/4460 were all already correct --
the collision was the sole defect.
Fix: the provisioner removes /usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp
at build (our dispatcher supersedes it in both modes); 20-chrony-onoffline is left
in place (harmless). Documents the finding in the dispatcher header,
docs/secure-ntp-design.md, and the CLAUDE.md secure_ntp note. BENCH-VERIFY on the
next clean build: a normal boot reaches NTS/cookies>0 with no manual step.
Hosts connected with 'nvme connect-all', which attaches EVERY subsystem in the
discovery log. Each host then mounts only its own by-UUID verity namespaces but
leaves a live controller on every other served version's subsystem. Observed via
spdk-release list: more controllers than workstations, each box on every version
(1/3/3/1/3 across 5 versions for ~3 boxes). Beyond wasted connections
(O(hosts x versions)), it defeated the rolling-update drain/reap: no version ever
reached 0 connections while any host was up, so spdk-loader's auto-reap could
never fire and old versions accumulated without bound.
Fix — targeted connect to this build's subsystem only:
- extract-artifacts.sh bakes the per-version SUBNQN (already in verity.json
.subnqn, matched by the serving side) into the signed UKI cmdline as
rd.nvmf.nqn=. NQN is version identity, not per-host state, and the cmdline is
already per-version + immutable-per-signed-UKI, so invariant #1 holds.
- nvmf-connect.sh (test path) uses 'nvme connect -n <nqn>'; legacy UKIs without
rd.nvmf.nqn fall back to connect-all with a warning so they still boot.
- dot1x-macsec-up.sh (secure path) gains an explicit step-6 targeted connect over
macsec0 (after it's addressed, per invariant #2), replacing the delegation to
the dracut nvmf module's discovery connect-all.
Now each host = 1 controller on its own subsystem; versions drain to 0 when their
hosts migrate, so auto-reap works and connections scale O(hosts). Requires a
rebuild + re-sign (cmdline change, invariant #6). BENCH-VERIFY both paths;
confirm the dracut nvmf module isn't also connect-all-ing on the secure path.
CLAUDE.md invariant #2 + TODO #6 updated.
The Dell 820 kept "finding a cert" and bypassing enrolment despite repeated BIOS
TPM clears. tpm2_nvreadpublic revealed 0x01800001 on that hardware is a FIRMWARE
platform index (platformcreate|policydelete|writelocked, 70 bytes — not our
cert; siblings at 0x01800003/4 + 0x01810000). platformcreate survives TPM2_Clear
and policydelete needs the platform's secret policy, so it's permanent and
unusable by us — the blind check false-positived on it, and enrolment could
never write there anyway. Same class of bug as the EK-handle collision.
Move the NV cert index default 0x01800001 -> 0x01800100 (free on the 820, owner
range, overridable via rd.dot1x.nvindex): cmdline (extract-artifacts.sh),
fleet-enrol -nvindex, the dracut hooks, bench scripts, CLAUDE.md invariant #1,
and design/SSP docs. The signing-key handle (0x81000100) is unchanged.
Harden WriteNVCert: refuse a platformcreate index with a clear "pick a free
owner index via rd.dot1x.nvindex" error instead of a confusing undefine
failure. Regression simtest/TestWriteNVCertRefusesPlatformIndex; NV roundtrip +
ReadNVCert tests moved to 0x01800100. All go tests + packer validate pass.
Bench (Proxmox vTPM, host cygnus-cruiser-99) confirmed via tpm2_readpublic that
0x81010001 — the handle our design used for the fleet signing key — is the
TCG-standard RSA Endorsement Key slot: the object there is the swtpm-provisioned
EK (restricted|decrypt|adminwithpolicy, RSA-2048/AES128CFB, canonical EK
authPolicy 837197…, EK cert at NV 0x1c00002). Most firmware/swtpm TPMs occupy
that slot, so EnsureKey could never create our key there — it errored reading
the EK as ECC and enrolment died before the CSR/POST.
Move the fleet signing key to 0x81000100, a non-reserved owner persistent
handle (clear of EK 0x81010001/2 and SRK 0x81000001/2). Threaded through:
- fleet-enrol -handle default + new rd.dot1x.handle= cmdline knob (baked by
extract-artifacts.sh, overridable via SMBIOS)
- files/etc/dot1x/wpa_supplicant-dot1x.conf private_key handle
- the dracut hook, bench tpm-provision.sh / tpm-selftest.sh
- CLAUDE.md invariant #1 + design/SSP docs + TODO #11
NV index 0x01800001 (owner range) is unchanged.
EnsureKey hardening: refuse to evict any restricted-decrypt object (EK/SRK) as
a safety net — never clobber the TPM endorsement/storage identity (also
protects the future EK/AK attestation path). Still self-heals a genuine stale
signing key. New regression TestEnsureKeyRefusesToEvictProtectedKey; the
stale-key regression and handle constants moved to 0x81000100.
Design note records the IDevID (0x81020001) option for the later EK/AK
attestation work. Follow-up items logged: content-aware NV presence check
(verify the cert chains to the fleet CA, so leftover/foreign NV can't suppress
enrolment) and a de-enrol/TPM-reset procedure.
Audiobuzz