secure-ntp: remove the chrony pkg NM dispatcher that clobbered the NTS source

NTS was inactive on the clean-build deploy (secure-ntp-d1621e9, comet-beacon-54): chronyc authdata showed a plain IP source in Mode '-', Cook 0, with no NTS-KE attempt at all -- chrony was on plain NTP. Root cause: NetworkManager runs dispatcher scripts from BOTH /etc/NetworkManager/dispatcher.d and /usr/lib/NetworkManager/dispatcher.d, merged and sorted by filename. The chrony RPM ships /usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp, which writes the DHCP option-42 server to the SAME /run/chrony-dhcp/<iface>.sources file our 10-chrony-dhcp-ntp uses. Sorting after 10-, it ran last and overwrote our `server ntp.fleet.lab iburst nts` line with `server 192.168.50.10 iburst`, then reloaded chrony -- last writer wins on the shared filename, silently defeating authenticated time. Isolation proof: removing the distro script + hand-writing the NTS source brought NTS up immediately (ntp.fleet.lab NTS, Cook 8, zero NAKs), confirming the host image, the fleet-signed NTS-KE cert, DNS, and tcp/4460 were all already correct -- the collision was the sole defect. Fix: the provisioner removes /usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp at build (our dispatcher supersedes it in both modes); 20-chrony-onoffline is left in place (harmless). Documents the finding in the dispatcher header, docs/secure-ntp-design.md, and the CLAUDE.md secure_ntp note. BENCH-VERIFY on the next clean build: a normal boot reaches NTS/cookies>0 with no manual step.
2026-07-11 14:02:40 +00:00
parent d1621e9b74
commit 5a3eaedec5
4 changed files with 55 additions and 3 deletions
@@ -276,7 +276,13 @@ instead of working around it.
  (`serving/chrony/nts-cert.sh`), and **tcp/4460** (NTS-KE) open + udp/123. Confirm
  on a booted host: `chronyc -N authdata` shows the source in `NTS` mode, cookies
  > 0. `nocerttimecheck 1` lets a cold boot clock (no RTC) bootstrap the first sync.
-  **Toggle:** `secure_ntp=false` reverts the dispatcher to the legacy
+  **Bench finding (2026-07-11, FIXED):** the chrony RPM ships its OWN NM dispatcher
+  `/usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp`; NM merges `/etc` + `/usr/lib`
+  dispatchers by name, so it ran AFTER our `10-chrony-dhcp-ntp` and overwrote the NTS
+  line in the shared `/run/chrony-dhcp/<iface>.sources` with the plain option-42
+  server — silently defeating NTS (authdata Mode `-`, Cook 0). The provisioner now
+  `rm`s it (NTS then reached Cook 8 on the bench). Don't re-add a competing writer to
+  that file. **Toggle:** `secure_ntp=false` reverts the dispatcher to the legacy
  unauthenticated **DHCP option 42** path (plain `server <ip> iburst`).
 - `fleet` Plymouth theme: confirm (a) the script plugin package actually
  installed (`/usr/lib64/plymouth/script.so` — else the build fell back to
@@ -448,6 +448,15 @@ build {
      "if [ \"${var.secure_ntp}\" = \"true\" ]; then NTP_MODE=nts; else NTP_MODE=dhcp; fi",
      "sed -i \"s/@NTP_MODE@/$NTP_MODE/; s/@NTP_PREFIX@/${var.ntp_prefix}/\" /etc/NetworkManager/dispatcher.d/10-chrony-dhcp-ntp",
      "restorecon /etc/NetworkManager/dispatcher.d/10-chrony-dhcp-ntp",
+      # Remove the chrony package's OWN NM dispatcher. NetworkManager merges
+      # dispatchers from /etc AND /usr/lib sorted by name, so /usr/lib/.../20-chrony-dhcp
+      # runs AFTER our 10-chrony-dhcp-ntp and writes the DHCP option-42 server to the
+      # SAME /run/chrony-dhcp/<iface>.sources — clobbering our NTS line and silently
+      # defeating authenticated time (bench 2026-07-11, comet-beacon-54: chrony fell
+      # back to plain NTP, authdata Mode '-', Cook 0; removing it → NTS Cook 8). Our
+      # dispatcher supersedes it in BOTH modes, so drop it unconditionally. Leave
+      # 20-chrony-onoffline (only marks sources online/offline — harmless).
+      "rm -f /usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp",
      "sed -i -E 's/^[[:space:]]*(pool|server)[[:space:]]/#&/' /etc/chrony.conf",
      "grep -q '^sourcedir /run/chrony-dhcp' /etc/chrony.conf || echo 'sourcedir /run/chrony-dhcp' >> /etc/chrony.conf",
      "if [ \"${var.secure_ntp}\" = \"true\" ]; then echo '==> secure NTP: NTS via <${var.ntp_prefix}>.<dhcp-domain> (fleet-CA-authenticated)'; { printf '\\n# Secure NTP (NTS, RFC 8915) — verify the server against the fleet CA. docs/secure-ntp-design.md\\n'; printf 'ntstrustedcerts /etc/dot1x/radius-ca.pem\\n'; printf 'nocerttimecheck 1\\n'; } >> /etc/chrony.conf; else echo '==> legacy NTP: unauthenticated DHCP option-42'; fi",
@@ -1,7 +1,9 @@
 # Secure NTP design — CA-authenticated time via NTS (F-03 / P-18)

-> Status: **implemented (host + serving scaffold), BENCH-VERIFY.** Branch
-> `secure-ntp`. Closes SAR finding **F-03** / POA&M **P-18**.
+> Status: **NTS proven end-to-end on the metal (2026-07-11, comet-beacon-54:
+> `chronyc authdata` → NTS, Cook 8), pending a clean-build re-verify of the
+> dispatcher-collision fix.** Branch `secure-ntp`. Closes SAR finding **F-03** /
+> POA&M **P-18**. See "Bench findings" below.

 ## Problem

@@ -120,6 +122,35 @@ and is the standard chrony guidance for RTC-less devices.
 7. STIG: confirm remediation doesn't strip `ntstrustedcerts`/`nocerttimecheck` or
   the `nts` option in the sourcedir (host is a client; STIG-compatible).

+## Bench findings
+
+**2026-07-11 (comet-beacon-54, `secure-ntp-d1621e9`) — the chrony package's own NM
+dispatcher clobbered the NTS source (FIXED).** On the first clean-build deploy NTS
+was inactive: `chronyc -N authdata` showed the source as a plain IP (`192.168.50.10`)
+in Mode `-`, Cook 0, and the journal showed **no NTS-KE attempt at all** — chrony was
+on plain NTP. Root cause: NetworkManager runs dispatcher scripts from **both**
+`/etc/NetworkManager/dispatcher.d/` and `/usr/lib/NetworkManager/dispatcher.d/`,
+merged and sorted by filename. The `chrony` RPM ships
+`/usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp`, which writes the DHCP
+option-42 server to the **same** `/run/chrony-dhcp/<iface>.sources` file our
+`10-chrony-dhcp-ntp` uses. Because `20-` sorts after `10-`, it ran last and
+**overwrote our `server ntp.fleet.lab iburst nts` line with `server 192.168.50.10
+iburst`**, then reloaded chrony — silently defeating authenticated time (last writer
+wins on a shared filename).
+
+Proof + isolation: removing the distro script and hand-writing the NTS source
+(`printf 'server ntp.fleet.lab iburst nts\n' > …`) + `chronyc reload sources`
+immediately brought NTS up — `ntp.fleet.lab NTS … Cook 8`, zero NAKs — confirming
+the host image, the fleet-signed NTS-KE cert, DNS (`ntp.fleet.lab`→serving host),
+and tcp/4460 were **all already correct**; the collision was the sole defect.
+
+Fix: the provisioner removes `/usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp`
+at build time (our dispatcher supersedes it in both modes); `20-chrony-onoffline`
+is left in place (harmless — it only marks sources online/offline). **BENCH-VERIFY
+on the next clean build:** with the distro script gone, a normal boot leaves
+`/run/chrony-dhcp/<iface>.sources` = the NTS line and `chronyc authdata` reaches
+`NTS`/cookies > 0 with no manual intervention.
+
 ## Toggle

 `secure_ntp=false` reverts the dispatcher to the legacy unauthenticated DHCP
@@ -22,6 +22,12 @@
 #
 # Installed 0755 root:root. Lenient on purpose: a hiccup here must never break
 # networking. @NTP_MODE@/@NTP_PREFIX@ are substituted at build time.
+#
+# NB: the chrony package ships its OWN dispatcher at
+# /usr/lib/NetworkManager/dispatcher.d/20-chrony-dhcp writing option-42 servers to
+# the SAME <iface>.sources. NM merges /etc + /usr/lib sorted by name, so it runs
+# AFTER us and would clobber the NTS line — the provisioner removes it (it defeated
+# NTS on the bench: chrony fell back to plain NTP). This script supersedes it.

 MODE="@NTP_MODE@"      # nts | dhcp
 PREFIX="@NTP_PREFIX@"  # nts: server name = <PREFIX>.<dhcp-domain>