Recent work added telemetry fields and a cert path not yet in the docs:
- files/agent/README.md: add uptime to the health list; document that chrony is
read from /var/log/chrony/tracking.log (not the command socket) and needs BOTH
the chrony supplementary group (DAC) and the fleet_agent_chronylog SELinux allow
(TE), plus a reachable lab NTP (123/udp firewall); note the chrony group on the
unit row.
- serving/fleet-manager/README.md: add uptime + chrony detail to the health list;
note the cert-auto agent-cert-renewer sidecar + GetCertificate hot-reload
(certreload.go) in the bring-up and checkpoint 7.
- docs/TODO.md #14: probe list gains uptime + chrony-from-tracking.log; new dated
block records uptime/reboot-pending, the 4-layer chrony-access chase (all
silent-failing; inventory.json is the isolator), and the agent-plane cert
auto-renewal, all bench-verified 2026-07-10.
- docs/fleet-management-design.md: health record gains uptime + chrony
source/stratum/offset detail.
The tracking.log read was still returning empty under enforcing with NO AVC —
the tell of a DAC (not SELinux) denial. Root cause: /var/log/chrony is mode
0750 chrony:chrony (others have no access), so the agent's DynamicUser (group
tss only) can't even traverse the directory to reach the 0644 tracking.log —
EACCES on open, which produces no audit record. Manager confirmed genuinely on
fleet-manager-1db3977 (chrony-capable), and uptime flowed while chrony didn't,
proving the agent was sending empty chrony (failed read), not a manager gap.
Earlier manual reads worked only as root (sudo) or because fleetadmin had been
usermod'd into chrony during the socket test (reverted on reboot — /etc/group
is on the volatile overlay).
Fix: SupplementaryGroups=tss chrony on the unit — the chrony group has r-x on
the 0750 dir so the agent can traverse it; the log file is world-readable. This
is the DAC half; the fleet_agent_chronylog SELinux module (02a6621) is the TE
half — BOTH are required (verified: group-only -> TE-denied; module-only ->
DAC-denied silently). Corrected the now-stale 'no group' code comment; recorded
the two-layer requirement in the .te and stig-deviations #16. Needs a
workstation image rebuild + reboot.
Bench on comet-beacon-54 proved every command-socket route is blocked for the
hardened agent: chronyd's unix socket is owner-write-only (0755, so even a
chrony-group member can't connect), and STIG closes the UDP command port
(cmdport 0). Rather than re-open a port or expose chrony's privileged socket,
enable chronyd's tracking log and have the agent READ it — no port, no socket,
no group, no SELinux domain transition, full hardening intact.
- chrony.conf: enable 'log tracking' (idempotent) so chronyd appends
source/stratum/offset to /var/log/chrony/tracking.log. Write volume is a ~130B
line per clock update (backing off to ~maxpoll when stable) and capped by the
chrony RPM's logrotate — negligible on the RAM overlay. cmdport 0 stays, so no
STIG deviation.
- agent: chronyTracking() now reads the last data line of tracking.log via
parseTrackingLog() (source f[2], stratum f[3], offset f[6], leap/sync f[7]);
replaces the chronyc exec + parseChrony. Unit test swapped accordingly.
- revert the two abandoned approaches: drop the bindcmdaddress line (394ad10)
and the fleet_agent_chronyc SELinux nnp_transition module + provisioner block
(e6fd4a2, .te deleted) — no chronyc exec means no transition to allow. Removed
stig-deviations #16 and #17 (both waivers no longer needed).
Needs a workstation image rebuild + reboot. BENCH-VERIFY: tracking.log is
world-readable so the DynamicUser agent can read it; under enforcing
unconfined_service_t can read chronyd_var_log_t (add a minimal file-read allow
if denied); and the offset sign matches chronyc on a synced host.
Root cause of the blank chrony fields (found on comet-beacon-54): chrony 4.x
(RHEL 9) defaults its command interface to the UNIX socket only
(/run/chrony/chronyd.sock), usable by root or the chrony group with a writable
/run/chrony. The fleet-agent unit is hardened — DynamicUser (group tss, not
chrony) + ProtectSystem=strict (/run read-only) — so it cannot open the unix
socket nor create chronyc's reply socket. Bench proof: normal-user 'chronyc
tracking' -> 506 Cannot talk to daemon; only sudo worked; 'chronyc -h 127.0.0.1'
failed and ss showed no UDP :323. So the probe always failed and every chrony_*
field stayed empty. SELinux permissive did NOT help — it was never (only) an
SELinux problem; the nnp_transition module (prev commit) is still needed for the
enforcing exec transition.
Fix: bind chronyd's command socket on 127.0.0.1 (bindcmdaddress in chrony.conf,
idempotent in the 2c provisioner) and point the probe at it (chronyc -n -h
127.0.0.1 tracking). UDP needs no fs write or group and rides the unit's
already-allowed AF_INET. Loopback + monitoring-only (state changes still need the
root unix socket), so exposure is minimal. Rejected the unix-socket route (chrony
group + ReadWritePaths=/run/chrony) because /run/chrony is 0750 (group has no
write). Documented in docs/stig-deviations.md #17; .te bench note updated for the
enforcing UDP path. Needs a workstation image rebuild + reboot.
The agent only reported chrony as synced|unsynced — nothing about how well the
clock tracks. Parse `chronyc tracking` for the source-relative metrics.
- proto: HealthSnapshot.chrony_offset_seconds (16, signed system-time offset,
fast=+), chrony_source (17, reference name/IP), chrony_stratum (18); regenerated.
- agent: replace chronyState() with chronyTracking()/parseChrony() — one exec,
parses Leap status + Reference ID + Stratum + System time offset (+ test).
- manager: Health.{ChronyOffset,ChronySource,ChronyStratum}; snapshotToHealth maps
them; influx writes chrony_offset_seconds/_stratum/_source and whitelists the
offset + stratum as graphable fields.
- UI: chronySummary() + offsetMs() — the chrony row now shows
"synced · 192.168.50.10 · s3 · +0.01ms" instead of just "synced".
The manager lets an admin schedule a reboot (desired.reboot_at) but never showed
whether it had happened. Add host uptime to the health snapshot and derive a
reboot status from it.
- proto: HealthSnapshot.uptime_seconds (15); regenerated pb.go (manager + agent copy).
- agent: probe /proc/uptime -> uptime_seconds (with parser + test).
- manager: Health.UptimeSeconds cached in inventory; snapshotToHealth maps it;
influx writes + whitelists uptime_seconds as a queryable field.
- UI: fmt.js dur() + rebootStatus() — boot_time = last_seen - uptime (manager clock
frame, no host/manager sync needed), compared to reboot_at. Monitor host detail
shows an "uptime" row and a "reboot" row: pending in X / overdue by X / rebooted.
Observability-only — works before the reboot ACTUATOR lands (deferred). Once a host
boots after the scheduled time (BMC/manual now, agent later), the badge flips to
"rebooted" automatically.
The agent's baked default pointed at the PUBLIC admin-UI name
(diskless-server.bitasylum.org), which resolves to the serving host's outbound IP
— unreachable from the fleet LAN. There's no split-horizon to "automate": the
manager agent plane is just another *.fleet.lab, served by the fleet dnsmasq like
boot/nvme/audit.
- serving/.env.example: MGMT_HOST=manager.fleet.lab.
- serving/docker-compose.yml: dnsmasq --address=/${MGMT_HOST}/${SERVER_IP}.
- files/agent/agent.env + main.go default: FLEET_AGENT_MANAGER=manager.fleet.lab:9600.
- server-pki.sh: AGENT_SANS default -> manager.fleet.lab (cert SAN matches the
baked name, so no FLEET_AGENT_SERVER_NAME override).
- docs/TODO #14 gotcha #4 updated: fleet name, not the IP+server-name workaround.
The public diskless-server.bitasylum.org stays the ADMIN-UI name (Traefik/:443).
Four fixes from the on-host agent bring-up:
1. Bake the real fleet CA into the image. New `fleet_ca` build var (default =
the committed placeholder) file-provisions the trust anchor to
/etc/dot1x/radius-ca.pem — used by wpa_supplicant (RADIUS) AND the fleet-agent
(verify the manager). CI materializes a FLEET_CA secret to a runner-temp file
and passes -var fleet_ca; unset => placeholder (lab). No CA material committed.
2. server-pki.sh installs the agent CLIENT CA as root+intermediate (not root
only). Host TPM certs are leaf-only in NV and signed by the step-ca
intermediate, so a root-only client CA makes the manager advertise only the
root in its CertificateRequest -> Go clients send no cert ("tls: certificate
required") and it couldn't verify a leaf-only cert anyway. bootstrap.sh now
exports intermediate_ca.crt so FLEET_CA_INTERMEDIATE resolves.
3. New `fleet-manager register-host <serial> [-name] [-hostnqn]` subcommand:
seeds/updates an inventory record for a host enrolled out of band (bench
broker, migrated fleet) through the real store, so its agent stream is
accepted. The manual counterpart to the authorise-approve flow.
4. The agent surfaces the authoritative recv-side gRPC status instead of the
bare io.EOF gRPC returns on the send side when the server ends the stream —
so "host is not enrolled" (PermissionDenied) shows up instead of "EOF".
The 2b-agent provisioner installed agent.env from /tmp/etc-stage, but the 2a
package step deletes /tmp/etc-stage before 2b runs, so the build failed with
"cannot stat '/tmp/etc-stage/fleet-agent/agent.env'". 2b-hostapi avoids this by
installing from its own upload dir; do the same: move agent.env into files/agent/
so it rides the agent's file upload (/tmp/agent), which survives until 2b-agent
cleans it. Runtime path (/etc/fleet-agent/agent.env) is unchanged.
Build the on-host agent that pairs with fleet-hostapi and feeds the
fleet-manager control plane (TODO #14 "STILL TO BUILD (a)"). Nothing on the
workstation talked to the manager before this; Monitor + InfluxDB stayed empty.
Scope: telemetry only. The agent reuses the host TPM identity for mTLS (read-only
tpmkey.LoadSigner at 0x81000100 + cert from NV 0x01800100; the private key never
leaves the TPM), opens the long-lived FleetAgent.Session bidi stream, sends Hello
+ periodic HealthSnapshot (load/cpu/temp/mem/tmpfs/nvme/auth/selinux/chrony/kiosk,
probed read-only from /proc+/sys plus hostapi facts), and LOGS but does not act on
the DesiredState/Command it receives. Reconcile actuators (kiosk URL, reboot,
drain) are deferred until the telemetry half is bench-verified.
- files/agent/: new module (go-tpm + grpc), static CGO_ENABLED=0 like enrol/hostapi.
Copies the enrol tpmkey (+ read-only LoadSigner) and the generated fleet.agent.v1
proto so it stays self-contained without importing the initramfs enrol module.
- files/agent/simtest/: proves the TPM-signed cert completes a real mTLS client-auth
handshake (RequireAndVerifyClientCert) against the go-tpm software simulator.
- files/etc/fleet-agent/agent.env: fleet-generic config (manager DNS name, fleet CA
reused from radius-ca.pem, TPM handle/nvindex, interval). No per-host state.
- fleet-agent.service: hardened unit (tss group for /dev/tpmrm0, INET egress).
- CI builds fleet-agent out-of-VM + runs vet/test + the simulator handshake test.
- Packer: enable_agent flag (default on) + 2b-agent provisioner + fapolicyd trust.
Post-switch-root only, so no dracut/initramfs changes.
- docs: TODO #14 + fleet-management-design as-built updated.
Requires the host already enrolled (broker approve births the inventory row) —
else the manager returns PermissionDenied, the correct "not enrolled yet" signal.
Audiobuzz