Audit / log collector (receive side)

An rsyslog container that receives logs forwarded by booted diskless hosts and writes them to durable, per-source-host files. Diskless hosts run on an immutable erofs root with a RAM-only volatile overlay, so their local /var/log/audit/audit.log is lost on every reboot — this gives auditd somewhere permanent to offload to. It implements the remote-audit-offload of STIG deviation #6 (docs/stig-deviations.md); since 2026-07-12 the hop is syslog-over-TLS and the rsyslog_encrypt_offload_* STIG rules are enforced (no longer waived).

booted host                                serving host
┌────────────────────────────┐            ┌──────────────────────────────┐
│ auditd → audisp-syslog      │  TLS/6514  │ audit (this container)        │
│      → rsyslog (omfwd gtls) ├───────────►│ rsyslogd (gtls, fleet-CA      │
│         x509/name vs the    │   syslog   │  server cert) →               │
│         baked fleet CA      │            │  /var/log/remote/             │
└────────────────────────────┘            │   <sender-ip>/audit.log       │
                (legacy plaintext :514     │   <sender-ip>/messages.log    │
                 kept for pre-TLS images)  └──────────────────────────────┘

What this provides (the receiver)

Listens on TCP/6514, syslog-over-TLS (rsyslog gtls driver). The server cert is issued from the fleet step-ca (below); senders verify it x509/name against the fleet CA baked in the image (/etc/dot1x/radius-ca.pem). Clients are not cert-authenticated (authMode="anon"): hosts are fleet-generic (no per-host state, invariant #1) and a baked fleet-wide client key would be shared secret material — sender access is constrained by the isolated boot LAN + the firewall below.
Still listens on TCP/514 plaintext as a legacy/migration intake for already-released pre-TLS images. Disable it (comment the input out + close the firewall port) once the fleet is on TLS-sending images.
Disk-assisted queue so bursts / brief stalls don't drop records.
Writes per sender IP (not the claimed hostname): audit records → audit.log, everything else → messages.log, under ./logs/<ip>/ (bind-mounted, inspectable on the host).
Host-networked, so hosts reach it at ${AUDIT_HOST} (resolved to the serving host by the stack's dnsmasq), matching the boot/NVMe DNS pattern.
Relays a copy of audit records to the Wazuh SIEM node (see rsyslog.conf) — that hop is still plaintext (tracked in docs/TODO.md; it must stay ptcp until wazuh1 has a TLS listener).

Decision: no client mTLS with the TPM identity (2026-07-12, considered + deferred)

Hosts already carry a fleet-CA-signed TPM identity (key at 0x81000100, usable for mTLS — the fleet-agent uses it). We deliberately do not use it to client-authenticate the log stream:

Low marginal gain. On the production/secure path the collector hop rides a LAN already gated by 802.1X EAP-TLS with the exact same TPM identity + MACsec — client mTLS at layer 7 would re-prove a credential the network proved at layer 2. It also doesn't close the residual threat: a compromised-but-enrolled host has a valid cert and can still inject records about itself. And rsyslog can validate a client cert but can't template on its identity, so per-host files would still be keyed by sender IP — no attribution upgrade. The STIG rsyslog_encrypt_offload_* rules require server-auth TLS only; no compliance delta.
Disproportionate cost. The TPM key is non-exportable by design and rsyslog's gtls driver only takes PEM key files (no PKCS#11 URI), so this needs either a boot-time delegated cert (online CA round-trip ahead of log offload; key lands in tmpfs anyway — "TPM-held" degrades to "TPM-issued") or a PKCS#11-capable fronting daemon (ghostunnel etc.: new non-RPM daemon in the immutable image + fapolicyd/SELinux/reproducibility surface).
Failure-mode asymmetry. Offload is the forensic safety net; hard-failing it on client-auth problems makes a host with a broken TPM/cert — exactly the host whose logs matter most — go dark (RAM-backed spool, lost on reboot).

Revisit trigger: if cryptographically-attributed log transport becomes a hard requirement (ISM/SSP), route logs through the fleet-agent's existing TPM-mTLS channel (go-tpm, no PKCS#11 gymnastics) instead of teaching rsyslog client certs. Until then: authMode="anon" + boot-LAN firewall scoping (+ $AllowedSender as cheap depth), with 802.1X/MACsec as the sender-authentication layer.

TLS server cert (required before first start)

Issue it from the fleet CA (the dot1x-bench step-ca) into ./tls/ (gitignored — key material never enters the repo):

cd serving/audit-collector
./issue-cert.sh          # SAN = ${AUDIT_HOST} from serving/.env; writes tls/{server.crt,server.key,ca.crt}

This MUST be the same CA whose root the image was built with (var.fleet_ca → /etc/dot1x/radius-ca.pem) or every sender rejects the handshake and spools. Certs expire (default 720h): re-run issue-cert.sh and docker compose up -d audit to renew. Expiry symptom: collector logs TLS handshake errors, sender dirs stop growing, hosts spool (action.resumeRetryCount="-1" — records drain after renewal, not lost).

Run it with the rest of the stack:

cd serving
docker compose build audit
docker compose up -d audit
docker compose logs -f audit          # startup + connection messages
tail -F audit-collector/logs/*/audit.log

Quick self-tests:

# TLS handshake + cert chain (expect Verify return code: 0 against the fleet CA)
openssl s_client -connect 127.0.0.1:6514 -CAfile audit-collector/tls/ca.crt </dev/null

# End-to-end record over the LEGACY plaintext intake (still simplest smoke test)
logger -n 127.0.0.1 -P 514 -T -t audispd "test audit record"
ls audit-collector/logs/                 # a 127.0.0.1/ dir appears

Serving-host firewall — open 6514/tcp (easy to miss)

This container uses host networking, so the serving host's own firewall governs port 6514 (and legacy 514) — Docker does not manage iptables for host-net containers (the usual "Docker bypasses the host firewall" behaviour only applies to published bridge ports, which insert rules into the DOCKER chain ahead of the host firewall). The symptom when it's closed is deceptive: the container is healthy, ss -tlnp shows 0.0.0.0:6514 listening, DNS/DHCP from the same host work, but booted hosts sit in SYN-SENT and rsyslog logs cannot connect to audit.fleet.lab:6514 ... Connection timed out (timed out = dropped, not refused). The logs aren't lost — the sender's action.resumeRetryCount="-1" buffers them and drains once the port opens. (This exact trap was hit live on :514 — 2026-06-19, wrong NIC in the ufw rule.)

Open 6514/tcp (+ 514/tcp while legacy senders remain) scoped to the boot LAN (don't expose it globally):

# ufw (Debian/Ubuntu serving host):
sudo ufw allow from 192.168.50.0/24 to any port 6514 proto tcp
sudo ufw allow from 192.168.50.0/24 to any port 514 proto tcp   # legacy, until migrated
sudo ufw reload && sudo ufw status numbered

# firewalld (RHEL/Rocky serving host):
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=192.168.50.0/24 port port=6514 protocol=tcp accept'
sudo firewall-cmd --reload

Verify SYNs now complete (SYN → SYN-ACK) and a sender dir appears:

sudo tcpdump -ni any 'tcp port 6514 and host <sender-ip>'   # SYN-ACK + TLS, no cleartext
ls -R audit-collector/logs/                                  # <sender-ip>/ appears within ~1 min

If you ever move the collector from host networking to published ports (6514:6514), this firewall rule stops applying — published Docker ports bypass ufw/firewalld and must be controlled via Docker (DOCKER-USER chain) instead.

The sender (in the image — wired)

Provisioner 2f in diskless.pkr.hcl (gated on the audit_host var; empty = local-only) configures each host to send — host-generic, no per-host state:

auditd → syslog: the audisp-syslog plugin is enabled on LOCAL6, so audit events also go to the local syslog socket.
rsyslog forward: /etc/rsyslog.d/60-fleet-remote.conf forwards *.* to ${AUDIT_HOST}:6514 over gtls (mode 1, x509/name, StreamDriverPermittedPeers=${AUDIT_HOST}), trusting /etc/dot1x/radius-ca.pem (the baked fleet CA), with a disk-assisted queue so a network blip / collector outage doesn't drop records.

Production hardening (remaining)

Disable the legacy :514 plaintext intake once no pre-TLS image is in service (comment out the input in rsyslog.conf, rebuild, close the port).
TLS the Wazuh relay hop (collector → wazuh1) — see serving/siem/ + docs/TODO.md.
Restrict senders by subnet ($AllowedSender) as defense-in-depth beyond the firewall scope.
Add log rotation/retention (logrotate sidecar or rsyslog outchannel) and long-term archival per ISM event-logging retention.

9.3 KiB Raw Blame History