Audit / log collector (receive side)
An rsyslog container that receives logs forwarded by booted diskless hosts
and writes them to durable, per-source-host files. Diskless hosts run on an
immutable erofs root with a RAM-only volatile overlay, so their local
/var/log/audit/audit.log is lost on every reboot — this gives auditd somewhere
permanent to offload to. It implements the remote-audit-offload that STIG
deviation #6 (docs/stig-deviations.md) lists as future hardening.
booted host serving host
┌───────────────────────────┐ ┌──────────────────────────────┐
│ auditd → audisp-syslog │ TCP/514 │ audit (this container) │
│ → rsyslog (forward) ├──────────►│ rsyslogd → /var/log/remote/ │
│ to ${AUDIT_HOST} │ syslog │ <sender-ip>/audit.log │
└───────────────────────────┘ │ <sender-ip>/messages.log │
└──────────────────────────────┘
What this provides (the receiver)
- Listens on TCP/514 (plain syslog). Disk-assisted queue so bursts / brief stalls don't drop records.
- Writes per sender IP (not the claimed hostname): audit records →
audit.log, everything else →messages.log, under./logs/<ip>/(bind-mounted, inspectable on the host). - Host-networked, so hosts reach it at
${AUDIT_HOST}(resolved to the serving host by the stack's dnsmasq), matching the boot/NVMe DNS pattern.
Run it with the rest of the stack:
cd serving
docker compose build audit
docker compose up -d audit
docker compose logs -f audit # startup + connection messages
tail -F audit-collector/logs/*/audit.log
Quick self-test (send a line as if from a host):
logger -n 127.0.0.1 -P 514 -T -t audispd "test audit record"
ls audit-collector/logs/ # a 127.0.0.1/ dir appears
Serving-host firewall — open 514/tcp (easy to miss)
This container uses host networking, so the serving host's own firewall
governs port 514 — Docker does not manage iptables for host-net containers
(the usual "Docker bypasses the host firewall" behaviour only applies to
published bridge ports, which insert rules into the DOCKER chain ahead of the
host firewall). The symptom when it's closed is deceptive: the container is
healthy, ss -tlnp shows 0.0.0.0:514 listening, DNS/DHCP from the same host
work, but booted hosts sit in SYN-SENT and rsyslog logs cannot connect to audit.fleet.lab:514 ... Connection timed out (timed out = dropped, not
refused). The logs aren't lost — the sender's action.resumeRetryCount="-1"
buffers them and drains once the port opens.
Open 514/tcp scoped to the boot LAN (don't expose it globally):
# ufw (Debian/Ubuntu serving host):
sudo ufw allow from 192.168.50.0/24 to any port 514 proto tcp
sudo ufw reload && sudo ufw status numbered
# firewalld (RHEL/Rocky serving host):
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=192.168.50.0/24 port port=514 protocol=tcp accept'
sudo firewall-cmd --reload
Verify SYNs now complete (SYN → SYN-ACK) and a sender dir appears:
sudo tcpdump -ni any 'tcp port 514 and host <sender-ip>' # SYN-ACK now returns
ls -R audit-collector/logs/ # <sender-ip>/ appears within ~1 min
If you ever move the collector from host networking to published ports (
514:514), this firewall rule stops applying — published Docker ports bypass ufw/firewalld and must be controlled via Docker (DOCKER-USERchain) instead.
What's still needed (the sender — in the image)
This container only receives. To make hosts actually send, the hardened
image needs (not yet wired — keep no per-host state, use the generic
${AUDIT_HOST} name, never a per-host address):
-
auditd → syslog: enable the
audisp-syslogplugin (/etc/audit/plugins.d/syslog.conf:active = yes), so audit events also go to the local syslog socket. -
rsyslog forward rule: a drop-in (e.g.
/etc/rsyslog.d/60-fleet-remote.conf) forwarding to the collector with a disk-assisted action queue, so a network blip doesn't drop records:*.* action(type="omfwd" target="audit.fleet.lab" port="514" protocol="tcp" queue.type="LinkedList" queue.fileName="fwdq" queue.maxDiskSpace="256m" queue.saveOnShutdown="on" action.resumeRetryCount="-1") -
Bake
audit.fleet.lab(the generic name) — DNS resolves it per environment.
Once the sender is wired, flip STIG deviation #6 in docs/stig-deviations.md
from "local-only" to "offloading to ${AUDIT_HOST}" and the
rsyslog_remote_loghost rule should pass.
Production hardening (lab simplifications to revisit)
The lab stack is plaintext TCP, mirroring the HTTP-boot/no-TLS posture. For a secured fleet:
- TLS: rsyslog
gtlsdriver (mutual auth with the boot-infra CA), or RELP + TLS (imrelp/omrelp) for guaranteed, acknowledged delivery. - Restrict senders by subnet (
$AllowedSender) or firewall :514 to the boot LAN. - Add log rotation/retention (logrotate sidecar or rsyslog
outchannel) and ship to long-term storage / SIEM.