179 lines
9.3 KiB
Markdown
179 lines
9.3 KiB
Markdown
# Audit / log collector (receive side)
|
|
|
|
An **rsyslog** container that receives logs forwarded by booted diskless hosts
|
|
and writes them to durable, per-source-host files. Diskless hosts run on an
|
|
immutable erofs root with a **RAM-only volatile overlay**, so their local
|
|
`/var/log/audit/audit.log` is lost on every reboot — this gives auditd somewhere
|
|
permanent to offload to. It implements the remote-audit-offload of STIG
|
|
deviation #6 (`docs/stig-deviations.md`); since 2026-07-12 the hop is
|
|
**syslog-over-TLS** and the `rsyslog_encrypt_offload_*` STIG rules are enforced
|
|
(no longer waived).
|
|
|
|
```
|
|
booted host serving host
|
|
┌────────────────────────────┐ ┌──────────────────────────────┐
|
|
│ auditd → audisp-syslog │ TLS/6514 │ audit (this container) │
|
|
│ → rsyslog (omfwd gtls) ├───────────►│ rsyslogd (gtls, fleet-CA │
|
|
│ x509/name vs the │ syslog │ server cert) → │
|
|
│ baked fleet CA │ │ /var/log/remote/ │
|
|
└────────────────────────────┘ │ <sender-ip>/audit.log │
|
|
(legacy plaintext :514 │ <sender-ip>/messages.log │
|
|
kept for pre-TLS images) └──────────────────────────────┘
|
|
```
|
|
|
|
## What this provides (the receiver)
|
|
|
|
- Listens on **TCP/6514, syslog-over-TLS** (rsyslog `gtls` driver). The server
|
|
cert is issued from the **fleet step-ca** (below); senders verify it
|
|
`x509/name` against the fleet CA baked in the image
|
|
(`/etc/dot1x/radius-ca.pem`). Clients are **not** cert-authenticated
|
|
(`authMode="anon"`): hosts are fleet-generic (no per-host state, invariant
|
|
#1) and a baked fleet-wide client key would be shared secret material —
|
|
sender access is constrained by the isolated boot LAN + the firewall below.
|
|
- Still listens on **TCP/514 plaintext** as a **legacy/migration** intake for
|
|
already-released pre-TLS images. Disable it (comment the input out + close
|
|
the firewall port) once the fleet is on TLS-sending images.
|
|
- Disk-assisted queue so bursts / brief stalls don't drop records.
|
|
- Writes per **sender IP** (not the claimed hostname): audit records →
|
|
`audit.log`, everything else → `messages.log`, under `./logs/<ip>/`
|
|
(bind-mounted, inspectable on the host).
|
|
- Host-networked, so hosts reach it at `${AUDIT_HOST}` (resolved to the serving
|
|
host by the stack's dnsmasq), matching the boot/NVMe DNS pattern.
|
|
- Relays a **copy** of audit records to the Wazuh SIEM node (see
|
|
`rsyslog.conf`) — that hop is still plaintext (tracked in docs/TODO.md; it
|
|
must stay `ptcp` until wazuh1 has a TLS listener).
|
|
|
|
## Decision: no client mTLS with the TPM identity (2026-07-12, considered + deferred)
|
|
|
|
Hosts already carry a fleet-CA-signed TPM identity (key at `0x81000100`,
|
|
usable for mTLS — the fleet-agent uses it). We deliberately do **not** use it
|
|
to client-authenticate the log stream:
|
|
|
|
- **Low marginal gain.** On the production/secure path the collector hop rides
|
|
a LAN already gated by **802.1X EAP-TLS with the exact same TPM identity** +
|
|
MACsec — client mTLS at layer 7 would re-prove a credential the network
|
|
proved at layer 2. It also doesn't close the residual threat: a
|
|
compromised-but-enrolled host has a valid cert and can still inject records
|
|
about itself. And rsyslog can *validate* a client cert but can't template on
|
|
its identity, so per-host files would still be keyed by sender IP — no
|
|
attribution upgrade. The STIG `rsyslog_encrypt_offload_*` rules require
|
|
server-auth TLS only; no compliance delta.
|
|
- **Disproportionate cost.** The TPM key is non-exportable by design and
|
|
rsyslog's gtls driver only takes PEM key *files* (no PKCS#11 URI), so this
|
|
needs either a boot-time delegated cert (online CA round-trip *ahead of* log
|
|
offload; key lands in tmpfs anyway — "TPM-held" degrades to "TPM-issued") or
|
|
a PKCS#11-capable fronting daemon (ghostunnel etc.: new non-RPM daemon in the
|
|
immutable image + fapolicyd/SELinux/reproducibility surface).
|
|
- **Failure-mode asymmetry.** Offload is the forensic safety net; hard-failing
|
|
it on client-auth problems makes a host with a broken TPM/cert — exactly the
|
|
host whose logs matter most — go dark (RAM-backed spool, lost on reboot).
|
|
|
|
**Revisit trigger:** if cryptographically-attributed log transport becomes a
|
|
hard requirement (ISM/SSP), route logs through the **fleet-agent's existing
|
|
TPM-mTLS channel** (go-tpm, no PKCS#11 gymnastics) instead of teaching rsyslog
|
|
client certs. Until then: `authMode="anon"` + boot-LAN firewall scoping (+
|
|
`$AllowedSender` as cheap depth), with 802.1X/MACsec as the sender-authentication
|
|
layer.
|
|
|
|
## TLS server cert (required before first start)
|
|
|
|
Issue it from the fleet CA (the dot1x-bench step-ca) into `./tls/`
|
|
(**gitignored** — key material never enters the repo):
|
|
|
|
```bash
|
|
cd serving/audit-collector
|
|
./issue-cert.sh # SAN = ${AUDIT_HOST} from serving/.env; writes tls/{server.crt,server.key,ca.crt}
|
|
```
|
|
|
|
This MUST be the same CA whose root the image was built with (`var.fleet_ca` →
|
|
`/etc/dot1x/radius-ca.pem`) or every sender rejects the handshake and spools.
|
|
Certs expire (default 720h): re-run `issue-cert.sh` and
|
|
`docker compose up -d audit` to renew. Expiry symptom: collector logs TLS
|
|
handshake errors, sender dirs stop growing, hosts spool
|
|
(`action.resumeRetryCount="-1"` — records drain after renewal, not lost).
|
|
|
|
Run it with the rest of the stack:
|
|
|
|
```bash
|
|
cd serving
|
|
docker compose build audit
|
|
docker compose up -d audit
|
|
docker compose logs -f audit # startup + connection messages
|
|
tail -F audit-collector/logs/*/audit.log
|
|
```
|
|
|
|
Quick self-tests:
|
|
|
|
```bash
|
|
# TLS handshake + cert chain (expect Verify return code: 0 against the fleet CA)
|
|
openssl s_client -connect 127.0.0.1:6514 -CAfile audit-collector/tls/ca.crt </dev/null
|
|
|
|
# End-to-end record over the LEGACY plaintext intake (still simplest smoke test)
|
|
logger -n 127.0.0.1 -P 514 -T -t audispd "test audit record"
|
|
ls audit-collector/logs/ # a 127.0.0.1/ dir appears
|
|
```
|
|
|
|
## Serving-host firewall — open 6514/tcp (easy to miss)
|
|
|
|
This container uses **host networking**, so the **serving host's own firewall
|
|
governs port 6514** (and legacy 514) — Docker does *not* manage iptables for
|
|
host-net containers (the usual "Docker bypasses the host firewall" behaviour
|
|
only applies to *published* bridge ports, which insert rules into the `DOCKER`
|
|
chain ahead of the host firewall). The symptom when it's closed is deceptive:
|
|
the container is healthy, `ss -tlnp` shows `0.0.0.0:6514` listening, DNS/DHCP
|
|
from the same host work, but booted hosts sit in `SYN-SENT` and rsyslog logs
|
|
`cannot connect to audit.fleet.lab:6514 ... Connection timed out` (timed out =
|
|
**dropped**, not refused). The logs aren't lost — the sender's
|
|
`action.resumeRetryCount="-1"` buffers them and drains once the port opens.
|
|
(This exact trap was hit live on :514 — 2026-06-19, wrong NIC in the ufw rule.)
|
|
|
|
Open 6514/tcp (+ 514/tcp while legacy senders remain) **scoped to the boot
|
|
LAN** (don't expose it globally):
|
|
|
|
```bash
|
|
# ufw (Debian/Ubuntu serving host):
|
|
sudo ufw allow from 192.168.50.0/24 to any port 6514 proto tcp
|
|
sudo ufw allow from 192.168.50.0/24 to any port 514 proto tcp # legacy, until migrated
|
|
sudo ufw reload && sudo ufw status numbered
|
|
|
|
# firewalld (RHEL/Rocky serving host):
|
|
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=192.168.50.0/24 port port=6514 protocol=tcp accept'
|
|
sudo firewall-cmd --reload
|
|
```
|
|
|
|
Verify SYNs now complete (SYN → SYN-ACK) and a sender dir appears:
|
|
|
|
```bash
|
|
sudo tcpdump -ni any 'tcp port 6514 and host <sender-ip>' # SYN-ACK + TLS, no cleartext
|
|
ls -R audit-collector/logs/ # <sender-ip>/ appears within ~1 min
|
|
```
|
|
|
|
> If you ever move the collector from host networking to **published ports**
|
|
> (`6514:6514`), this firewall rule stops applying — published Docker ports
|
|
> bypass ufw/firewalld and must be controlled via Docker (`DOCKER-USER` chain)
|
|
> instead.
|
|
|
|
## The sender (in the image — wired)
|
|
|
|
Provisioner **2f** in `diskless.pkr.hcl` (gated on the `audit_host` var; empty =
|
|
local-only) configures each host to send — host-generic, no per-host state:
|
|
|
|
1. **auditd → syslog**: the `audisp-syslog` plugin is enabled on LOCAL6, so
|
|
audit events also go to the local syslog socket.
|
|
2. **rsyslog forward**: `/etc/rsyslog.d/60-fleet-remote.conf` forwards `*.*` to
|
|
`${AUDIT_HOST}:6514` over `gtls` (mode 1, `x509/name`,
|
|
`StreamDriverPermittedPeers=${AUDIT_HOST}`), trusting
|
|
`/etc/dot1x/radius-ca.pem` (the baked fleet CA), with a disk-assisted queue
|
|
so a network blip / collector outage doesn't drop records.
|
|
|
|
## Production hardening (remaining)
|
|
|
|
- **Disable the legacy :514 plaintext intake** once no pre-TLS image is in
|
|
service (comment out the input in `rsyslog.conf`, rebuild, close the port).
|
|
- **TLS the Wazuh relay hop** (collector → wazuh1) — see `serving/siem/` +
|
|
docs/TODO.md.
|
|
- **Restrict senders** by subnet (`$AllowedSender`) as defense-in-depth beyond
|
|
the firewall scope.
|
|
- Add log rotation/retention (logrotate sidecar or rsyslog `outchannel`) and
|
|
long-term archival per ISM event-logging retention.
|