sunjizzle/diskless-workstation
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Adding rsyslog tls

Browse Source
This commit is contained in:
Audiobuzz
2026-07-12 14:04:19 +09:30
parent 28d31e436c
commit 5bc6d27f77
13 changed files with 542 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serving/audit-collector/Dockerfile
+5 -2
View File
@@ -6,12 +6,15 @@
FROM harbor.bitasylum.org/dockerhub/library/alpine:3.20
RUN apk add --no-cache rsyslog tzdata \
# rsyslog-tls ships lmnsd_gtls.so (gnutls netstream driver) for the :6514 TLS
# listener (verified present in Alpine 3.20's rsyslog-tls subpackage).
RUN apk add --no-cache rsyslog rsyslog-tls tzdata \
&& mkdir -p /var/log/remote /var/spool/rsyslog
COPY rsyslog.conf /etc/rsyslog.conf
EXPOSE 514/tcp
# 6514 = syslog-over-TLS (primary), 514 = legacy plaintext (migration only).
EXPOSE 6514/tcp 514/tcp
# Foreground (-n), our config only (-f). rsyslog logs its own startup to stderr,
# captured by `docker compose logs audit`.
ENTRYPOINT ["rsyslogd", "-n", "-f", "/etc/rsyslog.conf"]
serving/audit-collector/README.md
+122 -62
View File
@@ -4,28 +4,93 @@ An **rsyslog** container that receives logs forwarded by booted diskless hosts
and writes them to durable, per-source-host files. Diskless hosts run on an
immutable erofs root with a **RAM-only volatile overlay**, so their local
`/var/log/audit/audit.log` is lost on every reboot — this gives auditd somewhere
permanent to offload to. It implements the remote-audit-offload that STIG
deviation #6 (`docs/stig-deviations.md`) lists as future hardening.
permanent to offload to. It implements the remote-audit-offload of STIG
deviation #6 (`docs/stig-deviations.md`); since 2026-07-12 the hop is
**syslog-over-TLS** and the `rsyslog_encrypt_offload_*` STIG rules are enforced
(no longer waived).
```
booted host serving host
┌───────────────────────────┐ ┌──────────────────────────────┐
│ auditd → audisp-syslog │ TCP/514 │ audit (this container) │
│ → rsyslog (forward) ├──────────►│ rsyslogd → /var/log/remote/ │
│ to ${AUDIT_HOST} │ syslog │ <sender-ip>/audit.log │
└───────────────────────────┘ │ <sender-ip>/messages.log │
└──────────────────────────────┘
booted host serving host
┌────────────────────────────┐ ┌──────────────────────────────┐
│ auditd → audisp-syslog │ TLS/6514 │ audit (this container) │
│ → rsyslog (omfwd gtls) ├───────────►│ rsyslogd (gtls, fleet-CA │
│ x509/name vs the │ syslog │ server cert) → │
│ baked fleet CA │ │ /var/log/remote/ │
└────────────────────────────┘ │ <sender-ip>/audit.log │
(legacy plaintext :514 │ <sender-ip>/messages.log │
kept for pre-TLS images) └──────────────────────────────┘
```
## What this provides (the receiver)
- Listens on **TCP/514** (plain syslog). Disk-assisted queue so bursts / brief
stalls don't drop records.
- Listens on **TCP/6514, syslog-over-TLS** (rsyslog `gtls` driver). The server
cert is issued from the **fleet step-ca** (below); senders verify it
`x509/name` against the fleet CA baked in the image
(`/etc/dot1x/radius-ca.pem`). Clients are **not** cert-authenticated
(`authMode="anon"`): hosts are fleet-generic (no per-host state, invariant
#1) and a baked fleet-wide client key would be shared secret material —
sender access is constrained by the isolated boot LAN + the firewall below.
- Still listens on **TCP/514 plaintext** as a **legacy/migration** intake for
already-released pre-TLS images. Disable it (comment the input out + close
the firewall port) once the fleet is on TLS-sending images.
- Disk-assisted queue so bursts / brief stalls don't drop records.
- Writes per **sender IP** (not the claimed hostname): audit records →
`audit.log`, everything else → `messages.log`, under
`./logs/<ip>/` (bind-mounted, inspectable on the host).
`audit.log`, everything else → `messages.log`, under `./logs/<ip>/`
(bind-mounted, inspectable on the host).
- Host-networked, so hosts reach it at `${AUDIT_HOST}` (resolved to the serving
host by the stack's dnsmasq), matching the boot/NVMe DNS pattern.
- Relays a **copy** of audit records to the Wazuh SIEM node (see
`rsyslog.conf`) — that hop is still plaintext (tracked in docs/TODO.md; it
must stay `ptcp` until wazuh1 has a TLS listener).
## Decision: no client mTLS with the TPM identity (2026-07-12, considered + deferred)
Hosts already carry a fleet-CA-signed TPM identity (key at `0x81000100`,
usable for mTLS — the fleet-agent uses it). We deliberately do **not** use it
to client-authenticate the log stream:
- **Low marginal gain.** On the production/secure path the collector hop rides
a LAN already gated by **802.1X EAP-TLS with the exact same TPM identity** +
MACsec — client mTLS at layer 7 would re-prove a credential the network
proved at layer 2. It also doesn't close the residual threat: a
compromised-but-enrolled host has a valid cert and can still inject records
about itself. And rsyslog can *validate* a client cert but can't template on
its identity, so per-host files would still be keyed by sender IP — no
attribution upgrade. The STIG `rsyslog_encrypt_offload_*` rules require
server-auth TLS only; no compliance delta.
- **Disproportionate cost.** The TPM key is non-exportable by design and
rsyslog's gtls driver only takes PEM key *files* (no PKCS#11 URI), so this
needs either a boot-time delegated cert (online CA round-trip *ahead of* log
offload; key lands in tmpfs anyway — "TPM-held" degrades to "TPM-issued") or
a PKCS#11-capable fronting daemon (ghostunnel etc.: new non-RPM daemon in the
immutable image + fapolicyd/SELinux/reproducibility surface).
- **Failure-mode asymmetry.** Offload is the forensic safety net; hard-failing
it on client-auth problems makes a host with a broken TPM/cert — exactly the
host whose logs matter most — go dark (RAM-backed spool, lost on reboot).
**Revisit trigger:** if cryptographically-attributed log transport becomes a
hard requirement (ISM/SSP), route logs through the **fleet-agent's existing
TPM-mTLS channel** (go-tpm, no PKCS#11 gymnastics) instead of teaching rsyslog
client certs. Until then: `authMode="anon"` + boot-LAN firewall scoping (+
`$AllowedSender` as cheap depth), with 802.1X/MACsec as the sender-authentication
layer.
## TLS server cert (required before first start)
Issue it from the fleet CA (the dot1x-bench step-ca) into `./tls/`
(**gitignored** — key material never enters the repo):
```bash
cd serving/audit-collector
./issue-cert.sh # SAN = ${AUDIT_HOST} from serving/.env; writes tls/{server.crt,server.key,ca.crt}
```
This MUST be the same CA whose root the image was built with (`var.fleet_ca` →
`/etc/dot1x/radius-ca.pem`) or every sender rejects the handshake and spools.
Certs expire (default 720h): re-run `issue-cert.sh` and
`docker compose up -d audit` to renew. Expiry symptom: collector logs TLS
handshake errors, sender dirs stop growing, hosts spool
(`action.resumeRetryCount="-1"` — records drain after renewal, not lost).
Run it with the rest of the stack:
@@ -37,82 +102,77 @@ docker compose logs -f audit # startup + connection messages
tail -F audit-collector/logs/*/audit.log
```
Quick self-test (send a line as if from a host):
Quick self-tests:
```bash
# TLS handshake + cert chain (expect Verify return code: 0 against the fleet CA)
openssl s_client -connect 127.0.0.1:6514 -CAfile audit-collector/tls/ca.crt </dev/null
# End-to-end record over the LEGACY plaintext intake (still simplest smoke test)
logger -n 127.0.0.1 -P 514 -T -t audispd "test audit record"
ls audit-collector/logs/ # a 127.0.0.1/ dir appears
```
## Serving-host firewall — open 514/tcp (easy to miss)
## Serving-host firewall — open 6514/tcp (easy to miss)
This container uses **host networking**, so the **serving host's own firewall
governs port 514** — Docker does *not* manage iptables for host-net containers
(the usual "Docker bypasses the host firewall" behaviour only applies to
*published* bridge ports, which insert rules into the `DOCKER` chain ahead of the
host firewall). The symptom when it's closed is deceptive: the container is
healthy, `ss -tlnp` shows `0.0.0.0:514` listening, DNS/DHCP from the same host
work, but booted hosts sit in `SYN-SENT` and rsyslog logs `cannot connect to
audit.fleet.lab:514 ... Connection timed out` (timed out = **dropped**, not
refused). The logs aren't lost — the sender's `action.resumeRetryCount="-1"`
buffers them and drains once the port opens.
governs port 6514** (and legacy 514) — Docker does *not* manage iptables for
host-net containers (the usual "Docker bypasses the host firewall" behaviour
only applies to *published* bridge ports, which insert rules into the `DOCKER`
chain ahead of the host firewall). The symptom when it's closed is deceptive:
the container is healthy, `ss -tlnp` shows `0.0.0.0:6514` listening, DNS/DHCP
from the same host work, but booted hosts sit in `SYN-SENT` and rsyslog logs
`cannot connect to audit.fleet.lab:6514 ... Connection timed out` (timed out =
**dropped**, not refused). The logs aren't lost — the sender's
`action.resumeRetryCount="-1"` buffers them and drains once the port opens.
(This exact trap was hit live on :514 — 2026-06-19, wrong NIC in the ufw rule.)
Open 514/tcp **scoped to the boot LAN** (don't expose it globally):
Open 6514/tcp (+ 514/tcp while legacy senders remain) **scoped to the boot
LAN** (don't expose it globally):
```bash
# ufw (Debian/Ubuntu serving host):
sudo ufw allow from 192.168.50.0/24 to any port 514 proto tcp
sudo ufw allow from 192.168.50.0/24 to any port 6514 proto tcp
sudo ufw allow from 192.168.50.0/24 to any port 514 proto tcp # legacy, until migrated
sudo ufw reload && sudo ufw status numbered
# firewalld (RHEL/Rocky serving host):
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=192.168.50.0/24 port port=514 protocol=tcp accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=192.168.50.0/24 port port=6514 protocol=tcp accept'
sudo firewall-cmd --reload
```
Verify SYNs now complete (SYN → SYN-ACK) and a sender dir appears:
```bash
sudo tcpdump -ni any 'tcp port 514 and host <sender-ip>' # SYN-ACK now returns
ls -R audit-collector/logs/ # <sender-ip>/ appears within ~1 min
sudo tcpdump -ni any 'tcp port 6514 and host <sender-ip>' # SYN-ACK + TLS, no cleartext
ls -R audit-collector/logs/ # <sender-ip>/ appears within ~1 min
```
> If you ever move the collector from host networking to **published ports**
> (`514:514`), this firewall rule stops applying — published Docker ports bypass
> ufw/firewalld and must be controlled via Docker (`DOCKER-USER` chain) instead.
> (`6514:6514`), this firewall rule stops applying — published Docker ports
> bypass ufw/firewalld and must be controlled via Docker (`DOCKER-USER` chain)
> instead.
## What's still needed (the sender — in the image)
## The sender (in the image — wired)
This container only **receives**. To make hosts actually send, the hardened
image needs (not yet wired — keep `no per-host state`, use the generic
`${AUDIT_HOST}` name, never a per-host address):
Provisioner **2f** in `diskless.pkr.hcl` (gated on the `audit_host` var; empty =
local-only) configures each host to send — host-generic, no per-host state:
1. **auditd → syslog**: enable the `audisp-syslog` plugin
(`/etc/audit/plugins.d/syslog.conf`: `active = yes`), so audit events also go
to the local syslog socket.
2. **rsyslog forward rule**: a drop-in (e.g. `/etc/rsyslog.d/60-fleet-remote.conf`)
forwarding to the collector with a disk-assisted action queue, so a network
blip doesn't drop records:
1. **auditd → syslog**: the `audisp-syslog` plugin is enabled on LOCAL6, so
audit events also go to the local syslog socket.
2. **rsyslog forward**: `/etc/rsyslog.d/60-fleet-remote.conf` forwards `*.*` to
`${AUDIT_HOST}:6514` over `gtls` (mode 1, `x509/name`,
`StreamDriverPermittedPeers=${AUDIT_HOST}`), trusting
`/etc/dot1x/radius-ca.pem` (the baked fleet CA), with a disk-assisted queue
so a network blip / collector outage doesn't drop records.
```rsyslog
*.* action(type="omfwd" target="audit.fleet.lab" port="514" protocol="tcp"
queue.type="LinkedList" queue.fileName="fwdq"
queue.maxDiskSpace="256m" queue.saveOnShutdown="on"
action.resumeRetryCount="-1")
```
3. Bake `audit.fleet.lab` (the generic name) — DNS resolves it per environment.
## Production hardening (remaining)
Once the sender is wired, flip STIG deviation #6 in `docs/stig-deviations.md`
from "local-only" to "offloading to ${AUDIT_HOST}" and the
`rsyslog_remote_loghost` rule should pass.
## Production hardening (lab simplifications to revisit)
The lab stack is **plaintext TCP**, mirroring the HTTP-boot/no-TLS posture. For
a secured fleet:
- **TLS**: rsyslog `gtls` driver (mutual auth with the boot-infra CA), or
**RELP + TLS** (`imrelp`/`omrelp`) for guaranteed, acknowledged delivery.
- **Restrict senders** by subnet (`$AllowedSender`) or firewall :514 to the
boot LAN.
- **Disable the legacy :514 plaintext intake** once no pre-TLS image is in
service (comment out the input in `rsyslog.conf`, rebuild, close the port).
- **TLS the Wazuh relay hop** (collector → wazuh1) — see `serving/siem/` +
docs/TODO.md.
- **Restrict senders** by subnet (`$AllowedSender`) as defense-in-depth beyond
the firewall scope.
- Add log rotation/retention (logrotate sidecar or rsyslog `outchannel`) and
ship to long-term storage / SIEM.
long-term archival per ISM event-logging retention.
serving/audit-collector/issue-cert.sh
Executable
+59
View File
@@ -0,0 +1,59 @@
#!/usr/bin/env bash
# issue-cert.sh — issue the audit collector's syslog-over-TLS server cert from
# the fleet CA (the dot1x-bench step-ca) into ./tls/, where docker-compose
# mounts it read-only at /etc/rsyslog-tls for the :6514 gtls listener.
#
# The cert's SAN must be the name senders verify (x509/name): ${AUDIT_HOST},
# the same host-generic DNS name baked into the image's 2f forward and resolved
# by the serving dnsmasq. Hosts validate it against the fleet CA root already
# baked at /etc/dot1x/radius-ca.pem — so this MUST be the same CA whose root
# the image was built with (var.fleet_ca).
#
# Same in-container issuance pattern as dot1x-bench/step-ca/bootstrap.sh: `step`
# runs inside the step-ca container; /export = dot1x-bench/pki on the host.
#
# Certs expire (default 720h — see bootstrap.sh CA claims): re-run this script
# and `docker compose restart audit` to renew. If the collector starts logging
# TLS handshake errors and senders spool, check expiry first:
# openssl x509 -in tls/server.crt -noout -enddate
set -euo pipefail
cd "$(dirname "$0")"
COLLECTOR_DIR=$PWD
BENCH_DIR=../dot1x-bench
# AUDIT_HOST from the serving stack's .env (the name dnsmasq resolves for hosts).
[ -f ../.env ] && . ../.env
AUDIT_HOST=${AUDIT_HOST:-audit.fleet.lab}
cd "$BENCH_DIR"
[ -f .env ] && . ./.env
CA_PORT=${CA_PORT:-8443}
CA_CONNECT=${CA_CONNECT:-localhost} # see bootstrap.sh: SAN-valid AND resolvable
CA_URL="https://${CA_CONNECT}:${CA_PORT}"
CONTAINER_PW="/home/step/password" # bootstrap.sh: pki/stepca/password in-container
step() { docker compose exec -T step-ca step "$@"; }
incca() { docker compose exec -T step-ca "$@"; }
echo "==> issuing audit-collector cert (CN/SAN: $AUDIT_HOST) from the fleet CA"
incca mkdir -p /export/audit
step ca certificate "$AUDIT_HOST" \
/export/audit/server.crt /export/audit/server.key \
--ca-url "$CA_URL" --root /home/step/certs/root_ca.crt \
--provisioner enrol --provisioner-password-file "$CONTAINER_PW" \
--not-after "${AUDIT_CERT_DUR:-720h}" --force
incca cp /home/step/certs/root_ca.crt /export/audit/ca.crt
# Stage into ./tls/ (compose bind-mounts it). pki/ may be uid-1000/root-owned
# (see bootstrap.sh perm notes) — copy via the container's /export if needed.
mkdir -p "$COLLECTOR_DIR/tls" 2>/dev/null || { echo "==> fixing tls/ perms with sudo (lab)"; sudo mkdir -p "$COLLECTOR_DIR/tls" && sudo chown "$(id -u)" "$COLLECTOR_DIR/tls"; }
for f in server.crt server.key ca.crt; do
cp pki/audit/"$f" "$COLLECTOR_DIR/tls/$f" 2>/dev/null \
|| { echo "==> pki/audit unreadable by this user; fixing with sudo (lab)"; sudo cp pki/audit/"$f" "$COLLECTOR_DIR/tls/$f" && sudo chown "$(id -u)" "$COLLECTOR_DIR/tls/$f"; }
done
chmod 0600 "$COLLECTOR_DIR/tls/server.key"
chmod 0644 "$COLLECTOR_DIR/tls/server.crt" "$COLLECTOR_DIR/tls/ca.crt"
echo "==> done: $(openssl x509 -in "$COLLECTOR_DIR/tls/server.crt" -noout -subject -enddate 2>/dev/null || echo 'tls/server.crt staged')"
echo " apply: (cd .. && docker compose up -d audit) # or restart if already up"
serving/audit-collector/rsyslog.conf
+29 -9
View File
@@ -1,19 +1,39 @@
# Remote audit / syslog collector for the diskless fleet (LAB).
# Remote audit / syslog collector for the diskless fleet.
#
# Receives logs forwarded by booted diskless hosts over plain TCP/514 and writes
# them per-source-host. Diskless hosts have an immutable, RAM-only overlay root,
# Receives logs forwarded by booted diskless hosts and writes them
# per-source-host. Diskless hosts have an immutable, RAM-only overlay root,
# so their local audit log dies on reboot — this gives auditd somewhere durable
# to offload to (STIG remote-audit-offload, deviation #6 in docs/stig-deviations.md).
#
# Lab posture: PLAINTEXT TCP, mirroring the HTTP-boot/no-TLS lab stack. For
# production switch to TLS (gtls driver) or RELP+TLS and restrict senders by
# subnet — see README.md.
# Primary intake is syslog-over-TLS on 6514 (gtls; server cert issued by the
# fleet step-ca — see issue-cert.sh + README.md). Hosts authenticate the
# collector via the fleet CA baked in the image (x509/name); the collector does
# NOT require client certs (authMode "anon"): hosts are fleet-generic (no
# per-host state, invariant #1) and a baked fleet-wide client key would be
# shared secret material in the image. Sender access is instead constrained by
# the isolated boot LAN + the serving-host firewall scoping :6514 to it.
global(workDirectory="/var/spool/rsyslog")
global(
workDirectory="/var/spool/rsyslog"
# TLS material for the 6514 listener, mounted from ./tls (issue-cert.sh).
# NOTE: deliberately NOT setting DefaultNetstreamDriver="gtls" here — that
# would flip every plain `protocol="tcp"` omfwd (the Wazuh relay below) to
# TLS and break it. TLS is enabled per-listener instead.
DefaultNetstreamDriverCAFile="/etc/rsyslog-tls/ca.crt"
DefaultNetstreamDriverCertFile="/etc/rsyslog-tls/server.crt"
DefaultNetstreamDriverKeyFile="/etc/rsyslog-tls/server.key"
)
# --- intake: plain TCP syslog ---
# --- intake: syslog-over-TLS (the hardened image's 2f sender targets this) ---
module(load="imtcp" maxSessions="1000")
input(type="imtcp" port="514")
input(type="imtcp" port="6514"
streamDriver.name="gtls" streamDriver.mode="1" streamDriver.authMode="anon")
# --- intake: plain TCP syslog (LEGACY/migration) ---
# Kept for already-released images that still forward plaintext to :514.
# Disable (comment out + close the firewall port) once the fleet is on
# TLS-sending images — production must not accept plaintext audit records.
input(type="imtcp" port="514" streamDriver.name="ptcp")
# Some senders only speak UDP; uncomment to also accept it (lossy, no queue).
#module(load="imudp")
#input(type="imudp" port="514")