diff --git a/docs/TODO.md b/docs/TODO.md index bec6175..5ca9688 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -307,8 +307,13 @@ and cleanup. "Permissions 640 too open" and refused to start). RHEL's `sshd-keygen@.service` regenerates them into the overlay before sshd starts. **⚠ Caveat:** the ephemeral overlay means they regenerate **every boot** → the host key changes - per reboot (clients warn). Follow-up for a stable host identity: persist the - host keys or derive them from the TPM (ties to the hostnqn/identity model). + per reboot (clients warn). **RESOLVED (F-02/P-17, bench-verified 2026-07-11 on + comet-beacon-54):** rather than persist a key, each per-boot key is signed into + an OpenSSH host certificate by the fleet SSH CA via the enrol-broker host plane + at boot (`files/ssh-hostcert` → `fleet-ssh-hostcert`); admins trust one + `@cert-authority` line instead of TOFU. See + [ssh-host-certificate-design.md](ssh-host-certificate-design.md). Bench-surfaced + fixes on branch `poam-hardening` (`28d31e4`). ## [ ] 3. Roll the errata into design doc v3 diff --git a/docs/ssh-host-certificate-design.md b/docs/ssh-host-certificate-design.md index 015b97e..a905cc8 100644 --- a/docs/ssh-host-certificate-design.md +++ b/docs/ssh-host-certificate-design.md @@ -1,8 +1,11 @@ # SSH host-certificate design — killing per-boot TOFU (F-02 / P-17) -> Status: **implemented (host + broker + serving scaffold), BENCH-VERIFY.** Branch -> `poam-hardening`. Closes SAR finding **F-02** / POA&M **P-17**. Signing is -> **broker-mediated** (host → enrol-broker host plane → step-ca), not host-direct. +> Status: **BENCH-VERIFIED end-to-end 2026-07-11** (`comet-beacon-54`, real TPM): +> TPM-mTLS → broker → step-ca signing → cert install → admin login **by FQDN with +> no TOFU**, and per-boot auto-re-sign across a reboot. Closes SAR finding **F-02** +> / POA&M **P-17**. Signing is **broker-mediated** (host → enrol-broker host plane → +> step-ca), not host-direct. The defects the bench surfaced are fixed on branch +> `poam-hardening` (commit `28d31e4`) — see **Bench outcome (2026-07-11)** below. ## Problem @@ -81,17 +84,24 @@ broker's new **host plane** with its TPM identity over mTLS. enabled** and the provisioner allowed to issue SSH host certs (`ssh-ca-bootstrap.sh`). -**Principal authorisation** is the remaining sharp edge: step-ca / the provisioner -must *authorise* the requested principals (the host's hostname). Tighten this in -the broker (validate `principals` against the peer cert) and/or via an SSH host -template on step-ca. Today the broker forwards the requested principals and logs -the peer CN. +**Principals — name alignment (bench 2026-07-11).** Admins connect by **FQDN** +(`host.`), but the host defaulted its principal to the **short** hostname, +so `ssh host.fleet.lab` failed with *"name is not a listed principal"* even though +the cert was CA-valid. Fixed host-side: `fleet-ssh-hostcert` now also derives +`.` from the DHCP search domain (`/etc/resolv.conf`) at runtime — +never baked (invariant #1). step-ca's **default** SSH host template already passes +requested principals through, so **no custom template is needed** (verified). +**Still a sharp edge (defence-in-depth):** the broker forwards whatever principals +are requested; tightening `handleSSHHostCert` to validate them against the peer +cert (so a host can only obtain its own names) remains a TODO. ## Security properties - **Invariant #1 (no per-host state):** the image carries only a fleet-generic broker DNS name + CA path. The host identity, and therefore the cert, derive from - the TPM at runtime. Principals default to the runtime hostname (not baked). + the TPM at runtime. Principals are the runtime hostname **and** its + `.` FQDN derived from the DHCP search domain — both learned at + runtime, never baked. - **step-ca stays behind the front door:** hosts reach the broker, not step-ca. The broker holds the provisioner credential; a host can never self-sign. - **Fail-open:** never installs a missing/broken `HostCertificate` — a broker/CA @@ -101,9 +111,48 @@ the peer CN. - **Short-lived certs** + hourly renewal bound the blast radius of a leaked cert (the key is ephemeral anyway). +## Bench outcome (2026-07-11) + +Proven end-to-end on `comet-beacon-54` (poam-hardening image, real TPM): the full +chain — per-boot host key → TPM-mTLS to the broker → step-ca SSH signing → cert + +`60-hostcert.conf` install → sshd reload — works; an admin with one +`@cert-authority *.fleet.lab ` line logs in **by FQDN with no TOFU prompt**; +and after a reboot `ssh-hostcert.timer` re-signs the fresh per-boot key +automatically. **SELinux confinement is clean** (no AVCs writing `/etc/ssh` + +reloading sshd → **no local TE module needed**), and STIG crypto policy accepts the +`ecdsa-sha2-nistp256-cert-v01` host cert. + +Getting there surfaced defects, all fixed on `poam-hardening` (`28d31e4`): + +| # | Defect (as seen on the bench) | Fix | +|---|---|---| +| 3 | Broker dialed step-ca by the fleet dnsmasq name `ca.fleet.lab`, unresolvable from its host netns → `no such host` at sign time (502) | `STEP_CA_URL=https://localhost:8443` (co-located; `localhost` is in the CA cert SAN) | +| 4 | `ssh-ca-bootstrap.sh` **swallowed** a failed provisioner SSH-enable → the CA 502s every sign | Fail loudly + verify `enableSSHCA=true` | +| 6 | `ssh_host_ca.pub` exported in PEM/PKCS8 → the `@cert-authority` line was unparseable → silent TOFU fallback | Export in OpenSSH wire format (`ssh-keygen -i -m PKCS8`) | +| 7 | Cert principal was the **short** hostname; admins connect by FQDN → *"name is not a listed principal"* | Derive `.` from the DHCP domain at runtime | +| 1 | A second, mock-signer broker in `serving/` collided on 9090/9443 and issued untrusted certs | Removed; the real broker (dot1x-bench, co-located with step-ca) is the only one | +| 2 | Broker HTML admin console = a weaker, client-cert-only parallel human-approve path | Retired; the fleet-manager passkey SPA drives the mTLS JSON API | + +**Retracted finding:** an early read suggested the cert had *empty* principals +(valid for any host). That was a `grep` artifact hiding `ssh-keygen -L`'s indented +principal lines — the step-ca sign log confirmed the requested principal was +present. No over-scoping; #7 is the real (name-alignment) issue. + +**Operational note:** there is a brief post-boot window where sshd serves the bare +per-boot key before `ssh-hostcert.timer` (OnBootSec) fires; an admin connecting in +that window still sees TOFU. Tune `OnBootSec`/ordering to shrink it. (A client-side +PQ-KEX advisory also appears on connect — an OpenSSH warning, not an F-02 issue.) + +**Deploy caveat:** #7 is baked into the `fleet-ssh-hostcert` binary, so it only +takes effect on a **new image build** from poam-hardening — until then the reboot +no-TOFU test regresses (a short-name-only cert can't match an FQDN login). The +serving-side fixes (#1–#4, #6) apply on `docker compose up -d --build` + a re-run of +`ssh-ca-bootstrap.sh`. + ## Client rollout -One line per admin (or a system `known_hosts`), from `ssh-ca-bootstrap.sh` output: +One line per admin (or a system `known_hosts`), from `ssh-ca-bootstrap.sh` output +(now emitted in OpenSSH wire format — see fix #6 above): ``` @cert-authority *.fleet.lab ecdsa-sha2-nistp256 AAAA... # fleet SSH host CA @@ -111,6 +160,9 @@ One line per admin (or a system `known_hosts`), from `ssh-ca-bootstrap.sh` outpu ## Bench checklist +Items 1–6 were **verified 2026-07-11** (see Bench outcome); item 7 (broker-side +principal validation) is the remaining defence-in-depth TODO. + 1. Serving: run `ssh-ca-bootstrap.sh`; confirm `ca.json` has the `ssh` block, the `enrol` provisioner is SSH-enabled, and `pki/broker/{host-server.crt, fleet-client-ca.crt}` + `pki/ssh_host_ca.pub` exist. Bring up the broker with