SSH host-certificate design — killing per-boot TOFU (F-02 / P-17)

Status: implemented (host + broker + serving scaffold), BENCH-VERIFY. Branch poam-hardening. Closes SAR finding F-02 / POA&M P-17. Signing is broker-mediated (host → enrol-broker host plane → step-ca), not host-direct.

Problem

The diskless root is an immutable erofs + a RAM overlay, so sshd-keygen@.service regenerates the SSH host keys on every boot (see scripts/extract-artifacts.sh, which strips baked host keys — a baked key would be shared fleet-wide, invariant #1). Consequences:

Administrators can never pin a host key — every ssh to a fleet host is trust-on-first-use, and the key changes each reboot so even a cached entry triggers the "REMOTE HOST IDENTIFICATION HAS CHANGED" warning.
A LAN man-in-the-middle of admin SSH is undetectable client-side.

Approaches considered

A — per-boot key + boot-time CA-signed host cert (CHOSEN). Keep the ephemeral per-boot host key; at boot, get it signed by the fleet SSH CA over a TPM-authenticated request. Clients trust via one @cert-authority line. The TPM only authenticates the signing request; it is not the host key.
B — dedicated stable TPM host key + HostKeyAgent + cert at enrolment. Hardware-anchored, CA-independent, stable host identity — but needs a new persistent TPM handle, a host-key agent daemon (SELinux/fapolicyd/ordering), and more provisioning. Deferred; a follow-on if a CA-independent hardware host key is specifically wanted.

The certificate — not the key's origin — is what removes TOFU. A per-boot key with a cert is fine (short-lived keys are good hygiene), so A delivers the security outcome with the least new attack surface and no added cross-protocol key reuse. See the chat rationale; A was chosen 2026-07-11.

Architecture (Design A, broker-mediated)

The signing request goes through the enrol-broker, not directly to step-ca — so step-ca stays behind the single front door (consistent with enrolment), and the proven step CLI does the SSH signing server-side. The host authenticates to the broker's new host plane with its TPM identity over mTLS.

 boot ─▶ sshd-keygen@ecdsa ─▶ /etc/ssh/ssh_host_ecdsa_key(.pub)   (ephemeral)
                                     │
 ssh-hostcert.timer (OnBootSec + hourly)
        └─▶ fleet-ssh-hostcert:
              1. TPM: LoadSigner(0x81000100) + ReadNVCert(0x01800100) → mTLS client cert
              2. read ssh_host_ecdsa_key.pub
              3. POST {publicKey, principals, keyID} ─(mTLS)─▶ enrol-broker host plane
              4. write ssh_host_ecdsa_key-cert.pub + 60-hostcert.conf; reload sshd

 enrol-broker host plane (POST /ssh/hostcert):
        verify client cert to the fleet CA (⇒ enrolled host)
        └─▶ step ssh certificate --host --sign  (broker holds the provisioner) ─▶ step-ca SSH CA
            └─▶ returns {cert}

 client:  @cert-authority *.fleet.lab <ssh_host_ca.pub>   (one line, once)

Components

Piece	Where	Notes
`fleet-ssh-hostcert`	`files/ssh-hostcert/` (Go, static, go-tpm)	TPM mTLS client; out-of-VM CI build like fleet-enrol/agent; deps pinned by go.sum
`ssh-hostcert.service` + `.timer`	`files/ssh-hostcert/`	root oneshot (writes `/etc/ssh`, reloads sshd), sandboxed; timer = boot + hourly renew
`ssh-hostcert.env`	`files/ssh-hostcert/` → `/etc/ssh-hostcert/`	fleet-generic broker URL / CA / paths
Packer `2b-ssh-hostcert`	`diskless.pkr.hcl`	gated on `enable_ssh_hostcert`; fapolicyd-trusts the binary in 2c-STIG-postfix
broker host plane (`/ssh/hostcert`)	`serving/enrol-broker/host.go` + `sshsigner.go`	mTLS (fleet-CA host certs); `step ssh certificate` signer; enabled by `BROKER_HOST_ADDR`
step-ca SSH CA enablement	`serving/dot1x-bench/step-ca/ssh-ca-bootstrap.sh`	SSH CA keys + provisioner `--ssh` + broker host-plane cert; prints the admin `@cert-authority` line

Wire contract (the BENCH-VERIFY seam)

Host → broker (mTLS): POST /ssh/hostcert {publicKey, principals, keyID} → {cert}. The host's mTLS client cert is its TPM identity; the broker verifies it to the fleet CA (RequireAndVerifyClientCert) — that is the authorisation (an enrolled host). No token, no CA secret on the host.
Broker → step-ca: step ssh certificate --host --sign … with the broker's provisioner credential. The exact flags are the tunable part (serving/enrol-broker/sshsigner.go); the CA must have its SSH host CA enabled and the provisioner allowed to issue SSH host certs (ssh-ca-bootstrap.sh).

Principal authorisation is the remaining sharp edge: step-ca / the provisioner must authorise the requested principals (the host's hostname). Tighten this in the broker (validate principals against the peer cert) and/or via an SSH host template on step-ca. Today the broker forwards the requested principals and logs the peer CN.

Security properties

Invariant #1 (no per-host state): the image carries only a fleet-generic broker DNS name + CA path. The host identity, and therefore the cert, derive from the TPM at runtime. Principals default to the runtime hostname (not baked).
step-ca stays behind the front door: hosts reach the broker, not step-ca. The broker holds the provisioner credential; a host can never self-sign.
Fail-open: never installs a missing/broken HostCertificate — a broker/CA outage degrades to today's bare-key behaviour, not a boot/sshd failure.
Key reuse (cf. F-06): the TPM identity authenticates the mTLS request; the SSH host key is a separate ephemeral key.
Short-lived certs + hourly renewal bound the blast radius of a leaked cert (the key is ephemeral anyway).

Client rollout

One line per admin (or a system known_hosts), from ssh-ca-bootstrap.sh output:

@cert-authority *.fleet.lab ecdsa-sha2-nistp256 AAAA...   # fleet SSH host CA

Bench checklist

Serving: run ssh-ca-bootstrap.sh; confirm ca.json has the ssh block, the enrol provisioner is SSH-enabled, and pki/broker/{host-server.crt, fleet-client-ca.crt} + pki/ssh_host_ca.pub exist. Bring up the broker with BROKER_HOST_ADDR set; journalctl shows "host plane up".
Broker: from an enrolled host's identity, curl --cert/--key … https:// boot.fleet.lab:9444/ssh/hostcert returns a {cert}; an un-enrolled client is rejected at the TLS handshake (RequireAndVerifyClientCert).
Host: systemctl start ssh-hostcert.service; journalctl -u ssh-hostcert shows "installed SSH host certificate"; /etc/ssh/ssh_host_ecdsa_key-cert.pub present; sshd -T | grep hostcertificate.
SELinux: watch for AVCs on the root service writing /etc/ssh/* and running systemctl reload sshd under confinement — add a local TE module if needed (like fleet_agent_chronylog). Reproducibility: a new semodule would rebuild the policy store → re-check verity-hash stability across two builds.
Client: with the @cert-authority line set, ssh <host> shows no host-key prompt/warning across reboots.
Confirm STIG crypto policy accepts the ecdsa-sha2-nistp256-cert-v01 host cert (same family already accepted for admin keys).
Principal policy: confirm the broker/step-ca only issue the principals a host is entitled to (tighten handleSSHHostCert to validate against the peer cert).

Deferred (Design B follow-on)

Stable, CA-independent hardware host key: a dedicated TPM key at a new persistent handle (e.g. 0x81000101, avoiding OEM/reserved NV like the identity key) served to sshd via a small TPM HostKeyAgent, cert issued at enrolment. Only pursue if a host identity that survives a CA outage is required.

7.7 KiB Raw Blame History