diskless-workstation/docs/ssh-host-certificate-design.md

SSH host-certificate design — killing per-boot TOFU (F-02 / P-17)

Status: BENCH-VERIFIED end-to-end 2026-07-11 on clean build 28d31e4 (comet-beacon-54, real TPM): TPM-mTLS → broker → step-ca signing → cert install → admin login by FQDN with no TOFU, and per-boot auto-re-sign across a reboot. Closes SAR finding F-02 / POA&M P-17. Signing is broker-mediated (host → enrol-broker host plane → step-ca), not host-direct. The defects the bench surfaced are fixed on branch poam-hardening (commit 28d31e4) — see Bench outcome (2026-07-11) below.

Problem

The diskless root is an immutable erofs + a RAM overlay, so sshd-keygen@.service regenerates the SSH host keys on every boot (see scripts/extract-artifacts.sh, which strips baked host keys — a baked key would be shared fleet-wide, invariant #1). Consequences:

• Administrators can never pin a host key — every ssh to a fleet host is trust-on-first-use, and the key changes each reboot so even a cached entry triggers the "REMOTE HOST IDENTIFICATION HAS CHANGED" warning.
• A LAN man-in-the-middle of admin SSH is undetectable client-side.

Approaches considered

• A — per-boot key + boot-time CA-signed host cert (CHOSEN). Keep the ephemeral per-boot host key; at boot, get it signed by the fleet SSH CA over a TPM-authenticated request. Clients trust via one @cert-authority line. The TPM only authenticates the signing request; it is not the host key.
• B — dedicated stable TPM host key + HostKeyAgent + cert at enrolment. Hardware-anchored, CA-independent, stable host identity — but needs a new persistent TPM handle, a host-key agent daemon (SELinux/fapolicyd/ordering), and more provisioning. Deferred; a follow-on if a CA-independent hardware host key is specifically wanted.

The certificate — not the key's origin — is what removes TOFU. A per-boot key with a cert is fine (short-lived keys are good hygiene), so A delivers the security outcome with the least new attack surface and no added cross-protocol key reuse. See the chat rationale; A was chosen 2026-07-11.

Architecture (Design A, broker-mediated)

The signing request goes through the enrol-broker, not directly to step-ca — so step-ca stays behind the single front door (consistent with enrolment), and the proven step CLI does the SSH signing server-side. The host authenticates to the broker's new host plane with its TPM identity over mTLS.

 boot ─▶ sshd-keygen@ecdsa ─▶ /etc/ssh/ssh_host_ecdsa_key(.pub)   (ephemeral)
                                     │
 ssh-hostcert.timer (OnBootSec + hourly)
        └─▶ fleet-ssh-hostcert:
              1. TPM: LoadSigner(0x81000100) + ReadNVCert(0x01800100) → mTLS client cert
              2. read ssh_host_ecdsa_key.pub
              3. POST {publicKey, principals, keyID} ─(mTLS)─▶ enrol-broker host plane
              4. write ssh_host_ecdsa_key-cert.pub + 60-hostcert.conf; reload sshd

 enrol-broker host plane (POST /ssh/hostcert):
        verify client cert to the fleet CA (⇒ enrolled host)
        └─▶ step ssh certificate --host --sign  (broker holds the provisioner) ─▶ step-ca SSH CA
            └─▶ returns {cert}

 client:  @cert-authority *.fleet.lab <ssh_host_ca.pub>   (one line, once)

Components

Piece	Where	Notes
fleet-ssh-hostcert	files/ssh-hostcert/ (Go, static, go-tpm)	TPM mTLS client; out-of-VM CI build like fleet-enrol/agent; deps pinned by go.sum
ssh-hostcert.service + .timer	files/ssh-hostcert/	root oneshot (writes /etc/ssh, reloads sshd), sandboxed; timer = boot + hourly renew
ssh-hostcert.env	files/ssh-hostcert/ → /etc/ssh-hostcert/	fleet-generic broker URL / CA / paths
Packer 2b-ssh-hostcert	diskless.pkr.hcl	gated on enable_ssh_hostcert; fapolicyd-trusts the binary in 2c-STIG-postfix
broker host plane (/ssh/hostcert)	serving/enrol-broker/host.go + sshsigner.go	mTLS (fleet-CA host certs); step ssh certificate signer; enabled by BROKER_HOST_ADDR
step-ca SSH CA enablement	serving/dot1x-bench/step-ca/ssh-ca-bootstrap.sh	SSH CA keys + provisioner --ssh + broker host-plane cert; prints the admin @cert-authority line

Wire contract (the BENCH-VERIFY seam)

• Host → broker (mTLS): POST /ssh/hostcert {publicKey, principals, keyID} → {cert}. The host's mTLS client cert is its TPM identity; the broker verifies it to the fleet CA (RequireAndVerifyClientCert) — that is the authorisation (an enrolled host). No token, no CA secret on the host.
• Broker → step-ca: step ssh certificate --host --sign … with the broker's provisioner credential. The exact flags are the tunable part (serving/enrol-broker/sshsigner.go); the CA must have its SSH host CA enabled and the provisioner allowed to issue SSH host certs (ssh-ca-bootstrap.sh).

Principals — name alignment (bench 2026-07-11). Admins connect by FQDN (host.<domain>), but the host defaulted its principal to the short hostname, so ssh host.fleet.lab failed with "name is not a listed principal" even though the cert was CA-valid. Fixed host-side: fleet-ssh-hostcert now also derives <hostname>.<domain> from the DHCP search domain (/etc/resolv.conf) at runtime — never baked (invariant #1). step-ca's default SSH host template already passes requested principals through, so no custom template is needed (verified). Still a sharp edge (defence-in-depth): the broker forwards whatever principals are requested; tightening handleSSHHostCert to validate them against the peer cert (so a host can only obtain its own names) remains a TODO.

Security properties

• Invariant #1 (no per-host state): the image carries only a fleet-generic broker DNS name + CA path. The host identity, and therefore the cert, derive from the TPM at runtime. Principals are the runtime hostname and its <hostname>.<domain> FQDN derived from the DHCP search domain — both learned at runtime, never baked.
• step-ca stays behind the front door: hosts reach the broker, not step-ca. The broker holds the provisioner credential; a host can never self-sign.
• Fail-open: never installs a missing/broken HostCertificate — a broker/CA outage degrades to today's bare-key behaviour, not a boot/sshd failure.
• Key reuse (cf. F-06): the TPM identity authenticates the mTLS request; the SSH host key is a separate ephemeral key.
• Short-lived certs + hourly renewal bound the blast radius of a leaked cert (the key is ephemeral anyway).

Bench outcome (2026-07-11)

Proven end-to-end on comet-beacon-54 (poam-hardening image, real TPM): the full chain — per-boot host key → TPM-mTLS to the broker → step-ca SSH signing → cert + 60-hostcert.conf install → sshd reload — works; an admin with one @cert-authority *.fleet.lab <ca> line logs in by FQDN with no TOFU prompt; and after a reboot ssh-hostcert.timer re-signs the fresh per-boot key automatically. SELinux confinement is clean (no AVCs writing /etc/ssh + reloading sshd → no local TE module needed), and STIG crypto policy accepts the ecdsa-sha2-nistp256-cert-v01 host cert.

Re-verified on the rebuilt 28d31e4 image (2026-07-11), not hand-patched: the auto-derived FQDN principal (<host>.<domain>) is baked, so admin login by FQDN with no TOFU survives a reboot with zero manual steps — the exact case the hand-fixed host regressed on (its manual SSH_HOSTCERT_PRINCIPALS edit lived on the RAM overlay and evaporated). F-02/P-17 is closed on a real build, not just a hand-fixed host.

Getting there surfaced defects, all fixed on poam-hardening (28d31e4):

#	Defect (as seen on the bench)	Fix
3	Broker dialed step-ca by the fleet dnsmasq name ca.fleet.lab, unresolvable from its host netns → no such host at sign time (502)	STEP_CA_URL=https://localhost:8443 (co-located; localhost is in the CA cert SAN)
4	ssh-ca-bootstrap.sh swallowed a failed provisioner SSH-enable → the CA 502s every sign	Fail loudly + verify enableSSHCA=true
6	ssh_host_ca.pub exported in PEM/PKCS8 → the @cert-authority line was unparseable → silent TOFU fallback	Export in OpenSSH wire format (ssh-keygen -i -m PKCS8)
7	Cert principal was the short hostname; admins connect by FQDN → "name is not a listed principal"	Derive <hostname>.<domain> from the DHCP domain at runtime
1	A second, mock-signer broker in serving/ collided on 9090/9443 and issued untrusted certs	Removed; the real broker (dot1x-bench, co-located with step-ca) is the only one
2	Broker HTML admin console = a weaker, client-cert-only parallel human-approve path	Retired; the fleet-manager passkey SPA drives the mTLS JSON API

Retracted finding: an early read suggested the cert had empty principals (valid for any host). That was a grep artifact hiding ssh-keygen -L's indented principal lines — the step-ca sign log confirmed the requested principal was present. No over-scoping; #7 is the real (name-alignment) issue.

Operational note: there is a brief post-boot window where sshd serves the bare per-boot key before ssh-hostcert.timer (OnBootSec) fires; an admin connecting in that window still sees TOFU. Tune OnBootSec/ordering to shrink it. (A client-side PQ-KEX advisory also appears on connect — an OpenSSH warning, not an F-02 issue.)

Deploy caveat: #7 is baked into the fleet-ssh-hostcert binary, so it only takes effect on a new image build from poam-hardening — until then the reboot no-TOFU test regresses (a short-name-only cert can't match an FQDN login). The serving-side fixes (#1–#4, #6) apply on docker compose up -d --build + a re-run of ssh-ca-bootstrap.sh.

Client rollout

One line per admin (or a system known_hosts), from ssh-ca-bootstrap.sh output (now emitted in OpenSSH wire format — see fix #6 above):

@cert-authority *.fleet.lab ecdsa-sha2-nistp256 AAAA...   # fleet SSH host CA

Bench checklist

Items 1–6 were verified 2026-07-11 (see Bench outcome); item 7 (broker-side principal validation) is the remaining defence-in-depth TODO.

1. Serving: run ssh-ca-bootstrap.sh; confirm ca.json has the ssh block, the enrol provisioner is SSH-enabled, and pki/broker/{host-server.crt, fleet-client-ca.crt} + pki/ssh_host_ca.pub exist. Bring up the broker with BROKER_HOST_ADDR set; journalctl shows "host plane up".
2. Broker: from an enrolled host's identity, curl --cert/--key … https:// boot.fleet.lab:9444/ssh/hostcert returns a {cert}; an un-enrolled client is rejected at the TLS handshake (RequireAndVerifyClientCert).
3. Host: systemctl start ssh-hostcert.service; journalctl -u ssh-hostcert shows "installed SSH host certificate"; /etc/ssh/ssh_host_ecdsa_key-cert.pub present; sshd -T | grep hostcertificate.
4. SELinux: watch for AVCs on the root service writing /etc/ssh/* and running systemctl reload sshd under confinement — add a local TE module if needed (like fleet_agent_chronylog). Reproducibility: a new semodule would rebuild the policy store → re-check verity-hash stability across two builds.
5. Client: with the @cert-authority line set, ssh <host> shows no host-key prompt/warning across reboots.
6. Confirm STIG crypto policy accepts the ecdsa-sha2-nistp256-cert-v01 host cert (same family already accepted for admin keys).
7. Principal policy: confirm the broker/step-ca only issue the principals a host is entitled to (tighten handleSSHHostCert to validate against the peer cert).

Deferred (Design B follow-on)

Stable, CA-independent hardware host key: a dedicated TPM key at a new persistent handle (e.g. 0x81000101, avoiding OEM/reserved NV like the identity key) served to sshd via a small TPM HostKeyAgent, cert issued at enrolment. Only pursue if a host identity that survives a CA outage is required.