# SSH host-certificate design — killing per-boot TOFU (F-02 / P-17)

> Status: **BENCH-VERIFIED end-to-end 2026-07-11 on clean build `28d31e4`** (`comet-beacon-54`, real TPM):
> TPM-mTLS → broker → step-ca signing → cert install → admin login **by FQDN with
> no TOFU**, and per-boot auto-re-sign across a reboot. Closes SAR finding **F-02**
> / POA&M **P-17**. Signing is **broker-mediated** (host → enrol-broker host plane →
> step-ca), not host-direct. The defects the bench surfaced are fixed on branch
> `poam-hardening` (commit `28d31e4`) — see **Bench outcome (2026-07-11)** below.

## Problem

The diskless root is an immutable erofs + a **RAM overlay**, so
`sshd-keygen@.service` regenerates the SSH **host keys on every boot** (see
`scripts/extract-artifacts.sh`, which strips baked host keys — a baked key would
be shared fleet-wide, invariant #1). Consequences:

- Administrators can never pin a host key — every `ssh` to a fleet host is
  **trust-on-first-use**, and the key *changes each reboot* so even a cached entry
  triggers the "REMOTE HOST IDENTIFICATION HAS CHANGED" warning.
- A **LAN man-in-the-middle** of admin SSH is undetectable client-side.

## Approaches considered

- **A — per-boot key + boot-time CA-signed host cert (CHOSEN).** Keep the
  ephemeral per-boot host key; at boot, get it **signed by the fleet SSH CA** over
  a TPM-authenticated request. Clients trust via one `@cert-authority` line. The
  TPM only *authenticates the signing request*; it is not the host key.
- **B — dedicated stable TPM host key + `HostKeyAgent` + cert at enrolment.**
  Hardware-anchored, CA-independent, stable host identity — but needs a new
  persistent TPM handle, a host-key agent daemon (SELinux/fapolicyd/ordering), and
  more provisioning. Deferred; a follow-on if a CA-independent hardware host key is
  specifically wanted.

**The certificate — not the key's origin — is what removes TOFU.** A per-boot key
*with a cert* is fine (short-lived keys are good hygiene), so A delivers the
security outcome with the least new attack surface and no added cross-protocol key
reuse. See the chat rationale; A was chosen 2026-07-11.

## Architecture (Design A, broker-mediated)

The signing request goes **through the enrol-broker**, not directly to step-ca —
so step-ca stays behind the single front door (consistent with enrolment), and the
proven `step` CLI does the SSH signing server-side. The host authenticates to the
broker's new **host plane** with its TPM identity over mTLS.

```
 boot ─▶ sshd-keygen@ecdsa ─▶ /etc/ssh/ssh_host_ecdsa_key(.pub)   (ephemeral)
                                     │
 ssh-hostcert.timer (OnBootSec + hourly)
        └─▶ fleet-ssh-hostcert:
              1. TPM: LoadSigner(0x81000100) + ReadNVCert(0x01800100) → mTLS client cert
              2. read ssh_host_ecdsa_key.pub
              3. POST {publicKey, principals, keyID} ─(mTLS)─▶ enrol-broker host plane
              4. write ssh_host_ecdsa_key-cert.pub + 60-hostcert.conf; reload sshd

 enrol-broker host plane (POST /ssh/hostcert):
        verify client cert to the fleet CA (⇒ enrolled host)
        └─▶ step ssh certificate --host --sign  (broker holds the provisioner) ─▶ step-ca SSH CA
            └─▶ returns {cert}

 client:  @cert-authority *.fleet.lab <ssh_host_ca.pub>   (one line, once)
```

**Components**

| Piece | Where | Notes |
|---|---|---|
| `fleet-ssh-hostcert` | `files/ssh-hostcert/` (Go, static, go-tpm) | TPM mTLS client; out-of-VM CI build like fleet-enrol/agent; deps pinned by go.sum |
| `ssh-hostcert.service` + `.timer` | `files/ssh-hostcert/` | **root** oneshot (writes `/etc/ssh`, reloads sshd), sandboxed; timer = boot + hourly renew |
| `ssh-hostcert.env` | `files/ssh-hostcert/` → `/etc/ssh-hostcert/` | fleet-generic broker URL / CA / paths |
| Packer `2b-ssh-hostcert` | `diskless.pkr.hcl` | gated on `enable_ssh_hostcert`; fapolicyd-trusts the binary in 2c-STIG-postfix |
| broker **host plane** (`/ssh/hostcert`) | `serving/enrol-broker/host.go` + `sshsigner.go` | mTLS (fleet-CA host certs); `step ssh certificate` signer; enabled by `BROKER_HOST_ADDR` |
| step-ca SSH CA enablement | `serving/dot1x-bench/step-ca/ssh-ca-bootstrap.sh` | SSH CA keys + provisioner `--ssh` + broker host-plane cert; prints the admin `@cert-authority` line |

## Wire contract (the BENCH-VERIFY seam)

- **Host → broker** (mTLS): `POST /ssh/hostcert` `{publicKey, principals, keyID}`
  → `{cert}`. The host's mTLS client cert is its TPM identity; the broker verifies
  it to the fleet CA (`RequireAndVerifyClientCert`) — *that* is the authorisation
  (an enrolled host). No token, no CA secret on the host.
- **Broker → step-ca**: `step ssh certificate --host --sign …` with the broker's
  provisioner credential. The exact flags are the tunable part
  (`serving/enrol-broker/sshsigner.go`); the CA must have its **SSH host CA
  enabled** and the provisioner allowed to issue SSH host certs
  (`ssh-ca-bootstrap.sh`).

**Principals — name alignment (bench 2026-07-11).** Admins connect by **FQDN**
(`host.<domain>`), but the host defaulted its principal to the **short** hostname,
so `ssh host.fleet.lab` failed with *"name is not a listed principal"* even though
the cert was CA-valid. Fixed host-side: `fleet-ssh-hostcert` now also derives
`<hostname>.<domain>` from the DHCP search domain (`/etc/resolv.conf`) at runtime —
never baked (invariant #1). step-ca's **default** SSH host template already passes
requested principals through, so **no custom template is needed** (verified).
**Still a sharp edge (defence-in-depth):** the broker forwards whatever principals
are requested; tightening `handleSSHHostCert` to validate them against the peer
cert (so a host can only obtain its own names) remains a TODO.

## Security properties

- **Invariant #1 (no per-host state):** the image carries only a fleet-generic
  broker DNS name + CA path. The host identity, and therefore the cert, derive from
  the TPM at runtime. Principals are the runtime hostname **and** its
  `<hostname>.<domain>` FQDN derived from the DHCP search domain — both learned at
  runtime, never baked.
- **step-ca stays behind the front door:** hosts reach the broker, not step-ca. The
  broker holds the provisioner credential; a host can never self-sign.
- **Fail-open:** never installs a missing/broken `HostCertificate` — a broker/CA
  outage degrades to today's bare-key behaviour, not a boot/sshd failure.
- **Key reuse (cf. F-06):** the TPM identity authenticates the mTLS request; the
  SSH host key is a separate ephemeral key.
- **Short-lived certs** + hourly renewal bound the blast radius of a leaked cert
  (the key is ephemeral anyway).

## Bench outcome (2026-07-11)

Proven end-to-end on `comet-beacon-54` (poam-hardening image, real TPM): the full
chain — per-boot host key → TPM-mTLS to the broker → step-ca SSH signing → cert +
`60-hostcert.conf` install → sshd reload — works; an admin with one
`@cert-authority *.fleet.lab <ca>` line logs in **by FQDN with no TOFU prompt**;
and after a reboot `ssh-hostcert.timer` re-signs the fresh per-boot key
automatically. **SELinux confinement is clean** (no AVCs writing `/etc/ssh` +
reloading sshd → **no local TE module needed**), and STIG crypto policy accepts the
`ecdsa-sha2-nistp256-cert-v01` host cert.

**Re-verified on the rebuilt `28d31e4` image (2026-07-11), not hand-patched:** the
auto-derived FQDN principal (`<host>.<domain>`) is baked, so admin login by FQDN
with no TOFU **survives a reboot with zero manual steps** — the exact case the
hand-fixed host regressed on (its manual `SSH_HOSTCERT_PRINCIPALS` edit lived on the
RAM overlay and evaporated). F-02/P-17 is closed on a real build, not just a
hand-fixed host.

Getting there surfaced defects, all fixed on `poam-hardening` (`28d31e4`):

| # | Defect (as seen on the bench) | Fix |
|---|---|---|
| 3 | Broker dialed step-ca by the fleet dnsmasq name `ca.fleet.lab`, unresolvable from its host netns → `no such host` at sign time (502) | `STEP_CA_URL=https://localhost:8443` (co-located; `localhost` is in the CA cert SAN) |
| 4 | `ssh-ca-bootstrap.sh` **swallowed** a failed provisioner SSH-enable → the CA 502s every sign | Fail loudly + verify `enableSSHCA=true` |
| 6 | `ssh_host_ca.pub` exported in PEM/PKCS8 → the `@cert-authority` line was unparseable → silent TOFU fallback | Export in OpenSSH wire format (`ssh-keygen -i -m PKCS8`) |
| 7 | Cert principal was the **short** hostname; admins connect by FQDN → *"name is not a listed principal"* | Derive `<hostname>.<domain>` from the DHCP domain at runtime |
| 1 | A second, mock-signer broker in `serving/` collided on 9090/9443 and issued untrusted certs | Removed; the real broker (dot1x-bench, co-located with step-ca) is the only one |
| 2 | Broker HTML admin console = a weaker, client-cert-only parallel human-approve path | Retired; the fleet-manager passkey SPA drives the mTLS JSON API |

**Retracted finding:** an early read suggested the cert had *empty* principals
(valid for any host). That was a `grep` artifact hiding `ssh-keygen -L`'s indented
principal lines — the step-ca sign log confirmed the requested principal was
present. No over-scoping; #7 is the real (name-alignment) issue.

**Operational note:** there is a brief post-boot window where sshd serves the bare
per-boot key before `ssh-hostcert.timer` (OnBootSec) fires; an admin connecting in
that window still sees TOFU. Tune `OnBootSec`/ordering to shrink it. (A client-side
PQ-KEX advisory also appears on connect — an OpenSSH warning, not an F-02 issue.)

**Deploy caveat:** #7 is baked into the `fleet-ssh-hostcert` binary, so it only
takes effect on a **new image build** from poam-hardening — until then the reboot
no-TOFU test regresses (a short-name-only cert can't match an FQDN login). The
serving-side fixes (#1–#4, #6) apply on `docker compose up -d --build` + a re-run of
`ssh-ca-bootstrap.sh`.

## Client rollout

One line per admin (or a system `known_hosts`), from `ssh-ca-bootstrap.sh` output
(now emitted in OpenSSH wire format — see fix #6 above):

```
@cert-authority *.fleet.lab ecdsa-sha2-nistp256 AAAA...   # fleet SSH host CA
```

## Bench checklist

Items 1–6 were **verified 2026-07-11** (see Bench outcome); item 7 (broker-side
principal validation) is the remaining defence-in-depth TODO.

1. Serving: run `ssh-ca-bootstrap.sh`; confirm `ca.json` has the `ssh` block, the
   `enrol` provisioner is SSH-enabled, and `pki/broker/{host-server.crt,
   fleet-client-ca.crt}` + `pki/ssh_host_ca.pub` exist. Bring up the broker with
   `BROKER_HOST_ADDR` set; `journalctl` shows "host plane up".
2. Broker: from an enrolled host's identity, `curl --cert/--key … https://
   boot.fleet.lab:9444/ssh/hostcert` returns a `{cert}`; an un-enrolled client is
   rejected at the TLS handshake (`RequireAndVerifyClientCert`).
3. Host: `systemctl start ssh-hostcert.service`; `journalctl -u ssh-hostcert`
   shows "installed SSH host certificate"; `/etc/ssh/ssh_host_ecdsa_key-cert.pub`
   present; `sshd -T | grep hostcertificate`.
4. SELinux: watch for AVCs on the root service writing `/etc/ssh/*` and running
   `systemctl reload sshd` under confinement — add a local TE module if needed
   (like `fleet_agent_chronylog`). Reproducibility: a new `semodule` would rebuild
   the policy store → re-check verity-hash stability across two builds.
5. Client: with the `@cert-authority` line set, `ssh <host>` shows **no** host-key
   prompt/warning across reboots.
6. Confirm STIG crypto policy accepts the `ecdsa-sha2-nistp256-cert-v01` host cert
   (same family already accepted for admin keys).
7. Principal policy: confirm the broker/step-ca only issue the principals a host is
   entitled to (tighten `handleSSHHostCert` to validate against the peer cert).

## Deferred (Design B follow-on)

Stable, CA-independent hardware host key: a dedicated TPM key at a new persistent
handle (e.g. `0x81000101`, avoiding OEM/reserved NV like the identity key) served
to sshd via a small TPM `HostKeyAgent`, cert issued at enrolment. Only pursue if a
host identity that survives a CA outage is required.