Audiobuzz
0a3deca335
F-02/P-17 SSH host certs re-verified on the rebuilt 28d31e4 image (not the
hand-fixed host): the auto-derived FQDN principal is baked, so by-FQDN no-TOFU
admin login survives a reboot with zero manual steps. Cite the digest in the
design-doc status + bench-outcome, SSP v0.4 status line + R-16 row, and Annex A
2.4 row + R-16 register row.
200 lines
12 KiB
Markdown
200 lines
12 KiB
Markdown
# SSH host-certificate design — killing per-boot TOFU (F-02 / P-17)
|
||
|
||
> Status: **BENCH-VERIFIED end-to-end 2026-07-11 on clean build `28d31e4`** (`comet-beacon-54`, real TPM):
|
||
> TPM-mTLS → broker → step-ca signing → cert install → admin login **by FQDN with
|
||
> no TOFU**, and per-boot auto-re-sign across a reboot. Closes SAR finding **F-02**
|
||
> / POA&M **P-17**. Signing is **broker-mediated** (host → enrol-broker host plane →
|
||
> step-ca), not host-direct. The defects the bench surfaced are fixed on branch
|
||
> `poam-hardening` (commit `28d31e4`) — see **Bench outcome (2026-07-11)** below.
|
||
|
||
## Problem
|
||
|
||
The diskless root is an immutable erofs + a **RAM overlay**, so
|
||
`sshd-keygen@.service` regenerates the SSH **host keys on every boot** (see
|
||
`scripts/extract-artifacts.sh`, which strips baked host keys — a baked key would
|
||
be shared fleet-wide, invariant #1). Consequences:
|
||
|
||
- Administrators can never pin a host key — every `ssh` to a fleet host is
|
||
**trust-on-first-use**, and the key *changes each reboot* so even a cached entry
|
||
triggers the "REMOTE HOST IDENTIFICATION HAS CHANGED" warning.
|
||
- A **LAN man-in-the-middle** of admin SSH is undetectable client-side.
|
||
|
||
## Approaches considered
|
||
|
||
- **A — per-boot key + boot-time CA-signed host cert (CHOSEN).** Keep the
|
||
ephemeral per-boot host key; at boot, get it **signed by the fleet SSH CA** over
|
||
a TPM-authenticated request. Clients trust via one `@cert-authority` line. The
|
||
TPM only *authenticates the signing request*; it is not the host key.
|
||
- **B — dedicated stable TPM host key + `HostKeyAgent` + cert at enrolment.**
|
||
Hardware-anchored, CA-independent, stable host identity — but needs a new
|
||
persistent TPM handle, a host-key agent daemon (SELinux/fapolicyd/ordering), and
|
||
more provisioning. Deferred; a follow-on if a CA-independent hardware host key is
|
||
specifically wanted.
|
||
|
||
**The certificate — not the key's origin — is what removes TOFU.** A per-boot key
|
||
*with a cert* is fine (short-lived keys are good hygiene), so A delivers the
|
||
security outcome with the least new attack surface and no added cross-protocol key
|
||
reuse. See the chat rationale; A was chosen 2026-07-11.
|
||
|
||
## Architecture (Design A, broker-mediated)
|
||
|
||
The signing request goes **through the enrol-broker**, not directly to step-ca —
|
||
so step-ca stays behind the single front door (consistent with enrolment), and the
|
||
proven `step` CLI does the SSH signing server-side. The host authenticates to the
|
||
broker's new **host plane** with its TPM identity over mTLS.
|
||
|
||
```
|
||
boot ─▶ sshd-keygen@ecdsa ─▶ /etc/ssh/ssh_host_ecdsa_key(.pub) (ephemeral)
|
||
│
|
||
ssh-hostcert.timer (OnBootSec + hourly)
|
||
└─▶ fleet-ssh-hostcert:
|
||
1. TPM: LoadSigner(0x81000100) + ReadNVCert(0x01800100) → mTLS client cert
|
||
2. read ssh_host_ecdsa_key.pub
|
||
3. POST {publicKey, principals, keyID} ─(mTLS)─▶ enrol-broker host plane
|
||
4. write ssh_host_ecdsa_key-cert.pub + 60-hostcert.conf; reload sshd
|
||
|
||
enrol-broker host plane (POST /ssh/hostcert):
|
||
verify client cert to the fleet CA (⇒ enrolled host)
|
||
└─▶ step ssh certificate --host --sign (broker holds the provisioner) ─▶ step-ca SSH CA
|
||
└─▶ returns {cert}
|
||
|
||
client: @cert-authority *.fleet.lab <ssh_host_ca.pub> (one line, once)
|
||
```
|
||
|
||
**Components**
|
||
|
||
| Piece | Where | Notes |
|
||
|---|---|---|
|
||
| `fleet-ssh-hostcert` | `files/ssh-hostcert/` (Go, static, go-tpm) | TPM mTLS client; out-of-VM CI build like fleet-enrol/agent; deps pinned by go.sum |
|
||
| `ssh-hostcert.service` + `.timer` | `files/ssh-hostcert/` | **root** oneshot (writes `/etc/ssh`, reloads sshd), sandboxed; timer = boot + hourly renew |
|
||
| `ssh-hostcert.env` | `files/ssh-hostcert/` → `/etc/ssh-hostcert/` | fleet-generic broker URL / CA / paths |
|
||
| Packer `2b-ssh-hostcert` | `diskless.pkr.hcl` | gated on `enable_ssh_hostcert`; fapolicyd-trusts the binary in 2c-STIG-postfix |
|
||
| broker **host plane** (`/ssh/hostcert`) | `serving/enrol-broker/host.go` + `sshsigner.go` | mTLS (fleet-CA host certs); `step ssh certificate` signer; enabled by `BROKER_HOST_ADDR` |
|
||
| step-ca SSH CA enablement | `serving/dot1x-bench/step-ca/ssh-ca-bootstrap.sh` | SSH CA keys + provisioner `--ssh` + broker host-plane cert; prints the admin `@cert-authority` line |
|
||
|
||
## Wire contract (the BENCH-VERIFY seam)
|
||
|
||
- **Host → broker** (mTLS): `POST /ssh/hostcert` `{publicKey, principals, keyID}`
|
||
→ `{cert}`. The host's mTLS client cert is its TPM identity; the broker verifies
|
||
it to the fleet CA (`RequireAndVerifyClientCert`) — *that* is the authorisation
|
||
(an enrolled host). No token, no CA secret on the host.
|
||
- **Broker → step-ca**: `step ssh certificate --host --sign …` with the broker's
|
||
provisioner credential. The exact flags are the tunable part
|
||
(`serving/enrol-broker/sshsigner.go`); the CA must have its **SSH host CA
|
||
enabled** and the provisioner allowed to issue SSH host certs
|
||
(`ssh-ca-bootstrap.sh`).
|
||
|
||
**Principals — name alignment (bench 2026-07-11).** Admins connect by **FQDN**
|
||
(`host.<domain>`), but the host defaulted its principal to the **short** hostname,
|
||
so `ssh host.fleet.lab` failed with *"name is not a listed principal"* even though
|
||
the cert was CA-valid. Fixed host-side: `fleet-ssh-hostcert` now also derives
|
||
`<hostname>.<domain>` from the DHCP search domain (`/etc/resolv.conf`) at runtime —
|
||
never baked (invariant #1). step-ca's **default** SSH host template already passes
|
||
requested principals through, so **no custom template is needed** (verified).
|
||
**Still a sharp edge (defence-in-depth):** the broker forwards whatever principals
|
||
are requested; tightening `handleSSHHostCert` to validate them against the peer
|
||
cert (so a host can only obtain its own names) remains a TODO.
|
||
|
||
## Security properties
|
||
|
||
- **Invariant #1 (no per-host state):** the image carries only a fleet-generic
|
||
broker DNS name + CA path. The host identity, and therefore the cert, derive from
|
||
the TPM at runtime. Principals are the runtime hostname **and** its
|
||
`<hostname>.<domain>` FQDN derived from the DHCP search domain — both learned at
|
||
runtime, never baked.
|
||
- **step-ca stays behind the front door:** hosts reach the broker, not step-ca. The
|
||
broker holds the provisioner credential; a host can never self-sign.
|
||
- **Fail-open:** never installs a missing/broken `HostCertificate` — a broker/CA
|
||
outage degrades to today's bare-key behaviour, not a boot/sshd failure.
|
||
- **Key reuse (cf. F-06):** the TPM identity authenticates the mTLS request; the
|
||
SSH host key is a separate ephemeral key.
|
||
- **Short-lived certs** + hourly renewal bound the blast radius of a leaked cert
|
||
(the key is ephemeral anyway).
|
||
|
||
## Bench outcome (2026-07-11)
|
||
|
||
Proven end-to-end on `comet-beacon-54` (poam-hardening image, real TPM): the full
|
||
chain — per-boot host key → TPM-mTLS to the broker → step-ca SSH signing → cert +
|
||
`60-hostcert.conf` install → sshd reload — works; an admin with one
|
||
`@cert-authority *.fleet.lab <ca>` line logs in **by FQDN with no TOFU prompt**;
|
||
and after a reboot `ssh-hostcert.timer` re-signs the fresh per-boot key
|
||
automatically. **SELinux confinement is clean** (no AVCs writing `/etc/ssh` +
|
||
reloading sshd → **no local TE module needed**), and STIG crypto policy accepts the
|
||
`ecdsa-sha2-nistp256-cert-v01` host cert.
|
||
|
||
**Re-verified on the rebuilt `28d31e4` image (2026-07-11), not hand-patched:** the
|
||
auto-derived FQDN principal (`<host>.<domain>`) is baked, so admin login by FQDN
|
||
with no TOFU **survives a reboot with zero manual steps** — the exact case the
|
||
hand-fixed host regressed on (its manual `SSH_HOSTCERT_PRINCIPALS` edit lived on the
|
||
RAM overlay and evaporated). F-02/P-17 is closed on a real build, not just a
|
||
hand-fixed host.
|
||
|
||
Getting there surfaced defects, all fixed on `poam-hardening` (`28d31e4`):
|
||
|
||
| # | Defect (as seen on the bench) | Fix |
|
||
|---|---|---|
|
||
| 3 | Broker dialed step-ca by the fleet dnsmasq name `ca.fleet.lab`, unresolvable from its host netns → `no such host` at sign time (502) | `STEP_CA_URL=https://localhost:8443` (co-located; `localhost` is in the CA cert SAN) |
|
||
| 4 | `ssh-ca-bootstrap.sh` **swallowed** a failed provisioner SSH-enable → the CA 502s every sign | Fail loudly + verify `enableSSHCA=true` |
|
||
| 6 | `ssh_host_ca.pub` exported in PEM/PKCS8 → the `@cert-authority` line was unparseable → silent TOFU fallback | Export in OpenSSH wire format (`ssh-keygen -i -m PKCS8`) |
|
||
| 7 | Cert principal was the **short** hostname; admins connect by FQDN → *"name is not a listed principal"* | Derive `<hostname>.<domain>` from the DHCP domain at runtime |
|
||
| 1 | A second, mock-signer broker in `serving/` collided on 9090/9443 and issued untrusted certs | Removed; the real broker (dot1x-bench, co-located with step-ca) is the only one |
|
||
| 2 | Broker HTML admin console = a weaker, client-cert-only parallel human-approve path | Retired; the fleet-manager passkey SPA drives the mTLS JSON API |
|
||
|
||
**Retracted finding:** an early read suggested the cert had *empty* principals
|
||
(valid for any host). That was a `grep` artifact hiding `ssh-keygen -L`'s indented
|
||
principal lines — the step-ca sign log confirmed the requested principal was
|
||
present. No over-scoping; #7 is the real (name-alignment) issue.
|
||
|
||
**Operational note:** there is a brief post-boot window where sshd serves the bare
|
||
per-boot key before `ssh-hostcert.timer` (OnBootSec) fires; an admin connecting in
|
||
that window still sees TOFU. Tune `OnBootSec`/ordering to shrink it. (A client-side
|
||
PQ-KEX advisory also appears on connect — an OpenSSH warning, not an F-02 issue.)
|
||
|
||
**Deploy caveat:** #7 is baked into the `fleet-ssh-hostcert` binary, so it only
|
||
takes effect on a **new image build** from poam-hardening — until then the reboot
|
||
no-TOFU test regresses (a short-name-only cert can't match an FQDN login). The
|
||
serving-side fixes (#1–#4, #6) apply on `docker compose up -d --build` + a re-run of
|
||
`ssh-ca-bootstrap.sh`.
|
||
|
||
## Client rollout
|
||
|
||
One line per admin (or a system `known_hosts`), from `ssh-ca-bootstrap.sh` output
|
||
(now emitted in OpenSSH wire format — see fix #6 above):
|
||
|
||
```
|
||
@cert-authority *.fleet.lab ecdsa-sha2-nistp256 AAAA... # fleet SSH host CA
|
||
```
|
||
|
||
## Bench checklist
|
||
|
||
Items 1–6 were **verified 2026-07-11** (see Bench outcome); item 7 (broker-side
|
||
principal validation) is the remaining defence-in-depth TODO.
|
||
|
||
1. Serving: run `ssh-ca-bootstrap.sh`; confirm `ca.json` has the `ssh` block, the
|
||
`enrol` provisioner is SSH-enabled, and `pki/broker/{host-server.crt,
|
||
fleet-client-ca.crt}` + `pki/ssh_host_ca.pub` exist. Bring up the broker with
|
||
`BROKER_HOST_ADDR` set; `journalctl` shows "host plane up".
|
||
2. Broker: from an enrolled host's identity, `curl --cert/--key … https://
|
||
boot.fleet.lab:9444/ssh/hostcert` returns a `{cert}`; an un-enrolled client is
|
||
rejected at the TLS handshake (`RequireAndVerifyClientCert`).
|
||
3. Host: `systemctl start ssh-hostcert.service`; `journalctl -u ssh-hostcert`
|
||
shows "installed SSH host certificate"; `/etc/ssh/ssh_host_ecdsa_key-cert.pub`
|
||
present; `sshd -T | grep hostcertificate`.
|
||
4. SELinux: watch for AVCs on the root service writing `/etc/ssh/*` and running
|
||
`systemctl reload sshd` under confinement — add a local TE module if needed
|
||
(like `fleet_agent_chronylog`). Reproducibility: a new `semodule` would rebuild
|
||
the policy store → re-check verity-hash stability across two builds.
|
||
5. Client: with the `@cert-authority` line set, `ssh <host>` shows **no** host-key
|
||
prompt/warning across reboots.
|
||
6. Confirm STIG crypto policy accepts the `ecdsa-sha2-nistp256-cert-v01` host cert
|
||
(same family already accepted for admin keys).
|
||
7. Principal policy: confirm the broker/step-ca only issue the principals a host is
|
||
entitled to (tighten `handleSSHHostCert` to validate against the peer cert).
|
||
|
||
## Deferred (Design B follow-on)
|
||
|
||
Stable, CA-independent hardware host key: a dedicated TPM key at a new persistent
|
||
handle (e.g. `0x81000101`, avoiding OEM/reserved NV like the identity key) served
|
||
to sshd via a small TPM `HostKeyAgent`, cert issued at enrolment. Only pursue if a
|
||
host identity that survives a CA outage is required.
|