sunjizzle/diskless-workstation
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

ssh-hostcert + enrol-broker: fix defects surfaced by the F-02 bench

Browse Source 
F-02/P-17 SSH host-cert flow was proven end-to-end on comet-beacon-54
(2026-07-11); these fix the correctness/scoping/cleanup defects the bench
surfaced:

- ssh-hostcert: derive the FQDN principal <hostname>.<domain> from the DHCP
  search domain at runtime (resolv.conf), alongside the short hostname, so an
  admin `@cert-authority *.<domain>` line accepts by-FQDN logins instead of
  "name is not a listed principal". Never baked (invariant #1). + tests.
- ssh-ca-bootstrap.sh: export the SSH host CA public key in OpenSSH wire format
  (ssh-keygen -i -m PKCS8), not the PEM/PKCS8 that `step crypto keypair` writes
  -- a PEM @cert-authority anchor is unparseable and silently falls back to TOFU.
- ssh-ca-bootstrap.sh: fail loudly (and verify enableSSHCA=true) if the
  provisioner SSH-enable step fails, instead of swallowing it and shipping a CA
  that 502s every runtime sign.
- dot1x-bench: point the co-located broker at https://localhost:8443, not the
  fleet dnsmasq name ca.fleet.lab (unresolvable from the broker's host netns ->
  "no such host" at sign time). localhost is in the CA cert SAN.
- serving: remove the mock-signer enrol-broker copy -- two host-networked
  brokers collide on 9090/9443 and the mock issued untrusted certs. The real
  broker lives with step-ca (dot1x-bench); nginx /enrol/ still proxies to it on
  loopback.
- enrol-broker: retire the HTML admin console; the fleet-manager passkey SPA now
  drives the mTLS JSON API (/pending, /approve|deny). Removes a weaker,
  client-cert-only parallel human-approve path.

Design-doc bench note deferred: docs/ssh-host-certificate-design.md lives on the
secure-ntp branch (fbb8100), not here -- to be added there / on reconcile.
This commit is contained in:
Audiobuzz
2026-07-11 09:44:33 +00:00
parent 8325818c1b
commit 28d31e436c
9 changed files with 207 additions and 177 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serving/defaults.env
+7 -32
View File
@@ -104,38 +104,13 @@ SUBNQN=nqn.2026-06.org.example.fleet:rootfs
TARGET_PORT=4420
SPDK_CPUMASK=0x1 # one polling core is plenty for a read-only boot fs
# --- TPM enrolment broker (TEMPORARY in this stack — see docs/TODO.md #11) ---
# The broker is the one unauthenticated pre-auth endpoint (nginx proxies /enrol/
# to it). Defaults run the zero-dependency LAB path: a `mock` signer (ephemeral
# CA; certs NOT trusted by RADIUS/NVMe) + plaintext loopback admin — enough to
# exercise the request/approve/poll flow while the host side is built.
#
# For REAL enrolment: set ENROL_SIGNER=step, point ENROL_STEP_CA_URL at a
# reachable step-ca (e.g. the one in serving/dot1x-bench/), drop its root +
# provisioner password into ./enrol-pki/, generate the admin mTLS PKI
# (./enrol-broker/admin-pki.sh ./enrol-pki/broker <admin>), and set
# ENROL_ADMIN_INSECURE=false. Then it belongs in its own CA-co-located stack.
ENROL_SIGNER=mock # mock (lab, no CA) | step (real, needs a CA)
ENROL_ADMIN_INSECURE=true # true = plaintext loopback admin (LAB ONLY); false = mTLS
ENROL_PENDING_TTL=10m # time-box on unapproved requests
ENROL_WORDS=8 # fingerprint word count
ENROL_HOSTNQN_BASE=nqn.2026-06.org.example.fleet:host-
# step signer (only when ENROL_SIGNER=step) — paths are INSIDE the container
# (./enrol-pki is mounted at /pki):
ENROL_STEP_CA_URL= # e.g. https://ca.fleet.lab:8443
ENROL_STEP_CA_ROOT=/pki/root_ca.crt
ENROL_STEP_PROVISIONER=enrol
ENROL_STEP_PASSWORD_FILE=/pki/password
# Issued-cert lifetime. NOTE: the step-ca provisioner must ALLOW this — a stock
# step-ca caps certs at 24h, so 720h is rejected ("requested duration ... more
# than the authorized maximum certificate duration of 24h"). The bench
# step-ca/bootstrap.sh now raises authority.claims.maxTLSCertDuration; a
# different CA needs the same. Verified end-to-end against a real step-ca.
ENROL_CERT_NOT_AFTER=720h
# admin mTLS (only when ENROL_ADMIN_INSECURE=false):
ENROL_ADMIN_CERT=/pki/broker/server.crt
ENROL_ADMIN_KEY=/pki/broker/server.key
ENROL_ADMIN_CLIENT_CA=/pki/broker/admin-ca.crt
# --- TPM enrolment broker — configured in the CA-co-located stack -----------
# The broker no longer runs in this stack (a mock-signer copy here collided with
# the real one on 9090/9443 and issued untrusted certs — removed 2026-07-11). It
# lives in serving/dot1x-bench/ alongside step-ca (real signing needs the CA), and
# the `http` nginx here still proxies /enrol/ -> 127.0.0.1:9090 to reach it on the
# same host. Configure the broker (signer, provisioner, admin mTLS) in
# serving/dot1x-bench/.env, not here.
# --- SPDK version loader / reaper (declarative rolling updates) -------------
# The spdk-loader service stages + adds ${IMAGE_VERSION} to the long-lived target
serving/docker-compose.yml
+9 -44
View File
@@ -193,50 +193,15 @@ services:
# For HTTPS boot (production), terminate TLS here with the boot-infra CA's
# cert and switch the dhcp-boot URL above to https://. Left HTTP for the lab.
# --- TPM enrolment broker (TEMPORARY home — see docs/TODO.md #11) -----------
# The `http` nginx already reverse-proxies /enrol/ -> 127.0.0.1:9090 (this
# broker's public listener), so an empty-NV host can POST its CSR and poll for
# the signed cert via the normal serving host. For now the broker runs HERE so
# a single `compose up` in serving/ brings the enrolment endpoint up.
#
# ⚠ PROPER HOME is its own stack co-located with step-ca (the fleet CA), i.e.
# serving/dot1x-bench/, since REAL signing needs that CA. SPLIT THIS BACK OUT
# once the CA lives beside it. Do NOT run this AND dot1x-bench's enrol-broker on
# the same host — they collide on 9090/9443.
#
# Defaults below are the zero-dependency LAB path: `mock` signer (ephemeral
# in-process CA — certs are NOT trusted by RADIUS/NVMe) + plaintext loopback
# admin. For REAL enrolment, in .env set ENROL_SIGNER=step + point
# ENROL_STEP_CA_* at a reachable step-ca, and set ENROL_ADMIN_INSECURE=false
# after generating the admin mTLS PKI (a DEDICATED admin CA, not the fleet root):
# ./enrol-broker/admin-pki.sh ./enrol-pki/broker <admin-name>
# then import enrol-pki/broker/clients/<admin>.p12 and open https://127.0.0.1:9443/
enrol-broker:
build: ./enrol-broker
network_mode: host # public on 127.0.0.1:9090 (nginx proxies /enrol/)
restart: unless-stopped
environment:
BROKER_PUBLIC_ADDR: "127.0.0.1:9090" # nginx /enrol/ proxies here
BROKER_ADMIN_ADDR: "127.0.0.1:9443" # mgmt/loopback ONLY — never the boot LAN
BROKER_SIGNER: ${ENROL_SIGNER:-mock}
BROKER_PENDING_TTL: ${ENROL_PENDING_TTL:-10m}
BROKER_WORDS: ${ENROL_WORDS:-8}
HOSTNQN_BASE: ${ENROL_HOSTNQN_BASE:-nqn.2026-06.org.example.fleet:host-}
# --- step signer (only consulted when ENROL_SIGNER=step) ---
STEP_CA_URL: ${ENROL_STEP_CA_URL:-}
STEP_CA_ROOT: ${ENROL_STEP_CA_ROOT:-/pki/root_ca.crt}
STEP_PROVISIONER: ${ENROL_STEP_PROVISIONER:-enrol}
STEP_PROVISIONER_PASSWORD_FILE: ${ENROL_STEP_PASSWORD_FILE:-/pki/password}
STEP_CERT_NOT_AFTER: ${ENROL_CERT_NOT_AFTER:-720h}
# --- admin mTLS (set ENROL_ADMIN_INSECURE=false for real mTLS) ---
BROKER_ADMIN_INSECURE: ${ENROL_ADMIN_INSECURE:-true}
BROKER_ADMIN_CERT: ${ENROL_ADMIN_CERT:-/pki/broker/server.crt}
BROKER_ADMIN_KEY: ${ENROL_ADMIN_KEY:-/pki/broker/server.key}
BROKER_ADMIN_CLIENT_CA: ${ENROL_ADMIN_CLIENT_CA:-/pki/broker/admin-ca.crt}
volumes:
# CA root + provisioner password (step signer) and admin mTLS certs live
# here; unused/empty for the default mock+insecure path. Gitignored.
- ./enrol-pki:/pki:ro
# --- TPM enrolment broker — NOT defined here (lives in the CA stack) --------
# The broker's home is co-located with step-ca (the fleet CA), i.e.
# serving/dot1x-bench/, since real signing needs that CA. A `mock`-signer
# convenience copy used to run here too, but two host-networked brokers collide
# on 9090/9443 and the mock copy issued certs RADIUS/NVMe don't trust — so it was
# removed (bench 2026-07-11; see docs/ssh-host-certificate-design.md). The `http`
# nginx still reverse-proxies /enrol/ -> 127.0.0.1:9090; bring up
# serving/dot1x-bench on the same host (its broker binds that loopback port) to
# serve enrolment. Do NOT re-add a second broker here.
# --- Kiosk SPA (spaceship terminal shown by the diskless kiosk) ------------
# Svelte + Tailwind app built into a static bundle served by its own nginx.
serving/dot1x-bench/docker-compose.yml
+6 -1
View File
@@ -70,7 +70,12 @@ services:
BROKER_PUBLIC_ADDR: "127.0.0.1:9090"
BROKER_ADMIN_ADDR: "127.0.0.1:9443"
BROKER_SIGNER: "step"
STEP_CA_URL: "https://${CA_DNS:-ca.fleet.lab}:${CA_PORT:-8443}"
# Co-located with step-ca (both network_mode: host), so reach it over
# loopback — NOT the fleet DNS name. ca.fleet.lab is a dnsmasq name for
# boot-LAN clients; the broker's host resolver doesn't know it, so signing
# died with "lookup ca.fleet.lab: no such host" (bench 2026-07-11). localhost
# is always in the CA cert SAN (DOCKER_STEPCA_INIT_DNS_NAMES) so TLS verifies.
STEP_CA_URL: "https://localhost:${CA_PORT:-8443}"
STEP_CA_ROOT: "/pki/root_ca.crt"
STEP_PROVISIONER: "enrol"
STEP_PROVISIONER_PASSWORD_FILE: "/pki/stepca/secrets/password"
serving/dot1x-bench/step-ca/ssh-ca-bootstrap.sh
+40 -6
View File
@@ -82,10 +82,33 @@ fi
# The broker authenticates to step-ca with this JWK provisioner (signer.go). It
# must be allowed to issue SSH HOST certs. BENCH-VERIFY the exact subcommand/flag
# on your step version (some use `provisioner update <name> --ssh`).
echo "==> enabling SSH on provisioner '${PROVISIONER}' (BENCH-VERIFY the flag)"
incca step ca provisioner update "${PROVISIONER}" --ssh 2>/dev/null \
&& restart_ca \
|| echo " NOTE: could not auto-enable SSH on '${PROVISIONER}' — enable it by hand (step ca provisioner update ${PROVISIONER} --ssh) and restart step-ca."
echo "==> enabling SSH host-cert issuance on provisioner '${PROVISIONER}'"
# FAIL LOUDLY: a swallowed failure here leaves enableSSHCA=false, and every
# `step ssh certificate` at runtime dies with a 502 whose cause the operator
# can't see (bench 2026-07-11). If your `step` uses a different flag, fix it and
# re-run rather than shipping a half-configured CA.
if ! incca step ca provisioner update "${PROVISIONER}" --ssh; then
echo "ERROR: 'step ca provisioner update ${PROVISIONER} --ssh' failed. Your step" >&2
echo " version may use a different flag — check: step ca provisioner update --help" >&2
echo " Aborting so SSH host-cert signing isn't silently left disabled." >&2
exit 1
fi
restart_ca
# Verify it actually took — the CA only issues SSH host certs if enableSSHCA=true.
if ! python3 - "$CA_JSON" "$PROVISIONER" <<'PY'
import json,sys
c=json.load(open(sys.argv[1])); name=sys.argv[2]
for p in c["authority"]["provisioners"]:
if p.get("name")==name and p.get("claims",{}).get("enableSSHCA") is True:
sys.exit(0)
sys.exit(1)
PY
then
echo "ERROR: provisioner '${PROVISIONER}' still not SSH-enabled (enableSSHCA != true)" >&2
echo " after the update + restart. SSH host-cert signing would fail at runtime; aborting." >&2
exit 1
fi
echo " SSH host-cert issuance enabled on '${PROVISIONER}' (enableSSHCA=true)."
# --- 3. broker host-plane server cert + fleet client-CA bundle --------------
# The host plane is mTLS: its SERVER cert must be FLEET-signed (the booted host
@@ -104,8 +127,19 @@ cat "$PKI/root_ca.crt" > "$PKI/broker/fleet-client-ca.crt"
[ -f "$PKI/intermediate_ca.crt" ] && cat "$PKI/intermediate_ca.crt" >> "$PKI/broker/fleet-client-ca.crt"
# --- 4. export the SSH host CA public key (admin trust anchor) ---------------
echo "==> exporting SSH host CA public key -> $PKI/ssh_host_ca.pub"
incca cp /home/step/secrets/ssh_host_ca_key.pub /export/ssh_host_ca.pub
# `step crypto keypair` writes the public key in PEM/PKCS8, but a known_hosts
# `@cert-authority` line needs the single-line OpenSSH wire format
# (`ecdsa-sha2-nistp256 AAAA…`). Emitting the PEM verbatim yields an unparseable
# anchor → ssh silently falls back to TOFU (bench 2026-07-11). Convert it.
echo "==> exporting SSH host CA public key (OpenSSH format) -> $PKI/ssh_host_ca.pub"
incca cp /home/step/secrets/ssh_host_ca_key.pub /export/ssh_host_ca_key.pem
command -v ssh-keygen >/dev/null 2>&1 || {
echo "ERROR: ssh-keygen not found — cannot convert the CA key to OpenSSH format." >&2
echo " Install openssh-clients; the raw PKCS8 key is at $PKI/ssh_host_ca_key.pem." >&2
exit 1
}
ssh-keygen -i -m PKCS8 -f "$PKI/ssh_host_ca_key.pem" > "$PKI/ssh_host_ca.pub"
rm -f "$PKI/ssh_host_ca_key.pem"
echo
echo "Admin rollout — add ONE line to each admin's ~/.ssh/known_hosts (or a system"
echo "known_hosts), so every fleet host is trusted via the CA instead of TOFU:"
serving/enrol-broker/README.md
+6 -4
View File
@@ -19,7 +19,7 @@ PUBLIC (plain HTTP, behind the serving nginx /enrol/) UNAUTHENTICATED
GET /healthz
ADMIN (mTLS, management network / loopback ONLY) admin client cert
GET / the pending-queue web UI (fingerprint + approve/deny)
GET / plaintext pointer (API only; human UI is the fleet-manager SPA)
GET /pending JSON queue
POST /approve/{id} OOB-approve -> step ca sign -> store cert
POST /deny/{id} reject
@@ -74,9 +74,11 @@ docker compose up -d step-ca && ./step-ca/bootstrap.sh # CA + root_ca.crt + en
docker compose up -d enrol-broker
```
Then import `pki/broker/clients/alice.p12` (empty password) into your browser and
open `https://127.0.0.1:9443/`. Hosts reach the public side via the serving
stack's nginx: `http://boot.fleet.lab/enrol/`.
Approve/deny is driven by the **fleet-manager SPA** (passkey-gated), which calls
this plane's JSON API over mTLS; the broker no longer serves its own HTML console.
For a scripted/manual approve, use the admin client cert directly against the API,
e.g. `curl --cert/--key … https://127.0.0.1:9443/pending`. Hosts reach the public
side via the serving stack's nginx: `http://boot.fleet.lab/enrol/`.
### Admin mTLS PKI — important
serving/enrol-broker/admin.go
+21 -84
View File
@@ -1,4 +1,4 @@
// admin.go — the mTLS admin plane: the pending-queue web UI + approve/deny.
// admin.go — the mTLS admin plane: the pending-queue + approve/deny JSON API.
//
// This listener is the AUTHENTICATED half of the broker and must live on the
// management network / loopback only — NEVER the boot LAN (docs invariant: the
@@ -11,7 +11,6 @@ package main
import (
"crypto/tls"
"crypto/x509"
"html/template"
"log"
"net/http"
"os"
@@ -21,7 +20,7 @@ import (
func (b *broker) serveAdmin() {
mux := http.NewServeMux()
mux.HandleFunc("GET /", b.handleAdminIndex)
mux.HandleFunc("GET /", b.handleAdminRoot)
mux.HandleFunc("GET /pending", b.handleAdminPending)
mux.HandleFunc("POST /approve/{id}", b.handleApprove)
mux.HandleFunc("POST /deny/{id}", b.handleDeny)
@@ -121,94 +120,32 @@ func (b *broker) handleDeny(w http.ResponseWriter, r *http.Request) {
respondDecision(w, r, e)
}
// respondDecision replies to an approve/deny. A browser form gets redirected back
// to the queue; a caller that asks for JSON (Accept: application/json — e.g. the
// fleet-manager driving the broker as its authorise plane) gets the decided
// Enrolment (cert_serial, hostnqn, context, fingerprint — NOT the cert PEM, which
// is unexported and only reaches the requesting host); any other API caller keeps
// the original 204 (so verify-serving.sh and existing scripts are unaffected).
// respondDecision replies to an approve/deny. A caller that asks for JSON
// (Accept: application/json — the fleet-manager driving the broker as its
// authorise plane) gets the decided Enrolment (cert_serial, hostnqn, context,
// fingerprint — NOT the cert PEM, which is unexported and only reaches the
// requesting host); any other API caller keeps the original 204 (so
// verify-serving.sh and existing scripts are unaffected).
func respondDecision(w http.ResponseWriter, r *http.Request, e *Enrolment) {
switch {
case isBrowser(r):
http.Redirect(w, r, "/", http.StatusSeeOther)
case wantsJSON(r):
if wantsJSON(r) {
writeJSON(w, http.StatusOK, e)
default:
w.WriteHeader(http.StatusNoContent)
return
}
w.WriteHeader(http.StatusNoContent)
}
func wantsJSON(r *http.Request) bool {
return strings.Contains(r.Header.Get("Accept"), "application/json")
}
func isBrowser(r *http.Request) bool {
a := r.Header.Get("Accept")
return len(a) >= 9 && a[:9] == "text/html" || // explicit
r.Header.Get("Sec-Fetch-Mode") == "navigate"
// handleAdminRoot: the human approval UI has moved to the fleet-manager SPA
// (WebAuthn-passkey gated), which drives this plane's JSON API over mTLS. This
// listener is now API-only — the old embedded HTML console was retired to avoid a
// second, weaker (client-cert-only) human approve path parallel to the passkey
// UI. Kept as a plaintext pointer so a bookmark to the old console isn't a 404.
func (b *broker) handleAdminRoot(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
_, _ = w.Write([]byte("enrol-broker admin plane — API only.\n" +
" GET /pending\n POST /approve/{id}\n POST /deny/{id}\n" +
"The human approval UI is the fleet-manager SPA (passkey-gated).\n"))
}
func (b *broker) handleAdminIndex(w http.ResponseWriter, r *http.Request) {
data := struct {
Admin string
Pending []*Enrolment
Now time.Time
}{
Admin: adminCN(r),
Pending: b.store.Pending(),
Now: time.Now(),
}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
if err := adminTmpl.Execute(w, data); err != nil {
log.Printf("admin template: %v", err)
}
}
var adminTmpl = template.Must(template.New("admin").Funcs(template.FuncMap{
"age": func(t time.Time) string { return time.Since(t).Round(time.Second).String() },
}).Parse(adminHTML))
const adminHTML = `<!doctype html>
<html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>fleet enrolment — pending</title>
<style>
:root{color-scheme:dark}
body{background:#0b0f14;color:#cfe3ee;font:14px/1.5 ui-monospace,Menlo,Consolas,monospace;margin:0;padding:1.5rem}
h1{font-size:1.1rem;color:#7fd1ff;margin:0 0 .25rem}
.sub{color:#6b8294;margin:0 0 1.25rem}
.empty{color:#6b8294;padding:2rem 0}
.card{background:#111923;border:1px solid #1f2d3a;border-radius:8px;padding:1rem 1.1rem;margin:0 0 1rem}
.fp{font-size:1.25rem;color:#aef9b0;letter-spacing:.5px;word-break:break-word}
.meta{display:grid;grid-template-columns:max-content 1fr;gap:.1rem .8rem;margin:.6rem 0;color:#9fb6c4}
.meta b{color:#6b8294;font-weight:400}
.actions{margin-top:.7rem}
button{font:inherit;border:0;border-radius:6px;padding:.45rem .9rem;cursor:pointer}
.approve{background:#15602f;color:#d6ffe0}
.deny{background:#5e1b1b;color:#ffd9d9;margin-left:.5rem}
form{display:inline}
code{color:#cfe3ee}
</style></head><body>
<h1>fleet enrolment — pending approvals</h1>
<p class="sub">admin: <code>{{.Admin}}</code> · {{len .Pending}} pending · cross-check each fingerprint against the device's splash before approving</p>
{{if not .Pending}}<div class="empty">no pending enrolments.</div>{{end}}
{{range .Pending}}
<div class="card">
<div class="fp">{{.Fingerprint}}</div>
<div class="meta">
<b>enrol-id</b><code>{{.ID}}</code>
<b>CN</b><code>{{.CN}}</code>
{{if .Context.MAC}}<b>MAC</b><code>{{.Context.MAC}}</code>{{end}}
{{if .Context.DMISerial}}<b>DMI serial</b><code>{{.Context.DMISerial}}</code>{{end}}
{{if .Context.DMIUUID}}<b>DMI uuid</b><code>{{.Context.DMIUUID}}</code>{{end}}
{{if .Context.EKHash}}<b>TPM EK</b><code>{{.Context.EKHash}}</code>{{end}}
<b>pubkey</b><code>{{.PubKeyHash}}</code>
<b>age</b><code>{{age .CreatedAt}}</code>
</div>
<div class="actions">
<form method="post" action="/approve/{{.ID}}"><button class="approve" type="submit">approve</button></form>
<form method="post" action="/deny/{{.ID}}"><button class="deny" type="submit">deny</button></form>
</div>
</div>
{{end}}
</body></html>`