diskless-workstation/docs/secure-boot-keys.md

# Secure Boot key generation & enrollment — diskless fleet

The fleet boots a **signed UKI directly via systemd-stub** (no shim, no GRUB), so
UEFI Secure Boot verifies the UKI against keys **you** enrol in firmware. This
doc generates the three Secure Boot keys (the "trio": **PK, KEK, db**), enrols
them, and wires the `db` key into the build.

> **Invariant #7:** signing keys never enter the build VM or this repo. The
> commands below produce key material that lives **offline / in an HSM**. Only the
> **db** key signs UKIs routinely, and the build receives it as a *path*
> (`sb_key`/`sb_cert`) on the CI/HSM host — never committed, never echoed in logs.
> Treat every `*.key` file here as secret; never check one into git.

## The trio — roles and where each lives

| Key | Role | Used by | Where it lives |
|---|---|---|---|
| **PK** (Platform Key) | Root of trust; controls who may update the KEK. Setting it locks Secure Boot into User Mode. | Firmware enrolment only | **Offline / HSM**, used once (and for KEK rotation) |
| **KEK** (Key Exchange Key) | Signs updates to `db`/`dbx`. | Firmware enrolment only | **Offline / HSM** |
| **db** (Signature Database) | Signs the bootable UKI. | **The fleet build** — `sb_key`/`sb_cert` ([../scripts/build-oci.sh](../scripts/build-oci.sh) → ukify/sbsign) | HSM/CI signing host |

Trust chain at enrolment: **PK signs KEK → KEK signs db → db signs the UKI.**

Only `db` is online (it signs every release; the UKI cmdline is immutable once
signed — invariant #6 — so re-signing is part of each build). PK and KEK are only
needed to rotate keys later, so they stay offline.

## Tooling

```bash
# Fedora / RHEL:   dnf install openssl efitools sbsigntools
# Debian / Ubuntu: apt install openssl efitools sbsigntool
```

## 1. Generate the three key/cert pairs

```bash
GUID=$(uuidgen); echo "$GUID" > GUID.txt    # owner GUID for the EFI variables

# Platform Key
openssl req -newkey rsa:2048 -nodes -keyout PK.key -new -x509 -sha256 -days 3650 \
  -subj "/CN=Fleet Platform Key/" -out PK.crt
# Key Exchange Key
openssl req -newkey rsa:2048 -nodes -keyout KEK.key -new -x509 -sha256 -days 3650 \
  -subj "/CN=Fleet KEK/" -out KEK.crt
# Signature DB key  <-- this pair becomes sb_key / sb_cert
openssl req -newkey rsa:2048 -nodes -keyout db.key -new -x509 -sha256 -days 3650 \
  -subj "/CN=Fleet db/" -out db.crt
```

`rsa:2048` is the maximally firmware-compatible size for the enrolled variables;
`4096` works on most modern UEFI but some implementations reject it for
PK/KEK/db. The PEM `db.key`/`db.crt` are exactly what sbsign/ukify consume.

## 2. Convert to signed EFI variable updates (`.auth`)

```bash
for v in PK KEK db; do cert-to-efi-sig-list -g "$GUID" $v.crt $v.esl; done

sign-efi-sig-list -g "$GUID" -k PK.key  -c PK.crt  PK  PK.esl  PK.auth   # PK self-signs
sign-efi-sig-list -g "$GUID" -k PK.key  -c PK.crt  KEK KEK.esl KEK.auth  # PK signs KEK
sign-efi-sig-list -g "$GUID" -k KEK.key -c KEK.crt db  db.esl  db.auth   # KEK signs db
```

## 3. Enrol into firmware

Put the firmware in **Setup Mode** first (clear the existing keys in the UEFI
setup screen). Then enrol **db and KEK before PK** — setting PK last is what flips
the firmware out of Setup Mode and turns Secure Boot on:

```bash
efi-updatevar -f db.auth  db
efi-updatevar -f KEK.auth KEK
efi-updatevar -f PK.auth  PK      # do this LAST
```

Alternatively, enrol the `.auth`/`.esl` files via the firmware's own
"enrol key from file" UI or `KeyTool.efi` — useful for headless fleet hardware
where this is scripted into provisioning.

> The fleet uses a UKI with **no shim and no Microsoft keys**, so no MOK is
> needed. If a host must also boot vendor-signed media, *add* your `db` alongside
> Microsoft's UEFI CA instead of clearing it.

### On a fleet workstation: read state over SSH, enrol via firmware

> **`efitools` (`efi-readvar`/`efi-updatevar`/`KeyTool`) is NOT packaged for EL9**
> (confirmed on Rocky 9, 2026-06-20). The image therefore ships only
> `efivar`/`mokutil`/`efibootmgr` (base repos). That's enough to **read** Secure
> Boot state over SSH, but **enrolment of custom PK/KEK/db is done in the firmware
> setup UI**, not from the OS.

Read Secure Boot state over SSH (`mokutil` is the `efi-readvar` replacement; acts
on efivarfs, works on the read-only root, RPM-installed so fapolicyd-trusted):

```bash
mokutil --sb-state     # enabled / setup mode
mokutil --pk           # Platform Key
mokutil --kek          # KEK
mokutil --db           # signature db — your enrolled fleet db cert appears here
mokutil --dbx          # revocations
```

**Enrol — two paths:**

**(a) Baked helper over SSH (`fleet-sb-enroll`)** — the OS-side, efitools-free
path. The build bakes `/usr/local/bin/fleet-sb-enroll` plus the public
`PK/KEK/db.auth` at `/usr/share/fleet-secureboot/` (supplied via
`files/secureboot/keys/` or the `SB_*_AUTH_B64` CI secrets). On a workstation in
**Setup Mode**:
```bash
sudo fleet-sb-enroll --check     # show SB state + which keys are baked
sudo fleet-sb-enroll             # enrol db -> KEK -> PK (PK last; exits Setup Mode)
mokutil --db ; mokutil --sb-state   # verify
```
It writes the signed `.auth` updates to efivarfs directly (the EL9-native
equivalent of `efi-updatevar`), refuses to run outside Setup Mode without
`--force`, and is added to fapolicyd's trust DB at build (it's a non-RPM exec).
⚠ This is a deliberate trust-bootstrap action — run it only on a host you control,
on a trusted/isolated network (see TODO #10).

**(b) Firmware UI** — no OS tooling, the most robust for one-off laptop bring-up:
1. Put the public `db.crt` (and `KEK.crt`/`PK.crt`, or the `.esl`/`.auth`) on a
   **FAT USB stick** (or the ESP).
2. Firmware setup → Secure Boot → **Custom**, **Clear/Reset** keys (Setup Mode),
   then **Enrol / Load key from file** for db, then KEK, then PK. On **HP
   EliteBooks**: `F10` → *Security* → *Secure Boot Configuration*.
3. Save & exit; re-check over SSH with `mokutil --db`.

Only the **public** certs/`.esl`/`.auth` reach the workstation — never copy
`PK.key`/`KEK.key`/`db.key` onto a host.

### Troubleshooting `fleet-sb-enroll` (bench, 2026-06-20)

**`dd: ... Invalid argument` (EINVAL) on the `db` write** = the firmware's
`SetVariable` rejected the payload (EFI_INVALID_PARAMETER), *not* an auth/Setup
Mode failure (that surfaces as EACCES / "Permission denied"). Diagnose top-down:

- **Is the `.auth` actually a signed authenticated variable?** First 16 bytes are
  an `EFI_TIME`: `sudo od -An -tx1 -N16 db.auth` → a valid one starts with the
  year little-endian (2026 = `ea 07`). If it starts with `a1 59 c0 a5 …` it's a
  bare **ESL** (the `sign-efi-sig-list` step was skipped); `30 82 …` is a raw DER
  cert. Re-create the `.auth` (step 2) and re-bake.
- **The 4-byte attribute prefix must contain its NUL bytes.** A bash *variable*
  **cannot** hold NULs — `ATTR=$'\x27\x00\x00\x00'` silently truncates to the
  single byte `0x27`, so efivarfs reads a corrupt attribute u32 and the kernel
  returns EINVAL **before the firmware is even called**. This was a real bug in an
  early `fleet-sb-enroll`; the fix emits the prefix with `printf '\x27\x00\x00\x00'`
  at write time (never via a variable). If you hand-roll an enrolment, do the same.
- **Proxmox / OVMF red herring:** VM EFI disks default to `pre-enrolled-keys=1`,
  so `mokutil --db` shows the **Microsoft** cert (`Windows Production PCA 2011`,
  owner `77fa9abd-…`) *even in Setup Mode*. This looks like a "db not empty →
  rejected" problem but isn't — the write fails identically on a clean
  `pre-enrolled-keys=0` disk if the NUL bug is present. For a clean OVMF test bed:
  `qm set <vmid> --efidisk0 <storage>:1,efitype=4m,pre-enrolled-keys=0`.

**Verified:** with the fix, a Proxmox VM enrols db→KEK→PK from the baked `.auth`
files and boots the CI-signed UKI under **enforcing Secure Boot** over the HTTP
boot path (`mokutil --sb-state` = enabled). The PXE/iPXE front-end additionally
needs `snponly.efi` signed with `db` — see the iPXE section below.

## 4. Wire `db` into the build

```bash
packer build -var image_version=X.Y.Z \
  -var sb_key=/path/to/db.key \
  -var sb_cert=/path/to/db.crt \
  -var oci_image=<registry>/fleet-boot .
```

ukify signs `fleet-<ver>.efi` with `db`; the enrolled `db` cert lets firmware
verify it. Without these vars the build stays **unsigned** (lab-only, with the
loud warning). In CI, `sb_key`/`sb_cert` are secret paths on the HSM/signing host.

## Signing in CI (Gitea Actions)

`sb_key`/`sb_cert` **can** be Gitea repo secrets, with one wrinkle: ukify consumes
them as **file paths**, not values, and the signing post-processor runs
**shell-local on the runner** (not in the build VM — consistent with invariant
#7). So the job materializes the secrets to runner-temp files, passes the paths,
and shreds the key afterward. This is wired in
[../.gitea/workflows/gitea-main-ci.yaml](../.gitea/workflows/gitea-main-ci.yaml)
(steps: *Materialize Secure Boot signing material* → *packer build* → *Shred
Secure Boot signing key*). Set repo secrets **`SB_KEY`** (the `db` private key,
PEM) and **`SB_CERT`** (the `db` cert, PEM); leave them unset for an unsigned lab
build.

Caveats:
- **Only `db` belongs in CI.** PK and KEK never leave the offline/HSM store.
- `SB_CERT` is a public cert (not secret); kept as a secret only for convenience.
- A CI secret puts the private key **in plaintext on the runner FS** for the
  duration of the build — only as safe as the runner. Use a **trusted,
  ephemeral/self-hosted** runner, ensure secrets are **not exposed to fork/PR
  builds**, and rely on `RUNNER_TEMP` wipe + the shred step. Gitea masks secret
  values in logs, and `build-oci.sh` passes the *path* (not content) to ukify, so
  the key won't leak into logs.
- **Stronger (production target):** keep `db.key` in an **HSM/PKCS#11** token or
  KMS and sign *through* it, so the raw key never touches the runner disk. The
  CI-secret flow is a reasonable interim; the HSM flow is the invariant-#7 ideal
  (TODO #8).

## PXE-only hosts: iPXE chainload and Secure Boot

Firmware without UEFI HTTP boot (e.g. HP EliteBook 820 G4) falls back to PXE; the
serving stack hands those hosts **iPXE** over TFTP (`snponly.efi`), which then
chainloads the UKI over HTTP. dnsmasq selects per client automatically
(HTTP-capable → UKI direct; PXE-only → iPXE → UKI), so no per-host config.

**Packaging:** the signed `snponly.efi` is built + signed + pushed by CI as a
small FROM-scratch **`fleet-ipxe` image** (`.gitea/workflows/ipxe-image.yaml` →
`scripts/build-ipxe-oci.sh`), mounted into the dnsmasq container at `/ipxe`
(serving/docker-compose.yml). It is deliberately a **separate, per-Secure-Boot-
environment image** (its signature is `db`-key-specific) so the dnsmasq image
stays generic/key-agnostic — same reasoning as invariant #9 keeping serving
software out of the release-artifact image. For offline/lab use, drop an
`snponly.efi` in `serving/tftp/` and the entrypoint falls back to that bind mount.

Secure Boot across that chain needs **only the `db` key — the one already
enrolled for the UKI. No extra keys, no Microsoft CA, no shim** (the fleet owns
its keys):

```
firmware --(verify snponly.efi vs db)--> iPXE --(chain -> firmware LoadImage
                                                  verifies UKI vs db)--> UKI
```

- The firmware verifies the TFTP-downloaded NBP (`snponly.efi`) against `db` →
  **the iPXE binary must be signed by `db`** (`sbsign`; CI does this in
  `build-ipxe-oci.sh` when the `SB_KEY`/`SB_CERT` secrets are set, same key as the
  UKI). Stock/unsigned iPXE is rejected under Secure Boot.
- iPXE must load the UKI as an **EFI image** (`imgfetch` + `boot`, or `chain`) so
  the firmware's `LoadImage()` re-verifies it against `db` — do **not** use iPXE's
  `kernel`/`boot` on a raw vmlinuz, which bypasses SB.
- **cmdline gotcha (Secure Boot OFF only):** when iPXE executes the UKI it sets the
  EFI load options to the image's cmdline, or — if empty — to the image **name**.
  With SB off systemd-stub uses the load options *as the kernel cmdline*,
  **replacing the UKI's embedded cmdline** (roothash/nvmf/...) → dracut emergency
  shell. Clearing the args does NOT help (empty → iPXE falls back to the name). The
  serving stack's rendered `boot.ipxe` therefore feeds iPXE the **real cmdline**,
  which the dnsmasq entrypoint extracts from the UKI's `.cmdline` PE section
  (`objcopy`, artifact mounted read-only) — see `serving/tftp/README.md`. **Secure
  Boot ON is immune** — the stub ignores any loader-supplied cmdline — so this is
  purely a lab/SB-off concern.

**Posture:** the `fleet-ipxe` image is signed with `db` **whenever CI has the
`SB_KEY`/`SB_CERT` secrets** (same condition as the UKI), so the PXE path can stay
Secure-Boot-enforced end-to-end. An **unsigned** `fleet-ipxe` build (no secrets)
keeps its loud warning and requires Secure Boot OFF on the PXE target. The
HTTP-boot path (firmware → UKI) is unaffected and SB-clean regardless. Still
**BENCH-VERIFY** the signed PXE chain on the 820 (the VM uses HTTP boot, so the
signed iPXE stage is unverified on real hardware — TODO #8).

## Key hygiene & lifecycle

- **PK.key and KEK.key go offline / into an HSM and stay there.** They are only
  needed to rotate keys; routine signing uses **db.key** only.
- **Production ideal:** keep `db.key` in an HSM/PKCS#11 token and have
  sbsign/ukify sign *through* it rather than reading a file from disk.
- **Validity:** `-days 3650` is 10 years; pick a rotation policy. Rotating `db`
  = sign new `db.esl` with KEK, push the `db` variable update, re-sign UKIs.
- **dbx** (the revocation list) is a fourth, optional variable — manage it to
  blacklist a compromised `db` cert.
- **Lab shortcut:** [`sbctl`](https://github.com/Foxboron/sbctl)
  (`sbctl create-keys` → `sbctl enroll-keys`) generates and enrols the whole trio
  and tracks signed files — convenient for a dev box, but it keeps keys as local
  files, so for the fleet's HSM/offline-PK model use the manual flow above for
  PK/KEK.

## Related

- Invariants #6 (UKI cmdline immutable once signed), #7 (keys never in repo/VM),
  #5 (roothash↔UKI hard gate) — [../CLAUDE.md](../CLAUDE.md).
- Signing is the production-posture step in [TODO.md](TODO.md) #8.
- FIPS is currently off and is to be revisited alongside signing/Secure Boot
  ([stig-deviations.md](stig-deviations.md) #5).