280 lines
14 KiB
Markdown
280 lines
14 KiB
Markdown
# Secure Boot key generation & enrollment — diskless fleet
|
|
|
|
The fleet boots a **signed UKI directly via systemd-stub** (no shim, no GRUB), so
|
|
UEFI Secure Boot verifies the UKI against keys **you** enrol in firmware. This
|
|
doc generates the three Secure Boot keys (the "trio": **PK, KEK, db**), enrols
|
|
them, and wires the `db` key into the build.
|
|
|
|
> **Invariant #7:** signing keys never enter the build VM or this repo. The
|
|
> commands below produce key material that lives **offline / in an HSM**. Only the
|
|
> **db** key signs UKIs routinely, and the build receives it as a *path*
|
|
> (`sb_key`/`sb_cert`) on the CI/HSM host — never committed, never echoed in logs.
|
|
> Treat every `*.key` file here as secret; never check one into git.
|
|
|
|
## The trio — roles and where each lives
|
|
|
|
| Key | Role | Used by | Where it lives |
|
|
|---|---|---|---|
|
|
| **PK** (Platform Key) | Root of trust; controls who may update the KEK. Setting it locks Secure Boot into User Mode. | Firmware enrolment only | **Offline / HSM**, used once (and for KEK rotation) |
|
|
| **KEK** (Key Exchange Key) | Signs updates to `db`/`dbx`. | Firmware enrolment only | **Offline / HSM** |
|
|
| **db** (Signature Database) | Signs the bootable UKI. | **The fleet build** — `sb_key`/`sb_cert` ([../scripts/build-oci.sh](../scripts/build-oci.sh) → ukify/sbsign) | HSM/CI signing host |
|
|
|
|
Trust chain at enrolment: **PK signs KEK → KEK signs db → db signs the UKI.**
|
|
|
|
Only `db` is online (it signs every release; the UKI cmdline is immutable once
|
|
signed — invariant #6 — so re-signing is part of each build). PK and KEK are only
|
|
needed to rotate keys later, so they stay offline.
|
|
|
|
## Tooling
|
|
|
|
```bash
|
|
# Fedora / RHEL: dnf install openssl efitools sbsigntools
|
|
# Debian / Ubuntu: apt install openssl efitools sbsigntool
|
|
```
|
|
|
|
## 1. Generate the three key/cert pairs
|
|
|
|
```bash
|
|
GUID=$(uuidgen); echo "$GUID" > GUID.txt # owner GUID for the EFI variables
|
|
|
|
# Platform Key
|
|
openssl req -newkey rsa:2048 -nodes -keyout PK.key -new -x509 -sha256 -days 3650 \
|
|
-subj "/CN=Fleet Platform Key/" -out PK.crt
|
|
# Key Exchange Key
|
|
openssl req -newkey rsa:2048 -nodes -keyout KEK.key -new -x509 -sha256 -days 3650 \
|
|
-subj "/CN=Fleet KEK/" -out KEK.crt
|
|
# Signature DB key <-- this pair becomes sb_key / sb_cert
|
|
openssl req -newkey rsa:2048 -nodes -keyout db.key -new -x509 -sha256 -days 3650 \
|
|
-subj "/CN=Fleet db/" -out db.crt
|
|
```
|
|
|
|
`rsa:2048` is the maximally firmware-compatible size for the enrolled variables;
|
|
`4096` works on most modern UEFI but some implementations reject it for
|
|
PK/KEK/db. The PEM `db.key`/`db.crt` are exactly what sbsign/ukify consume.
|
|
|
|
## 2. Convert to signed EFI variable updates (`.auth`)
|
|
|
|
```bash
|
|
for v in PK KEK db; do cert-to-efi-sig-list -g "$GUID" $v.crt $v.esl; done
|
|
|
|
sign-efi-sig-list -g "$GUID" -k PK.key -c PK.crt PK PK.esl PK.auth # PK self-signs
|
|
sign-efi-sig-list -g "$GUID" -k PK.key -c PK.crt KEK KEK.esl KEK.auth # PK signs KEK
|
|
sign-efi-sig-list -g "$GUID" -k KEK.key -c KEK.crt db db.esl db.auth # KEK signs db
|
|
```
|
|
|
|
## 3. Enrol into firmware
|
|
|
|
Put the firmware in **Setup Mode** first (clear the existing keys in the UEFI
|
|
setup screen). Then enrol **db and KEK before PK** — setting PK last is what flips
|
|
the firmware out of Setup Mode and turns Secure Boot on:
|
|
|
|
```bash
|
|
efi-updatevar -f db.auth db
|
|
efi-updatevar -f KEK.auth KEK
|
|
efi-updatevar -f PK.auth PK # do this LAST
|
|
```
|
|
|
|
Alternatively, enrol the `.auth`/`.esl` files via the firmware's own
|
|
"enrol key from file" UI or `KeyTool.efi` — useful for headless fleet hardware
|
|
where this is scripted into provisioning.
|
|
|
|
> The fleet uses a UKI with **no shim and no Microsoft keys**, so no MOK is
|
|
> needed. If a host must also boot vendor-signed media, *add* your `db` alongside
|
|
> Microsoft's UEFI CA instead of clearing it.
|
|
|
|
### On a fleet workstation: read state over SSH, enrol via firmware
|
|
|
|
> **`efitools` (`efi-readvar`/`efi-updatevar`/`KeyTool`) is NOT packaged for EL9**
|
|
> (confirmed on Rocky 9, 2026-06-20). The image therefore ships only
|
|
> `efivar`/`mokutil`/`efibootmgr` (base repos). That's enough to **read** Secure
|
|
> Boot state over SSH, but **enrolment of custom PK/KEK/db is done in the firmware
|
|
> setup UI**, not from the OS.
|
|
|
|
Read Secure Boot state over SSH (`mokutil` is the `efi-readvar` replacement; acts
|
|
on efivarfs, works on the read-only root, RPM-installed so fapolicyd-trusted):
|
|
|
|
```bash
|
|
mokutil --sb-state # enabled / setup mode
|
|
mokutil --pk # Platform Key
|
|
mokutil --kek # KEK
|
|
mokutil --db # signature db — your enrolled fleet db cert appears here
|
|
mokutil --dbx # revocations
|
|
```
|
|
|
|
**Enrol — two paths:**
|
|
|
|
**(a) Baked helper over SSH (`fleet-sb-enroll`)** — the OS-side, efitools-free
|
|
path. The build bakes `/usr/local/bin/fleet-sb-enroll` plus the public
|
|
`PK/KEK/db.auth` at `/usr/share/fleet-secureboot/` (supplied via
|
|
`files/secureboot/keys/` or the `SB_*_AUTH_B64` CI secrets). On a workstation in
|
|
**Setup Mode**:
|
|
```bash
|
|
sudo fleet-sb-enroll --check # show SB state + which keys are baked
|
|
sudo fleet-sb-enroll # enrol db -> KEK -> PK (PK last; exits Setup Mode)
|
|
mokutil --db ; mokutil --sb-state # verify
|
|
```
|
|
It writes the signed `.auth` updates to efivarfs directly (the EL9-native
|
|
equivalent of `efi-updatevar`), refuses to run outside Setup Mode without
|
|
`--force`, and is added to fapolicyd's trust DB at build (it's a non-RPM exec).
|
|
⚠ This is a deliberate trust-bootstrap action — run it only on a host you control,
|
|
on a trusted/isolated network (see TODO #10).
|
|
|
|
**(b) Firmware UI** — no OS tooling, the most robust for one-off laptop bring-up:
|
|
1. Put the public `db.crt` (and `KEK.crt`/`PK.crt`, or the `.esl`/`.auth`) on a
|
|
**FAT USB stick** (or the ESP).
|
|
2. Firmware setup → Secure Boot → **Custom**, **Clear/Reset** keys (Setup Mode),
|
|
then **Enrol / Load key from file** for db, then KEK, then PK. On **HP
|
|
EliteBooks**: `F10` → *Security* → *Secure Boot Configuration*.
|
|
3. Save & exit; re-check over SSH with `mokutil --db`.
|
|
|
|
Only the **public** certs/`.esl`/`.auth` reach the workstation — never copy
|
|
`PK.key`/`KEK.key`/`db.key` onto a host.
|
|
|
|
### Troubleshooting `fleet-sb-enroll` (bench, 2026-06-20)
|
|
|
|
**`dd: ... Invalid argument` (EINVAL) on the `db` write** = the firmware's
|
|
`SetVariable` rejected the payload (EFI_INVALID_PARAMETER), *not* an auth/Setup
|
|
Mode failure (that surfaces as EACCES / "Permission denied"). Diagnose top-down:
|
|
|
|
- **Is the `.auth` actually a signed authenticated variable?** First 16 bytes are
|
|
an `EFI_TIME`: `sudo od -An -tx1 -N16 db.auth` → a valid one starts with the
|
|
year little-endian (2026 = `ea 07`). If it starts with `a1 59 c0 a5 …` it's a
|
|
bare **ESL** (the `sign-efi-sig-list` step was skipped); `30 82 …` is a raw DER
|
|
cert. Re-create the `.auth` (step 2) and re-bake.
|
|
- **The 4-byte attribute prefix must contain its NUL bytes.** A bash *variable*
|
|
**cannot** hold NULs — `ATTR=$'\x27\x00\x00\x00'` silently truncates to the
|
|
single byte `0x27`, so efivarfs reads a corrupt attribute u32 and the kernel
|
|
returns EINVAL **before the firmware is even called**. This was a real bug in an
|
|
early `fleet-sb-enroll`; the fix emits the prefix with `printf '\x27\x00\x00\x00'`
|
|
at write time (never via a variable). If you hand-roll an enrolment, do the same.
|
|
- **Proxmox / OVMF red herring:** VM EFI disks default to `pre-enrolled-keys=1`,
|
|
so `mokutil --db` shows the **Microsoft** cert (`Windows Production PCA 2011`,
|
|
owner `77fa9abd-…`) *even in Setup Mode*. This looks like a "db not empty →
|
|
rejected" problem but isn't — the write fails identically on a clean
|
|
`pre-enrolled-keys=0` disk if the NUL bug is present. For a clean OVMF test bed:
|
|
`qm set <vmid> --efidisk0 <storage>:1,efitype=4m,pre-enrolled-keys=0`.
|
|
|
|
**Verified end-to-end on real hardware (2026-06-22):** `fleet-sb-enroll` (the
|
|
baked, fixed script) enrols db→KEK→PK cleanly, and both boot transports verify
|
|
under **enforcing Secure Boot** (`mokutil --sb-state` = enabled):
|
|
- **Proxmox VM — HTTP boot:** firmware → signed UKI (2026-06-20).
|
|
- **HP EliteBook 820 G4 — PXE boot:** firmware → signed `snponly.efi` (fleet-ipxe)
|
|
→ signed UKI (2026-06-22).
|
|
So the full signing pipeline (UKI + `fleet-ipxe`, both against the same enrolled
|
|
`db`) is proven on real silicon, not just in the VM.
|
|
|
|
## 4. Wire `db` into the build
|
|
|
|
```bash
|
|
packer build -var image_version=X.Y.Z \
|
|
-var sb_key=/path/to/db.key \
|
|
-var sb_cert=/path/to/db.crt \
|
|
-var oci_image=<registry>/fleet-boot .
|
|
```
|
|
|
|
ukify signs `fleet-<ver>.efi` with `db`; the enrolled `db` cert lets firmware
|
|
verify it. Without these vars the build stays **unsigned** (lab-only, with the
|
|
loud warning). In CI, `sb_key`/`sb_cert` are secret paths on the HSM/signing host.
|
|
|
|
## Signing in CI (Gitea Actions)
|
|
|
|
`sb_key`/`sb_cert` **can** be Gitea repo secrets, with one wrinkle: ukify consumes
|
|
them as **file paths**, not values, and the signing post-processor runs
|
|
**shell-local on the runner** (not in the build VM — consistent with invariant
|
|
#7). So the job materializes the secrets to runner-temp files, passes the paths,
|
|
and shreds the key afterward. This is wired in
|
|
[../.gitea/workflows/gitea-main-ci.yaml](../.gitea/workflows/gitea-main-ci.yaml)
|
|
(steps: *Materialize Secure Boot signing material* → *packer build* → *Shred
|
|
Secure Boot signing key*). Set repo secrets **`SB_KEY`** (the `db` private key,
|
|
PEM) and **`SB_CERT`** (the `db` cert, PEM); leave them unset for an unsigned lab
|
|
build.
|
|
|
|
Caveats:
|
|
- **Only `db` belongs in CI.** PK and KEK never leave the offline/HSM store.
|
|
- `SB_CERT` is a public cert (not secret); kept as a secret only for convenience.
|
|
- A CI secret puts the private key **in plaintext on the runner FS** for the
|
|
duration of the build — only as safe as the runner. Use a **trusted,
|
|
ephemeral/self-hosted** runner, ensure secrets are **not exposed to fork/PR
|
|
builds**, and rely on `RUNNER_TEMP` wipe + the shred step. Gitea masks secret
|
|
values in logs, and `build-oci.sh` passes the *path* (not content) to ukify, so
|
|
the key won't leak into logs.
|
|
- **Stronger (production target):** keep `db.key` in an **HSM/PKCS#11** token or
|
|
KMS and sign *through* it, so the raw key never touches the runner disk. The
|
|
CI-secret flow is a reasonable interim; the HSM flow is the invariant-#7 ideal
|
|
(TODO #8).
|
|
|
|
## PXE-only hosts: iPXE chainload and Secure Boot
|
|
|
|
Firmware without UEFI HTTP boot (e.g. HP EliteBook 820 G4) falls back to PXE; the
|
|
serving stack hands those hosts **iPXE** over TFTP (`snponly.efi`), which then
|
|
chainloads the UKI over HTTP. dnsmasq selects per client automatically
|
|
(HTTP-capable → UKI direct; PXE-only → iPXE → UKI), so no per-host config.
|
|
|
|
**Packaging:** the signed `snponly.efi` is built + signed + pushed by CI as a
|
|
small FROM-scratch **`fleet-ipxe` image** (`.gitea/workflows/ipxe-image.yaml` →
|
|
`scripts/build-ipxe-oci.sh`), mounted into the dnsmasq container at `/ipxe`
|
|
(serving/docker-compose.yml). It is deliberately a **separate, per-Secure-Boot-
|
|
environment image** (its signature is `db`-key-specific) so the dnsmasq image
|
|
stays generic/key-agnostic — same reasoning as invariant #9 keeping serving
|
|
software out of the release-artifact image. For offline/lab use, drop an
|
|
`snponly.efi` in `serving/tftp/` and the entrypoint falls back to that bind mount.
|
|
|
|
Secure Boot across that chain needs **only the `db` key — the one already
|
|
enrolled for the UKI. No extra keys, no Microsoft CA, no shim** (the fleet owns
|
|
its keys):
|
|
|
|
```
|
|
firmware --(verify snponly.efi vs db)--> iPXE --(chain -> firmware LoadImage
|
|
verifies UKI vs db)--> UKI
|
|
```
|
|
|
|
- The firmware verifies the TFTP-downloaded NBP (`snponly.efi`) against `db` →
|
|
**the iPXE binary must be signed by `db`** (`sbsign`; CI does this in
|
|
`build-ipxe-oci.sh` when the `SB_KEY`/`SB_CERT` secrets are set, same key as the
|
|
UKI). Stock/unsigned iPXE is rejected under Secure Boot.
|
|
- iPXE must load the UKI as an **EFI image** (`imgfetch` + `boot`, or `chain`) so
|
|
the firmware's `LoadImage()` re-verifies it against `db` — do **not** use iPXE's
|
|
`kernel`/`boot` on a raw vmlinuz, which bypasses SB.
|
|
- **cmdline gotcha (Secure Boot OFF only):** when iPXE executes the UKI it sets the
|
|
EFI load options to the image's cmdline, or — if empty — to the image **name**.
|
|
With SB off systemd-stub uses the load options *as the kernel cmdline*,
|
|
**replacing the UKI's embedded cmdline** (roothash/nvmf/...) → dracut emergency
|
|
shell. Clearing the args does NOT help (empty → iPXE falls back to the name). The
|
|
serving stack's rendered `boot.ipxe` therefore feeds iPXE the **real cmdline**,
|
|
which the dnsmasq entrypoint extracts from the UKI's `.cmdline` PE section
|
|
(`objcopy`, artifact mounted read-only) — see `serving/tftp/README.md`. **Secure
|
|
Boot ON is immune** — the stub ignores any loader-supplied cmdline — so this is
|
|
purely a lab/SB-off concern.
|
|
|
|
**Posture:** the `fleet-ipxe` image is signed with `db` **whenever CI has the
|
|
`SB_KEY`/`SB_CERT` secrets** (same condition as the UKI), so the PXE path can stay
|
|
Secure-Boot-enforced end-to-end. An **unsigned** `fleet-ipxe` build (no secrets)
|
|
keeps its loud warning and requires Secure Boot OFF on the PXE target. The
|
|
HTTP-boot path (firmware → UKI) is unaffected and SB-clean regardless. Still
|
|
**BENCH-VERIFY** the signed PXE chain on the 820 (the VM uses HTTP boot, so the
|
|
signed iPXE stage is unverified on real hardware — TODO #8).
|
|
|
|
## Key hygiene & lifecycle
|
|
|
|
- **PK.key and KEK.key go offline / into an HSM and stay there.** They are only
|
|
needed to rotate keys; routine signing uses **db.key** only.
|
|
- **Production ideal:** keep `db.key` in an HSM/PKCS#11 token and have
|
|
sbsign/ukify sign *through* it rather than reading a file from disk.
|
|
- **Validity:** `-days 3650` is 10 years; pick a rotation policy. Rotating `db`
|
|
= sign new `db.esl` with KEK, push the `db` variable update, re-sign UKIs.
|
|
- **dbx** (the revocation list) is a fourth, optional variable — manage it to
|
|
blacklist a compromised `db` cert.
|
|
- **Lab shortcut:** [`sbctl`](https://github.com/Foxboron/sbctl)
|
|
(`sbctl create-keys` → `sbctl enroll-keys`) generates and enrols the whole trio
|
|
and tracks signed files — convenient for a dev box, but it keeps keys as local
|
|
files, so for the fleet's HSM/offline-PK model use the manual flow above for
|
|
PK/KEK.
|
|
|
|
## Related
|
|
|
|
- Invariants #6 (UKI cmdline immutable once signed), #7 (keys never in repo/VM),
|
|
#5 (roothash↔UKI hard gate) — [../CLAUDE.md](../CLAUDE.md).
|
|
- Signing is the production-posture step in [TODO.md](TODO.md) #8.
|
|
- FIPS is currently off and is to be revisited alongside signing/Secure Boot
|
|
([stig-deviations.md](stig-deviations.md) #5).
|