diff --git a/.gitignore b/.gitignore index 18f209c..ff59b1a 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,8 @@ keys /files/secureboot/keys/*.auth /files/secureboot/keys/*.esl /files/secureboot/keys/*.crt + +# iPXE network bootloader binaries for the PXE fallback (serving/tftp). Binary, +# and the signed one is db-derived/environment-specific — supply per environment. +/serving/tftp/*.efi +/serving/tftp/*.kpxe diff --git a/docs/TODO.md b/docs/TODO.md index 471a1ba..c18e4f6 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -626,6 +626,24 @@ smoke test. Secure Boot on targets. Until then the loud unsigned warning must stay. **Key generation + enrolment procedure: [secure-boot-keys.md](secure-boot-keys.md)** (the PK/KEK/db trio; `db` = `sb_key`/`sb_cert`, PK/KEK stay offline). +- **PXE-only iPXE fallback — BENCH-VERIFIED on the EliteBook 820 G4 (2026-06-20).** + Firmware without UEFI HTTP boot chainloads `serving/tftp/snponly.efi` → renders + `autoexec.ipxe` (ifopen + DHCP-retry + `imgargs `) → UKI over HTTP + → NVMe/verity root. dnsmasq tags `pxe`/`ipxe`/`httpboot` give graceful per-client + fallback; HTTP-boot hosts are unaffected (firmware → UKI direct). Gotchas that + cost real time (all captured in [../serving/tftp/README.md](../serving/tftp/README.md)): + (i) the I219-LM needs **`snponly.efi`** (native iPXE driver won't bind); + (ii) **PortFast/edge on the switch port is REQUIRED** — iPXE's PHY bounce + triggers STP forwarding delay that drops DHCP (this was the real blocker); + (iii) iPXE aborts the script on any unguarded command failure (use `||`); + (iv) Secure Boot OFF makes systemd-stub take iPXE's load options as the cmdline, + so the entrypoint extracts the UKI's real `.cmdline` and feeds it via `imgargs`. +- **Sign iPXE for the PXE-only fallback (Secure Boot, production).** The lab + `snponly.efi` is **UNSIGNED** → that path needs **Secure Boot OFF**. For + production: `sbsign` the iPXE binary with the fleet **db** key (only `db` needed — + no MS CA / shim), so SB stays enforced firmware → iPXE → UKI; then the cmdline + contraption becomes a no-op (the stub ignores loader cmdline under SB). See + [secure-boot-keys.md](secure-boot-keys.md) "PXE-only hosts". - **Bench-verify on the secure path** (swtpm + hostapd + FreeRADIUS, `disable_dot1x=false`) — the items deferred this session: - NM port secure path: `en*` UNMANAGED + `macsec0` MANAGED/renewing, MKA diff --git a/docs/secure-boot-keys.md b/docs/secure-boot-keys.md index c68595d..e774b3e 100644 --- a/docs/secure-boot-keys.md +++ b/docs/secure-boot-keys.md @@ -170,6 +170,44 @@ Caveats: CI-secret flow is a reasonable interim; the HSM flow is the invariant-#7 ideal (TODO #8). +## PXE-only hosts: iPXE chainload and Secure Boot + +Firmware without UEFI HTTP boot (e.g. HP EliteBook 820 G4) falls back to PXE; the +serving stack hands those hosts **iPXE** over TFTP (`serving/tftp/ipxe.efi`), +which then chainloads the UKI over HTTP. dnsmasq selects per client automatically +(HTTP-capable → UKI direct; PXE-only → iPXE → UKI), so no per-host config. + +Secure Boot across that chain needs **only the `db` key — the one already +enrolled for the UKI. No extra keys, no Microsoft CA, no shim** (the fleet owns +its keys): + +``` +firmware --(verify snponly.efi vs db)--> iPXE --(chain -> firmware LoadImage + verifies UKI vs db)--> UKI +``` + +- The firmware verifies the TFTP-downloaded NBP (`ipxe.efi`) against `db` → + **the iPXE binary must be signed by `db`** (`sbsign`, same key/step as the UKI). + Stock/unsigned iPXE is rejected under Secure Boot. +- iPXE must load the UKI as an **EFI image** (`imgfetch` + `boot`, or `chain`) so + the firmware's `LoadImage()` re-verifies it against `db` — do **not** use iPXE's + `kernel`/`boot` on a raw vmlinuz, which bypasses SB. +- **cmdline gotcha (Secure Boot OFF only):** when iPXE executes the UKI it sets the + EFI load options to the image's cmdline, or — if empty — to the image **name**. + With SB off systemd-stub uses the load options *as the kernel cmdline*, + **replacing the UKI's embedded cmdline** (roothash/nvmf/...) → dracut emergency + shell. Clearing the args does NOT help (empty → iPXE falls back to the name). The + serving stack's rendered `boot.ipxe` therefore feeds iPXE the **real cmdline**, + which the dnsmasq entrypoint extracts from the UKI's `.cmdline` PE section + (`objcopy`, artifact mounted read-only) — see `serving/tftp/README.md`. **Secure + Boot ON is immune** — the stub ignores any loader-supplied cmdline — so this is + purely a lab/SB-off concern. + +**Lab posture (now): iPXE signing OFF → run the iPXE path with Secure Boot OFF.** +`serving/tftp/ipxe.efi` is unsigned. **Production upgrade:** sign it with `db` +and Secure Boot stays enforced end-to-end (tracked in TODO #8). The HTTP-boot path +(firmware → UKI) is unaffected and remains SB-clean regardless. + ## Key hygiene & lifecycle - **PK.key and KEK.key go offline / into an HSM and stay there.** They are only diff --git a/serving/dnsmasq/Dockerfile b/serving/dnsmasq/Dockerfile index 6b40ae8..1600a61 100644 --- a/serving/dnsmasq/Dockerfile +++ b/serving/dnsmasq/Dockerfile @@ -2,10 +2,18 @@ # the version is pinned and the entrypoint is predictable). Config is passed # entirely as --flags from docker-compose.yml. FROM harbor.bitasylum.org/dockerhub/library/alpine:3.20 -RUN apk add --no-cache dnsmasq +# binutils (objcopy): the entrypoint extracts the UKI's .cmdline PE section so it +# can feed the REAL cmdline to iPXE (Secure Boot OFF chainload) — see entrypoint.sh. +RUN apk add --no-cache dnsmasq binutils # fleet-namer: --dhcp-script that assigns creative starship hostnames per MAC, # server-side (keeps the workstation image neutral). COPY'd in root-owned, 0755 # so dnsmasq will execute it. Uses busybox sha256sum/cut/wc — all in the image. COPY fleet-namer /usr/local/bin/fleet-namer RUN chmod 0755 /usr/local/bin/fleet-namer -ENTRYPOINT ["dnsmasq"] +# entrypoint assembles the TFTP root (copies the user's ipxe.efi from the ro +# source mount + renders boot.ipxe, which clears the iPXE load options so the +# UKI's embedded cmdline survives the chainload — see entrypoint.sh) then execs +# dnsmasq with the compose args. +COPY entrypoint.sh /usr/local/bin/dnsmasq-entrypoint +RUN chmod 0755 /usr/local/bin/dnsmasq-entrypoint +ENTRYPOINT ["/usr/local/bin/dnsmasq-entrypoint"] diff --git a/serving/dnsmasq/entrypoint.sh b/serving/dnsmasq/entrypoint.sh new file mode 100644 index 0000000..59a7358 --- /dev/null +++ b/serving/dnsmasq/entrypoint.sh @@ -0,0 +1,101 @@ +#!/bin/sh +# dnsmasq entrypoint: assemble the TFTP root, then exec dnsmasq with the compose +# args ("$@"). +# +# iPXE + Secure Boot OFF cmdline problem: +# When iPXE executes the UKI it sets the EFI LoadOptions to the image's command +# line, or — if that's empty — to the image NAME. With Secure Boot OFF, +# systemd-stub uses LoadOptions AS THE KERNEL CMDLINE, replacing the UKI's +# EMBEDDED cmdline (root/roothash/rd.nvmf.discover/rd.dot1x=0/volatile). So +# booting the UKI URL directly (or clearing the args) makes the kernel come up +# with just "fleet-.efi" as its cmdline -> dracut emergency shell. +# Fix: feed iPXE the REAL cmdline via `imgargs`. That cmdline lives only inside +# the UKI on the serving host (not in the OCI image, not as a file), so we read +# it out of the UKI's .cmdline PE section and bake it into boot.ipxe. +# (Secure Boot ON ignores any loader cmdline and uses the embedded one, so this +# whole dance is a lab/SB-OFF concern; harmless when SB is on.) +# +# Extraction: objdump -h gives the .cmdline section's size + file offset; we dd +# those bytes out of the (read-only) UKI. We deliberately avoid `objcopy`: +# --dump-section rewrites the input in place (fails on the read-only /artifacts +# mount), and `-O binary --only-section` lays the section out at its huge VMA +# (multi-GB sparse file). +set -eu + +mkdir -p /srv/tftp +# User-supplied iPXE binary (ipxe.efi) from the read-only source mount. +cp -f /tftp-src/*.efi /srv/tftp/ 2>/dev/null || true + +extract_cmdline() { # $1 = UKI path; echoes the cleaned cmdline (empty on failure) + uki="$1" + command -v objdump >/dev/null 2>&1 || return 0 + # objdump -h columns: Idx Name Size VMA LMA File-off Algn + line=$(objdump -h "$uki" 2>/dev/null | awk '$2==".cmdline"{print $3" "$6; exit}') + [ -n "$line" ] || return 0 + sz=$((0x${line% *})); off=$((0x${line#* })) + [ "$sz" -gt 0 ] || return 0 + dd if="$uki" bs=1 skip="$off" count="$sz" 2>/dev/null \ + | tr -d '\000' | tr '\n\t' ' ' | sed 's/ */ /g; s/^ //; s/ $//' +} + +render_boot_ipxe() { + uki="fleet-${IMAGE_VERSION}.efi" + cmdline="" + [ -f "/artifacts/${uki}" ] && cmdline=$(extract_cmdline "/artifacts/${uki}") + if [ -z "$cmdline" ]; then + echo "WARN: could not extract .cmdline from /artifacts/${uki} — iPXE boot will" >&2 + echo " hit a dracut shell under Secure Boot OFF (no embedded cmdline)." >&2 + echo " Check the /artifacts mount + binutils, or enable Secure Boot." >&2 + fi + # Render the SAME boot logic to two files: + # autoexec.ipxe — iPXE fetches + runs this at STARTUP (before autoboot). The + # leading `ifopen net0` brings the link UP on NICs whose NII/UNDI link iPXE + # can't establish during autoboot — e.g. EliteBook 820 G4 / Intel I219-LM, + # which sits at link:down ("No configurable methods") until ifopen. Without + # this, autoboot fails before it can even DHCP for boot.ipxe. + # boot.ipxe — the DHCP-bootfile fallback (tag:ipxe) for firmware that DOES + # autoboot-DHCP fine; harmless to carry the same ifopen. + # Both feed iPXE the UKI's REAL cmdline via imgargs (Secure Boot OFF; see top). + for f in /srv/tftp/autoexec.ipxe /srv/tftp/boot.ipxe; do + cat > "$f" < iPXE exits -> the +# firmware restarts PXE), so EVERY step that can fail is guarded with '||'. +# +# The link is already up + forwarding here (the TFTP fetch of this script just +# succeeded), so try DHCP FIRST without touching the PHY — on a PortFast/edge +# port this succeeds immediately with no delay. +dhcp net0 && goto have_ip || echo initial DHCP failed; will reset link and retry +# Fallback: reset the link, then retry DHCP long enough to outlast the switch's +# STP forwarding delay (~30s) that the ifopen bounce triggers. A single ifopen +# (held open across the loop) means the PHY bounces ONCE; STP then converges and +# a later DHCP gets through. PROPER fix is PortFast/edge on the switch port so +# there's no delay at all — see serving/tftp/README.md. +ifopen net0 || goto err +set tries 0 +:dhcp_retry +dhcp net0 && goto have_ip || echo DHCP retry \${tries} +inc tries || goto err +iseq \${tries} 30 && goto no_ip || sleep 3 +goto dhcp_retry +:have_ip +imgfetch http://${SERVER_IP}/${uki} || goto err +imgargs ${uki} ${cmdline} +boot ${uki} || goto err +:no_ip +echo No DHCP after ~90s on net0 - enable PortFast/edge on the switch port (STP delay). +:err +echo Fleet iPXE chain failed (\${errno}) - dropping to iPXE shell +shell +EOF + done + echo "rendered autoexec.ipxe + boot.ipxe -> ${uki} (cmdline: ${cmdline:-})" +} + +if [ -n "${SERVER_IP:-}" ] && [ -n "${IMAGE_VERSION:-}" ]; then + render_boot_ipxe +else + echo "WARN: SERVER_IP/IMAGE_VERSION unset; /srv/tftp/boot.ipxe NOT rendered" >&2 +fi + +exec dnsmasq "$@" diff --git a/serving/docker-compose.yml b/serving/docker-compose.yml index d702e8e..ab0e658 100644 --- a/serving/docker-compose.yml +++ b/serving/docker-compose.yml @@ -48,6 +48,26 @@ services: # + DNS A/PTR. Auto-populated by fleet-namer (creative names per MAC); add # manual lines to pin specific hosts. RW so fleet-namer can append. - ./dnsmasq/dhcp-hosts:/etc/dnsmasq-hosts + # iPXE binary source (read-only) for the PXE-only chainload (firmware without + # UEFI HTTP boot, e.g. EliteBook 820 G4). Drop ipxe.efi here — see + # serving/tftp/README. The entrypoint copies it into /srv/tftp and renders + # boot.ipxe alongside it (the actual TFTP root is container-internal). + - ./tftp:/tftp-src:ro + # The release artifacts (same image volume as the http service), read-only, + # so the entrypoint can objcopy the UKI's .cmdline section into boot.ipxe + # (Secure Boot OFF needs the real cmdline fed to iPXE — see entrypoint.sh). + - type: image + source: ${OCI_IMAGE}:${IMAGE_VERSION} + target: /artifacts + image: + subpath: artifacts + # Fallback (Docker < 28 / offline): bind the local Packer output dir instead: + # - ${ARTIFACT_SRC}:/artifacts:ro + environment: + # Consumed by the entrypoint to render /srv/tftp/boot.ipxe (the iPXE chain + # script that clears load options so the UKI's embedded cmdline survives). + - SERVER_IP=${SERVER_IP} + - IMAGE_VERSION=${IMAGE_VERSION} command: - --keep-in-foreground # Log to stderr so `docker compose logs` captures output. Without this, @@ -108,6 +128,32 @@ services: - --dhcp-vendorclass=set:httpboot,HTTPClient - --dhcp-option=tag:httpboot,60,HTTPClient - --dhcp-boot=tag:httpboot,http://${SERVER_IP}/fleet-${IMAGE_VERSION}.efi + # ---- PXE -> iPXE -> HTTP chainload (for firmware WITHOUT UEFI HTTP boot) ---- + # Graceful fallback: HTTP-capable firmware matches 'httpboot' above and gets + # the UKI directly; PXE-only firmware (PXEClient) gets ipxe.efi over TFTP; + # once iPXE is running it re-DHCPs with user-class "iPXE" and is handed the + # SAME UKI HTTP URL, which it chainloads (.efi -> LoadImage). Each client + # gets the right thing based on what it advertises — no per-host config. + # snponly.efi = UEFI x64 iPXE that rides the FIRMWARE's NIC driver (UEFI SNP) + # instead of iPXE's own — works on virtio VMs AND real hardware where the + # native driver won't bind (e.g. EliteBook 820 G4 / Intel I219-LM, which + # fails ipxe.efi with "No configurable methods"). Ships in Ubuntu's `ipxe` + # package; see serving/tftp/README.md. + # NOTE (lab): the iPXE binary is UNSIGNED, so this path needs Secure Boot OFF. + # For production, sign it with the fleet db key (then SB stays enforced + # end-to-end) — see docs/secure-boot-keys.md. + - --enable-tftp + - --tftp-root=/srv/tftp + - --dhcp-vendorclass=set:pxe,PXEClient + - --dhcp-userclass=set:ipxe,iPXE + # firmware PXE, not yet iPXE -> hand iPXE (UEFI x64) over TFTP + - --dhcp-boot=tag:pxe,tag:!ipxe,snponly.efi + # iPXE running -> hand boot.ipxe (NOT the UKI URL directly). The script + # imgfetch'es the UKI then `imgargs`-clears the load options before boot, so + # systemd-stub uses the UKI's EMBEDDED cmdline. Booting the UKI URL directly + # makes iPXE pass the image name as the cmdline, which (Secure Boot OFF) + # REPLACES the embedded roothash/nvmf/... args -> dracut emergency shell. + - --dhcp-boot=tag:ipxe,boot.ipxe # --- HTTP boot server (serves the UKI) ------------------------------------- http: diff --git a/serving/tftp/.gitkeep b/serving/tftp/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/serving/tftp/README.md b/serving/tftp/README.md new file mode 100644 index 0000000..ba4ee0f --- /dev/null +++ b/serving/tftp/README.md @@ -0,0 +1,118 @@ +# TFTP root — iPXE chainload for PXE-only firmware + +This directory is mounted read-only at `/srv/tftp` in the dnsmasq container and +serves the iPXE network bootloader over TFTP to firmware that **lacks UEFI HTTP +boot** (e.g. HP EliteBook 820 G4). HTTP-boot-capable firmware never touches this +path — it gets the UKI straight over HTTP (see the `httpboot` tag in +`docker-compose.yml`). The fallback is automatic, driven by what each client +advertises (DHCP vendor-class / client-arch). + +Flow for a PXE-only host: + +``` +firmware PXE (PXEClient) --TFTP--> snponly.efi (iPXE) + iPXE re-DHCPs (user-class "iPXE") --> handed boot.ipxe (TFTP) + boot.ipxe: imgfetch UKI over HTTP -> imgargs -> boot +``` + +`boot.ipxe` is **rendered at container start** by the dnsmasq entrypoint; only the +iPXE binary (`snponly.efi`) is supplied here. The script exists because of a sharp +edge: + +> **Why not chainload the UKI URL directly?** When iPXE executes the UKI it sets +> the EFI load options to the image's command line — or, if that's empty, to the +> image **name**. With **Secure Boot OFF**, systemd-stub uses the load options *as +> the kernel cmdline*, **replacing** the UKI's embedded cmdline +> (`roothash`/`rd.nvmf.discover`/`rd.dot1x=0`/`systemd.volatile=overlay`). So +> booting the URL directly — *or clearing the args* — makes the kernel come up +> with just `fleet-.efi` as its cmdline → **dracut emergency shell**. +> +> The fix: feed iPXE the **real cmdline**. On the serving host that cmdline only +> lives *inside the UKI* (it's not in the OCI image — invariant #10 — nor shipped +> as a file), so the entrypoint **extracts the UKI's `.cmdline` PE section with +> `objcopy`** (the artifact image is mounted read-only at `/artifacts`) and bakes +> it into `boot.ipxe` as `imgargs `. Needs `binutils` in the image +> (added) and the artifact mount. +> +> **Secure Boot ON makes all of this moot** — the stub ignores any loader-supplied +> cmdline and uses the embedded one — so this is purely a lab/SB-OFF concern; the +> rendered script is harmless once SB is on. + +## What to drop here + +A UEFI x86-64 iPXE binary named **`snponly.efi`** (the name the dnsmasq config +serves). `snponly` rides the **firmware's** NIC driver via the UEFI SNP protocol +rather than iPXE's own, so it works on virtio VMs *and* real hardware where the +native driver won't bind. + +```bash +# Ubuntu/Debian package (simplest — confirmed present): +sudo apt-get install -y ipxe # then copy whatever snponly it ships, e.g.: +# dpkg -L ipxe | grep -i snponly +cp /usr/lib/ipxe/snponly.efi serving/tftp/snponly.efi +# Fedora/EL: dnf install ipxe-bootimgs -> /usr/share/ipxe/ipxe-snponly-x86_64.efi +# or build: git clone https://github.com/ipxe/ipxe && cd ipxe/src +# make bin-x86_64-efi/snponly.efi +``` + +> **Why `snponly`, not the full `ipxe.efi`?** iPXE's own NIC drivers don't bind on +> some hardware — the **EliteBook 820 G4 (Intel I219-LM)** fails `ipxe.efi` with +> *"net0: No configurable methods"*. `snponly` inherits the firmware's working +> driver (the one that just loaded iPXE over PXE), so it's the robust default. +> `boot.ipxe.org` only publishes the full `ipxe.efi`, not `snponly` — get `snponly` +> from a distro package or build it. + +> **Updating the binary requires a dnsmasq restart** — the entrypoint copies it +> from this (read-only) mount into the container's TFTP root at startup: +> `docker compose restart dnsmasq`. + +(BIOS/legacy hosts would need `undionly.kpxe` + client-arch matching; this fleet +is UEFI x64.) + +## Secure Boot + +**Lab (now): Secure Boot OFF for the iPXE path.** `snponly.efi` here is +**unsigned**, so a Secure-Boot-enforcing machine would reject it. + +**Production upgrade:** sign `snponly.efi` with the fleet **db** key (the same key +that signs the UKI), then Secure Boot stays enforced end-to-end +(firmware → verifies iPXE vs db → iPXE `chain`s the UKI → firmware LoadImage +verifies UKI vs db). Only `db` needs enrolling — no Microsoft keys / shim, since +the fleet owns its keys. See [../../docs/secure-boot-keys.md](../../docs/secure-boot-keys.md). + +```bash +# production (where db.key lives — CI/HSM): +sbsign --key db.key --cert db.crt --output snponly.efi snponly.efi +``` + +## Bench notes / gotchas (EliteBook 820 G4 / Intel I219-LM, 2026-06-20) + +Hard-won findings from bringing up the first PXE-only physical host: + +- **Use `snponly.efi`, not the full `ipxe.efi`.** The 820 G4's NIC is an Intel + **I219-LM** (PCI `00:1f.6`). iPXE's native `intel` driver in the prebuilt full + build doesn't bind it (`net0: No configurable methods` / no `net0`). `snponly` + rides the firmware's driver via NII/SNP, so `net0` appears. Ubuntu's `ipxe` + package ships a usable `snponly.efi` (no build needed). +- **PortFast/edge on the access port is REQUIRED** (this was the real blocker). + iPXE's `ifopen` resets the PHY → the link bounces → a managed switch restarts + its STP forwarding delay (~30 s) and silently drops packets → iPXE's DHCP never + reaches the server (`ifstat` shows `link:down` / `Link status: Unknown`; symptom + is `No configurable methods` or a dracut shell). The firmware's own PXE works + only because it keeps the link up continuously from POST so STP already + converged. **Fix: enable PortFast/edge (or disable STP) on the workstation + port** — VERIFIED to fix it 2026-06-20. (Cisco `spanning-tree portfast`; + Aruba/HP `admin-edge-port`; UniFi "Fast"; or a dumb unmanaged switch.) The + `autoexec.ipxe` retry loop (below) is a fallback that outlasts STP but adds + ~30 s to every boot — PortFast removes the delay entirely. +- **iPXE aborts the script on any unguarded command failure** (→ iPXE exits → the + firmware restarts the whole PXE process, looking like "it doesn't retry"). Every + fallible step in `boot.ipxe`/`autoexec.ipxe` is therefore `||`-guarded. +- **`autoexec.ipxe`** (rendered by the dnsmasq entrypoint) is fetched + run at + iPXE startup, *replacing* the default autoboot; that's where the `ifopen` + + DHCP-retry + `imgargs ` chain lives. + +## Note + +`*.efi` here is **gitignored** (binary, and the signed one is environment-specific +/ produced from db). Keep this README + `.gitkeep`.