Update pxe boot flow

2026-06-20 09:17:05 +00:00
parent e3978bb210
commit fb69068980
8 changed files with 336 additions and 2 deletions
@@ -9,3 +9,8 @@ keys
 /files/secureboot/keys/*.auth
 /files/secureboot/keys/*.esl
 /files/secureboot/keys/*.crt
+
+# iPXE network bootloader binaries for the PXE fallback (serving/tftp). Binary,
+# and the signed one is db-derived/environment-specific — supply per environment.
+/serving/tftp/*.efi
+/serving/tftp/*.kpxe
@@ -626,6 +626,24 @@ smoke test.
  Secure Boot on targets. Until then the loud unsigned warning must stay.
  **Key generation + enrolment procedure: [secure-boot-keys.md](secure-boot-keys.md)**
  (the PK/KEK/db trio; `db` = `sb_key`/`sb_cert`, PK/KEK stay offline).
+- **PXE-only iPXE fallback — BENCH-VERIFIED on the EliteBook 820 G4 (2026-06-20).**
+  Firmware without UEFI HTTP boot chainloads `serving/tftp/snponly.efi` → renders
+  `autoexec.ipxe` (ifopen + DHCP-retry + `imgargs <real cmdline>`) → UKI over HTTP
+  → NVMe/verity root. dnsmasq tags `pxe`/`ipxe`/`httpboot` give graceful per-client
+  fallback; HTTP-boot hosts are unaffected (firmware → UKI direct). Gotchas that
+  cost real time (all captured in [../serving/tftp/README.md](../serving/tftp/README.md)):
+  (i) the I219-LM needs **`snponly.efi`** (native iPXE driver won't bind);
+  (ii) **PortFast/edge on the switch port is REQUIRED** — iPXE's PHY bounce
+  triggers STP forwarding delay that drops DHCP (this was the real blocker);
+  (iii) iPXE aborts the script on any unguarded command failure (use `||`);
+  (iv) Secure Boot OFF makes systemd-stub take iPXE's load options as the cmdline,
+  so the entrypoint extracts the UKI's real `.cmdline` and feeds it via `imgargs`.
+- **Sign iPXE for the PXE-only fallback (Secure Boot, production).** The lab
+  `snponly.efi` is **UNSIGNED** → that path needs **Secure Boot OFF**. For
+  production: `sbsign` the iPXE binary with the fleet **db** key (only `db` needed —
+  no MS CA / shim), so SB stays enforced firmware → iPXE → UKI; then the cmdline
+  contraption becomes a no-op (the stub ignores loader cmdline under SB). See
+  [secure-boot-keys.md](secure-boot-keys.md) "PXE-only hosts".
 - **Bench-verify on the secure path** (swtpm + hostapd + FreeRADIUS,
  `disable_dot1x=false`) — the items deferred this session:
  - NM port secure path: `en*` UNMANAGED + `macsec0` MANAGED/renewing, MKA
@@ -170,6 +170,44 @@ Caveats:
  CI-secret flow is a reasonable interim; the HSM flow is the invariant-#7 ideal
  (TODO #8).

+## PXE-only hosts: iPXE chainload and Secure Boot
+
+Firmware without UEFI HTTP boot (e.g. HP EliteBook 820 G4) falls back to PXE; the
+serving stack hands those hosts **iPXE** over TFTP (`serving/tftp/ipxe.efi`),
+which then chainloads the UKI over HTTP. dnsmasq selects per client automatically
+(HTTP-capable → UKI direct; PXE-only → iPXE → UKI), so no per-host config.
+
+Secure Boot across that chain needs **only the `db` key — the one already
+enrolled for the UKI. No extra keys, no Microsoft CA, no shim** (the fleet owns
+its keys):
+
+```
+firmware --(verify snponly.efi vs db)--> iPXE --(chain -> firmware LoadImage
+                                                  verifies UKI vs db)--> UKI
+```
+
+- The firmware verifies the TFTP-downloaded NBP (`ipxe.efi`) against `db` →
+  **the iPXE binary must be signed by `db`** (`sbsign`, same key/step as the UKI).
+  Stock/unsigned iPXE is rejected under Secure Boot.
+- iPXE must load the UKI as an **EFI image** (`imgfetch` + `boot`, or `chain`) so
+  the firmware's `LoadImage()` re-verifies it against `db` — do **not** use iPXE's
+  `kernel`/`boot` on a raw vmlinuz, which bypasses SB.
+- **cmdline gotcha (Secure Boot OFF only):** when iPXE executes the UKI it sets the
+  EFI load options to the image's cmdline, or — if empty — to the image **name**.
+  With SB off systemd-stub uses the load options *as the kernel cmdline*,
+  **replacing the UKI's embedded cmdline** (roothash/nvmf/...) → dracut emergency
+  shell. Clearing the args does NOT help (empty → iPXE falls back to the name). The
+  serving stack's rendered `boot.ipxe` therefore feeds iPXE the **real cmdline**,
+  which the dnsmasq entrypoint extracts from the UKI's `.cmdline` PE section
+  (`objcopy`, artifact mounted read-only) — see `serving/tftp/README.md`. **Secure
+  Boot ON is immune** — the stub ignores any loader-supplied cmdline — so this is
+  purely a lab/SB-off concern.
+
+**Lab posture (now): iPXE signing OFF → run the iPXE path with Secure Boot OFF.**
+`serving/tftp/ipxe.efi` is unsigned. **Production upgrade:** sign it with `db`
+and Secure Boot stays enforced end-to-end (tracked in TODO #8). The HTTP-boot path
+(firmware → UKI) is unaffected and remains SB-clean regardless.
+
 ## Key hygiene & lifecycle

 - **PK.key and KEK.key go offline / into an HSM and stay there.** They are only
@@ -2,10 +2,18 @@
 # the version is pinned and the entrypoint is predictable). Config is passed
 # entirely as --flags from docker-compose.yml.
 FROM harbor.bitasylum.org/dockerhub/library/alpine:3.20
-RUN apk add --no-cache dnsmasq
+# binutils (objcopy): the entrypoint extracts the UKI's .cmdline PE section so it
+# can feed the REAL cmdline to iPXE (Secure Boot OFF chainload) — see entrypoint.sh.
+RUN apk add --no-cache dnsmasq binutils
 # fleet-namer: --dhcp-script that assigns creative starship hostnames per MAC,
 # server-side (keeps the workstation image neutral). COPY'd in root-owned, 0755
 # so dnsmasq will execute it. Uses busybox sha256sum/cut/wc — all in the image.
 COPY fleet-namer /usr/local/bin/fleet-namer
 RUN chmod 0755 /usr/local/bin/fleet-namer
-ENTRYPOINT ["dnsmasq"]
+# entrypoint assembles the TFTP root (copies the user's ipxe.efi from the ro
+# source mount + renders boot.ipxe, which clears the iPXE load options so the
+# UKI's embedded cmdline survives the chainload — see entrypoint.sh) then execs
+# dnsmasq with the compose args.
+COPY entrypoint.sh /usr/local/bin/dnsmasq-entrypoint
+RUN chmod 0755 /usr/local/bin/dnsmasq-entrypoint
+ENTRYPOINT ["/usr/local/bin/dnsmasq-entrypoint"]
@@ -0,0 +1,101 @@
+#!/bin/sh
+# dnsmasq entrypoint: assemble the TFTP root, then exec dnsmasq with the compose
+# args ("$@").
+#
+# iPXE + Secure Boot OFF cmdline problem:
+#   When iPXE executes the UKI it sets the EFI LoadOptions to the image's command
+#   line, or — if that's empty — to the image NAME. With Secure Boot OFF,
+#   systemd-stub uses LoadOptions AS THE KERNEL CMDLINE, replacing the UKI's
+#   EMBEDDED cmdline (root/roothash/rd.nvmf.discover/rd.dot1x=0/volatile). So
+#   booting the UKI URL directly (or clearing the args) makes the kernel come up
+#   with just "fleet-<ver>.efi" as its cmdline -> dracut emergency shell.
+#   Fix: feed iPXE the REAL cmdline via `imgargs`. That cmdline lives only inside
+#   the UKI on the serving host (not in the OCI image, not as a file), so we read
+#   it out of the UKI's .cmdline PE section and bake it into boot.ipxe.
+#   (Secure Boot ON ignores any loader cmdline and uses the embedded one, so this
+#   whole dance is a lab/SB-OFF concern; harmless when SB is on.)
+#
+#   Extraction: objdump -h gives the .cmdline section's size + file offset; we dd
+#   those bytes out of the (read-only) UKI. We deliberately avoid `objcopy`:
+#   --dump-section rewrites the input in place (fails on the read-only /artifacts
+#   mount), and `-O binary --only-section` lays the section out at its huge VMA
+#   (multi-GB sparse file).
+set -eu
+
+mkdir -p /srv/tftp
+# User-supplied iPXE binary (ipxe.efi) from the read-only source mount.
+cp -f /tftp-src/*.efi /srv/tftp/ 2>/dev/null || true
+
+extract_cmdline() {  # $1 = UKI path; echoes the cleaned cmdline (empty on failure)
+  uki="$1"
+  command -v objdump >/dev/null 2>&1 || return 0
+  # objdump -h columns: Idx Name Size VMA LMA File-off Algn
+  line=$(objdump -h "$uki" 2>/dev/null | awk '$2==".cmdline"{print $3" "$6; exit}')
+  [ -n "$line" ] || return 0
+  sz=$((0x${line% *})); off=$((0x${line#* }))
+  [ "$sz" -gt 0 ] || return 0
+  dd if="$uki" bs=1 skip="$off" count="$sz" 2>/dev/null \
+    | tr -d '\000' | tr '\n\t' '  ' | sed 's/  */ /g; s/^ //; s/ $//'
+}
+
+render_boot_ipxe() {
+  uki="fleet-${IMAGE_VERSION}.efi"
+  cmdline=""
+  [ -f "/artifacts/${uki}" ] && cmdline=$(extract_cmdline "/artifacts/${uki}")
+  if [ -z "$cmdline" ]; then
+    echo "WARN: could not extract .cmdline from /artifacts/${uki} — iPXE boot will" >&2
+    echo "      hit a dracut shell under Secure Boot OFF (no embedded cmdline)." >&2
+    echo "      Check the /artifacts mount + binutils, or enable Secure Boot." >&2
+  fi
+  # Render the SAME boot logic to two files:
+  #   autoexec.ipxe — iPXE fetches + runs this at STARTUP (before autoboot). The
+  #     leading `ifopen net0` brings the link UP on NICs whose NII/UNDI link iPXE
+  #     can't establish during autoboot — e.g. EliteBook 820 G4 / Intel I219-LM,
+  #     which sits at link:down ("No configurable methods") until ifopen. Without
+  #     this, autoboot fails before it can even DHCP for boot.ipxe.
+  #   boot.ipxe — the DHCP-bootfile fallback (tag:ipxe) for firmware that DOES
+  #     autoboot-DHCP fine; harmless to carry the same ifopen.
+  # Both feed iPXE the UKI's REAL cmdline via imgargs (Secure Boot OFF; see top).
+  for f in /srv/tftp/autoexec.ipxe /srv/tftp/boot.ipxe; do
+    cat > "$f" <<EOF
+#!ipxe
+# NOTE: in iPXE a bare failing command aborts the script (-> iPXE exits -> the
+# firmware restarts PXE), so EVERY step that can fail is guarded with '||'.
+#
+# The link is already up + forwarding here (the TFTP fetch of this script just
+# succeeded), so try DHCP FIRST without touching the PHY — on a PortFast/edge
+# port this succeeds immediately with no delay.
+dhcp net0 && goto have_ip || echo initial DHCP failed; will reset link and retry
+# Fallback: reset the link, then retry DHCP long enough to outlast the switch's
+# STP forwarding delay (~30s) that the ifopen bounce triggers. A single ifopen
+# (held open across the loop) means the PHY bounces ONCE; STP then converges and
+# a later DHCP gets through. PROPER fix is PortFast/edge on the switch port so
+# there's no delay at all — see serving/tftp/README.md.
+ifopen net0 || goto err
+set tries 0
+:dhcp_retry
+dhcp net0 && goto have_ip || echo DHCP retry \${tries}
+inc tries || goto err
+iseq \${tries} 30 && goto no_ip || sleep 3
+goto dhcp_retry
+:have_ip
+imgfetch http://${SERVER_IP}/${uki} || goto err
+imgargs ${uki} ${cmdline}
+boot ${uki} || goto err
+:no_ip
+echo No DHCP after ~90s on net0 - enable PortFast/edge on the switch port (STP delay).
+:err
+echo Fleet iPXE chain failed (\${errno}) - dropping to iPXE shell
+shell
+EOF
+  done
+  echo "rendered autoexec.ipxe + boot.ipxe -> ${uki} (cmdline: ${cmdline:-<MISSING>})"
+}
+
+if [ -n "${SERVER_IP:-}" ] && [ -n "${IMAGE_VERSION:-}" ]; then
+  render_boot_ipxe
+else
+  echo "WARN: SERVER_IP/IMAGE_VERSION unset; /srv/tftp/boot.ipxe NOT rendered" >&2
+fi
+
+exec dnsmasq "$@"
@@ -48,6 +48,26 @@ services:
      # + DNS A/PTR. Auto-populated by fleet-namer (creative names per MAC); add
      # manual lines to pin specific hosts. RW so fleet-namer can append.
      - ./dnsmasq/dhcp-hosts:/etc/dnsmasq-hosts
+      # iPXE binary source (read-only) for the PXE-only chainload (firmware without
+      # UEFI HTTP boot, e.g. EliteBook 820 G4). Drop ipxe.efi here — see
+      # serving/tftp/README. The entrypoint copies it into /srv/tftp and renders
+      # boot.ipxe alongside it (the actual TFTP root is container-internal).
+      - ./tftp:/tftp-src:ro
+      # The release artifacts (same image volume as the http service), read-only,
+      # so the entrypoint can objcopy the UKI's .cmdline section into boot.ipxe
+      # (Secure Boot OFF needs the real cmdline fed to iPXE — see entrypoint.sh).
+      - type: image
+        source: ${OCI_IMAGE}:${IMAGE_VERSION}
+        target: /artifacts
+        image:
+          subpath: artifacts
+      # Fallback (Docker < 28 / offline): bind the local Packer output dir instead:
+      # - ${ARTIFACT_SRC}:/artifacts:ro
+    environment:
+      # Consumed by the entrypoint to render /srv/tftp/boot.ipxe (the iPXE chain
+      # script that clears load options so the UKI's embedded cmdline survives).
+      - SERVER_IP=${SERVER_IP}
+      - IMAGE_VERSION=${IMAGE_VERSION}
    command:
      - --keep-in-foreground
      # Log to stderr so `docker compose logs` captures output. Without this,
@@ -108,6 +128,32 @@ services:
      - --dhcp-vendorclass=set:httpboot,HTTPClient
      - --dhcp-option=tag:httpboot,60,HTTPClient
      - --dhcp-boot=tag:httpboot,http://${SERVER_IP}/fleet-${IMAGE_VERSION}.efi
+      # ---- PXE -> iPXE -> HTTP chainload (for firmware WITHOUT UEFI HTTP boot) ----
+      # Graceful fallback: HTTP-capable firmware matches 'httpboot' above and gets
+      # the UKI directly; PXE-only firmware (PXEClient) gets ipxe.efi over TFTP;
+      # once iPXE is running it re-DHCPs with user-class "iPXE" and is handed the
+      # SAME UKI HTTP URL, which it chainloads (.efi -> LoadImage). Each client
+      # gets the right thing based on what it advertises — no per-host config.
+      # snponly.efi = UEFI x64 iPXE that rides the FIRMWARE's NIC driver (UEFI SNP)
+      # instead of iPXE's own — works on virtio VMs AND real hardware where the
+      # native driver won't bind (e.g. EliteBook 820 G4 / Intel I219-LM, which
+      # fails ipxe.efi with "No configurable methods"). Ships in Ubuntu's `ipxe`
+      # package; see serving/tftp/README.md.
+      # NOTE (lab): the iPXE binary is UNSIGNED, so this path needs Secure Boot OFF.
+      # For production, sign it with the fleet db key (then SB stays enforced
+      # end-to-end) — see docs/secure-boot-keys.md.
+      - --enable-tftp
+      - --tftp-root=/srv/tftp
+      - --dhcp-vendorclass=set:pxe,PXEClient
+      - --dhcp-userclass=set:ipxe,iPXE
+      # firmware PXE, not yet iPXE -> hand iPXE (UEFI x64) over TFTP
+      - --dhcp-boot=tag:pxe,tag:!ipxe,snponly.efi
+      # iPXE running -> hand boot.ipxe (NOT the UKI URL directly). The script
+      # imgfetch'es the UKI then `imgargs`-clears the load options before boot, so
+      # systemd-stub uses the UKI's EMBEDDED cmdline. Booting the UKI URL directly
+      # makes iPXE pass the image name as the cmdline, which (Secure Boot OFF)
+      # REPLACES the embedded roothash/nvmf/... args -> dracut emergency shell.
+      - --dhcp-boot=tag:ipxe,boot.ipxe

  # --- HTTP boot server (serves the UKI) -------------------------------------
  http:
@@ -0,0 +1,118 @@
+# TFTP root — iPXE chainload for PXE-only firmware
+
+This directory is mounted read-only at `/srv/tftp` in the dnsmasq container and
+serves the iPXE network bootloader over TFTP to firmware that **lacks UEFI HTTP
+boot** (e.g. HP EliteBook 820 G4). HTTP-boot-capable firmware never touches this
+path — it gets the UKI straight over HTTP (see the `httpboot` tag in
+`docker-compose.yml`). The fallback is automatic, driven by what each client
+advertises (DHCP vendor-class / client-arch).
+
+Flow for a PXE-only host:
+
+```
+firmware PXE (PXEClient) --TFTP--> snponly.efi (iPXE)
+   iPXE re-DHCPs (user-class "iPXE") --> handed boot.ipxe (TFTP)
+   boot.ipxe: imgfetch UKI over HTTP -> imgargs <real cmdline> -> boot
+```
+
+`boot.ipxe` is **rendered at container start** by the dnsmasq entrypoint; only the
+iPXE binary (`snponly.efi`) is supplied here. The script exists because of a sharp
+edge:
+
+> **Why not chainload the UKI URL directly?** When iPXE executes the UKI it sets
+> the EFI load options to the image's command line — or, if that's empty, to the
+> image **name**. With **Secure Boot OFF**, systemd-stub uses the load options *as
+> the kernel cmdline*, **replacing** the UKI's embedded cmdline
+> (`roothash`/`rd.nvmf.discover`/`rd.dot1x=0`/`systemd.volatile=overlay`). So
+> booting the URL directly — *or clearing the args* — makes the kernel come up
+> with just `fleet-<ver>.efi` as its cmdline → **dracut emergency shell**.
+>
+> The fix: feed iPXE the **real cmdline**. On the serving host that cmdline only
+> lives *inside the UKI* (it's not in the OCI image — invariant #10 — nor shipped
+> as a file), so the entrypoint **extracts the UKI's `.cmdline` PE section with
+> `objcopy`** (the artifact image is mounted read-only at `/artifacts`) and bakes
+> it into `boot.ipxe` as `imgargs <uki> <cmdline>`. Needs `binutils` in the image
+> (added) and the artifact mount.
+>
+> **Secure Boot ON makes all of this moot** — the stub ignores any loader-supplied
+> cmdline and uses the embedded one — so this is purely a lab/SB-OFF concern; the
+> rendered script is harmless once SB is on.
+
+## What to drop here
+
+A UEFI x86-64 iPXE binary named **`snponly.efi`** (the name the dnsmasq config
+serves). `snponly` rides the **firmware's** NIC driver via the UEFI SNP protocol
+rather than iPXE's own, so it works on virtio VMs *and* real hardware where the
+native driver won't bind.
+
+```bash
+# Ubuntu/Debian package (simplest — confirmed present):
+sudo apt-get install -y ipxe        # then copy whatever snponly it ships, e.g.:
+# dpkg -L ipxe | grep -i snponly
+cp /usr/lib/ipxe/snponly.efi serving/tftp/snponly.efi
+# Fedora/EL: dnf install ipxe-bootimgs -> /usr/share/ipxe/ipxe-snponly-x86_64.efi
+# or build:  git clone https://github.com/ipxe/ipxe && cd ipxe/src
+#            make bin-x86_64-efi/snponly.efi
+```
+
+> **Why `snponly`, not the full `ipxe.efi`?** iPXE's own NIC drivers don't bind on
+> some hardware — the **EliteBook 820 G4 (Intel I219-LM)** fails `ipxe.efi` with
+> *"net0: No configurable methods"*. `snponly` inherits the firmware's working
+> driver (the one that just loaded iPXE over PXE), so it's the robust default.
+> `boot.ipxe.org` only publishes the full `ipxe.efi`, not `snponly` — get `snponly`
+> from a distro package or build it.
+
+> **Updating the binary requires a dnsmasq restart** — the entrypoint copies it
+> from this (read-only) mount into the container's TFTP root at startup:
+> `docker compose restart dnsmasq`.
+
+(BIOS/legacy hosts would need `undionly.kpxe` + client-arch matching; this fleet
+is UEFI x64.)
+
+## Secure Boot
+
+**Lab (now): Secure Boot OFF for the iPXE path.** `snponly.efi` here is
+**unsigned**, so a Secure-Boot-enforcing machine would reject it.
+
+**Production upgrade:** sign `snponly.efi` with the fleet **db** key (the same key
+that signs the UKI), then Secure Boot stays enforced end-to-end
+(firmware → verifies iPXE vs db → iPXE `chain`s the UKI → firmware LoadImage
+verifies UKI vs db). Only `db` needs enrolling — no Microsoft keys / shim, since
+the fleet owns its keys. See [../../docs/secure-boot-keys.md](../../docs/secure-boot-keys.md).
+
+```bash
+# production (where db.key lives — CI/HSM):
+sbsign --key db.key --cert db.crt --output snponly.efi snponly.efi
+```
+
+## Bench notes / gotchas (EliteBook 820 G4 / Intel I219-LM, 2026-06-20)
+
+Hard-won findings from bringing up the first PXE-only physical host:
+
+- **Use `snponly.efi`, not the full `ipxe.efi`.** The 820 G4's NIC is an Intel
+  **I219-LM** (PCI `00:1f.6`). iPXE's native `intel` driver in the prebuilt full
+  build doesn't bind it (`net0: No configurable methods` / no `net0`). `snponly`
+  rides the firmware's driver via NII/SNP, so `net0` appears. Ubuntu's `ipxe`
+  package ships a usable `snponly.efi` (no build needed).
+- **PortFast/edge on the access port is REQUIRED** (this was the real blocker).
+  iPXE's `ifopen` resets the PHY → the link bounces → a managed switch restarts
+  its STP forwarding delay (~30 s) and silently drops packets → iPXE's DHCP never
+  reaches the server (`ifstat` shows `link:down` / `Link status: Unknown`; symptom
+  is `No configurable methods` or a dracut shell). The firmware's own PXE works
+  only because it keeps the link up continuously from POST so STP already
+  converged. **Fix: enable PortFast/edge (or disable STP) on the workstation
+  port** — VERIFIED to fix it 2026-06-20. (Cisco `spanning-tree portfast`;
+  Aruba/HP `admin-edge-port`; UniFi "Fast"; or a dumb unmanaged switch.) The
+  `autoexec.ipxe` retry loop (below) is a fallback that outlasts STP but adds
+  ~30 s to every boot — PortFast removes the delay entirely.
+- **iPXE aborts the script on any unguarded command failure** (→ iPXE exits → the
+  firmware restarts the whole PXE process, looking like "it doesn't retry"). Every
+  fallible step in `boot.ipxe`/`autoexec.ipxe` is therefore `||`-guarded.
+- **`autoexec.ipxe`** (rendered by the dnsmasq entrypoint) is fetched + run at
+  iPXE startup, *replacing* the default autoboot; that's where the `ifopen` +
+  DHCP-retry + `imgargs <real cmdline>` chain lives.
+
+## Note
+
+`*.efi` here is **gitignored** (binary, and the signed one is environment-specific
+/ produced from db). Keep this README + `.gitkeep`.