diff --git a/CLAUDE.md b/CLAUDE.md index b22cd05..da68f48 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -100,10 +100,12 @@ instead of working around it. When adding a runtime dependency to the hook, add it to the dracut module’s `install()` *and* extend the lsinitrd assertion. 1. **The OCI artifact image stays FROM scratch.** Only - `/artifacts/{fleet-.efi, rootfs.erofs, verity.json, SHA256SUMS}`. - No shell, no tools, no base image. Serving software (SPDK, nginx) lives - in separate, independently versioned images — never merge them into the - artifact image. + `/artifacts/{fleet-.efi, rootfs.erofs, rootfs.hash, verity.json, + SHA256SUMS}`. `rootfs.erofs` (erofs data) and `rootfs.hash` (dm-verity hash + tree) are **separate** namespaces because the as-built Rocky 9 systemd does + not support the in-file `hash-offset` verity layout. No shell, no tools, no + base image. Serving software (SPDK, nginx) lives in separate, independently + versioned images — never merge them into the artifact image. 1. **Deployments are pinned by digest.** One commit per fleet release in the deployment manifest; that manifest is the single source coupling UKI ↔ namespace. Rollback = revert the commit. @@ -128,8 +130,12 @@ instead of working around it. build; the supplicant config carries both options. - erofs module availability on the target RHEL kernel (fallback: journal-less ext4 in `extract-artifacts.sh`). -- `systemd.verity_root_options=hash-offset` support in the shipped systemd - (fallback: split data/hash into two namespaces). +- `systemd.verity_root_options=hash-offset` is **NOT supported** by Rocky 9.8 + systemd-veritysetup (verified: "unknown option hash-offset=" → "Failed to load + verity superblock"). Resolved by the documented fallback — data and hash are + **two separate NVMe namespaces** (`rootfs.erofs` + `rootfs.hash`, distinct + pinned UUIDs), so the cmdline uses plain `systemd.verity_root_data=` / + `systemd.verity_root_hash=` by-id devices with no offset. - `network-legacy` dracut module needs `dhclient`; RHEL 9 omits it from the minimal install, so the provisioner installs `dhcp-client` (without it dracut fails "module network-legacy can't be installed"). dhclient is deprecated — diff --git a/docs/TODO.md b/docs/TODO.md index 363e979..37739d3 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -173,6 +173,18 @@ boot LAN. `/etc/udev/rules.d/95-nvmf-initqueue.rules`) — so the fully-native `rd.nvmf.discover` autoconnect path is available as a fallback if the unit doesn't fire on boot. + - **dm-verity: `hash-offset` is unsupported → two namespaces.** With NVMe + connecting and the right by-id device present, verity still failed: + `systemd-veritysetup` logged `unknown option 'hash-offset=…'` then `Failed + to load verity superblock` — exactly the CLAUDE.md bench-verify item. The + in-file (data+hash in one namespace, split by offset) layout is not + supported by Rocky 9.8 systemd. **Fixed** via the documented fallback: + `extract-artifacts.sh` now emits `rootfs.erofs` (data) + `rootfs.hash` (hash + tree) as separate files, served as **two NVMe namespaces** with distinct + pinned UUIDs (`f1ee7000-…-01` data, `…-02` hash); the cmdline uses + `systemd.verity_root_data=`/`systemd.verity_root_hash=` by-id devices with no + offset. Touches `extract-artifacts.sh`, `build-oci.sh` (artifact list + + SHA256SUMS), `serving/spdk-setup.sh` (2nd bdev/namespace), and invariant #9. **⚠ The secure path has the same dependency on the (absent) nvmf autoconnect** — when 802.1X is re-enabled, the macsec0 path must likewise drive `connect-all` (or the network-manager port #1 must wire diff --git a/docs/design-v2-errata.md b/docs/design-v2-errata.md index f7fecd4..c11f2dd 100644 --- a/docs/design-v2-errata.md +++ b/docs/design-v2-errata.md @@ -4,7 +4,7 @@ Records where the as-built implementation diverges from `diskless-fleet-design-v2.pdf`. The PDF remains authoritative for the architecture; this note captures deltas discovered bringing the build up on **Rocky 9.8** until they are folded into a consolidated **v3**. Last updated -2026-06-13. +2026-06-15. The architecture is unchanged: 802.1X EAP-TLS with a TPM-held key, MACsec on the access link, dm-verity-protected erofs root over NVMe/TCP, signed UKI, and the @@ -30,6 +30,33 @@ failure mode is NFS root — not applicable here (NVMe block root). Set in [../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh); module list in [../files/etc/dracut.conf.d/90-diskless.conf](../files/etc/dracut.conf.d/90-diskless.conf). +## 1b. Design-level change — dm-verity uses two namespaces, not in-file offset + +**PDF says (§5):** the erofs data and the dm-verity hash tree share a single +NVMe/TCP namespace, appended in one file and split on the host with +`systemd.verity_root_options=hash-offset=`. + +**As built:** data and hash are **two separate namespaces** — `rootfs.erofs` +(data) and `rootfs.hash` (hash tree) — each a file in the OCI artifact, served as +its own NVMe namespace with a distinct pinned UUID. The cmdline uses +`systemd.verity_root_data=` / `systemd.verity_root_hash=` by-id devices with no +offset. + +**Why:** Rocky 9.8 `systemd-veritysetup` does not support `hash-offset=` — it +logs `Encountered unknown option 'hash-offset=…', ignoring` then `Failed to load +verity superblock: invalid argument`, so the single-namespace layout never maps +`/dev/mapper/root`. Two namespaces is the design's own documented fallback and +uses systemd's native verity (no custom activation code; keeps root-hash +signature support for production). Devices are referenced by stable +`/dev/disk/by-id/nvme-uuid.` (not `/dev/nvmeNnM`) because controller +indices shift with the transient discovery controller. + +**Touches:** [../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh) +(emits `rootfs.hash`, two UUIDs, offset-free cmdline), +[../scripts/build-oci.sh](../scripts/build-oci.sh) (artifact list + SHA256SUMS), +[../serving/spdk-setup.sh](../serving/spdk-setup.sh) (2nd bdev/namespace), and +CLAUDE.md invariant #9. + ## 2. Implementation notes (not architectural) These don't contradict the PDF (it doesn't specify them) but are worth recording. diff --git a/scripts/build-oci.sh b/scripts/build-oci.sh index c9bce60..4764116 100755 --- a/scripts/build-oci.sh +++ b/scripts/build-oci.sh @@ -10,7 +10,8 @@ # verity.json -- a mismatched pair is an unbootable fleet # 4. builds a FROM-scratch OCI image containing ONLY: # /artifacts/fleet-.efi (the signed UKI) -# /artifacts/rootfs.erofs (erofs + appended verity tree) +# /artifacts/rootfs.erofs (erofs DATA namespace) +# /artifacts/rootfs.hash (dm-verity HASH namespace) # /artifacts/verity.json # /artifacts/SHA256SUMS # Deploy as an initContainer (or k8s image volume) feeding the @@ -79,7 +80,7 @@ echo "==> roothash coupling verified: $WANT" STAGE=$(mktemp -d) trap 'rm -rf "$STAGE"' EXIT mkdir "$STAGE/artifacts" -cp "$UKI" "$ART/rootfs.erofs" "$ART/verity.json" "$STAGE/artifacts/" +cp "$UKI" "$ART/rootfs.erofs" "$ART/rootfs.hash" "$ART/verity.json" "$STAGE/artifacts/" ( cd "$STAGE/artifacts" && sha256sum -- * > SHA256SUMS ) REF="${IMAGE}:${VERSION}" diff --git a/scripts/extract-artifacts.sh b/scripts/extract-artifacts.sh index f083e13..cec4094 100755 --- a/scripts/extract-artifacts.sh +++ b/scripts/extract-artifacts.sh @@ -4,9 +4,9 @@ # Pulls the diskless deliverables out of the Packer-built disk image: # vmlinuz -> HTTPS boot server # initramfs.img -> HTTPS boot server (the fleet-generic image) -# rootfs.erofs -> nvmet target: shared READ-ONLY NVMe/TCP namespace -# (erofs data + dm-verity hash tree appended in-file) -# verity.json -> root hash + hash offset (feeds the kernel cmdline) +# rootfs.erofs -> nvmet target: READ-ONLY NVMe/TCP DATA namespace (erofs) +# rootfs.hash -> nvmet target: READ-ONLY NVMe/TCP dm-verity HASH namespace +# verity.json -> root hash + data/hash namespace UUIDs (feed the cmdline) # cmdline.txt -> ready-made kernel cmdline fragment for this build # # Requires on the build host: libguestfs-tools, erofs-utils (mkfs.erofs), @@ -23,14 +23,19 @@ DISABLE_DOT1X=${3:-false} # environments. Defaults to the literal placeholder so build-oci.sh's warning # fires if it was never set. NVMET_TRADDR=${NVMET_TRADDR:-NVMET_TRADDR} -# Stable root-namespace UUID. The root device is referenced by-id -# (/dev/disk/by-id/nvme-uuid.$ROOT_NS_UUID) rather than /dev/nvmeNn1 because -# NVMe controller indices are NOT stable: a transient discovery controller can -# take nvme0, pushing the I/O controller to nvme1, so a hardcoded node is a -# latent brick in an immutable signed cmdline. MUST match ROOT_NS_UUID in -# serving/spdk-setup.sh (which assigns it on the target via add_ns -u). -ROOT_NS_UUID=${ROOT_NS_UUID:-f1ee7000-0000-4000-8000-000000000001} -ROOT_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_NS_UUID}" +# Stable namespace UUIDs. Devices are referenced by-id +# (/dev/disk/by-id/nvme-uuid.) rather than /dev/nvmeNnM because NVMe +# controller/namespace indices are NOT stable (a transient discovery controller +# can shift them), so a hardcoded node is a latent brick in an immutable signed +# cmdline. dm-verity needs TWO namespaces — erofs DATA and the verity HASH tree +# — because this distro's systemd-veritysetup does not support the in-file +# hash-offset layout (it ignores hash-offset= and fails to load the superblock). +# MUST match ROOT_DATA_UUID / ROOT_HASH_UUID in serving/spdk-setup.sh (assigned +# on the target via add_ns -u). +ROOT_DATA_UUID=${ROOT_DATA_UUID:-f1ee7000-0000-4000-8000-000000000001} +ROOT_HASH_UUID=${ROOT_HASH_UUID:-f1ee7000-0000-4000-8000-000000000002} +ROOT_DATA_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_DATA_UUID}" +ROOT_HASH_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_HASH_UUID}" for bin in virt-copy-out virt-tar-out virt-ls mkfs.erofs veritysetup jq; do command -v "$bin" >/dev/null || { echo "ERROR: $bin not found on build host" >&2; exit 1; } @@ -69,21 +74,24 @@ FIXED_UUID="6c10a7f0-1d2e-4f5a-9b3c-000000000001" mkfs.erofs -T 0 -U "$FIXED_UUID" --all-root \ "$WORK/rootfs.erofs" "$WORK/root" -echo "==> Appending dm-verity hash tree (single-file / single-namespace layout)" -DATA_BYTES=$(stat -c %s "$WORK/rootfs.erofs") -# Align hash area to 4 KiB after the data -HASH_OFFSET=$(( (DATA_BYTES + 4095) / 4096 * 4096 )) -VERITY_OUT=$(veritysetup format "$WORK/rootfs.erofs" "$WORK/rootfs.erofs" \ - --hash-offset="$HASH_OFFSET" \ +echo "==> Generating dm-verity hash tree as a SEPARATE file (two-namespace layout)" +# Hash tree in its OWN file (served as a 2nd namespace), NOT appended in-file: +# this distro's systemd-veritysetup ignores hash-offset= ("unknown option") and +# fails to load the verity superblock. With separate data/hash devices the +# cmdline needs no offset. Deterministic for identical input trees (fixed erofs +# UUID + -T0 => stable data => stable hash tree => stable root hash). +VERITY_OUT=$(veritysetup format "$WORK/rootfs.erofs" "$WORK/rootfs.hash" \ --data-block-size=4096 --hash-block-size=4096) ROOT_HASH=$(printf '%s\n' "$VERITY_OUT" | awk '/Root hash:/ {print $3}') [ -n "$ROOT_HASH" ] || { echo "ERROR: veritysetup produced no root hash" >&2; exit 1; } mv "$WORK/rootfs.erofs" "$OUT/rootfs.erofs" +mv "$WORK/rootfs.hash" "$OUT/rootfs.hash" -jq -n --arg hash "$ROOT_HASH" --argjson offset "$HASH_OFFSET" \ - --arg uuid "$FIXED_UUID" \ - '{root_hash: $hash, hash_offset: $offset, erofs_uuid: $uuid, +jq -n --arg hash "$ROOT_HASH" --arg erofs_uuid "$FIXED_UUID" \ + --arg data_uuid "$ROOT_DATA_UUID" --arg hash_uuid "$ROOT_HASH_UUID" \ + '{root_hash: $hash, erofs_uuid: $erofs_uuid, + data_ns_uuid: $data_uuid, hash_ns_uuid: $hash_uuid, data_block_size: 4096, hash_block_size: 4096}' \ > "$OUT/verity.json" @@ -110,21 +118,20 @@ printf '%s' \ "rd.nvmf.discover=tcp,${NVMET_TRADDR},4420 " \ "root=/dev/mapper/root rootfstype=erofs rootflags=ro " \ "systemd.verity=yes roothash=${ROOT_HASH} " \ - "systemd.verity_root_data=${ROOT_DEV} " \ - "systemd.verity_root_hash=${ROOT_DEV} " \ - "systemd.verity_root_options=hash-offset=${HASH_OFFSET}" \ + "systemd.verity_root_data=${ROOT_DATA_DEV} " \ + "systemd.verity_root_hash=${ROOT_HASH_DEV}" \ > "$OUT/cmdline.txt" echo "==> Checksums" -( cd "$OUT" && sha256sum vmlinuz initramfs.img rootfs.erofs | tee SHA256SUMS ) +( cd "$OUT" && sha256sum vmlinuz initramfs.img rootfs.erofs rootfs.hash | tee SHA256SUMS ) cat < publish to the HTTPS boot server (versioned dir, flip a symlink for atomic rollout/rollback) - rootfs.erofs -> expose as a READ-ONLY nvmet namespace; version - rollout = publish new namespace, repoint hosts + rootfs.erofs -> expose as a READ-ONLY nvmet DATA namespace + rootfs.hash -> expose as a READ-ONLY nvmet HASH namespace (verity) cmdline.txt -> kernel cmdline for this build. The roothash binds the root fs to the boot artifacts: serve via a signed UKI or signed GRUB config so Secure Boot diff --git a/serving/spdk-setup.sh b/serving/spdk-setup.sh index 4e7d3e1..006a471 100644 --- a/serving/spdk-setup.sh +++ b/serving/spdk-setup.sh @@ -1,29 +1,32 @@ #!/usr/bin/env bash -# Configure SPDK nvmf_tgt to serve the diskless root filesystem as a single -# READ-ONLY NVMe/TCP namespace (design doc §5, §7). +# Configure SPDK nvmf_tgt to serve the diskless root filesystem as TWO READ-ONLY +# NVMe/TCP namespaces — erofs DATA and the dm-verity HASH tree (design doc §5, §7). # -# rootfs.erofs is the erofs image with the dm-verity hash tree appended in-file; -# we expose the WHOLE file as one namespace, and the host splits data/hash with -# systemd.verity_root_options=hash-offset= from the kernel cmdline. Integrity is -# enforced end-to-end by dm-verity on the host; the target serves it read-only. +# Two namespaces (not one with an in-file hash-offset): the target Rocky 9 +# systemd-veritysetup does not support the hash-offset layout, so the host needs +# the verity data and hash on separate block devices. Integrity is enforced +# end-to-end by dm-verity on the host; the target serves both read-only. set -euo pipefail ROOTFS="/artifacts/rootfs.erofs" +ROOTHASH="/artifacts/rootfs.hash" SUBNQN="${SUBNQN:-nqn.2026-06.org.example.fleet:rootfs}" -# Fixed, fleet-generic namespace UUID. The host references the root device as -# /dev/disk/by-id/nvme-uuid.$ROOT_NS_UUID on the kernel cmdline, which is stable -# regardless of NVMe controller enumeration order (the I/O controller can land -# on nvme0/nvme1/... depending on whether a transient discovery controller took -# a slot first — so a hardcoded /dev/nvme0n1 is unsafe). MUST match -# ROOT_NS_UUID in scripts/extract-artifacts.sh, which bakes it into the UKI -# cmdline. NOTE: for multi-version serving (TODO #6) make this version-specific -# alongside the per-version NQN, so a host can't connect the wrong rootfs. -ROOT_NS_UUID="${ROOT_NS_UUID:-f1ee7000-0000-4000-8000-000000000001}" +# Fixed, fleet-generic namespace UUIDs. The host references each device as +# /dev/disk/by-id/nvme-uuid. on the kernel cmdline, stable regardless of +# NVMe controller/namespace enumeration order (a transient discovery controller +# can shift indices — so a hardcoded /dev/nvmeNnM is unsafe). MUST match +# ROOT_DATA_UUID / ROOT_HASH_UUID in scripts/extract-artifacts.sh, which bake +# them into the UKI cmdline. NOTE: for multi-version serving (TODO #6) make +# these version-specific alongside the per-version NQN, so a host can't connect +# the wrong rootfs. +ROOT_DATA_UUID="${ROOT_DATA_UUID:-f1ee7000-0000-4000-8000-000000000001}" +ROOT_HASH_UUID="${ROOT_HASH_UUID:-f1ee7000-0000-4000-8000-000000000002}" TADDR="${TARGET_ADDR:-0.0.0.0}" TPORT="${TARGET_PORT:-4420}" CPUMASK="${SPDK_CPUMASK:-0x1}" -[ -s "$ROOTFS" ] || { echo "ERROR: $ROOTFS not found/empty — is ARTIFACT_SRC populated?" >&2; exit 1; } +[ -s "$ROOTFS" ] || { echo "ERROR: $ROOTFS not found/empty — is ARTIFACT_SRC populated?" >&2; exit 1; } +[ -s "$ROOTHASH" ] || { echo "ERROR: $ROOTHASH not found/empty — rebuild artifacts with the two-namespace layout." >&2; exit 1; } # Binary locations differ across SPDK images; fall back to the common path. TGT=$(command -v nvmf_tgt || echo /usr/libexec/spdk/spdk/build/bin/nvmf_tgt) @@ -59,21 +62,24 @@ done echo "==> creating TCP transport" "$RPC" nvmf_create_transport -t TCP -u 16384 -m 8 -c 8192 -echo "==> exposing $ROOTFS as bdev 'rootfs' (4 KiB blocks)" +echo "==> exposing $ROOTFS (data) + $ROOTHASH (verity hash) as bdevs (4 KiB blocks)" # -r opens the backing file read-only where supported; harmless if ignored. "$RPC" bdev_aio_create "$ROOTFS" rootfs 4096 -r \ || "$RPC" bdev_aio_create "$ROOTFS" rootfs 4096 +"$RPC" bdev_aio_create "$ROOTHASH" roothash 4096 -r \ + || "$RPC" bdev_aio_create "$ROOTHASH" roothash 4096 echo "==> creating subsystem $SUBNQN (allow any host — lab/no-auth)" "$RPC" nvmf_create_subsystem "$SUBNQN" -a -s SPDKLAB0000000001 -d FleetRootfs -echo "==> adding namespace 1 (uuid $ROOT_NS_UUID)" # Read-only is enforced by the bdev_aio '-r' open above (this SPDK's add_ns has # no --read-only flag), plus the :ro image-volume mount and the client's erofs -# ro + dm-verity. -# -u pins the namespace UUID so the host can reference it as -# /dev/disk/by-id/nvme-uuid.$ROOT_NS_UUID (stable across controller indices). -"$RPC" nvmf_subsystem_add_ns "$SUBNQN" rootfs -u "$ROOT_NS_UUID" +# ro + dm-verity. -u pins each namespace UUID so the host references them as +# /dev/disk/by-id/nvme-uuid. (stable across controller indices). +echo "==> adding namespace 1: data (uuid $ROOT_DATA_UUID)" +"$RPC" nvmf_subsystem_add_ns "$SUBNQN" rootfs -u "$ROOT_DATA_UUID" +echo "==> adding namespace 2: verity hash (uuid $ROOT_HASH_UUID)" +"$RPC" nvmf_subsystem_add_ns "$SUBNQN" roothash -u "$ROOT_HASH_UUID" echo "==> adding TCP listener ${TADDR}:${TPORT} to $SUBNQN" "$RPC" nvmf_subsystem_add_listener "$SUBNQN" -t tcp -a "$TADDR" -s "$TPORT"