split out hash because rocky doesnt support it inline

CLAUDE.md

@@ -100,10 +100,12 @@ instead of working around it.
    When adding a runtime dependency to the hook, add it to the dracut
    module’s `install()` *and* extend the lsinitrd assertion.
 1. **The OCI artifact image stays FROM scratch.** Only
-   `/artifacts/{fleet-<ver>.efi, rootfs.erofs, verity.json, SHA256SUMS}`.
+   `/artifacts/{fleet-<ver>.efi, rootfs.erofs, rootfs.hash, verity.json,
-   No shell, no tools, no base image. Serving software (SPDK, nginx) lives
+   SHA256SUMS}`. `rootfs.erofs` (erofs data) and `rootfs.hash` (dm-verity hash
-   in separate, independently versioned images — never merge them into the
+   tree) are **separate** namespaces because the as-built Rocky 9 systemd does
-   artifact image.
+   not support the in-file `hash-offset` verity layout. No shell, no tools, no
+   base image. Serving software (SPDK, nginx) lives in separate, independently
+   versioned images — never merge them into the artifact image.
 1. **Deployments are pinned by digest.** One commit per fleet release in
    the deployment manifest; that manifest is the single source coupling
    UKI ↔ namespace. Rollback = revert the commit.
@@ -128,8 +130,12 @@ instead of working around it.
   build; the supplicant config carries both options.
 - erofs module availability on the target RHEL kernel (fallback:
   journal-less ext4 in `extract-artifacts.sh`).
-- `systemd.verity_root_options=hash-offset` support in the shipped systemd
+- `systemd.verity_root_options=hash-offset` is **NOT supported** by Rocky 9.8
-  (fallback: split data/hash into two namespaces).
+  systemd-veritysetup (verified: "unknown option hash-offset=" → "Failed to load
+  verity superblock"). Resolved by the documented fallback — data and hash are
+  **two separate NVMe namespaces** (`rootfs.erofs` + `rootfs.hash`, distinct
+  pinned UUIDs), so the cmdline uses plain `systemd.verity_root_data=` /
+  `systemd.verity_root_hash=` by-id devices with no offset.
 - `network-legacy` dracut module needs `dhclient`; RHEL 9 omits it from the
   minimal install, so the provisioner installs `dhcp-client` (without it dracut
   fails "module network-legacy can't be installed"). dhclient is deprecated —

docs/TODO.md

@@ -173,6 +173,18 @@ boot LAN.
     `/etc/udev/rules.d/95-nvmf-initqueue.rules`) — so the fully-native
     `rd.nvmf.discover` autoconnect path is available as a fallback if the unit
     doesn't fire on boot.
+  - **dm-verity: `hash-offset` is unsupported → two namespaces.** With NVMe
+    connecting and the right by-id device present, verity still failed:
+    `systemd-veritysetup` logged `unknown option 'hash-offset=…'` then `Failed
+    to load verity superblock` — exactly the CLAUDE.md bench-verify item. The
+    in-file (data+hash in one namespace, split by offset) layout is not
+    supported by Rocky 9.8 systemd. **Fixed** via the documented fallback:
+    `extract-artifacts.sh` now emits `rootfs.erofs` (data) + `rootfs.hash` (hash
+    tree) as separate files, served as **two NVMe namespaces** with distinct
+    pinned UUIDs (`f1ee7000-…-01` data, `…-02` hash); the cmdline uses
+    `systemd.verity_root_data=`/`systemd.verity_root_hash=` by-id devices with no
+    offset. Touches `extract-artifacts.sh`, `build-oci.sh` (artifact list +
+    SHA256SUMS), `serving/spdk-setup.sh` (2nd bdev/namespace), and invariant #9.
     **⚠ The secure path has the same dependency on the
     (absent) nvmf autoconnect** — when 802.1X is re-enabled, the macsec0 path
     must likewise drive `connect-all` (or the network-manager port #1 must wire

docs/design-v2-errata.md

@@ -4,7 +4,7 @@ Records where the as-built implementation diverges from
 `diskless-fleet-design-v2.pdf`. The PDF remains authoritative for the
 architecture; this note captures deltas discovered bringing the build up on
 **Rocky 9.8** until they are folded into a consolidated **v3**. Last updated
-2026-06-13.
+2026-06-15.
 
 The architecture is unchanged: 802.1X EAP-TLS with a TPM-held key, MACsec on the
 access link, dm-verity-protected erofs root over NVMe/TCP, signed UKI, and the
@@ -30,6 +30,33 @@ failure mode is NFS root — not applicable here (NVMe block root). Set in
 [../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh); module list
 in [../files/etc/dracut.conf.d/90-diskless.conf](../files/etc/dracut.conf.d/90-diskless.conf).
 
+## 1b. Design-level change — dm-verity uses two namespaces, not in-file offset
+
+**PDF says (§5):** the erofs data and the dm-verity hash tree share a single
+NVMe/TCP namespace, appended in one file and split on the host with
+`systemd.verity_root_options=hash-offset=`.
+
+**As built:** data and hash are **two separate namespaces** — `rootfs.erofs`
+(data) and `rootfs.hash` (hash tree) — each a file in the OCI artifact, served as
+its own NVMe namespace with a distinct pinned UUID. The cmdline uses
+`systemd.verity_root_data=` / `systemd.verity_root_hash=` by-id devices with no
+offset.
+
+**Why:** Rocky 9.8 `systemd-veritysetup` does not support `hash-offset=` — it
+logs `Encountered unknown option 'hash-offset=…', ignoring` then `Failed to load
+verity superblock: invalid argument`, so the single-namespace layout never maps
+`/dev/mapper/root`. Two namespaces is the design's own documented fallback and
+uses systemd's native verity (no custom activation code; keeps root-hash
+signature support for production). Devices are referenced by stable
+`/dev/disk/by-id/nvme-uuid.<uuid>` (not `/dev/nvmeNnM`) because controller
+indices shift with the transient discovery controller.
+
+**Touches:** [../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh)
+(emits `rootfs.hash`, two UUIDs, offset-free cmdline),
+[../scripts/build-oci.sh](../scripts/build-oci.sh) (artifact list + SHA256SUMS),
+[../serving/spdk-setup.sh](../serving/spdk-setup.sh) (2nd bdev/namespace), and
+CLAUDE.md invariant #9.
+
 ## 2. Implementation notes (not architectural)
 
 These don't contradict the PDF (it doesn't specify them) but are worth recording.

scripts/build-oci.sh

@@ -10,7 +10,8 @@
 #      verity.json -- a mismatched pair is an unbootable fleet
 #   4. builds a FROM-scratch OCI image containing ONLY:
 #        /artifacts/fleet-<version>.efi   (the signed UKI)
-#        /artifacts/rootfs.erofs          (erofs + appended verity tree)
+#        /artifacts/rootfs.erofs          (erofs DATA namespace)
+#        /artifacts/rootfs.hash           (dm-verity HASH namespace)
 #        /artifacts/verity.json
 #        /artifacts/SHA256SUMS
 #      Deploy as an initContainer (or k8s image volume) feeding the
@@ -79,7 +80,7 @@ echo "==> roothash coupling verified: $WANT"
 STAGE=$(mktemp -d)
 trap 'rm -rf "$STAGE"' EXIT
 mkdir "$STAGE/artifacts"
-cp "$UKI" "$ART/rootfs.erofs" "$ART/verity.json" "$STAGE/artifacts/"
+cp "$UKI" "$ART/rootfs.erofs" "$ART/rootfs.hash" "$ART/verity.json" "$STAGE/artifacts/"
 ( cd "$STAGE/artifacts" && sha256sum -- * > SHA256SUMS )
 
 REF="${IMAGE}:${VERSION}"

scripts/extract-artifacts.sh

@@ -4,9 +4,9 @@
 # Pulls the diskless deliverables out of the Packer-built disk image:
 #   vmlinuz           -> HTTPS boot server
 #   initramfs.img     -> HTTPS boot server (the fleet-generic image)
-#   rootfs.erofs      -> nvmet target: shared READ-ONLY NVMe/TCP namespace
+#   rootfs.erofs      -> nvmet target: READ-ONLY NVMe/TCP DATA namespace (erofs)
-#                        (erofs data + dm-verity hash tree appended in-file)
+#   rootfs.hash       -> nvmet target: READ-ONLY NVMe/TCP dm-verity HASH namespace
-#   verity.json       -> root hash + hash offset (feeds the kernel cmdline)
+#   verity.json       -> root hash + data/hash namespace UUIDs (feed the cmdline)
 #   cmdline.txt       -> ready-made kernel cmdline fragment for this build
 #
 # Requires on the build host: libguestfs-tools, erofs-utils (mkfs.erofs),
@@ -23,14 +23,19 @@ DISABLE_DOT1X=${3:-false}
 # environments. Defaults to the literal placeholder so build-oci.sh's warning
 # fires if it was never set.
 NVMET_TRADDR=${NVMET_TRADDR:-NVMET_TRADDR}
-# Stable root-namespace UUID. The root device is referenced by-id
+# Stable namespace UUIDs. Devices are referenced by-id
-# (/dev/disk/by-id/nvme-uuid.$ROOT_NS_UUID) rather than /dev/nvmeNn1 because
+# (/dev/disk/by-id/nvme-uuid.<uuid>) rather than /dev/nvmeNnM because NVMe
-# NVMe controller indices are NOT stable: a transient discovery controller can
+# controller/namespace indices are NOT stable (a transient discovery controller
-# take nvme0, pushing the I/O controller to nvme1, so a hardcoded node is a
+# can shift them), so a hardcoded node is a latent brick in an immutable signed
-# latent brick in an immutable signed cmdline. MUST match ROOT_NS_UUID in
+# cmdline. dm-verity needs TWO namespaces — erofs DATA and the verity HASH tree
-# serving/spdk-setup.sh (which assigns it on the target via add_ns -u).
+# — because this distro's systemd-veritysetup does not support the in-file
-ROOT_NS_UUID=${ROOT_NS_UUID:-f1ee7000-0000-4000-8000-000000000001}
+# hash-offset layout (it ignores hash-offset= and fails to load the superblock).
-ROOT_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_NS_UUID}"
+# MUST match ROOT_DATA_UUID / ROOT_HASH_UUID in serving/spdk-setup.sh (assigned
+# on the target via add_ns -u).
+ROOT_DATA_UUID=${ROOT_DATA_UUID:-f1ee7000-0000-4000-8000-000000000001}
+ROOT_HASH_UUID=${ROOT_HASH_UUID:-f1ee7000-0000-4000-8000-000000000002}
+ROOT_DATA_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_DATA_UUID}"
+ROOT_HASH_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_HASH_UUID}"
 
 for bin in virt-copy-out virt-tar-out virt-ls mkfs.erofs veritysetup jq; do
     command -v "$bin" >/dev/null || { echo "ERROR: $bin not found on build host" >&2; exit 1; }
@@ -69,21 +74,24 @@ FIXED_UUID="6c10a7f0-1d2e-4f5a-9b3c-000000000001"
 mkfs.erofs -T 0 -U "$FIXED_UUID" --all-root \
     "$WORK/rootfs.erofs" "$WORK/root"
 
-echo "==> Appending dm-verity hash tree (single-file / single-namespace layout)"
+echo "==> Generating dm-verity hash tree as a SEPARATE file (two-namespace layout)"
-DATA_BYTES=$(stat -c %s "$WORK/rootfs.erofs")
+# Hash tree in its OWN file (served as a 2nd namespace), NOT appended in-file:
-# Align hash area to 4 KiB after the data
+# this distro's systemd-veritysetup ignores hash-offset= ("unknown option") and
-HASH_OFFSET=$(( (DATA_BYTES + 4095) / 4096 * 4096 ))
+# fails to load the verity superblock. With separate data/hash devices the
-VERITY_OUT=$(veritysetup format "$WORK/rootfs.erofs" "$WORK/rootfs.erofs" \
+# cmdline needs no offset. Deterministic for identical input trees (fixed erofs
-    --hash-offset="$HASH_OFFSET" \
+# UUID + -T0 => stable data => stable hash tree => stable root hash).
+VERITY_OUT=$(veritysetup format "$WORK/rootfs.erofs" "$WORK/rootfs.hash" \
     --data-block-size=4096 --hash-block-size=4096)
 ROOT_HASH=$(printf '%s\n' "$VERITY_OUT" | awk '/Root hash:/ {print $3}')
 [ -n "$ROOT_HASH" ] || { echo "ERROR: veritysetup produced no root hash" >&2; exit 1; }
 
 mv "$WORK/rootfs.erofs" "$OUT/rootfs.erofs"
+mv "$WORK/rootfs.hash"  "$OUT/rootfs.hash"
 
-jq -n --arg hash "$ROOT_HASH" --argjson offset "$HASH_OFFSET" \
+jq -n --arg hash "$ROOT_HASH" --arg erofs_uuid "$FIXED_UUID" \
-      --arg uuid "$FIXED_UUID" \
+      --arg data_uuid "$ROOT_DATA_UUID" --arg hash_uuid "$ROOT_HASH_UUID" \
-      '{root_hash: $hash, hash_offset: $offset, erofs_uuid: $uuid,
+      '{root_hash: $hash, erofs_uuid: $erofs_uuid,
+        data_ns_uuid: $data_uuid, hash_ns_uuid: $hash_uuid,
         data_block_size: 4096, hash_block_size: 4096}' \
       > "$OUT/verity.json"
 
@@ -110,21 +118,20 @@ printf '%s' \
   "rd.nvmf.discover=tcp,${NVMET_TRADDR},4420 " \
   "root=/dev/mapper/root rootfstype=erofs rootflags=ro " \
   "systemd.verity=yes roothash=${ROOT_HASH} " \
-  "systemd.verity_root_data=${ROOT_DEV} " \
+  "systemd.verity_root_data=${ROOT_DATA_DEV} " \
-  "systemd.verity_root_hash=${ROOT_DEV} " \
+  "systemd.verity_root_hash=${ROOT_HASH_DEV}" \
-  "systemd.verity_root_options=hash-offset=${HASH_OFFSET}" \
   > "$OUT/cmdline.txt"
 
 echo "==> Checksums"
-( cd "$OUT" && sha256sum vmlinuz initramfs.img rootfs.erofs | tee SHA256SUMS )
+( cd "$OUT" && sha256sum vmlinuz initramfs.img rootfs.erofs rootfs.hash | tee SHA256SUMS )
 
 cat <<EOF
 
 Artifacts in $OUT:
   vmlinuz, initramfs.img -> publish to the HTTPS boot server (versioned dir,
                             flip a symlink for atomic rollout/rollback)
-  rootfs.erofs           -> expose as a READ-ONLY nvmet namespace; version
+  rootfs.erofs           -> expose as a READ-ONLY nvmet DATA namespace
-                            rollout = publish new namespace, repoint hosts
+  rootfs.hash            -> expose as a READ-ONLY nvmet HASH namespace (verity)
   cmdline.txt            -> kernel cmdline for this build. The roothash binds
                             the root fs to the boot artifacts: serve via a
                             signed UKI or signed GRUB config so Secure Boot

serving/spdk-setup.sh

@@ -1,29 +1,32 @@
 #!/usr/bin/env bash
-# Configure SPDK nvmf_tgt to serve the diskless root filesystem as a single
+# Configure SPDK nvmf_tgt to serve the diskless root filesystem as TWO READ-ONLY
-# READ-ONLY NVMe/TCP namespace (design doc §5, §7).
+# NVMe/TCP namespaces — erofs DATA and the dm-verity HASH tree (design doc §5, §7).
 #
-# rootfs.erofs is the erofs image with the dm-verity hash tree appended in-file;
+# Two namespaces (not one with an in-file hash-offset): the target Rocky 9
-# we expose the WHOLE file as one namespace, and the host splits data/hash with
+# systemd-veritysetup does not support the hash-offset layout, so the host needs
-# systemd.verity_root_options=hash-offset= from the kernel cmdline. Integrity is
+# the verity data and hash on separate block devices. Integrity is enforced
-# enforced end-to-end by dm-verity on the host; the target serves it read-only.
+# end-to-end by dm-verity on the host; the target serves both read-only.
 set -euo pipefail
 
 ROOTFS="/artifacts/rootfs.erofs"
+ROOTHASH="/artifacts/rootfs.hash"
 SUBNQN="${SUBNQN:-nqn.2026-06.org.example.fleet:rootfs}"
-# Fixed, fleet-generic namespace UUID. The host references the root device as
+# Fixed, fleet-generic namespace UUIDs. The host references each device as
-# /dev/disk/by-id/nvme-uuid.$ROOT_NS_UUID on the kernel cmdline, which is stable
+# /dev/disk/by-id/nvme-uuid.<uuid> on the kernel cmdline, stable regardless of
-# regardless of NVMe controller enumeration order (the I/O controller can land
+# NVMe controller/namespace enumeration order (a transient discovery controller
-# on nvme0/nvme1/... depending on whether a transient discovery controller took
+# can shift indices — so a hardcoded /dev/nvmeNnM is unsafe). MUST match
-# a slot first — so a hardcoded /dev/nvme0n1 is unsafe). MUST match
+# ROOT_DATA_UUID / ROOT_HASH_UUID in scripts/extract-artifacts.sh, which bake
-# ROOT_NS_UUID in scripts/extract-artifacts.sh, which bakes it into the UKI
+# them into the UKI cmdline. NOTE: for multi-version serving (TODO #6) make
-# cmdline. NOTE: for multi-version serving (TODO #6) make this version-specific
+# these version-specific alongside the per-version NQN, so a host can't connect
-# alongside the per-version NQN, so a host can't connect the wrong rootfs.
+# the wrong rootfs.
-ROOT_NS_UUID="${ROOT_NS_UUID:-f1ee7000-0000-4000-8000-000000000001}"
+ROOT_DATA_UUID="${ROOT_DATA_UUID:-f1ee7000-0000-4000-8000-000000000001}"
+ROOT_HASH_UUID="${ROOT_HASH_UUID:-f1ee7000-0000-4000-8000-000000000002}"
 TADDR="${TARGET_ADDR:-0.0.0.0}"
 TPORT="${TARGET_PORT:-4420}"
 CPUMASK="${SPDK_CPUMASK:-0x1}"
 
 [ -s "$ROOTFS" ]   || { echo "ERROR: $ROOTFS not found/empty — is ARTIFACT_SRC populated?" >&2; exit 1; }
+[ -s "$ROOTHASH" ] || { echo "ERROR: $ROOTHASH not found/empty — rebuild artifacts with the two-namespace layout." >&2; exit 1; }
 
 # Binary locations differ across SPDK images; fall back to the common path.
 TGT=$(command -v nvmf_tgt || echo /usr/libexec/spdk/spdk/build/bin/nvmf_tgt)
@@ -59,21 +62,24 @@ done
 echo "==> creating TCP transport"
 "$RPC" nvmf_create_transport -t TCP -u 16384 -m 8 -c 8192
 
-echo "==> exposing $ROOTFS as bdev 'rootfs' (4 KiB blocks)"
+echo "==> exposing $ROOTFS (data) + $ROOTHASH (verity hash) as bdevs (4 KiB blocks)"
 # -r opens the backing file read-only where supported; harmless if ignored.
 "$RPC" bdev_aio_create "$ROOTFS" rootfs 4096 -r \
     || "$RPC" bdev_aio_create "$ROOTFS" rootfs 4096
+"$RPC" bdev_aio_create "$ROOTHASH" roothash 4096 -r \
+    || "$RPC" bdev_aio_create "$ROOTHASH" roothash 4096
 
 echo "==> creating subsystem $SUBNQN (allow any host — lab/no-auth)"
 "$RPC" nvmf_create_subsystem "$SUBNQN" -a -s SPDKLAB0000000001 -d FleetRootfs
 
-echo "==> adding namespace 1 (uuid $ROOT_NS_UUID)"
 # Read-only is enforced by the bdev_aio '-r' open above (this SPDK's add_ns has
 # no --read-only flag), plus the :ro image-volume mount and the client's erofs
-# ro + dm-verity.
+# ro + dm-verity. -u pins each namespace UUID so the host references them as
-# -u pins the namespace UUID so the host can reference it as
+# /dev/disk/by-id/nvme-uuid.<uuid> (stable across controller indices).
-# /dev/disk/by-id/nvme-uuid.$ROOT_NS_UUID (stable across controller indices).
+echo "==> adding namespace 1: data (uuid $ROOT_DATA_UUID)"
-"$RPC" nvmf_subsystem_add_ns "$SUBNQN" rootfs -u "$ROOT_NS_UUID"
+"$RPC" nvmf_subsystem_add_ns "$SUBNQN" rootfs -u "$ROOT_DATA_UUID"
+echo "==> adding namespace 2: verity hash (uuid $ROOT_HASH_UUID)"
+"$RPC" nvmf_subsystem_add_ns "$SUBNQN" roothash -u "$ROOT_HASH_UUID"
 
 echo "==> adding TCP listener ${TADDR}:${TPORT} to $SUBNQN"
 "$RPC" nvmf_subsystem_add_listener "$SUBNQN" -t tcp -a "$TADDR" -s "$TPORT"