split out hash because rocky doesnt support it inline
This commit is contained in:
Audiobuzz
@@ -100,10 +100,12 @@ instead of working around it.
|
||||
When adding a runtime dependency to the hook, add it to the dracut
|
||||
module’s `install()` *and* extend the lsinitrd assertion.
|
||||
1. **The OCI artifact image stays FROM scratch.** Only
|
||||
`/artifacts/{fleet-<ver>.efi, rootfs.erofs, verity.json, SHA256SUMS}`.
|
||||
No shell, no tools, no base image. Serving software (SPDK, nginx) lives
|
||||
in separate, independently versioned images — never merge them into the
|
||||
artifact image.
|
||||
`/artifacts/{fleet-<ver>.efi, rootfs.erofs, rootfs.hash, verity.json,
|
||||
SHA256SUMS}`. `rootfs.erofs` (erofs data) and `rootfs.hash` (dm-verity hash
|
||||
tree) are **separate** namespaces because the as-built Rocky 9 systemd does
|
||||
not support the in-file `hash-offset` verity layout. No shell, no tools, no
|
||||
base image. Serving software (SPDK, nginx) lives in separate, independently
|
||||
versioned images — never merge them into the artifact image.
|
||||
1. **Deployments are pinned by digest.** One commit per fleet release in
|
||||
the deployment manifest; that manifest is the single source coupling
|
||||
UKI ↔ namespace. Rollback = revert the commit.
|
||||
@@ -128,8 +130,12 @@ instead of working around it.
|
||||
build; the supplicant config carries both options.
|
||||
- erofs module availability on the target RHEL kernel (fallback:
|
||||
journal-less ext4 in `extract-artifacts.sh`).
|
||||
- `systemd.verity_root_options=hash-offset` support in the shipped systemd
|
||||
(fallback: split data/hash into two namespaces).
|
||||
- `systemd.verity_root_options=hash-offset` is **NOT supported** by Rocky 9.8
|
||||
systemd-veritysetup (verified: "unknown option hash-offset=" → "Failed to load
|
||||
verity superblock"). Resolved by the documented fallback — data and hash are
|
||||
**two separate NVMe namespaces** (`rootfs.erofs` + `rootfs.hash`, distinct
|
||||
pinned UUIDs), so the cmdline uses plain `systemd.verity_root_data=` /
|
||||
`systemd.verity_root_hash=` by-id devices with no offset.
|
||||
- `network-legacy` dracut module needs `dhclient`; RHEL 9 omits it from the
|
||||
minimal install, so the provisioner installs `dhcp-client` (without it dracut
|
||||
fails "module network-legacy can't be installed"). dhclient is deprecated —
|
||||
|
||||
@@ -173,6 +173,18 @@ boot LAN.
|
||||
`/etc/udev/rules.d/95-nvmf-initqueue.rules`) — so the fully-native
|
||||
`rd.nvmf.discover` autoconnect path is available as a fallback if the unit
|
||||
doesn't fire on boot.
|
||||
- **dm-verity: `hash-offset` is unsupported → two namespaces.** With NVMe
|
||||
connecting and the right by-id device present, verity still failed:
|
||||
`systemd-veritysetup` logged `unknown option 'hash-offset=…'` then `Failed
|
||||
to load verity superblock` — exactly the CLAUDE.md bench-verify item. The
|
||||
in-file (data+hash in one namespace, split by offset) layout is not
|
||||
supported by Rocky 9.8 systemd. **Fixed** via the documented fallback:
|
||||
`extract-artifacts.sh` now emits `rootfs.erofs` (data) + `rootfs.hash` (hash
|
||||
tree) as separate files, served as **two NVMe namespaces** with distinct
|
||||
pinned UUIDs (`f1ee7000-…-01` data, `…-02` hash); the cmdline uses
|
||||
`systemd.verity_root_data=`/`systemd.verity_root_hash=` by-id devices with no
|
||||
offset. Touches `extract-artifacts.sh`, `build-oci.sh` (artifact list +
|
||||
SHA256SUMS), `serving/spdk-setup.sh` (2nd bdev/namespace), and invariant #9.
|
||||
**⚠ The secure path has the same dependency on the
|
||||
(absent) nvmf autoconnect** — when 802.1X is re-enabled, the macsec0 path
|
||||
must likewise drive `connect-all` (or the network-manager port #1 must wire
|
||||
|
||||
@@ -4,7 +4,7 @@ Records where the as-built implementation diverges from
|
||||
`diskless-fleet-design-v2.pdf`. The PDF remains authoritative for the
|
||||
architecture; this note captures deltas discovered bringing the build up on
|
||||
**Rocky 9.8** until they are folded into a consolidated **v3**. Last updated
|
||||
2026-06-13.
|
||||
2026-06-15.
|
||||
|
||||
The architecture is unchanged: 802.1X EAP-TLS with a TPM-held key, MACsec on the
|
||||
access link, dm-verity-protected erofs root over NVMe/TCP, signed UKI, and the
|
||||
@@ -30,6 +30,33 @@ failure mode is NFS root — not applicable here (NVMe block root). Set in
|
||||
[../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh); module list
|
||||
in [../files/etc/dracut.conf.d/90-diskless.conf](../files/etc/dracut.conf.d/90-diskless.conf).
|
||||
|
||||
## 1b. Design-level change — dm-verity uses two namespaces, not in-file offset
|
||||
|
||||
**PDF says (§5):** the erofs data and the dm-verity hash tree share a single
|
||||
NVMe/TCP namespace, appended in one file and split on the host with
|
||||
`systemd.verity_root_options=hash-offset=`.
|
||||
|
||||
**As built:** data and hash are **two separate namespaces** — `rootfs.erofs`
|
||||
(data) and `rootfs.hash` (hash tree) — each a file in the OCI artifact, served as
|
||||
its own NVMe namespace with a distinct pinned UUID. The cmdline uses
|
||||
`systemd.verity_root_data=` / `systemd.verity_root_hash=` by-id devices with no
|
||||
offset.
|
||||
|
||||
**Why:** Rocky 9.8 `systemd-veritysetup` does not support `hash-offset=` — it
|
||||
logs `Encountered unknown option 'hash-offset=…', ignoring` then `Failed to load
|
||||
verity superblock: invalid argument`, so the single-namespace layout never maps
|
||||
`/dev/mapper/root`. Two namespaces is the design's own documented fallback and
|
||||
uses systemd's native verity (no custom activation code; keeps root-hash
|
||||
signature support for production). Devices are referenced by stable
|
||||
`/dev/disk/by-id/nvme-uuid.<uuid>` (not `/dev/nvmeNnM`) because controller
|
||||
indices shift with the transient discovery controller.
|
||||
|
||||
**Touches:** [../scripts/extract-artifacts.sh](../scripts/extract-artifacts.sh)
|
||||
(emits `rootfs.hash`, two UUIDs, offset-free cmdline),
|
||||
[../scripts/build-oci.sh](../scripts/build-oci.sh) (artifact list + SHA256SUMS),
|
||||
[../serving/spdk-setup.sh](../serving/spdk-setup.sh) (2nd bdev/namespace), and
|
||||
CLAUDE.md invariant #9.
|
||||
|
||||
## 2. Implementation notes (not architectural)
|
||||
|
||||
These don't contradict the PDF (it doesn't specify them) but are worth recording.
|
||||
|
||||
@@ -10,7 +10,8 @@
|
||||
# verity.json -- a mismatched pair is an unbootable fleet
|
||||
# 4. builds a FROM-scratch OCI image containing ONLY:
|
||||
# /artifacts/fleet-<version>.efi (the signed UKI)
|
||||
# /artifacts/rootfs.erofs (erofs + appended verity tree)
|
||||
# /artifacts/rootfs.erofs (erofs DATA namespace)
|
||||
# /artifacts/rootfs.hash (dm-verity HASH namespace)
|
||||
# /artifacts/verity.json
|
||||
# /artifacts/SHA256SUMS
|
||||
# Deploy as an initContainer (or k8s image volume) feeding the
|
||||
@@ -79,7 +80,7 @@ echo "==> roothash coupling verified: $WANT"
|
||||
STAGE=$(mktemp -d)
|
||||
trap 'rm -rf "$STAGE"' EXIT
|
||||
mkdir "$STAGE/artifacts"
|
||||
cp "$UKI" "$ART/rootfs.erofs" "$ART/verity.json" "$STAGE/artifacts/"
|
||||
cp "$UKI" "$ART/rootfs.erofs" "$ART/rootfs.hash" "$ART/verity.json" "$STAGE/artifacts/"
|
||||
( cd "$STAGE/artifacts" && sha256sum -- * > SHA256SUMS )
|
||||
|
||||
REF="${IMAGE}:${VERSION}"
|
||||
|
||||
@@ -4,9 +4,9 @@
|
||||
# Pulls the diskless deliverables out of the Packer-built disk image:
|
||||
# vmlinuz -> HTTPS boot server
|
||||
# initramfs.img -> HTTPS boot server (the fleet-generic image)
|
||||
# rootfs.erofs -> nvmet target: shared READ-ONLY NVMe/TCP namespace
|
||||
# (erofs data + dm-verity hash tree appended in-file)
|
||||
# verity.json -> root hash + hash offset (feeds the kernel cmdline)
|
||||
# rootfs.erofs -> nvmet target: READ-ONLY NVMe/TCP DATA namespace (erofs)
|
||||
# rootfs.hash -> nvmet target: READ-ONLY NVMe/TCP dm-verity HASH namespace
|
||||
# verity.json -> root hash + data/hash namespace UUIDs (feed the cmdline)
|
||||
# cmdline.txt -> ready-made kernel cmdline fragment for this build
|
||||
#
|
||||
# Requires on the build host: libguestfs-tools, erofs-utils (mkfs.erofs),
|
||||
@@ -23,14 +23,19 @@ DISABLE_DOT1X=${3:-false}
|
||||
# environments. Defaults to the literal placeholder so build-oci.sh's warning
|
||||
# fires if it was never set.
|
||||
NVMET_TRADDR=${NVMET_TRADDR:-NVMET_TRADDR}
|
||||
# Stable root-namespace UUID. The root device is referenced by-id
|
||||
# (/dev/disk/by-id/nvme-uuid.$ROOT_NS_UUID) rather than /dev/nvmeNn1 because
|
||||
# NVMe controller indices are NOT stable: a transient discovery controller can
|
||||
# take nvme0, pushing the I/O controller to nvme1, so a hardcoded node is a
|
||||
# latent brick in an immutable signed cmdline. MUST match ROOT_NS_UUID in
|
||||
# serving/spdk-setup.sh (which assigns it on the target via add_ns -u).
|
||||
ROOT_NS_UUID=${ROOT_NS_UUID:-f1ee7000-0000-4000-8000-000000000001}
|
||||
ROOT_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_NS_UUID}"
|
||||
# Stable namespace UUIDs. Devices are referenced by-id
|
||||
# (/dev/disk/by-id/nvme-uuid.<uuid>) rather than /dev/nvmeNnM because NVMe
|
||||
# controller/namespace indices are NOT stable (a transient discovery controller
|
||||
# can shift them), so a hardcoded node is a latent brick in an immutable signed
|
||||
# cmdline. dm-verity needs TWO namespaces — erofs DATA and the verity HASH tree
|
||||
# — because this distro's systemd-veritysetup does not support the in-file
|
||||
# hash-offset layout (it ignores hash-offset= and fails to load the superblock).
|
||||
# MUST match ROOT_DATA_UUID / ROOT_HASH_UUID in serving/spdk-setup.sh (assigned
|
||||
# on the target via add_ns -u).
|
||||
ROOT_DATA_UUID=${ROOT_DATA_UUID:-f1ee7000-0000-4000-8000-000000000001}
|
||||
ROOT_HASH_UUID=${ROOT_HASH_UUID:-f1ee7000-0000-4000-8000-000000000002}
|
||||
ROOT_DATA_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_DATA_UUID}"
|
||||
ROOT_HASH_DEV="/dev/disk/by-id/nvme-uuid.${ROOT_HASH_UUID}"
|
||||
|
||||
for bin in virt-copy-out virt-tar-out virt-ls mkfs.erofs veritysetup jq; do
|
||||
command -v "$bin" >/dev/null || { echo "ERROR: $bin not found on build host" >&2; exit 1; }
|
||||
@@ -69,21 +74,24 @@ FIXED_UUID="6c10a7f0-1d2e-4f5a-9b3c-000000000001"
|
||||
mkfs.erofs -T 0 -U "$FIXED_UUID" --all-root \
|
||||
"$WORK/rootfs.erofs" "$WORK/root"
|
||||
|
||||
echo "==> Appending dm-verity hash tree (single-file / single-namespace layout)"
|
||||
DATA_BYTES=$(stat -c %s "$WORK/rootfs.erofs")
|
||||
# Align hash area to 4 KiB after the data
|
||||
HASH_OFFSET=$(( (DATA_BYTES + 4095) / 4096 * 4096 ))
|
||||
VERITY_OUT=$(veritysetup format "$WORK/rootfs.erofs" "$WORK/rootfs.erofs" \
|
||||
--hash-offset="$HASH_OFFSET" \
|
||||
echo "==> Generating dm-verity hash tree as a SEPARATE file (two-namespace layout)"
|
||||
# Hash tree in its OWN file (served as a 2nd namespace), NOT appended in-file:
|
||||
# this distro's systemd-veritysetup ignores hash-offset= ("unknown option") and
|
||||
# fails to load the verity superblock. With separate data/hash devices the
|
||||
# cmdline needs no offset. Deterministic for identical input trees (fixed erofs
|
||||
# UUID + -T0 => stable data => stable hash tree => stable root hash).
|
||||
VERITY_OUT=$(veritysetup format "$WORK/rootfs.erofs" "$WORK/rootfs.hash" \
|
||||
--data-block-size=4096 --hash-block-size=4096)
|
||||
ROOT_HASH=$(printf '%s\n' "$VERITY_OUT" | awk '/Root hash:/ {print $3}')
|
||||
[ -n "$ROOT_HASH" ] || { echo "ERROR: veritysetup produced no root hash" >&2; exit 1; }
|
||||
|
||||
mv "$WORK/rootfs.erofs" "$OUT/rootfs.erofs"
|
||||
mv "$WORK/rootfs.hash" "$OUT/rootfs.hash"
|
||||
|
||||
jq -n --arg hash "$ROOT_HASH" --argjson offset "$HASH_OFFSET" \
|
||||
--arg uuid "$FIXED_UUID" \
|
||||
'{root_hash: $hash, hash_offset: $offset, erofs_uuid: $uuid,
|
||||
jq -n --arg hash "$ROOT_HASH" --arg erofs_uuid "$FIXED_UUID" \
|
||||
--arg data_uuid "$ROOT_DATA_UUID" --arg hash_uuid "$ROOT_HASH_UUID" \
|
||||
'{root_hash: $hash, erofs_uuid: $erofs_uuid,
|
||||
data_ns_uuid: $data_uuid, hash_ns_uuid: $hash_uuid,
|
||||
data_block_size: 4096, hash_block_size: 4096}' \
|
||||
> "$OUT/verity.json"
|
||||
|
||||
@@ -110,21 +118,20 @@ printf '%s' \
|
||||
"rd.nvmf.discover=tcp,${NVMET_TRADDR},4420 " \
|
||||
"root=/dev/mapper/root rootfstype=erofs rootflags=ro " \
|
||||
"systemd.verity=yes roothash=${ROOT_HASH} " \
|
||||
"systemd.verity_root_data=${ROOT_DEV} " \
|
||||
"systemd.verity_root_hash=${ROOT_DEV} " \
|
||||
"systemd.verity_root_options=hash-offset=${HASH_OFFSET}" \
|
||||
"systemd.verity_root_data=${ROOT_DATA_DEV} " \
|
||||
"systemd.verity_root_hash=${ROOT_HASH_DEV}" \
|
||||
> "$OUT/cmdline.txt"
|
||||
|
||||
echo "==> Checksums"
|
||||
( cd "$OUT" && sha256sum vmlinuz initramfs.img rootfs.erofs | tee SHA256SUMS )
|
||||
( cd "$OUT" && sha256sum vmlinuz initramfs.img rootfs.erofs rootfs.hash | tee SHA256SUMS )
|
||||
|
||||
cat <<EOF
|
||||
|
||||
Artifacts in $OUT:
|
||||
vmlinuz, initramfs.img -> publish to the HTTPS boot server (versioned dir,
|
||||
flip a symlink for atomic rollout/rollback)
|
||||
rootfs.erofs -> expose as a READ-ONLY nvmet namespace; version
|
||||
rollout = publish new namespace, repoint hosts
|
||||
rootfs.erofs -> expose as a READ-ONLY nvmet DATA namespace
|
||||
rootfs.hash -> expose as a READ-ONLY nvmet HASH namespace (verity)
|
||||
cmdline.txt -> kernel cmdline for this build. The roothash binds
|
||||
the root fs to the boot artifacts: serve via a
|
||||
signed UKI or signed GRUB config so Secure Boot
|
||||
|
||||
+27
-21
@@ -1,29 +1,32 @@
|
||||
#!/usr/bin/env bash
|
||||
# Configure SPDK nvmf_tgt to serve the diskless root filesystem as a single
|
||||
# READ-ONLY NVMe/TCP namespace (design doc §5, §7).
|
||||
# Configure SPDK nvmf_tgt to serve the diskless root filesystem as TWO READ-ONLY
|
||||
# NVMe/TCP namespaces — erofs DATA and the dm-verity HASH tree (design doc §5, §7).
|
||||
#
|
||||
# rootfs.erofs is the erofs image with the dm-verity hash tree appended in-file;
|
||||
# we expose the WHOLE file as one namespace, and the host splits data/hash with
|
||||
# systemd.verity_root_options=hash-offset= from the kernel cmdline. Integrity is
|
||||
# enforced end-to-end by dm-verity on the host; the target serves it read-only.
|
||||
# Two namespaces (not one with an in-file hash-offset): the target Rocky 9
|
||||
# systemd-veritysetup does not support the hash-offset layout, so the host needs
|
||||
# the verity data and hash on separate block devices. Integrity is enforced
|
||||
# end-to-end by dm-verity on the host; the target serves both read-only.
|
||||
set -euo pipefail
|
||||
|
||||
ROOTFS="/artifacts/rootfs.erofs"
|
||||
ROOTHASH="/artifacts/rootfs.hash"
|
||||
SUBNQN="${SUBNQN:-nqn.2026-06.org.example.fleet:rootfs}"
|
||||
# Fixed, fleet-generic namespace UUID. The host references the root device as
|
||||
# /dev/disk/by-id/nvme-uuid.$ROOT_NS_UUID on the kernel cmdline, which is stable
|
||||
# regardless of NVMe controller enumeration order (the I/O controller can land
|
||||
# on nvme0/nvme1/... depending on whether a transient discovery controller took
|
||||
# a slot first — so a hardcoded /dev/nvme0n1 is unsafe). MUST match
|
||||
# ROOT_NS_UUID in scripts/extract-artifacts.sh, which bakes it into the UKI
|
||||
# cmdline. NOTE: for multi-version serving (TODO #6) make this version-specific
|
||||
# alongside the per-version NQN, so a host can't connect the wrong rootfs.
|
||||
ROOT_NS_UUID="${ROOT_NS_UUID:-f1ee7000-0000-4000-8000-000000000001}"
|
||||
# Fixed, fleet-generic namespace UUIDs. The host references each device as
|
||||
# /dev/disk/by-id/nvme-uuid.<uuid> on the kernel cmdline, stable regardless of
|
||||
# NVMe controller/namespace enumeration order (a transient discovery controller
|
||||
# can shift indices — so a hardcoded /dev/nvmeNnM is unsafe). MUST match
|
||||
# ROOT_DATA_UUID / ROOT_HASH_UUID in scripts/extract-artifacts.sh, which bake
|
||||
# them into the UKI cmdline. NOTE: for multi-version serving (TODO #6) make
|
||||
# these version-specific alongside the per-version NQN, so a host can't connect
|
||||
# the wrong rootfs.
|
||||
ROOT_DATA_UUID="${ROOT_DATA_UUID:-f1ee7000-0000-4000-8000-000000000001}"
|
||||
ROOT_HASH_UUID="${ROOT_HASH_UUID:-f1ee7000-0000-4000-8000-000000000002}"
|
||||
TADDR="${TARGET_ADDR:-0.0.0.0}"
|
||||
TPORT="${TARGET_PORT:-4420}"
|
||||
CPUMASK="${SPDK_CPUMASK:-0x1}"
|
||||
|
||||
[ -s "$ROOTFS" ] || { echo "ERROR: $ROOTFS not found/empty — is ARTIFACT_SRC populated?" >&2; exit 1; }
|
||||
[ -s "$ROOTHASH" ] || { echo "ERROR: $ROOTHASH not found/empty — rebuild artifacts with the two-namespace layout." >&2; exit 1; }
|
||||
|
||||
# Binary locations differ across SPDK images; fall back to the common path.
|
||||
TGT=$(command -v nvmf_tgt || echo /usr/libexec/spdk/spdk/build/bin/nvmf_tgt)
|
||||
@@ -59,21 +62,24 @@ done
|
||||
echo "==> creating TCP transport"
|
||||
"$RPC" nvmf_create_transport -t TCP -u 16384 -m 8 -c 8192
|
||||
|
||||
echo "==> exposing $ROOTFS as bdev 'rootfs' (4 KiB blocks)"
|
||||
echo "==> exposing $ROOTFS (data) + $ROOTHASH (verity hash) as bdevs (4 KiB blocks)"
|
||||
# -r opens the backing file read-only where supported; harmless if ignored.
|
||||
"$RPC" bdev_aio_create "$ROOTFS" rootfs 4096 -r \
|
||||
|| "$RPC" bdev_aio_create "$ROOTFS" rootfs 4096
|
||||
"$RPC" bdev_aio_create "$ROOTHASH" roothash 4096 -r \
|
||||
|| "$RPC" bdev_aio_create "$ROOTHASH" roothash 4096
|
||||
|
||||
echo "==> creating subsystem $SUBNQN (allow any host — lab/no-auth)"
|
||||
"$RPC" nvmf_create_subsystem "$SUBNQN" -a -s SPDKLAB0000000001 -d FleetRootfs
|
||||
|
||||
echo "==> adding namespace 1 (uuid $ROOT_NS_UUID)"
|
||||
# Read-only is enforced by the bdev_aio '-r' open above (this SPDK's add_ns has
|
||||
# no --read-only flag), plus the :ro image-volume mount and the client's erofs
|
||||
# ro + dm-verity.
|
||||
# -u pins the namespace UUID so the host can reference it as
|
||||
# /dev/disk/by-id/nvme-uuid.$ROOT_NS_UUID (stable across controller indices).
|
||||
"$RPC" nvmf_subsystem_add_ns "$SUBNQN" rootfs -u "$ROOT_NS_UUID"
|
||||
# ro + dm-verity. -u pins each namespace UUID so the host references them as
|
||||
# /dev/disk/by-id/nvme-uuid.<uuid> (stable across controller indices).
|
||||
echo "==> adding namespace 1: data (uuid $ROOT_DATA_UUID)"
|
||||
"$RPC" nvmf_subsystem_add_ns "$SUBNQN" rootfs -u "$ROOT_DATA_UUID"
|
||||
echo "==> adding namespace 2: verity hash (uuid $ROOT_HASH_UUID)"
|
||||
"$RPC" nvmf_subsystem_add_ns "$SUBNQN" roothash -u "$ROOT_HASH_UUID"
|
||||
|
||||
echo "==> adding TCP listener ${TADDR}:${TPORT} to $SUBNQN"
|
||||
"$RPC" nvmf_subsystem_add_listener "$SUBNQN" -t tcp -a "$TADDR" -s "$TPORT"
|
||||
|
||||
Reference in New Issue
Block a user