From 0e139f5346dced3ebff7663a08eb9a28f27cad2a Mon Sep 17 00:00:00 2001 From: Audiobuzz Date: Fri, 3 Jul 2026 01:07:16 +0000 Subject: [PATCH] serving: make the SPDK nvmeof target version-agnostic (stop recreating it on IMAGE_VERSION bump) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The nvmeof (spdk-target) service carried a versioned bootstrap mount (type: image, source ${OCI_IMAGE}:${IMAGE_VERSION} -> /artifacts) plus an IMAGE_VERSION env. Bumping IMAGE_VERSION therefore changed the service config, so `docker compose up -d` recreated the container -> nvmf_tgt restarted -> every connected host's NVMe/TCP root dropped at once. That is precisely the fleet-wide outage the long-lived-target design exists to prevent. Remove the /artifacts image-volume and IMAGE_VERSION env from nvmeof so its config no longer depends on the version — compose leaves it running across a bump. Bootstrap and rolling updates are already fully covered by spdk-loader (stages the current release into ./spdk/releases and adds it online via JSON-RPC); spdk-target.sh's seed block is guarded, so it just no-ops when /artifacts isn't mounted. Verified nvmeof no longer references IMAGE_VERSION (only spdk-loader does); compose YAML validates. BENCH-VERIFY: bump IMAGE_VERSION + up -d with a host connected -> nvmeof container not recreated, host root survives, loader adds the new version. --- docs/TODO.md | 17 +++++++++++++++++ serving/docker-compose.yml | 24 +++++++++++------------- 2 files changed, 28 insertions(+), 13 deletions(-) diff --git a/docs/TODO.md b/docs/TODO.md index 51ce208..7a21b37 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -675,6 +675,23 @@ must be built around **never restarting the process that holds live connections. the current UKI), so auto-reap is safe; removal is still drain-gated + grace-debounced. The loader + target share the RPC socket on a named volume (`nvmf_tgt -r`); the loader reuses the SPDK image (has python3 + rpc.py). + - **BENCH FINDING 2026-07-03 (FIXED) — the target WAS still recreated on an + `IMAGE_VERSION` bump.** Despite the above, the `nvmeof` (spdk-target) service + still carried a versioned `type: image` bootstrap mount + (`source: ${OCI_IMAGE}:${IMAGE_VERSION}` → `/artifacts`) **and** an + `IMAGE_VERSION` env, so bumping the version changed THIS service's config and + `compose up -d` recreated it → `nvmf_tgt` restarted → **every connected host's + NVMe/TCP root dropped** (the exact fleet-wide outage the long-lived-target + design exists to prevent). Fix: **made the target version-agnostic** — removed + the `/artifacts` image-volume + `IMAGE_VERSION` env from `nvmeof` in + `serving/docker-compose.yml`. Bootstrap AND rolling updates are now solely the + `spdk-loader`'s job (stages the current release into `./spdk/releases`, adds it + online); `spdk-target.sh`'s seed block is guarded + (`[ -s /artifacts/verity.json ] && [ -n IMAGE_VERSION ]`) so it no-ops when + unmounted. Verified `nvmeof` no longer references `IMAGE_VERSION` (only + `spdk-loader` does). **BENCH-VERIFY:** bump `IMAGE_VERSION` + `up -d` with a + host connected → `nvmeof` container NOT recreated (`docker ps` ID/uptime + unchanged), host root survives, loader adds the new version online. - **No jq dependency:** `spdk-release.sh` parses RPC JSON with **python3** (the SPDK image ships no jq); legacy artifacts (no per-version UUIDs in verity.json) fall back to the fixed UUID pair with a warning. diff --git a/serving/docker-compose.yml b/serving/docker-compose.yml index f073952..84c9ff8 100644 --- a/serving/docker-compose.yml +++ b/serving/docker-compose.yml @@ -306,23 +306,21 @@ services: # Multi-version release store: each ./spdk/releases// holds that # version's rootfs.erofs + rootfs.hash + verity.json. Staged by the # spdk-loader (or spdk-ctl.sh on the host); reconciled into the target at - # (re)start and added/removed live thereafter. RW so the bootstrap seed + - # loader can write here. Adding a version does NOT recreate this service. + # (re)start and added/removed live thereafter. RW so the spdk-loader can + # stage here. Adding a version does NOT recreate this service. - ./spdk/releases:/releases - # Bootstrap convenience (optional): a single OCI release mounted as an - # image volume. spdk-target seeds it into /releases as ${IMAGE_VERSION} if - # not already staged — preserves the simple one-version `compose up` flow. - # Remove this block for a pure multi-version (spdk-ctl-managed) deployment. - - type: image - source: ${OCI_IMAGE}:${IMAGE_VERSION} - target: /artifacts - image: - subpath: artifacts - # Fallback (Docker < 28 / offline): - ${ARTIFACT_SRC}:/artifacts:ro + # NOTE: the target is deliberately VERSION-AGNOSTIC. It mounts NO + # ${IMAGE_VERSION} artifact image and carries no IMAGE_VERSION env, so a + # version bump never changes THIS service's config — `docker compose up -d` + # after an IMAGE_VERSION bump recreates the loader/http/dnsmasq but leaves + # nvmf_tgt running (recreating it would tear down EVERY host's NVMe/TCP root + # at once). Bootstrap AND rolling updates are the spdk-loader's job: it + # stages the current release into ./spdk/releases and adds it online via + # JSON-RPC. (spdk-target still seeds from /artifacts IF one is manually + # mounted + IMAGE_VERSION set — a legacy/standalone convenience, off here.) # Legacy single-version entrypoint (configure-at-start, restart-to-change): # - ./spdk-setup.sh:/spdk-setup.sh:ro environment: - IMAGE_VERSION: ${IMAGE_VERSION} # bootstrap-seed label for /artifacts RELEASES_DIR: /releases RPC_SOCK: /var/run/spdk/spdk.sock # shared with spdk-loader (-r / -s) SUBNQN_BASE: ${SUBNQN_BASE} # per-version NQN base (falls back per release to verity.json .subnqn)