serving: make the SPDK nvmeof target version-agnostic (stop recreating it on IMAGE_VERSION bump)
The nvmeof (spdk-target) service carried a versioned bootstrap mount
(type: image, source ${OCI_IMAGE}:${IMAGE_VERSION} -> /artifacts) plus an
IMAGE_VERSION env. Bumping IMAGE_VERSION therefore changed the service config,
so `docker compose up -d` recreated the container -> nvmf_tgt restarted ->
every connected host's NVMe/TCP root dropped at once. That is precisely the
fleet-wide outage the long-lived-target design exists to prevent.
Remove the /artifacts image-volume and IMAGE_VERSION env from nvmeof so its
config no longer depends on the version — compose leaves it running across a
bump. Bootstrap and rolling updates are already fully covered by spdk-loader
(stages the current release into ./spdk/releases and adds it online via
JSON-RPC); spdk-target.sh's seed block is guarded, so it just no-ops when
/artifacts isn't mounted. Verified nvmeof no longer references IMAGE_VERSION
(only spdk-loader does); compose YAML validates.
BENCH-VERIFY: bump IMAGE_VERSION + up -d with a host connected -> nvmeof
container not recreated, host root survives, loader adds the new version.
This commit is contained in:
@@ -675,6 +675,23 @@ must be built around **never restarting the process that holds live connections.
|
||||
the current UKI), so auto-reap is safe; removal is still drain-gated +
|
||||
grace-debounced. The loader + target share the RPC socket on a named volume
|
||||
(`nvmf_tgt -r`); the loader reuses the SPDK image (has python3 + rpc.py).
|
||||
- **BENCH FINDING 2026-07-03 (FIXED) — the target WAS still recreated on an
|
||||
`IMAGE_VERSION` bump.** Despite the above, the `nvmeof` (spdk-target) service
|
||||
still carried a versioned `type: image` bootstrap mount
|
||||
(`source: ${OCI_IMAGE}:${IMAGE_VERSION}` → `/artifacts`) **and** an
|
||||
`IMAGE_VERSION` env, so bumping the version changed THIS service's config and
|
||||
`compose up -d` recreated it → `nvmf_tgt` restarted → **every connected host's
|
||||
NVMe/TCP root dropped** (the exact fleet-wide outage the long-lived-target
|
||||
design exists to prevent). Fix: **made the target version-agnostic** — removed
|
||||
the `/artifacts` image-volume + `IMAGE_VERSION` env from `nvmeof` in
|
||||
`serving/docker-compose.yml`. Bootstrap AND rolling updates are now solely the
|
||||
`spdk-loader`'s job (stages the current release into `./spdk/releases`, adds it
|
||||
online); `spdk-target.sh`'s seed block is guarded
|
||||
(`[ -s /artifacts/verity.json ] && [ -n IMAGE_VERSION ]`) so it no-ops when
|
||||
unmounted. Verified `nvmeof` no longer references `IMAGE_VERSION` (only
|
||||
`spdk-loader` does). **BENCH-VERIFY:** bump `IMAGE_VERSION` + `up -d` with a
|
||||
host connected → `nvmeof` container NOT recreated (`docker ps` ID/uptime
|
||||
unchanged), host root survives, loader adds the new version online.
|
||||
- **No jq dependency:** `spdk-release.sh` parses RPC JSON with **python3** (the
|
||||
SPDK image ships no jq); legacy artifacts (no per-version UUIDs in
|
||||
verity.json) fall back to the fixed UUID pair with a warning.
|
||||
|
||||
+11
-13
@@ -306,23 +306,21 @@ services:
|
||||
# Multi-version release store: each ./spdk/releases/<ver>/ holds that
|
||||
# version's rootfs.erofs + rootfs.hash + verity.json. Staged by the
|
||||
# spdk-loader (or spdk-ctl.sh on the host); reconciled into the target at
|
||||
# (re)start and added/removed live thereafter. RW so the bootstrap seed +
|
||||
# loader can write here. Adding a version does NOT recreate this service.
|
||||
# (re)start and added/removed live thereafter. RW so the spdk-loader can
|
||||
# stage here. Adding a version does NOT recreate this service.
|
||||
- ./spdk/releases:/releases
|
||||
# Bootstrap convenience (optional): a single OCI release mounted as an
|
||||
# image volume. spdk-target seeds it into /releases as ${IMAGE_VERSION} if
|
||||
# not already staged — preserves the simple one-version `compose up` flow.
|
||||
# Remove this block for a pure multi-version (spdk-ctl-managed) deployment.
|
||||
- type: image
|
||||
source: ${OCI_IMAGE}:${IMAGE_VERSION}
|
||||
target: /artifacts
|
||||
image:
|
||||
subpath: artifacts
|
||||
# Fallback (Docker < 28 / offline): - ${ARTIFACT_SRC}:/artifacts:ro
|
||||
# NOTE: the target is deliberately VERSION-AGNOSTIC. It mounts NO
|
||||
# ${IMAGE_VERSION} artifact image and carries no IMAGE_VERSION env, so a
|
||||
# version bump never changes THIS service's config — `docker compose up -d`
|
||||
# after an IMAGE_VERSION bump recreates the loader/http/dnsmasq but leaves
|
||||
# nvmf_tgt running (recreating it would tear down EVERY host's NVMe/TCP root
|
||||
# at once). Bootstrap AND rolling updates are the spdk-loader's job: it
|
||||
# stages the current release into ./spdk/releases and adds it online via
|
||||
# JSON-RPC. (spdk-target still seeds from /artifacts IF one is manually
|
||||
# mounted + IMAGE_VERSION set — a legacy/standalone convenience, off here.)
|
||||
# Legacy single-version entrypoint (configure-at-start, restart-to-change):
|
||||
# - ./spdk-setup.sh:/spdk-setup.sh:ro
|
||||
environment:
|
||||
IMAGE_VERSION: ${IMAGE_VERSION} # bootstrap-seed label for /artifacts
|
||||
RELEASES_DIR: /releases
|
||||
RPC_SOCK: /var/run/spdk/spdk.sock # shared with spdk-loader (-r / -s)
|
||||
SUBNQN_BASE: ${SUBNQN_BASE} # per-version NQN base (falls back per release to verity.json .subnqn)
|
||||
|
||||
Reference in New Issue
Block a user