frank-primary: R9V radiance cutover wiring (launcher-driven serve + state gates)

Replace the vLLM 0.30 UVA-offload wiring (never deployed; llama.cpp b11030 endpoint replaced by Frank's order, downtime window open) with the R9V radiance-vLLM shape on the same root unit: - serve-frank-primary.sh: layer the on-host qualified setup state (104 tuned R9V_* keys, /home/dank/r9v-state/uncensored/setup.json) under repo production overrides, then exec the fork's own launcher through the docker->podman shim and block on docker wait so a crashed engine is a nonzero ExecStart (launch.sh --detach alone would fake green). Gates: pinned image by OCI manifest digest in RepoDigests, PLE at pinned byte count (stat -c %s), all GGUF shards + mmproj. - frank-primary.env: thin production overrides only (LAN bind 0.0.0.0, :7770, alias frank-primary, container frank-primary-r9v, GPU BDFs); tuned engine shape stays in the setup state by design. - frank-primary.service: same root-unit shape (User=dank, LimitMEMLOCK=infinity for the tiered-expert pin), plus ExecStop podman stop — docker wait cannot stop the container child. - deploy-remote.sh deploy_llm_primary: state-driven gates (state ready, image digest present in rootless storage, PLE bytes, shard files) replace the fp030 shard-index gate; false-green MainPID check kept. - tests/frank_primary_unit.bats: seam rewritten for the R9V shape.
2026-09-27 04:05:46 -07:00
1 parent baba84fc90
commit 095128b669
@@ -1,21 +1,29 @@
 #!/usr/bin/env bash
-# serve-frank-primary.sh — frank-primary endpoint (vLLM 0.30,
-# Qwen3.8-Flash-Next-FP8 on the 2x R9700 pair, dankserver :7770).
+# serve-frank-primary.sh — frank-primary endpoint (R9V radiance-vLLM fork,
+# Qwen3.8-Flash-Next-Uncensored IQ4_XS + MTP4 drafter, 2x R9700, dankserver
+# :7770).
 #
-# Validates the image, the model payload and the env tunables, then execs the
-# container. The unit (scripts/frank-primary.service, root systemd, User=dank,
-# LimitMEMLOCK=infinity) is what makes the UVA pin budget legal — a plain
-# shell here carries the 8 MiB session limit and the pinned offload fails.
+# The engine is the upstream R9V launcher (scripts/launch.sh in the fork
+# checkout), NOT a hand-frozen podman argv: launch.sh owns the resolved
+# docker/podman args (binds, overlays, device passthrough, publish lines)
+# and the fork's qualification was measured through it. This script's job:
 #
-# OFFLOAD SHAPE (the only legal fit for a ~125B-param FP8 model on 2x 32 GiB
-# cards): --offload-backend uva pins the PLE n-gram table (51.2B params) and
-# the routed experts in host RAM and reads them in place over PCIe. The
-# first start MUST be watched for the UVA engagement log (not just /health):
-# a silent fallback to the copy path turns the endpoint into a slideshow.
+#   1. layer the on-host R9V setup state (the 104-key tuned shape the
+#      profile setup qualified) under this repo's frank-primary.env
+#      overrides (bind/port/alias/container);
+#   2. gate on the pinned runtime image (by its OCI manifest digest, the
+#      podman RepoDigest), the exact-size PLE payload, and the GGUF shards;
+#   3. exec the launcher, then wait on the container: launch.sh --detach
+#      exits as soon as the container is created, so without the wait the
+#      unit would be "green" while the engine crashes — the wait is what
+#      keeps systemd's Restart=on-failure self-healing real.
 #
-# NO --model-draft, no draft GGUF: vLLM's MTP drafting comes from the model's
-# own mtp.* weights via --speculative-config, and only when SPEC_CONFIG is
-# non-empty. Bring-up runs with it empty on purpose.
+# The root unit (scripts/frank-primary.service, User=dank,
+# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert UVA
+# host copy (~43 GiB pinned) and the 16 GiB PLE pinned reserve. A plain
+# shell session also works (proven 2026-09-27: podman rootless inherits
+# usable limits here), but the unit is the fleet-parity boot-persistent
+# shape.
 set -euo pipefail

 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -30,100 +38,101 @@ set -a
 source "$ENV_FILE"
 set +a

-: "${SERVED_NAME:?SERVED_NAME must be set}"
-: "${BIND_HOST:?BIND_HOST must be set}"
-: "${PORT:?PORT must be set}"
-: "${MODEL_DIR:?MODEL_DIR must be set}"
-: "${TP:?TP must be set}"
-: "${GPU_UTIL:?GPU_UTIL must be set}"
-: "${CTX_SIZE:?CTX_SIZE must be set}"
-: "${MAX_SEQS:?MAX_SEQS must be set}"
-: "${OFFLOAD_BACKEND:?OFFLOAD_BACKEND must be set}"
-: "${OFFLOAD_GB:?OFFLOAD_GB must be set}"
-: "${OFFLOAD_PARAMS:?OFFLOAD_PARAMS must be set}"
-: "${VLLM_ROCM_USE_AITER:?VLLM_ROCM_USE_AITER must be set}"
-: "${IMAGE:?IMAGE must be set}"
-: "${CONTAINER_NAME:?CONTAINER_NAME must be set}"
-: "${ROCR_VISIBLE_DEVICES:?ROCR_VISIBLE_DEVICES must be set}"
-: "${TOOL_PARSER:?TOOL_PARSER must be set}"
-: "${REASONING_PARSER:?REASONING_PARSER must be set}"
-: "${CHAT_TEMPLATE_KWARGS:?CHAT_TEMPLATE_KWARGS must be set}"
-: "${CACHE_ROOT:?CACHE_ROOT must be set}"
+: "${R9V_HOME:?R9V_HOME must name the fork checkout}"
+: "${R9V_STATE_FILE:?R9V_STATE_FILE must name the setup state}"
+: "${R9V_HOST_BIND:?R9V_HOST_BIND must be set}"
+: "${R9V_HOST_PORT:?R9V_HOST_PORT must be set}"
+: "${R9V_SERVED_MODEL_NAME:?R9V_SERVED_MODEL_NAME must be set}"
+: "${R9V_CONTAINER_NAME:?R9V_CONTAINER_NAME must be set}"

-if [ "$OFFLOAD_BACKEND" != "uva" ]; then
-    echo "serve-frank-primary: OFFLOAD_BACKEND=$OFFLOAD_BACKEND — this deployment is only qualified for uva" >&2
+LAUNCHER="$R9V_HOME/scripts/launch.sh"
+if [ ! -x "$LAUNCHER" ]; then
+    echo "serve-frank-primary: R9V launcher missing: $LAUNCHER" >&2
    exit 1
 fi

-if ! podman image inspect "$IMAGE" > /dev/null 2>&1; then
-    echo "serve-frank-primary: image missing: $IMAGE — build it before serving" >&2
+# The docker->podman shim is REQUIRED: the launcher and its doctor speak
+# docker (OCI-manifest-digest image IDs, LogConfig fields) that podman's
+# native CLI answers differently. The shim carries those laws.
+SHIM_DIR="$R9V_HOME/.podman-shim"
+if [ ! -x "$SHIM_DIR/docker" ]; then
+    echo "serve-frank-primary: docker->podman shim missing: $SHIM_DIR/docker" >&2
+    exit 1
+fi
+export PATH="$SHIM_DIR:$PATH"
+
+# --- layer 1: the qualified setup state (production overrides never come
+# from here; keys already defined in the env file are skipped) -------------
+STATE_FILE="$R9V_STATE_FILE"
+if [ ! -f "$STATE_FILE" ]; then
+    echo "serve-frank-primary: R9V setup state missing: $STATE_FILE" >&2
+    echo "  run the profile setup first (r9v setup qwen38-mtp4-uncensored ...)" >&2
+    exit 1
+fi
+STATE_EXPORTS="$(python3 - "$STATE_FILE" <<'PY'
+import json, shlex, sys, os
+state = json.load(open(sys.argv[1]))
+config = state.get("config") or {}
+if state.get("ready") is not True:
+    sys.exit("R9V setup state is not ready; rerun setup before serving")
+env_keys = {k for k in os.environ if k.startswith("R9V_")}
+for key in sorted(config):
+    if key.startswith("R9V_") and key not in env_keys:
+        print(f"export {key}={shlex.quote(config[key])}")
+PY
+)"
+eval "$STATE_EXPORTS"
+
+: "${R9V_IMAGE:?state must carry R9V_IMAGE}"
+: "${R9V_MODEL_DIR:?state must carry R9V_MODEL_DIR}"
+: "${R9V_PLE_PATH:?state must carry R9V_PLE_PATH}"
+
+# --- gates: never serve a half-finished payload or a missing image --------
+# Runtime image by pinned OCI manifest digest (podman keys images by config
+# digest but preserves the manifest digest in RepoDigests — the shim's ID law).
+if ! podman images --format '{{.RepoDigests}}' | grep -qF "${R9V_IMAGE#sha256:}"; then
+    echo "serve-frank-primary: pinned R9V image absent: $R9V_IMAGE" >&2
+    echo "  load the bundle: ./r9v setup qwen38-mtp4-uncensored --state-dir ..." >&2
    exit 1
 fi

-# The model payload is verified by the fetch/verify step (131 shards, index
-# cross-checked). The serve-time check is the load-bearing minimum: the
-# wrapper config and the shard index must exist, or the container start is a
-# guaranteed 20-minute timeout instead of a fast failure.
-HOST_MODEL_DIR="/home/dank/models/${MODEL_DIR#/models/}"
-if [ ! -f "$HOST_MODEL_DIR/config.json" ] || [ ! -f "$HOST_MODEL_DIR/model.safetensors.index.json" ]; then
-    echo "serve-frank-primary: model payload incomplete at $HOST_MODEL_DIR (config.json or index missing)" >&2
-    echo "serve-frank-primary: run scripts/download-frank-primary-fp8.sh on this host" >&2
-    exit 1
+# PLE payload at its pinned byte count (stat -c %s, never wc -c), plus the
+# three target GGUF shards and the mmproj the launcher mounts.
+PLE_BYTES="${R9V_PLE_EXPECTED_BYTES:-}"
+if [ -n "$PLE_BYTES" ]; then
+    ACTUAL="$(stat -c %s "$R9V_PLE_PATH" 2>/dev/null || echo 0)"
+    if [ "$ACTUAL" != "$PLE_BYTES" ]; then
+        echo "serve-frank-primary: PLE payload wrong size: $R9V_PLE_PATH" >&2
+        echo "  expected $PLE_BYTES, found $ACTUAL" >&2
+        exit 1
+    fi
+else
+    if [ ! -f "$R9V_PLE_PATH" ]; then
+        echo "serve-frank-primary: PLE payload missing: $R9V_PLE_PATH" >&2
+        exit 1
+    fi
 fi
+for rel in "$R9V_TARGET_REL" "$R9V_TARGET_SHARD2_REL" "$R9V_TARGET_SHARD3_REL" "$R9V_MMPROJ_REL"; do
+    if [ -z "$rel" ]; then continue; fi
+    if [ ! -f "$R9V_MODEL_DIR/$rel" ]; then
+        echo "serve-frank-primary: payload file missing: $R9V_MODEL_DIR/$rel" >&2
+        exit 1
+    fi
+done

 if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
-    echo "serve-frank-primary: /dev/kfd or /dev/dri missing — GPU stack not up" >&2
+    echo "serve-frank-primary: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
    exit 1
 fi

-mkdir -p /home/dank/.cache/frank-primary-vllm
-
-SERVE_ARGS=("$MODEL_DIR"
-    --served-model-name "$SERVED_NAME"
-    --host "$BIND_HOST"
-    --port "$PORT"
-    --tensor-parallel-size "$TP"
-    --gpu-memory-utilization "$GPU_UTIL"
-    --max-model-len "$CTX_SIZE"
-    --max-num-seqs "$MAX_SEQS"
-    --offload-backend "$OFFLOAD_BACKEND"
-    --cpu-offload-gb "$OFFLOAD_GB"
-    --cpu-offload-params "$OFFLOAD_PARAMS"
-    --enable-auto-tool-choice --tool-call-parser "$TOOL_PARSER"
-    --reasoning-parser "$REASONING_PARSER"
-    --default-chat-template-kwargs "$CHAT_TEMPLATE_KWARGS"
-)
-
-# SPEC_CONFIG (optional, JSON) is how MTP drafting is enabled after the
-# bring-up measurement: --speculative-config '{"method":"mtp",...}'.
-if [ -n "${SPEC_CONFIG:-}" ]; then
-    SERVE_ARGS+=(--speculative-config "$SPEC_CONFIG")
-fi
-# A crashed/killed unit leaves the container name registered (systemd can
-# SIGKILL conmon before the --rm cleanup runs); podman run would then die
-# on name-in-use forever. Clear any orphan of OUR name first — idempotent.
-podman rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true
-
-exec podman run --rm \
-    --name "$CONTAINER_NAME" \
-    --userns=keep-id \
-    --network=host \
-    --ipc=host \
-    --security-opt seccomp=unconfined \
-    --cap-add SYS_PTRACE \
-    --device /dev/kfd \
-    --device /dev/dri \
-    --group-add keep-groups \
-    -e ROCR_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES" \
-    -e HIP_VISIBLE_DEVICES="$HIP_VISIBLE_DEVICES" \
-    -e VLLM_ROCM_USE_AITER="$VLLM_ROCM_USE_AITER" \
-    -e NCCL_P2P_DISABLE="$NCCL_P2P_DISABLE" \
-    -e TRITON_CACHE_DIR="$CACHE_ROOT/triton" \
-    -e TORCHINDUCTOR_CACHE_DIR="$CACHE_ROOT/inductor" \
-    -e HOME="$CACHE_ROOT" \
-    -e TRITON_ALLOW_NON_CONST_GLOBALS=1 \
-    --volume /home/dank/models:/models \
-    --volume /home/dank/.cache/frank-primary-vllm:"$CACHE_ROOT" \
-    "$IMAGE" \
-    "${SERVE_ARGS[@]}"
+# A crashed/killed unit can leave the container name registered (systemd
+# SIGKILLs conmon before cleanup runs); the launcher refuses an existing
+# name forever after. Clear any orphan of OUR name first — idempotent.
+docker rm -f "$R9V_CONTAINER_NAME" >/dev/null 2>&1 || true

+# launch.sh detaches after preflight; from here on, this script's lifetime
+# IS the container's: wait returns the container's exit code, so a crash is
+# a nonzero ExecStart and Restart=on-failure fires. (systemd stop goes
+# through ExecStop in the unit — docker wait alone cannot stop the child.)
+"$LAUNCHER"
+exec docker wait "$R9V_CONTAINER_NAME"