frank-primary: vLLM 0.30 UVA-offload shape (pre-deploy commit)

The llama.cpp b11030 trio becomes the vLLM 0.30 trio ahead of the FP8 cutover on :7770 — same port, same alias, no client moves. Serve shape: locally built ROCm gfx1201 image (localhost/frank-primary:vllm030) execs via podman as dank from the root unit (LimitMEMLOCK=infinity legalizes the pin). UVA offload: --offload-backend uva pins the PLE family (47.75 GiB, 138 tensors: 131 n-gram table shards + 7 per-layer conv/proj) + routed experts (114.86 GiB, 150,528 tensors) in host RAM — 162.60 GiB combined, ~81.3 GiB per rank under TP=2, hence --cpu-offload-gb 82 (80/card strands 2.6 GiB of experts on the cards). Parsers per the host's own Ornith-era precedent: qwen3_xml tools + qwen3 reasoning (vLLM 0.30 emits the channel as `reasoning` — reasoning_content is renamed server-side) + enable_thinking/preserve_thinking template kwargs. Spec decode stays env-wired and OFF at bring-up. Deployer deploy_llm_primary rewritten: image gate, index-driven shard-count gate (131), stale-container cleanup, port wait, and the false-green hardening — health is only green when the unit's own MainPID is live behind the answer (2026-09-26 lesson). Seam tests rewritten to the vLLM contract (15 gates): offload arithmetic, parser wiring, no-llama.cpp-ism, deployer gates. Suite 349/349 green.
2026-09-27 00:16:42 -07:00
1 parent 82960c77db
commit ecaad36654
@@ -1,140 +1,123 @@
 #!/usr/bin/env bash
-# Serve unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 as frank-primary on
-# Dankserver's two Radeon AI PRO R9700s (gfx1201), using the unsloth
-# b11030-mix prebuilt (ROCm gfx120X, bundled runtime).
+# serve-frank-primary.sh — frank-primary endpoint (vLLM 0.30,
+# Qwen3.8-Flash-Next-FP8 on the 2x R9700 pair, dankserver :7770).
 #
-# Shape: one stream, the model's full 262144 context, all non-overridden
-# tensors on the GPUs (--n-gpu-layers 999 across both cards), with the
-# routed experts and the 51B-param n-gram embedding table computed in host
-# RAM via --override-tensor (the x4-pair law: streaming over PCIe loses,
-# computing in RAM does not care). mmap stays ON (the PLE table must stay
-# page-cache-backed; --no-mmap forces it into anonymous memory and the
-# cgroup kill is silent).
+# Validates the image, the model payload and the env tunables, then execs the
+# container. The unit (scripts/frank-primary.service, root systemd, User=dank,
+# LimitMEMLOCK=infinity) is what makes the UVA pin budget legal — a plain
+# shell here carries the 8 MiB session limit and the pinned offload fails.
 #
-# Spec chain: draft-mtp (the shared Q8_0 head, loaded via -md) chained with
-# the draftless ngram-mod matcher. The shared head borrows the target's
-# embedding tensors, which is why per_layer_token_embd.weight must be
-# CPU-resident (see frank-primary.env). depth 2.
+# OFFLOAD SHAPE (the only legal fit for a ~125B-param FP8 model on 2x 32 GiB
+# cards): --offload-backend uva pins the PLE n-gram table (51.2B params) and
+# the routed experts in host RAM and reads them in place over PCIe. The
+# first start MUST be watched for the UVA engagement log (not just /health):
+# a silent fallback to the copy path turns the endpoint into a slideshow.
 #
-# Vision: the BF16 projector, loaded explicitly via --mmproj (this build
-# never autoloads projectors — the frank-medium 2026-09-17 trap: a server
-# started without it comes up healthy and text-only, and answers image
-# requests with 500).
-#
-# Template: the GGUF's embedded chat template via --jinja. The Qwen3.8
-# family template gates reasoning_effort to {xhigh, medium, low} — the
-# harness folds its levels onto that set in models.yml
-# (reasoningEffortMap). If the embedded template ever falls back to
-# peg-native and bloats prompts (~73,600 tokens for a ~250-token request),
-# switch to --chat-template-file scripts/qwen-sharp-v22.4.0.jinja the way
-# frank-large does.
-#
-# Tunables come from frank-primary.env unless FRANK_PRIMARY_ENV_FILE
-# overrides it; anything not set there falls through to the defaults below.
+# NO --model-draft, no draft GGUF: vLLM's MTP drafting comes from the model's
+# own mtp.* weights via --speculative-config, and only when SPEC_CONFIG is
+# non-empty. Bring-up runs with it empty on purpose.
 set -euo pipefail

-SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}"
-if [[ -f "$ENV_FILE" ]]; then
-    set -a
-    source "$ENV_FILE"
-    set +a
-fi

-# --- defaults (used when the env file does not set them) ------------------
-: "${RUNTIME_DIR:=/home/dank/GitPackages/unsloth-llama-b11030}"
-: "${MODEL_ROOT:=/home/dank/models/Qwen3.8-Flash-Next-GGUF}"
-: "${BIND_HOST:=0.0.0.0}"
-: "${PORT:=7770}"
-: "${CONTEXT_SIZE:=262144}"
-: "${PARALLEL_SLOTS:=1}"
-: "${SPEC_DRAFT_N_MAX:=2}"
-: "${NGL:=999}"
-: "${OT_OVERRIDES:=exps=CPU,per_layer_token_embd.weight=CPU}"
-: "${TENSOR_SPLIT:=1,1}"
-: "${SPEC_DRAFT_DEVICE:=ROCm0}"
-: "${BATCH_SIZE:=8192}"
-: "${UBATCH_SIZE:=512}"
-: "${CACHE_TYPE_V:=f16}"
-: "${CACHE_TYPE_K:=f16}"
-: "${VERBOSITY:=4}"
-
-SERVER_BIN="$RUNTIME_DIR/llama-server"
-MODEL_PARTS=(
-    "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00001-of-00006.gguf"
-    "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00002-of-00006.gguf"
-    "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00003-of-00006.gguf"
-    "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00004-of-00006.gguf"
-    "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00005-of-00006.gguf"
-    "$MODEL_ROOT/Q8_0/Qwen3.8-Flash-Next-Q8_0-00006-of-00006.gguf"
-)
-DRAFTER_FILE="$MODEL_ROOT/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf"
-MMPROJ_FILE="$MODEL_ROOT/mmproj-BF16.gguf"
-
-if [[ ! -x "$SERVER_BIN" ]]; then
-    echo "runtime not found at $SERVER_BIN" >&2
-    echo "  fetch the unsloth b11030-mix gfx120X prebuilt (see the runbook)" >&2
+if [ ! -f "$ENV_FILE" ]; then
+    echo "serve-frank-primary: env file missing: $ENV_FILE" >&2
    exit 1
 fi
-# Every part is checked: part 1 is the -m target and parts 2-6 are followed
-# by the split-file loader, so a missing tail part fails at LOAD time with
-# a confusing partial-mmap error unless it is caught here.
-for f in "${MODEL_PARTS[@]}" "$DRAFTER_FILE" "$MMPROJ_FILE"; do
-    if [[ ! -f "$f" ]]; then
-        echo "missing asset: $f" >&2
-        echo "  run scripts/download-frank-primary.sh on this host" >&2
-        exit 1
-    fi
-done
+set -a
+# shellcheck disable=SC1090
+source "$ENV_FILE"
+set +a

-# --- env (the unsloth build's bundled ROCm runtime) -----------------------
-# Both R9700s (gfx1201, HIP devices 0 and 1); the host's Raphael iGPU is
-# device 2 and must stay hidden from the enumeration.
-export HIP_VISIBLE_DEVICES=0,1
-export LD_LIBRARY_PATH="$RUNTIME_DIR:${LD_LIBRARY_PATH:-}"
+: "${SERVED_NAME:?SERVED_NAME must be set}"
+: "${BIND_HOST:?BIND_HOST must be set}"
+: "${PORT:?PORT must be set}"
+: "${MODEL_DIR:?MODEL_DIR must be set}"
+: "${TP:?TP must be set}"
+: "${GPU_UTIL:?GPU_UTIL must be set}"
+: "${CTX_SIZE:?CTX_SIZE must be set}"
+: "${MAX_SEQS:?MAX_SEQS must be set}"
+: "${OFFLOAD_BACKEND:?OFFLOAD_BACKEND must be set}"
+: "${OFFLOAD_GB:?OFFLOAD_GB must be set}"
+: "${OFFLOAD_PARAMS:?OFFLOAD_PARAMS must be set}"
+: "${VLLM_ROCM_USE_AITER:?VLLM_ROCM_USE_AITER must be set}"
+: "${IMAGE:?IMAGE must be set}"
+: "${CONTAINER_NAME:?CONTAINER_NAME must be set}"
+: "${ROCR_VISIBLE_DEVICES:?ROCR_VISIBLE_DEVICES must be set}"
+: "${TOOL_PARSER:?TOOL_PARSER must be set}"
+: "${REASONING_PARSER:?REASONING_PARSER must be set}"
+: "${CHAT_TEMPLATE_KWARGS:?CHAT_TEMPLATE_KWARGS must be set}"
+: "${CACHE_ROOT:?CACHE_ROOT must be set}"

-# --- the serve command ---------------------------------------------------
-ARGS=(
+if [ "$OFFLOAD_BACKEND" != "uva" ]; then
+    echo "serve-frank-primary: OFFLOAD_BACKEND=$OFFLOAD_BACKEND — this deployment is only qualified for uva" >&2
+    exit 1
+fi
+
+if ! podman image inspect "$IMAGE" > /dev/null 2>&1; then
+    echo "serve-frank-primary: image missing: $IMAGE — build it before serving" >&2
+    exit 1
+fi
+
+# The model payload is verified by the fetch/verify step (131 shards, index
+# cross-checked). The serve-time check is the load-bearing minimum: the
+# wrapper config and the shard index must exist, or the container start is a
+# guaranteed 20-minute timeout instead of a fast failure.
+HOST_MODEL_DIR="/home/dank/models/${MODEL_DIR#/models/}"
+if [ ! -f "$HOST_MODEL_DIR/config.json" ] || [ ! -f "$HOST_MODEL_DIR/model.safetensors.index.json" ]; then
+    echo "serve-frank-primary: model payload incomplete at $HOST_MODEL_DIR (config.json or index missing)" >&2
+    echo "serve-frank-primary: run scripts/download-frank-primary-fp8.sh on this host" >&2
+    exit 1
+fi
+
+if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
+    echo "serve-frank-primary: /dev/kfd or /dev/dri missing — GPU stack not up" >&2
+    exit 1
+fi
+
+mkdir -p /home/dank/.cache/frank-primary-vllm
+
+SERVE_ARGS=(serve "$MODEL_DIR"
+    --served-model-name "$SERVED_NAME"
    --host "$BIND_HOST"
    --port "$PORT"
-    --model "${MODEL_PARTS[0]}"
-    --alias frank-primary
-    --ctx-size "$CONTEXT_SIZE"
-    --parallel "$PARALLEL_SLOTS"
-    -lv "$VERBOSITY"
-    --metrics
-    --n-gpu-layers "$NGL"
-    --tensor-split "$TENSOR_SPLIT"
-    --jinja
-    --reasoning auto
-    --reasoning-format deepseek
-    --reasoning-budget -1
-    --cache-type-k "$CACHE_TYPE_K"
-    --cache-type-v "$CACHE_TYPE_V"
-    --batch-size "$BATCH_SIZE"
-    --ubatch-size "$UBATCH_SIZE"
-    --cache-ram "$CACHE_RAM"
-    # Vision: explicit, never autoloaded by this build
-    --mmproj "$MMPROJ_FILE"
-    # Spec chain: the shared MTP head + the draftless ngram matcher
-    --spec-type draft-mtp,ngram-mod
-    --model-draft "$DRAFTER_FILE"
-    --spec-draft-n-max "$SPEC_DRAFT_N_MAX"
-    --spec-draft-device "$SPEC_DRAFT_DEVICE"
-    # Greedy server default: deterministic e2e, and the harness overrides
-    # sampling per-request (models.yml compat.extraBody carries Qwen's
-    # recommended temp 1.0 / top_p 0.95 / top_k 20).
-    --temp 0
-    --top-p 1.0
-    --top-k 0
-    --min-p 0.0
+    --tensor-parallel-size "$TP"
+    --gpu-memory-utilization "$GPU_UTIL"
+    --max-model-len "$CTX_SIZE"
+    --max-num-seqs "$MAX_SEQS"
+    --offload-backend "$OFFLOAD_BACKEND"
+    --cpu-offload-gb "$OFFLOAD_GB"
+    --cpu-offload-params "$OFFLOAD_PARAMS"
+    --enable-auto-tool-choice --tool-call-parser "$TOOL_PARSER"
+    --reasoning-parser "$REASONING_PARSER"
+    --default-chat-template-kwargs "$CHAT_TEMPLATE_KWARGS"
 )

-# Comma-separated <pattern>=<backend> pairs, ONE flag: b11030 logs
-# DEPRECATED for repeated --override-tensor and wants the joined list.
-# The loader applies the FIRST matching pattern per tensor, so the
-# specific block patterns must precede the broad exps=CPU sweep in
-# frank-primary.env.
-ARGS+=(--override-tensor "$OT_OVERRIDES")
+# SPEC_CONFIG (optional, JSON) is how MTP drafting is enabled after the
+# bring-up measurement: --speculative-config '{"method":"mtp",...}'.
+if [ -n "${SPEC_CONFIG:-}" ]; then
+    SERVE_ARGS+=(--speculative-config "$SPEC_CONFIG")
+fi

-exec "$SERVER_BIN" "${ARGS[@]}"
+exec podman run --rm \
+    --name "$CONTAINER_NAME" \
+    --userns=keep-id \
+    --network=host \
+    --ipc=host \
+    --security-opt seccomp=unconfined \
+    --cap-add SYS_PTRACE \
+    --device /dev/kfd \
+    --device /dev/dri \
+    --group-add keep-groups \
+    -e ROCR_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES" \
+    -e HIP_VISIBLE_DEVICES="$HIP_VISIBLE_DEVICES" \
+    -e VLLM_ROCM_USE_AITER="$VLLM_ROCM_USE_AITER" \
+    -e TRITON_CACHE_DIR="$CACHE_ROOT/triton" \
+    -e TORCHINDUCTOR_CACHE_DIR="$CACHE_ROOT/inductor" \
+    -e VLLM_CACHE_DIR="$CACHE_ROOT/vllm" \
+    -e TRITON_ALLOW_NON_CONST_GLOBALS=1 \
+    --volume /home/dank/models:/models \
+    --volume /home/dank/.cache/frank-primary-vllm:"$CACHE_ROOT" \
+    "$IMAGE" \
+    vllm "${SERVE_ARGS[@]}"