frank-primary: vLLM 0.30 UVA-offload shape (pre-deploy commit)

The llama.cpp b11030 trio becomes the vLLM 0.30 trio ahead of the
FP8 cutover on :7770 — same port, same alias, no client moves.

Serve shape: locally built ROCm gfx1201 image
(localhost/frank-primary:vllm030) execs via podman as dank from the
root unit (LimitMEMLOCK=infinity legalizes the pin). UVA offload:
--offload-backend uva pins the PLE family (47.75 GiB, 138 tensors:
131 n-gram table shards + 7 per-layer conv/proj) + routed experts
(114.86 GiB, 150,528 tensors) in host RAM — 162.60 GiB combined,
~81.3 GiB per rank under TP=2, hence --cpu-offload-gb 82 (80/card
strands 2.6 GiB of experts on the cards). Parsers per the host's
own Ornith-era precedent: qwen3_xml tools + qwen3 reasoning (vLLM
0.30 emits the channel as `reasoning` — reasoning_content is
renamed server-side) + enable_thinking/preserve_thinking template
kwargs. Spec decode stays env-wired and OFF at bring-up.

Deployer deploy_llm_primary rewritten: image gate, index-driven
shard-count gate (131), stale-container cleanup, port wait, and the
false-green hardening — health is only green when the unit's own
MainPID is live behind the answer (2026-09-26 lesson).

Seam tests rewritten to the vLLM contract (15 gates): offload
arithmetic, parser wiring, no-llama.cpp-ism, deployer gates. Suite
349/349 green.

scripts/frank-primary.env

@@ -1,113 +1,96 @@
-# frank-primary — unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X)
-# serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 on the two Radeon AI PRO
-# R9700s (gfx1201), with the shared Q8_0 MTP drafter head, the BF16 vision
-# projector, and the draft-mtp + ngram-mod spec chain.
+# frank-primary.env — tunables for the vLLM 0.30 frank-primary endpoint
+# (Qwen3.8-Flash-Next-FP8, 2x R9700 gfx1201, dankserver :7770).
 #
-# These are the tunables read by scripts/serve-frank-primary.sh. The flag
-# list is owned by the serve script; this file only sets fleet values.
+# Paths and numbers only — NEVER credentials. Do not name a tunable TEMP, HOST
+# or GPU_MAX_HEAP_SIZE: the HIP runtime reads those bare names (2026-08-13).
 #
-# DO NOT name a new tunable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
-# runtime reads those bare (the 2026-08-13 TEMP incident). The bind address
-# is BIND_HOST, not HOST — frank-large's env predates that lesson and gets
-# away with it; do not copy the shape.
+# Container paths: the model tree is mounted at /models (host
+# /home/dank/models); the compile caches live under /cache (host
+# /home/dank/.cache/frank-primary-vllm, keyed on model + stack + graph knobs —
+# never share it across configurations).
 
-# The payload (scripts/download-frank-primary.sh): the serve script refuses
-# to start without all six Q8_0 parts, the MTP head and the projector.
-MODEL_ROOT=/home/dank/models/Qwen3.8-Flash-Next-GGUF
-
-# The runtime: the unsloth b11030-mix prebuilt for gfx120X (the same
-# b11030-mix build frank-large runs on gfx1151 — it bundles its own ROCm
-# runtime, no host SDK gate).
-RUNTIME_DIR=/home/dank/GitPackages/unsloth-llama-b11030
-
-# Fleet endpoint: the port and alias the lucebox DS4.1 quadlet held before
-# the 2026-09-26 swap, so no client, probe or models.yml entry changes
-# address.
+# --- identity ---------------------------------------------------------------
+SERVED_NAME=frank-primary
 BIND_HOST=0.0.0.0
 PORT=7770
 
-# Four conversations, each the model's full native 262144 window.
-# --parallel divides --ctx-size into slots — the pool is 4x262144 cells
-# (~23 GiB at the V-quantized law below, split across both cards). Slots
-# DIVIDE the pool; they never multiply it. Per-slot window stays the
-# full native 262144, so no client or models.yml change is implied.
-CONTEXT_SIZE=1048576
-PARALLEL_SLOTS=4
+# --- model ------------------------------------------------------------------
+# Host-side dir: /home/dank/models/Qwen3.8-Flash-Next-FP8 (131 safetensors
+# shards, 172.78 GiB, verified against model.safetensors.index.json).
+MODEL_DIR=/models/Qwen3.8-Flash-Next-FP8
 
-# Spec chain: draft-mtp (the shared Q8_0 head) at depth 2, with the
-# draftless ngram-mod matcher chained behind it. Depth 2 is the measured
-# optimum class for this pair at long contexts (the retired Dankclaw
-# Flash-Next deployment ran exactly this head at depth 2 with 0.83-0.92
-# acceptance); each extra verified position re-batches the CPU expert
-# reads, so depth buys less than bandwidth here.
-SPEC_DRAFT_N_MAX=2
+# --- placement --------------------------------------------------------------
+# Fixed 2-card tensor parallel (the R9700 pair; the Raphael iGPU is index 2
+# and excluded by the visibility pins).
+ROCR_VISIBLE_DEVICES=0,1
+HIP_VISIBLE_DEVICES=0,1
+TP=2
+# 0.97 of 31.86 GiB/card (34,208,743,424 B) — leaves ~1 GiB for the ROCm
+# baseline. Raise only with a measured reason.
+GPU_UTIL=0.97
+# The model's native window.
+CTX_SIZE=262144
+# Sequences the scheduler may hold. 8 is the proven batch shape on this host
+# (radiance era); revisit with a bench after bring-up.
+MAX_SEQS=8
 
-# Device placement. -ngl 999 puts every layer's non-overridden tensors on
-# the GPUs; the big overrides then move the bandwidth-huge tensors to host
-# RAM (computed there — the x4-pair measurement: streaming over PCIe loses,
-# computing in RAM does not care). OT_OVERRIDES is ONE comma-separated
-# <tensor-pattern>=<backend> list passed as a single --override-tensor
-# flag (b11030 logs DEPRECATED for repeated flags and wants the joined
-# form), and the loader applies the FIRST matching pattern per tensor —
-# so the block-specific patterns MUST precede the broad exps=CPU sweep.
-#
-# Hot expert blocks stay GPU-resident (2026-09-26): -ot granularity is the
-# whole per-block expert tensor (512 experts share one tensor), so "critical
-# experts" = blocks co-located on the card that holds their layers.
-# Measured cost: 2.42 GiB/block on ROCm0, 2.49 GiB/block on ROCm1.
-# Blocks 0-4 on ROCm0 + 45-47 on ROCm1: card0 runs at ~2.5 GiB free at
-# boot / ~2.1 GB under load (vision graph included) — the operator
-# directed the fill PAST the 3 GB checkpoint floor on 2026-09-26
-# ("shift a few more experts"); measured, battery-verified. Card1 keeps
-# ~3.4 GB. Card1 CANNOT take another block: 3.31 GiB free − 2.49 GiB =
-# 0.82 GiB, and the vision graph + boot drift alone exceed that.
-#   * exps=CPU — the remaining routed experts (blk.*.ffn_*_exps.weight,
-#     ~100 GiB at Q8_0 even with 7 blocks GPU-side), read once per token;
-#     the CPU backend computes them from page cache.
-#   * per_layer_token_embd.weight=CPU — the 51B-element PLE n-gram table
-#     (dims [160, 320001536], Q8_0, 54.4 GiB — the whole of part 3):
-#     sparse row lookups, tiny PCIe traffic from CPU RAM. It doubles as
-#     the shared MTP head's borrow tensor — GPU-resident, the draft load
-#     crashes with "invalid vector subscript" (llama.cpp PR #28243) — so
-#     this override is REQUIRED, not just a placement choice. NEVER give
-#     it a GPU target (the 23 -> 13.4 t/s regression) and never --no-mmap
-#     (the table must stay page-cache-backed; anonymous memory
-#     cgroup-kills silently).
-NGL=999
-OT_OVERRIDES='blk\.[0-4]\.ffn_(up|gate|down)_exps=ROCm0,blk\.4[5-7]\.ffn_(up|gate|down)_exps=ROCm1,exps=CPU,per_layer_token_embd.weight=CPU'
-TENSOR_SPLIT=1,1
+# --- weight offloading (the UVA shape) ---------------------------------------
+# --offload-backend uva: matched weights live in PINNED host RAM and are read
+#   in place over PCIe by the kernels (zero-copy), instead of being resident
+#   on the cards. is_uva_available() = is_pin_memory_available() on ROCm; the
+#   pin needs the unit's LimitMEMLOCK=infinity (a root unit grants it via
+#   PID1 — the dankserver user session defaults to 8 MiB and cannot raise it).
+#   If pinning is unavailable the offloader falls back to a copy path — the
+#   load log MUST be checked for the UVA path, not just /health.
+# --cpu-offload-gb is PER GPU (GiB). 2x R9700 total pinned budget ≈ 160 GiB of
+#   the 246 GiB host RAM; the rest covers the OS, llama.cpp-era page cache
+#   drain and the pinned staging buffers. Do not raise without checking
+#   MemAvailable at load time.
+# --cpu-offload-params: name SEGMENTS (dot-delimited components) of tensors
+#   to offload. Measured from the payload headers (2026-09-26): `ple` matches
+#   138 tensors = 131 n-gram table shards + 7 per-layer conv/proj weights,
+#   47.75 GiB; `experts` matches 150,528 tensors, 114.86 GiB. Combined
+#   162.60 GiB — with TP=2 each rank needs ~81.3 GiB of pinned host RAM,
+#   hence OFFLOAD_GB=82 per GPU (164 GiB total; 80/card left 2.6 GiB of
+#   experts resident, inflating the on-card weight share). Resident after
+#   the full offload: 10.16 GiB of dense/attention/MTP/ViT/lm_head weights.
+#   Matching is greedy-up-to-budget in traversal order; unmatched params are
+#   never offloaded.
+OFFLOAD_BACKEND=uva
+OFFLOAD_GB=82
+OFFLOAD_PARAMS=ple,experts
 
-# The drafter MUST sit on the SECOND R9700 (ROCm1): with -ts 1,1 the main
-# model's output head lands on the last-layers card, and the shared MTP head
-# BORROWS it. --override-tensor sets has_tensor_overrides(), which pins
-# n_copies==1 in the draft's scheduler — a borrowed output.weight sitting in
-# a ROCm1 buffer then aborts the draft graph with ggml-backend.cpp:941
-# "pre-allocated tensor (output.weight) in a buffer (ROCm1) that cannot run
-# the operation" when the draft is elsewhere (verified live 2026-09-26: the
-# ROCm0 placement crash-looped 4x at ~12 s, ROCm1 loaded and served).
-SPEC_DRAFT_DEVICE=ROCm1
+# --- engine -----------------------------------------------------------------
+# No aiter wheel in this image (gfx9-only upstream); the AMD qwen4_exp path
+# runs its in-tree triton kernels. Deterministic rather than relying on the
+# availability probe.
+VLLM_ROCM_USE_AITER=0
+# Compile/inductor/triton caches (container /cache, TRITON_CACHE_DIR etc.)
+CACHE_ROOT=/cache
 
-# Batch geometry: logical batch 8192 with ubatch 512 — the measured optimum
-# class on this fleet's ROCm stack (a bigger logical batch is free; compute
-# buffers reserve at max graph size). ubatch 512 balances prefill
-# throughput against the asymmetric compute-buffer cost on the card that
-# holds the output layers and the drafter.
-BATCH_SIZE=8192
-UBATCH_SIZE=512
+# --- parsers / template (the harness wire contract) ---------------------------
+# vLLM 0.30 emits the thinking channel as `reasoning` (the deprecated
+# `reasoning_content` is renamed server-side, chat_completion/protocol.py:554).
+# The qwen3 reasoning parser extracts the template's think-block tags into
+# that channel; the qwen3_xml tool parser structures tool-call XML output into
+# tool_calls (auto-tool-choice enables the structured path for API clients).
+# The family precedent is the host's own Ornith-era serve scripts
+# (~/vllm/failedAttempts/OrnithFP8.sh); the FP8 tokenizer's template
+# consumes enable_thinking + preserve_thinking (verified in tokenizer_config).
+TOOL_PARSER=qwen3_xml
+REASONING_PARSER=qwen3
+CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
 
-# KV cache types: V quantized to q8_0, K held at the f16 default. The V
-# half is what quantizes on this build: main KV drops 24,576 -> 18,816
-# B/token (12 of 48 layers carry KV; the 36 SSM layers are constant-state
-# and never scale with ctx). NOTE the shared MTP drafter's KV ignores the
-# V type (measured: draft pool 2,048 MiB = the f16 law at 1,048,576
-# cells) — one layer's worth, so the miss costs ~0.5 GiB on ROCm1.
-CACHE_TYPE_K=f16
-CACHE_TYPE_V=q8_0
+# --- speculative decode ------------------------------------------------------
+# The hybrid stack (GatedDeltaNet + QSA short-conv) uses the v1 engine's
+# mamba-capable attention backends, negotiated by the model, not by env.
+# MTP drafting comes from the model's own mtp.* weights via
+# --speculative-config, only when SPEC_CONFIG is non-empty. Bring-up runs
+# with it empty on purpose: measure the base shape clean, then A/B; never
+# change two knobs at once.
+SPEC_CONFIG=
 
-# Prompt-cache budget: 16 GiB. The V-quantized KV pool at four full
-# windows is ~23 GiB spread across both cards (the hybrid DeltaNet/QSA
-# pattern keeps even 4x262144 a fraction of a full-ctx x all-layers
-# pool); the prompt cache is the RAM-side spend that actually moves
-# prefill on repeat traffic. Host RAM budget: ~90 GiB of expert pages +
-# this cache + the server's own arenas inside 246 GiB.
-CACHE_RAM=16384
+
+# --- container ---------------------------------------------------------------
+IMAGE=localhost/frank-primary:vllm030
+CONTAINER_NAME=frank-primary-vllm