# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770). # # 2026-09-30 cutover from orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (MTP4) to # Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet has served # through the fork's W4A16 expert-offload path. ROLLBACK: the pre-cutover qwen # state is commit 2339fae ("frank-primary: fix the SIGPIPE race in both # image-digest gates") — restore scripts/ from it and deploy; the qwen payload # (173 GiB) and the R9V state dirs are untouched on disk. # # QUALIFIED 2026-09-30 by boot probe + bench against the live engine: # * THE 35 tok/s TARGET IS NOT MET: decode 9.9-10.0 tok/s, prefill 918 tok/s # (warm, 2.6k-token prompt) at 131,072 ctx x 2 seqs. See the runbook's # "Why 35 tok/s is out of reach on this hardware" — the bound is the fork's # expert-gather read path (8.2 GB/s effective vs the link's 22.74 GB/s), # the same structural gap the 2026-09-29 qwen campaign measured. # * Arena 14.51 GiB = 2,383 slots x 6.234 MiB per rank; 12,096 host-backed # experts per rank, 73.64 GiB pinned. Live per-layer telemetry # (R4D_LRU_TELEMETRY=1, 2026-09-30): hit rate 0.53-0.66, ~500-730 installs # vs ~2,000 routed per layer over the window. # * --expert-precise-memory is ON: it replaces the 6.25 GiB default # compile-headroom figure with the registry's fitted 5.46 GiB, buying # 1.43 GiB of cache (13.08 -> 14.51 GiB). Decode measured flat across that # change (10.03 -> 9.88 tok/s), which is the evidence that the hit rate # saturates near 0.6 and residency is no longer the binding term. # * KV dtype stays bf16/AUTO — fp8 KV is NOT legal on this model. It disables # _use_rocm_sparse_triton() in the ROCM_AITER_MLA_SPARSE backend (the # predicate requires a non-fp8 cache), and that rope-free Triton path is the # only MLA decode available on gfx12: aiter's MLA decode op is registered # under @if_aiter_supported, i.e. gated on CDNA>2. With fp8 KV the MTP draft # step dies with "'_OpNamespace' 'vllm' object has no attribute # 'rocm_aiter_mla_decode_fwd'". # * MTP4 spec decode is ABSENT DELIBERATELY, unlike the qwen shape. It WORKS # here (the MTP head loads, architecture resolves to Glm5NextMTPModel, and # acceptance is excellent: mean accepted length 3.76-4.90, per-position # 0.92/0.85/0.69/0.58, 69-97%) — but it buys nothing on an offload-bound # deployment: measured decode 9.6-11.2 tok/s (vs 10.0 without) while prefill # COLLAPSED 918 -> 255 tok/s. The verification forward gathers the union of # the batch's experts, so spec decode does not amortize a residency-bound # gather the way it amortizes compute. It also costs 2.6 GiB of arena. # Reinstate only with a fresh A/B on a non-offload-bound shape. # * PREFETCH IS ABSENT BY LAW, unchanged from the qwen campaign: expert-cache # telemetry measured the prefetcher at 36% precision and deleting the flag # bought +40-51% decode. Never re-add --expert-prefetch without a fresh A/B. # * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert # offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts # numerics (repetition loops, early halts, invalid measurements). # * KV is NOT capped explicitly: the fork's plan reserve (1.76 GiB at 131,072) # already covers what max_model_len needs for this shape. It does NOT for an # MTP shape — see the runbook's fork-bug section and # scripts/patches/gpu_worker-kv-cap.patch. # # HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is # prefixed FRANK_PRIMARY_ so none can leak. # --- runtime ---------------------------------------------------------------- # Digest-pinned: the floating :latest tag moved once already. Local alias # qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same # bytes in rootless storage. Same image as the qwen cutover — the W4A16 path # (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM) ships in it. FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea # --- payload ---------------------------------------------------------------- # Intel AutoRound W4A16: symmetric int4, group 128, packing "auto_round:auto_gptq". # quantization_config.quant_method is "auto-round", which the fork overrides to # "inc" (inc.py:364) — that is what routes the MoE to R4dW4a16MoEMethod instead # of the compressed-tensors path's Triton experts, which has no offload recipe. # 34 numbered shards + model_extra_{conv,tensors}.safetensors, 113,074 tensors. FRANK_PRIMARY_MODEL_DIR=/home/dank/models/GLM-5.3-Flash-W4A16-AutoRound FRANK_PRIMARY_MODEL_SHARDS=34 # --- identity --------------------------------------------------------------- # The harness models.yml id stays frank-primary (provider 192.168.0.235:7770). FRANK_PRIMARY_CONTAINER=frank-primary-glm FRANK_PRIMARY_SERVED_NAME=frank-primary FRANK_PRIMARY_PUBLISH=7770:7770 # Containers removed before every start: the pre-cutover qwen identity (it holds # the same port and GPUs), the retired R9V identity, and the ad hoc experiment # identity. FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe" # --- qualified engine shape --------------------------------------------------- # 131,072 x 2 seqs. The window is NOT chosen for capacity: the engine's own # plan reserve is one max_model_len request, and below ~92k tokens the fork's # reserve model (13,600 B/token + a small mamba term) falls short of what the # engine actually needs for this model's per-sequence KDA state, which makes the # boot refuse ("2.07 GiB needed vs 1.74 GiB available"). 131,072 clears it with # the plan's own cap intact. KV cache as built: 138,561 tokens, 1.06x # concurrency for a 131,072-token request. FRANK_PRIMARY_MAXLEN=131072 FRANK_PRIMARY_SEQS=2 # 0.99 fails boot ("Free memory on device cuda:1 ..."); 0.98 leaves the # compile/capture headroom the plan models. FRANK_PRIMARY_UTIL=0.98 # Prefill chunk. 2,048 is the qualified value; the expert union a chunk routes # to saturates quickly, so the chunk size is an install-amortization knob, not a # residency one. Measured 2026-09-30: NBATCH 8,192 boots (arena 12.51 GiB) and # leads to the SAME warm prefill (922 vs 918 tok/s) — no reason to pay its # 2 GiB of profile buffer. FRANK_PRIMARY_NBATCH=2048 # GLM's own tool-call dialect. Without --enable-auto-tool-choice the endpoint # rejects tool_choice "auto" outright ("\"auto\" tool choice requires # --enable-auto-tool-choice and --tool-call-parser to be set"), which breaks # every tool-using agent on the endpoint. Verified 2026-09-30: a get_weather # prompt returns tool_calls with finish_reason "tool_calls". FRANK_PRIMARY_TOOLPARSER=glm47 # GLM's reasoning dialect. Without it the model's thinking leaks into `content` # (measured: content began with the answer, then "Let me verify: ... = 391" # followed by a bare "