serverAppConfigs/scripts/frank-primary.env

# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
#
# 2026-10-03 shape G (as shipped, after the 1.00 revert): the CUDA-graph plan
# term dropped from the analytic 0.55 GiB to the measured 0.07 GiB (0 base +
# 12 MiB/graph), and UTIL at the 0.982 law. MEASURED on the live deploy: arena
# 17.77 GiB / 7,762 slots, KV 401,408 tokens / 2.00x concurrency (the first
# shape holding two full-length conversations with real spare), driver used
# 30.97 GiB of 31.86 — 0.89 GiB of real free VRAM per card, held through a
# live 4,096-token generation. UTIL stays 0.982 ALWAYS: 1.00 is unbootable on
# this fork (worker init refuses budget >= device total) and 0.13-GiB-margin
# boots die under load.
# `budget - non-expert - KV reserve - graphs - overhead` and the KV reserve is a
# registry constant, so those two terms are the whole flag surface.
# 2026-10-03 shape F (absorbed): MNBT 2048 + the graph term pinned. Measured on
# the live deploy: arena 17.52 GiB / 7,653 slots, budget 31.29, graphs 0.32.
# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
# --expert-precise-memory ON. The window shrink from the native 262144 and the
# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
# the measured 2x2 is below); shape E adds the precise-overhead plan, which
# trades the default 7.60 GiB compile headroom for the fitted 5.25 GiB and puts
# the 2.35 GiB difference into the expert arena: 6,496 -> 7,522 slots (+16%).
# The pool is deliberately NOT cut (option A): the pool is not an arena lever
# (measured), so 3.15e9 keeps shape D's 1.99x concurrency while the arena grows.
# Same digest-pinned tcclaviger RDNA4 fork. Rollback for the fp8+MTP4 era is git
# tag frank-primary-fp8-mtp-20261001.
#
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
# prefixed FRANK_PRIMARY_ so none can leak. The one bare name we WANT inside the
# container (GPU_MAX_HW_QUEUES) is held here under the prefix and passed on the
# run line as `-e GPU_MAX_HW_QUEUES=…`.
#

# --- runtime ----------------------------------------------------------------
# Digest-pinned: the floating :latest tag moved once already. Local alias
# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
# bytes in rootless storage.
FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea

# --- payload ----------------------------------------------------------------
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
FRANK_PRIMARY_MODEL_SHARDS=131

# --- identity ---------------------------------------------------------------
# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
FRANK_PRIMARY_CONTAINER=frank-primary-qwen
FRANK_PRIMARY_SERVED_NAME=frank-primary
FRANK_PRIMARY_PUBLISH=7770:7770
# Containers removed before every start: the retired R9V identity and the ad
# hoc experiment identity (both hold the GPUs / name registrations).
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"

# --- qualified engine shape (2026-10-02, shape E) ---------------------------
# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
# (SEQS 3), but the POOL, not the slot count, decides how many full-length
# conversations are resident. MEASURED LIVE 2026-10-02 at this shape:
#   GPU KV cache size: 398,658 tokens,
#   Maximum concurrency for 200,704 tokens per request: 1.99x
# (3,146,129,408 B reserved — 7,892 B/token). The per-token price is
# SHAPE-DEPENDENT, not a constant: the 262144 shape priced at 7,701 B/token
# (6,100,000,000 B -> 792,069 tokens), ~2.5% cheaper, so never extrapolate a
# concurrency figure across geometries. Shorter requests use the spare blocks,
# and a third LONG request queues rather than being refused. Verified: three
# concurrent 67-token requests answered in parallel, ~4.1 s each.
#
# WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
# against the then-deployed known-good shape, production stopped for each):
#   SEQS 2 + MNBT 1024 -> Engine core initialization failed
#   SEQS 2 + MNBT 2048 -> Engine core initialization failed
#   SEQS 3 + MNBT 1024 -> healthy (792,069 tokens, 3.02x, arena 6,248)
#   SEQS 3 + MNBT 2048 -> healthy (the deployed incumbent)
# The failure is torch.fx ConstraintViolationError: query_start_loc.size()[0]
# "marked dynamic but your code specialized it to be a constant (3)", raised
# from AOT build_guards(strict_error=True) during profile_run -> _dummy_run.
# model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets
# num_reqs = min(max_num_batched_tokens, max_num_reqs), so the width is
# SEQS + 1 for ANY MNBT >= SEQS: 3 at SEQS 2, 4 at SEQS 3. MNBT only moves the
# compile RANGE ((1,1024) vs (1,2048)) and cannot dodge it — the 2x2 above is
# why MNBT is not a lever here. --enforce-eager bypasses the path but loses
# CUDA graphs. Same failure class killed the 2026-10-01 bf16 2 x 262144 shape.
#
# THE ARENA LEVER IS max_model_len, NOT KV_BYTES (measured 2026-10-02). The
# planner sizes expert cache as budget - non-expert - KV reserve - graphs -
# overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len:
#   MAXLEN 262144 -> reserve 2.48 GiB -> arena 14.30 GiB / 6,248 slots
#   MAXLEN 200704 -> reserve 1.91 GiB -> arena 14.87 GiB / 6,496 slots
# d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
# pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
# died with 0 bytes free on a real 995-token request). To grow the arena,
# lower MAXLEN — not KV_BYTES. The OTHER arena lever is the overhead term:
# --expert-precise-memory (shape E) takes it from 7.60 to 5.25 GiB and lands
# those 2.35 GiB in the arena — 14.87 -> 17.22 GiB / 6,496 -> 7,522 slots.
#
# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
# measured +18.7% decode vs default geometry and the win grows with depth.
# Do not drop it.
# MNBT 2048 (Frank's call, 2026-10-03; the lock was 1024). The measured law
# still holds: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s), 1024-4096
# are interchangeable on decode, 12288 costs 2.5% plus arena. 2048 is ARENA-FREE
# by construction: the registry's overhead model is
# base + 0.22 GiB x max(0, (nbt - 4096)/1024) with the anchor at nbt 4096, so
# every rung up to and including 4096 prices the SAME 5.25 GiB overhead — and
# the fitted 5.25 was fitted at nbt 4096, i.e. our 2048 sits UNDER the anchor,
# so the activation term is covered by a figure measured at a bigger batch.
# SEQS 3 boots at both 1024 and 2048 (measured 2026-10-02; the SEQS=2
# ConstraintViolationError is a SEQS property, not an MNBT one). Costs exactly
# one recompilation: the compile RANGE is part of the cache key, so
# FRANK_PRIMARY_CACHE_DIR pays for whichever rung is chosen once.
# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
# TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
# unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
# --expert-precise-memory is ON (shape E). It selects the fitted overhead
# (5.25 GiB) over the default (7.60 GiB = fitted + compile/graph-capture
# headroom); the 2.35 GiB difference lands in the expert arena. MEASURED
# 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
# booted AS the service, plan read off the engine's own log):
#   runtime overhead  7.60 -> 5.25 GiB   (registry Qwen4Exp…, precise
#                                         gfx1201 TP2 nbt4096/seqs16/len262144)
#   expert cache      14.87 -> 17.22 GiB
#   slots             6,496 -> 7,522     (+1,026 = +16%, 2.344 MiB each)
#   swap region                 13.7 GiB / 5,986 slots, 133-134 per layer
#   boot                        healthy in 420 s, no OOM, no knife-edge
# The flag was called a non-option after it crash-looped the 2 x 262144 bf16
# boot (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each) — that shape
# was already at the ceiling, so precise-memory was the last 2.35 GiB over, not
# the cause. At 200704 x 3 it fits with room; the run line also reports
# 'expert_precise_memory': True in non-default args, so a silent argparse drop
# would be visible in the boot log.
FRANK_PRIMARY_PRECISE_MEMORY=1
# GPU_MAX_HW_QUEUES caps the HSA hardware queues per GPU (AMD's own tuning
# guide recommends 2 for this class of workload; the runtime default on Linux
# is 4). Verified a REAL knob, not a no-op: the string is present in
# /opt/rocm/core-10.0/lib/libhsa-runtime64.so. NOT A/B MEASURED — Frank's call,
# shipped as a deliberate CPU-side tune alongside the 32x64 gather geometry.
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2
# Persistent compile cache. MEASURED 2026-10-02: the image ships NO
# /root/.cache at all (`ls: cannot access '/root/.cache'` in a fresh
# container), and after our boot it holds /root/.cache/vllm at 1,154,714,333 B
# in 4,665 files plus /root/.cache/comgr at 229 MB — both inside the
# container's ephemeral layer, so every `podman rm -f` (every deploy, restart
# and reboot) throws them away and re-pays the build. The engine's own log
# names the loss: `torch.compile took 181.20 s in total`, of which 'Compiling a
# graph for compile range (1, 1024) takes 173.19 s', followed by 'saved AOT
# compiled function to /root/.cache/vllm/torch_compile_cache/…'. Mounting a
# host dir at /root/.cache persists the vLLM AOT/torch_compile tree AND the
# comgr HIP-kernel JIT cache, and hides nothing (the image has no such dir).
# PROVEN LIVE 2026-10-02, two boots of this exact shape:
#   cold  `Compiling a graph for compile range (1, 1024) takes 179.24 s`
#         `torch.compile took 188.26 s in total`
#   warm  `Directly load the compiled graph(s) for compile range (1, 1024)
#          from the cache, took 2.143 s`
#         `torch.compile took 3.97 s in total`
#         `init engine (profile, create kv cache, warmup model) took 41.13 s
#          (compilation: 3.98 s)`
# (Total time to /health is dominated by the 172.8 GiB payload load, not the
# compile — the saving is the ~184 s of inductor/AOT build.)
# Safe to grow/prune: entries are keyed by version + config hash, so a stale
# entry is a miss, never a wrong answer.
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary
FRANK_PRIMARY_MAXLEN=200704
FRANK_PRIMARY_SEQS=3
# The POOL is deliberately kept at shape D's 3.15e9 (option A): because the pool
# is not an arena lever, raising it back from a cut buys concurrency only, and
# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A
# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected
# as the shipped value; it remains the conservative arm if free VRAM ever binds.
# KV forced to 400,700 tokens (Frank, 2026-10-03). The engine prices this pool
# at 7,891.8 B/token at this shape and loses ~3.87 MB to block alignment, so
# 3,166,500,000 B is the ask that lands at/above 400,700 (verify the reported
# "GPU KV cache size" line on every deploy; it is not a constant ratio across
# geometries — 7,701 B/token at 262144).
FRANK_PRIMARY_KV_BYTES=3166500000
FRANK_PRIMARY_KV_DTYPE=fp8
FRANK_PRIMARY_MNBT=2048
FRANK_PRIMARY_GATHER_CHUNKS=32
FRANK_PRIMARY_GATHER_LANES=64
# PREFIX CACHING IS OFF — THE FORK'S OWN NAMED MITIGATION (2026-10-05).
# MEASURED 2026-10-04, 18:46:07 PDT (5.6 h into the live boot, one running
# request, prefix hit rate 48.1%): 48 gdn_hip state faults in one 0.6 s
# burst, then the process-lifetime silence cap:
#   [gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
#   with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
#   [gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
#   [gdn_hip] ERROR: state fault report limit (32) reached; further faults
#   are counted silently
# One physical SSM slot, two heads, BOTH ranks (heads 3 and 8), 23 layers
# each. The guard's own message in this build: "known Qwen3.8-Flash-Next
# (QFN) flaw under prefix caching and is being actively worked; output
# quality for this request may degrade. Mitigation:
# --no-enable-prefix-caching".
# MECHANISM: with prefix caching on, this fork puts QFN on
# mamba_cache_mode 'align' (the default-for-this-arch branch in
# config/vllm.py; 'all' is deprecated and 'none' is the no-prefix-cache
# mode, so align is the only sound mode WITH the flag). Align mode
# checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
# mamba page every block, where block = 3136 tokens (the attention block
# size is raised to 3136 so the page sizes match exactly — the boot log's
# "Padding mamba page size by 0.13% ..."), and RE-SEEDS the state from that
# page whenever a request resumes from a cached prefix. The corrupt state
# landed at the 13th boundary: 13 x 3136 = 40,768 tokens — the "~40k" mark
# where the request's output turned repetitive/corrupted. The NaN stayed
# in the mamba page, so every later turn of that conversation re-seeded the
# same non-finite state for those two heads (silent: the fault-report cap
# was already spent), while fresh short sessions looked fine.
# RAM RULED OUT THE SAME DAY: zero MCE/EDAC kernel records since the boot
# under test, an 8 GiB in-band pattern check stable over 60 s under live
# load, 4800 MT/s is the JEDEC DDR5 floor on 5600-rated DIMMs (not an
# overclock), and the signature — a two-head, single-slot NaN state in one
# burst — is a state-path fault, not scattered finite bit corruption.
# COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole
# history instead of at most 3135 tail tokens (a full 40k context costs
# tens of seconds of TTFT per turn at this shape's prefill rate). A silently
# corrupted conversation is worse than a slower first token.
# RE-ENABLE PATH: if the fork ships a fixed align path, set this to empty,
# redeploy, and re-verify by taking one conversation past 40,768 tokens
# (decode must stay coherent and podman logs must show zero gdn_hip lines).
FRANK_PRIMARY_NO_PREFIX_CACHING=1
# UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
# 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and
# 0.982 is Frank's standing law: NEVER 1.00. The 1.00 attempt (2026-10-03,
# shape G v1) died in worker init BEFORE loading a byte of weights — the fork
# refuses budget >= device total with
#   ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)
# on BOTH ranks at 09:53 and 09:54, NRestarts=2 — so 1.0 is not even a runtime
# knife-edge here, it is unbootable. The two historical ceiling deaths (the
# 2x262144 bf16 boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a
# real 995-token request) remain the reason the margin exists. Rollback note
# for the other direction: if a request ever dies with
# `Tried to allocate ... 0 bytes free`, the first knob back is THIS number
# down a notch, not the window.
FRANK_PRIMARY_UTIL=0.982
# CUDA-graph plan term, priced at the measured capture. expert_plan.py charges
# estimate_cudagraph_memory() = BASE_MB + PER_GRAPH_MB x n_graphs (defaults
# 256/52 = 0.55 GiB for our 6-graph plan), and the estimator's own docstring
# says to tune these against the post-capture line. MEASURED twice on this
# shape: shape E reported "Graph capturing finished ... took 0.31 GiB" (TP0) and
# 0.17 GiB (TP1); shape F, with the 0.32 GiB we then charged, reported
# "took -0.06 GiB" — a NEGATIVE delta, i.e. capture fits inside memory the
# profiler had already counted. So the base pool (256 MiB) is the over-charge:
# charge 0 base + 12 MiB/graph = 0.07 GiB for the shipped no-spec 6-graph plan,
# which returns 0.25 GiB more to the arena while still paying per graph, so a
# future spec-decode shape (18 graphs) is charged 0.21 GiB rather than zero.
# Verify every deploy against the "Estimated CUDA graph memory" vs "Graph
# capturing finished ... took X GiB" lines; if capture ever goes positive and
# exceeds the charge, raise FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52.
# A THIRD env var exists and is the only way to state the graph term outright:
# VLLM_CUDAGRAPH_MEM_ESTIMATE_MB pins the WHOLE term to an exact MiB total in
# one knob — estimate_cudagraph_memory() checks it FIRST and returns base/
# per-graph untouched when set (verified in the shipped image,
# v1/worker/gpu/cudagraph_mem.py). We ship the BASE/PER_GRAPH pair instead so
# a future spec-decode shape re-prices automatically; switch to ESTIMATE_MB
# only if the drift check below ever shows the pair cannot track capture.
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0
FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12

# HOST-region cap — the last arena-plan term that still autofit per boot. With
# no cap the planner's host bound is MemAvailable AT BOOT minus PLE 47.68 -
# 2 x 7.0 process - 5.0 api/engine - 5% margin (measured boot 2026-10-05:
# MemAvailable 237.97 -> 159.39 GiB usable across 2 ranks; the demand is 105.48
# = 52.74/rank x 2 ranks, 23,040 swap-target experts). The same host reads 75.4
# GiB available while running — the bound tracks whatever the co-tenants hold at
# boot, and a boot where that drops near the demand silently pins next-hottest
# experts VRAM-only instead (smaller swap region; the pinned_extra warning is
# the only trace). Pin 120 GiB: measured demand + 14% headroom, on a host with
# 246.1 GiB total. If a boot ever logs "expert-offload-mem … exceeds
# MemAvailable at plan time", THAT is the contract being broken — look at the
# co-tenant RAM before touching this number. Plan semantics verified in the
# shipped image (offloader/expert_plan.py): the cap binds as the bound, an
# exceedance only warns.
FRANK_PRIMARY_EXPERT_OFFLOAD_MEM=120

# --- optional engine knobs (unset == the flag is absent) ---------------------
# FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds
# the flag. THE RETIREMENT VERDICT WAS INVALID: c1248d4 (2026-10-02) retired MTP
# by comparing fp8+MTP4 (37.4-40.6 tok/s, 54% acceptance, 09-29) against the
# bf16 no-spec boot (36.4-39.4) — and bf16 then proved unbootable, so fp8 came
# back without MTP and fp8+spec was NEVER compared against fp8+no-spec on the
# same shape. The no-spec incumbent measures ~30 tok/s single-stream (below the
# fleet's 35 tok/s target); every vendor flash-next recipe ships MTP3; and our
# MTP4 run used the fork's default GREEDY draft sampling, while the recipes pass
# draft_sample_method=probabilistic (greedy drafts are rejected more often under
# temp>0 — the plausible cause of the 54%). MTP also re-adds drafter workspace:
# the old note says the 7.2 GiB/rank KV ceiling WAS the MTP4 workspace, and
# --expert-precise-memory spends that same budget, so an MTP arm must be tested
# with precise memory BOTH on and off.
#   MTP3:  '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}'
FRANK_PRIMARY_SPEC=
# FRANK_PRIMARY_COMPCONFIG holds the whole --compilation-config JSON. The
# capture-ladder law: the piecewise cudagraph ladder must contain
# num_speculative_tokens + 1 (MTP3 -> 4 -> ladder 4,8,12; MTP4 -> 5,10,15;
# MTP5 -> 6,12,18,24), because every draft step proposes n+1 tokens. The default
# ladder is multiples of 1+4 up to 512 then powers of two, so an MTP3 shape can
# miss its own step size. Pinning the ladder is ALSO the candidate fix for the
# SEQS>3 wall: query_start_loc is specialised to SEQS+1 and that constant must
# land on a captured graph size (3 is not in the default ladder, 4 is —
# measured 2026-10-02, and the vendor runs SEQS 16 on this model/TP2).
# combo_kernels false/true are BOTH in the shipped recipes, so it is not a law.
#   MTP3 at SEQS 3:  '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,4,8,12],"max_cudagraph_capture_size":12}'
FRANK_PRIMARY_COMPCONFIG=
# FRANK_PRIMARY_GEN_CONFIG holds the whole --override-generation-config JSON.
# The payload's HF generation_config.json sets temperature 1.0 / top_p 0.95 /
# top_k 20 and the engine WARNS about it on every boot, so a client that sends
# no sampling fields gets creativity it never asked for. This is the one knob
# that changes ANSWERS, not speed.
#   '{"max_tokens":65536,"temperature":0.7,"top_p":0.95,"top_k":20}'
FRANK_PRIMARY_GEN_CONFIG=

# --- client contract: reasoning split, tools, preserve_thinking ----------------
# Without --reasoning-parser the engine leaves the thinking text INLINE in
# `content` with a dangling `</think>` and a null `reasoning` field (measured
# 2026-10-01: content = 'We need answer...Brief.\n</think>\n\n391') — the
# malformed thinking block clients choke on. The qwen3 parser splits it
# properly. The GLM rollback (d91379d) had dropped all three of these; the
# working values are the ones ecaad36 shipped for this model.
FRANK_PRIMARY_REASONING_PARSER=qwen3
# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
# tool_calls work here (without it, tool_choice auto returns 400).
FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
# preserve_thinking keeps prior thinking blocks in the rendered prompt across
# turns; enable_thinking makes thinking the server-side default. Verified the
# payload's tokenizer_config template consumes both keys.
FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'