# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
#
# 2026-10-01 cutover to the BF16-KV no-spec incumbent (the overnight redo
# campaign's verdict, Frank's call): same digest-pinned engine and payload as
# the 2026-09-29 cutover, but the KV cache is UNQUANTIZED (auto = bf16) and
# speculative decode is GONE. Git tag frank-primary-fp8-mtp-20261001 is the
# last pre-cutover commit; the old shape (fp8 KV + MTP4) is in history and
# restorable from the tag.
#
# Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited):
#   * MAXLEN 241664 x SEQS 3 (724,992 tokens) at bf16 KV is the DEPLOYED
#     shape (2026-10-01 night ladder): KV_BYTES 10,184,000,000 PER RANK,
#     engine measured "KV cache size: 732,359 tokens; Maximum concurrency
#     for 241,664 tokens per request: 3.03x". 262144 and 245760 both OOMed
#     at graph capture (76 MiB short / 0 bytes free); 237568 passed at
#     718,606 tokens and the +4096 bonus step passed and won.
#   * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was
#     the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because
#     the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes
#     11.05 GiB/rank and boots clean — the ceiling was spec's, not VRAM's.
#   * bf16 KV doubles the attention-KV bytes/token (measured 14,047 vs
#     7,926) and costs ~14% decode vs fp8 at 4064 ctx (49.25 -> 36.42 with
#     the gather win) — traded deliberately for clean cache + 262k windows.
#   * GATHER 32x64 is the night's biggest lever: R4D_LRU_GATHER_CHUNKS=32 /
#     R4D_LRU_GATHER_LANES=64 measured +18.7% decode vs default geometry,
#     and the win grows with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 fp8
#     era). Do not drop it.
#   * MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s
#     vs 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5%
#     and arena. 1024 keeps the workspace floor so the arena stays max.
#   * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
#     prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
#     roughly half the gather traffic was waste. Never re-add
#     --expert-prefetch without a fresh A/B.
#   * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
#     offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
#     numerics.
#
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
# prefixed FRANK_PRIMARY_ so none can leak.
#

# --- runtime ----------------------------------------------------------------
# Digest-pinned: the floating :latest tag moved once already. Local alias
# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
# bytes in rootless storage.
FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea

# --- payload ----------------------------------------------------------------
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
FRANK_PRIMARY_MODEL_SHARDS=131

# --- identity ---------------------------------------------------------------
# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
FRANK_PRIMARY_CONTAINER=frank-primary-qwen
FRANK_PRIMARY_SERVED_NAME=frank-primary
FRANK_PRIMARY_PUBLISH=7770:7770
# Containers removed before every start: the retired R9V identity and the ad
# hoc experiment identity (both hold the GPUs / name registrations).
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"

# --- qualified engine shape (2026-10-01, RESTORED KNOWN-GOOD: fp8 3 windows) ---
# bf16 at 2 x 262144 did NOT stay up: with and without --expert-precise-memory
# the boot failed (`Engine core initialization failed`, NRestarts climbing).
# The last shape measured serving was fp8 3 x 262144 / MNBT 2048 / KV 6.1e9
# (KV 792,069 tokens, 3.02x, arena 6,248, ~3.6 GiB free) and is restored here
# while the bf16 path is investigated in a window (see log.md 2026-10-01).
# Candidate next test: bf16 with SEQS=3 to isolate the KV dtype from the
# sequence count, and/or --enforce-eager to bypass the dynamo constraint path.
# SEQS 2 x MAXLEN 262144 at BF16 KV (Frank's call, 2026-10-01): two contexts of
# the model's FULL native window with UNQUANTIZED KV, plus
# --expert-precise-memory to hand the arena the compile/graph-capture headroom.
# Pool: bf16 costs 13,839 B/token (measured: 11,047,010,304 B -> 798,280 tokens
# on the 3x241664 bf16 boot; decomposes as 12,276 attention + 1,563 mamba/side
# caches, against fp8's 6,138 + 1,563 = 7,701). 2 x 262144 = 524,288 tokens
# -> 7.256e9 B floor, pinned at 7,300,000,000 (~527,500 tokens, 3,206 margin).
# Budget (per rank): 31.22 GiB total; non-expert weights 6.28, graphs 0.55,
# non-torch ~1.45, pool 6.76 -> ~29.35 used, ~2.5 GiB free (measured basis:
# the fp8 3-window shape used 28.27 and left 3.59).
# PRECISE MEMORY: DROPPED 2026-10-01 after it crashed the boot in combination
# with bf16. The flag selects the fitted 5.25 GiB overhead instead of the
# 7.60 GiB default, moving the plan's `room` and therefore the arena
# (6,248 -> ~7,278 slots). But the arena is DEVICE-resident, and the budget
# for this shape does not have 2.35 GiB to spare:
#   weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena 16.65
#   + non-torch ~1.45 = 31.73 of 31.86 GiB  =>  ~0.13 GiB free at boot
# measured as a CrashLoop: every attempt reached graph capture, the engine
# exited cleanly, systemd restarted it (NRestarts=3 at ~6.7 min each).
# With the default overhead the arena is 14.31 GiB and the same shape totals
# ~29.4 GiB, leaving ~2.4 GiB for capture. Re-enable the flag only with a
# pool/context reduction that pays for it.
# MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so
# 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096.
FRANK_PRIMARY_MAXLEN=262144
FRANK_PRIMARY_SEQS=3
FRANK_PRIMARY_KV_BYTES=6100000000
FRANK_PRIMARY_KV_DTYPE=fp8
FRANK_PRIMARY_MNBT=2048
FRANK_PRIMARY_GATHER_CHUNKS=32
FRANK_PRIMARY_GATHER_LANES=64
FRANK_PRIMARY_UTIL=0.98

# --- client contract: reasoning split, tools, preserve_thinking ----------------
# Without --reasoning-parser the engine leaves the thinking text INLINE in
# `content` with a dangling `</think>` and a null `reasoning` field (measured
# 2026-10-01: content = 'We need answer...Brief.\n</think>\n\n391') — the
# malformed thinking block clients choke on. The qwen3 parser splits it
# properly. The GLM rollback (d91379d) had dropped all three of these; the
# working values are the ones ecaad36 shipped for this model.
FRANK_PRIMARY_REASONING_PARSER=qwen3
# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
# tool_calls work here (without it, tool_choice auto returns 400).
FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
# preserve_thinking keeps prior thinking blocks in the rendered prompt across
# turns; enable_thinking makes thinking the server-side default. Verified the
# payload's tokenizer_config template consumes both keys.
FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'