Comrade Chief Designer
30904bac1b
The 2026-10-01 bf16 shape (2 x 262144, --expert-precise-memory) crash-looped at graph capture — the arena is DEVICE-resident and that shape had 0.13 GiB to spare. The plain bf16 2 x 262144 also did not stay up: 0 bytes free at runtime, and a real client request (995-token prompt, max_tokens=261149) killed the worker with . The correction is a pool cut, not a downgrade. FP8 KV (7,701 B/token), two 200704 windows (401,408 tokens -> 3.091e9 B floor, pinned 3.15e9), MNBT 1024. The fork's expert planner sizes the arena AFTER the KV override, so the ~2.95 GiB/rank reclaimed becomes arena: ~7,500 slots vs 6,248. Two contexts at 200704 is a capacity contract, not a throughput one (2-way concurrency summed BELOW parity at 4064 ctx), so the slot buys the VRAM headroom long generations need. Refs [[dankserver-qwen38-fp8-expert-offload-tuning]]. - scripts/frank-primary.env: MAXLEN 200704, SEQS 2, KV_BYTES 3.15e9, MNBT 1024, rewritten shape comment - scripts/serve-frank-primary.sh: header comment updated - tests/frank_primary_unit.bats: assertions updated, new seam pins the arena-floor pool law (the pool must move with a context change or the reduction buys nothing) - docs + AGENTS.md: fleet topology and deployed shape corrected
130 lines
7.2 KiB
Bash
130 lines
7.2 KiB
Bash
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
|
|
#
|
|
# 2026-10-01 cutover to the BF16-KV no-spec incumbent (the overnight redo
|
|
# campaign's verdict, Frank's call): same digest-pinned engine and payload as
|
|
# the 2026-09-29 cutover, but the KV cache is UNQUANTIZED (auto = bf16) and
|
|
# speculative decode is GONE. Git tag frank-primary-fp8-mtp-20261001 is the
|
|
# last pre-cutover commit; the old shape (fp8 KV + MTP4) is in history and
|
|
# restorable from the tag.
|
|
#
|
|
# Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited):
|
|
# * MAXLEN 241664 x SEQS 3 (724,992 tokens) at bf16 KV is the DEPLOYED
|
|
# shape (2026-10-01 night ladder): KV_BYTES 10,184,000,000 PER RANK,
|
|
# engine measured "KV cache size: 732,359 tokens; Maximum concurrency
|
|
# for 241,664 tokens per request: 3.03x". 262144 and 245760 both OOMed
|
|
# at graph capture (76 MiB short / 0 bytes free); 237568 passed at
|
|
# 718,606 tokens and the +4096 bonus step passed and won.
|
|
# * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was
|
|
# the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because
|
|
# the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes
|
|
# 11.05 GiB/rank and boots clean — the ceiling was spec's, not VRAM's.
|
|
# * bf16 KV doubles the attention-KV bytes/token (measured 14,047 vs
|
|
# 7,926) and costs ~14% decode vs fp8 at 4064 ctx (49.25 -> 36.42 with
|
|
# the gather win) — traded deliberately for clean cache + 262k windows.
|
|
# * GATHER 32x64 is the night's biggest lever: R4D_LRU_GATHER_CHUNKS=32 /
|
|
# R4D_LRU_GATHER_LANES=64 measured +18.7% decode vs default geometry,
|
|
# and the win grows with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 fp8
|
|
# era). Do not drop it.
|
|
# * MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s
|
|
# vs 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5%
|
|
# and arena. 1024 keeps the workspace floor so the arena stays max.
|
|
# * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
|
|
# prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
|
|
# roughly half the gather traffic was waste. Never re-add
|
|
# --expert-prefetch without a fresh A/B.
|
|
# * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
|
|
# offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
|
|
# numerics.
|
|
#
|
|
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
|
|
# prefixed FRANK_PRIMARY_ so none can leak.
|
|
#
|
|
|
|
# --- runtime ----------------------------------------------------------------
|
|
# Digest-pinned: the floating :latest tag moved once already. Local alias
|
|
# qwen-fp8-runtime:tuned-20260929 (image ID 3322dbbbc96d) points at the same
|
|
# bytes in rootless storage.
|
|
FRANK_PRIMARY_IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
|
|
|
|
# --- payload ----------------------------------------------------------------
|
|
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8
|
|
FRANK_PRIMARY_MODEL_SHARDS=131
|
|
|
|
# --- identity ---------------------------------------------------------------
|
|
# The harness models.yml id stays frank-primary (provider 192.168.0.235:7770).
|
|
FRANK_PRIMARY_CONTAINER=frank-primary-qwen
|
|
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
FRANK_PRIMARY_PUBLISH=7770:7770
|
|
# Containers removed before every start: the retired R9V identity and the ad
|
|
# hoc experiment identity (both hold the GPUs / name registrations).
|
|
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
|
|
|
# --- qualified engine shape (2026-10-02, Frank's own correction) ---
|
|
# 2026-10-01 left a shape that did not stay up: bf16 KV at 2 x 262144 with
|
|
# --expert-precise-memory crash-looped at graph capture (the arena is
|
|
# DEVICE-resident and the shape had 0.13 GiB to spare), and the plain bf16
|
|
# 2 x 262144 boot booted but left 0 bytes free at runtime. This is the
|
|
# correction: FP8 KV, TWO windows, each 200704 tokens, and the pool cut so
|
|
# the expert arena gets the room back.
|
|
#
|
|
# LADDER (2026-10-01 night, subtract-16k method): 262144 and 245760 OOM at
|
|
# graph capture; 237568 passed at 718,606 tokens; 241664 x 3 passed at
|
|
# 732,359 tokens / 3.03x but is the shape this cutover retired. 200704 was
|
|
# chosen to sit below the 241664 rung while leaving a full 2.95 GiB/rank of
|
|
# pool to cut loose for the arena — the two contexts are a capacity
|
|
# contract, not a throughput one (2-way concurrency summed BELOW parity at
|
|
# 4064 ctx), so the slot buys the VRAM headroom long generations need.
|
|
#
|
|
# KV_BYTES is PER RANK. fp8 costs 7,701 B/token (measured 6,138 attention
|
|
# + 1,563 mamba/side, dtype-independent). 2 x 200704 = 401,408 tokens
|
|
# -> 3.091e9 B floor, pinned at 3.15e9 (~405,000 tokens, ~44 MB margin —
|
|
# the same margin convention the 6.1e9 shape used).
|
|
#
|
|
# THE ARENA IS WHAT REMAINS: the fork's expert planner sizes it AFTER the
|
|
# KV override, so every byte cut here is arena. The 6.1e9 shape ran an
|
|
# arena of 6,248 slots; this one reclaims ~2.95 GiB/rank, roughly 1,260
|
|
# more slots at 2.344 MiB each. That is the point of the change — the
|
|
# gather read path is the measured decode bound, and a bigger arena means
|
|
# more experts resident instead of streamed.
|
|
#
|
|
# MNBT 1024 (down from 2048): 1024-4096 are interchangeable on decode AND
|
|
# identical on arena (the sweep measured the same slot count at every
|
|
# rung <= 4096), so this is a pure workspace-floor choice, not a decode
|
|
# trade. The floor matters because the arena budget is what is left.
|
|
#
|
|
# PRECISE MEMORY stays DROPPED: it crashed the 2 x 262144 bf16 boot
|
|
# (weights 6.28 + pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45
|
|
# = 31.73 of 31.86 GiB, ~0.13 GiB free, NRestarts=3). The flag survives in
|
|
# the serve script as an opt-in; re-enable it only with a pool/context
|
|
# reduction that actually pays for the extra overhead.
|
|
#
|
|
# TUNABLEOP stays OFF (silently skipped under offload unless
|
|
# CLAV_TUNABLEOP_WITH_OFFLOAD=1, and then corrupts numerics).
|
|
#
|
|
# SEQS 2 x MAXLEN 200704 at FP8 KV, pool cut for the arena. No
|
|
# speculative decode, no prefetch.
|
|
FRANK_PRIMARY_MAXLEN=200704
|
|
FRANK_PRIMARY_SEQS=2
|
|
FRANK_PRIMARY_KV_BYTES=3150000000
|
|
FRANK_PRIMARY_KV_DTYPE=fp8
|
|
FRANK_PRIMARY_MNBT=1024
|
|
FRANK_PRIMARY_GATHER_CHUNKS=32
|
|
FRANK_PRIMARY_GATHER_LANES=64
|
|
FRANK_PRIMARY_UTIL=0.98
|
|
|
|
# --- client contract: reasoning split, tools, preserve_thinking ----------------
|
|
# Without --reasoning-parser the engine leaves the thinking text INLINE in
|
|
# `content` with a dangling `</think>` and a null `reasoning` field (measured
|
|
# 2026-10-01: content = 'We need answer...Brief.\n</think>\n\n391') — the
|
|
# malformed thinking block clients choke on. The qwen3 parser splits it
|
|
# properly. The GLM rollback (d91379d) had dropped all three of these; the
|
|
# working values are the ones ecaad36 shipped for this model.
|
|
FRANK_PRIMARY_REASONING_PARSER=qwen3
|
|
# Tools: --enable-auto-tool-choice + the qwen3_xml parser is what makes
|
|
# tool_calls work here (without it, tool_choice auto returns 400).
|
|
FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
|
|
# preserve_thinking keeps prior thinking blocks in the rendered prompt across
|
|
# turns; enable_thinking makes thinking the server-side default. Verified the
|
|
# payload's tokenizer_config template consumes both keys.
|
|
FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
|