dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: 3 slots x 200704, fp8, pool sized for 2 contexts

Browse Source 
Re-applies the desired shape incrementally on the proven-good fp8 3x262144
base, after bisecting why the first attempt (30904ba) crash-looped.

Root cause: `--max-num-seqs 2` cannot compile on this model. AOT
build_guards(strict_error=True) inside profile_run -> _dummy_run specializes
`query_start_loc`'s width to SEQS + 1 (3 at SEQS 2, 4 at SEQS 3) and the
ConstraintViolationError kills engine init ~7 min in. MNBT is orthogonal:
SEQS 2 failed at MNBT 1024 AND 2048; SEQS 3 passed at both. KV dtype was a
red herring. Measured by four single-variable boot probes.

Arena lever measured: the expert arena is sized from `max_model_len` via the
registry-priced KV reserve (9190 B/token x max_model_len), NOT from
--kv-cache-memory-bytes. 262144 -> 14.30 GiB / 6,248 slots; 200704 ->
14.87 GiB / 6,496 slots. The 6.1e9 -> 3.15e9 pool cut bought zero arena and
is a runtime-headroom lever.

Shape: MAXLEN 200704, SEQS 3 (unchanged - the only slot count that compiles),
KV_BYTES 3.15e9/rank = 2.04x full-length concurrency at 7,701 B/token fp8,
MNBT 1024, gather 32x64, no spec decode, no prefetch, no precise-memory.
This commit is contained in:
Frank
2026-10-02 11:33:27 -07:00
parent e460bbbab5
commit c769ff2f63
6 changed files with 218 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-primary.env
+61 -71
View File
@@ -1,40 +1,10 @@
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
#
# 2026-10-01 cutover to the BF16-KV no-spec incumbent (the overnight redo
# campaign's verdict, Frank's call): same digest-pinned engine and payload as
# the 2026-09-29 cutover, but the KV cache is UNQUANTIZED (auto = bf16) and
# speculative decode is GONE. Git tag frank-primary-fp8-mtp-20261001 is the
# last pre-cutover commit; the old shape (fp8 KV + MTP4) is in history and
# restorable from the tag.
#
# Every number here was QUALIFIED by boot probe on 2026-10-01 (not inherited):
# * MAXLEN 241664 x SEQS 3 (724,992 tokens) at bf16 KV is the DEPLOYED
# shape (2026-10-01 night ladder): KV_BYTES 10,184,000,000 PER RANK,
# engine measured "KV cache size: 732,359 tokens; Maximum concurrency
# for 241,664 tokens per request: 3.03x". 262144 and 245760 both OOMed
# at graph capture (76 MiB short / 0 bytes free); 237568 passed at
# 718,606 tokens and the +4096 bonus step passed and won.
# * KV_BYTES is PER RANK. The old "7.2 GiB ceiling, 9/16 GiB OOM" law was
# the MTP4 WORKSPACE: the spec-era pool measured 729,545 tokens because
# the drafter reserved ~2.6 GiB/rank beside the pool. No-spec bf16 takes
# 11.05 GiB/rank and boots clean — the ceiling was spec's, not VRAM's.
# * bf16 KV doubles the attention-KV bytes/token (measured 14,047 vs
# 7,926) and costs ~14% decode vs fp8 at 4064 ctx (49.25 -> 36.42 with
# the gather win) — traded deliberately for clean cache + 262k windows.
# * GATHER 32x64 is the night's biggest lever: R4D_LRU_GATHER_CHUNKS=32 /
# R4D_LRU_GATHER_LANES=64 measured +18.7% decode vs default geometry,
# and the win grows with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 fp8
# era). Do not drop it.
# * MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s
# vs 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5%
# and arena. 1024 keeps the workspace floor so the arena stays max.
# * PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the
# prefetcher at 36% precision (36,720 slabs prefetched, 13,369 served) —
# roughly half the gather traffic was waste. Never re-add
# --expert-prefetch without a fresh A/B.
# * TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert
# offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts
# numerics.
# 2026-10-02 shape D: fp8 KV, MAXLEN 200704 x SEQS 3, the KV pool cut to ~2
# full-length contexts. The window shrinks from the native 262144 to 200704 and
# the pool from 6.1e9 to 3.15e9; --max-num-seqs stays 3 because SEQS=2 CANNOT
# boot this model (measured 2x2 below). Same digest-pinned tcclaviger RDNA4
# fork. Rollback for the fp8+MTP4 era is git tag frank-primary-fp8-mtp-20261001.
#
# HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE — every key here is
# prefixed FRANK_PRIMARY_ so none can leak.
@@ -59,43 +29,63 @@ FRANK_PRIMARY_PUBLISH=7770:7770
# hoc experiment identity (both hold the GPUs / name registrations).
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
# --- qualified engine shape (2026-10-01, RESTORED KNOWN-GOOD: fp8 3 windows) ---
# bf16 at 2 x 262144 did NOT stay up: with and without --expert-precise-memory
# the boot failed (`Engine core initialization failed`, NRestarts climbing).
# The last shape measured serving was fp8 3 x 262144 / MNBT 2048 / KV 6.1e9
# (KV 792,069 tokens, 3.02x, arena 6,248, ~3.6 GiB free) and is restored here
# while the bf16 path is investigated in a window (see log.md 2026-10-01).
# Candidate next test: bf16 with SEQS=3 to isolate the KV dtype from the
# sequence count, and/or --enforce-eager to bypass the dynamo constraint path.
# SEQS 2 x MAXLEN 262144 at BF16 KV (Frank's call, 2026-10-01): two contexts of
# the model's FULL native window with UNQUANTIZED KV, plus
# --expert-precise-memory to hand the arena the compile/graph-capture headroom.
# Pool: bf16 costs 13,839 B/token (measured: 11,047,010,304 B -> 798,280 tokens
# on the 3x241664 bf16 boot; decomposes as 12,276 attention + 1,563 mamba/side
# caches, against fp8's 6,138 + 1,563 = 7,701). 2 x 262144 = 524,288 tokens
# -> 7.256e9 B floor, pinned at 7,300,000,000 (~527,500 tokens, 3,206 margin).
# Budget (per rank): 31.22 GiB total; non-expert weights 6.28, graphs 0.55,
# non-torch ~1.45, pool 6.76 -> ~29.35 used, ~2.5 GiB free (measured basis:
# the fp8 3-window shape used 28.27 and left 3.59).
# PRECISE MEMORY: DROPPED 2026-10-01 after it crashed the boot in combination
# with bf16. The flag selects the fitted 5.25 GiB overhead instead of the
# 7.60 GiB default, moving the plan's `room` and therefore the arena
# (6,248 -> ~7,278 slots). But the arena is DEVICE-resident, and the budget
# for this shape does not have 2.35 GiB to spare:
# weights 6.28 + pool 6.80 (bf16 2 x 262144) + graphs 0.55 + arena 16.65
# + non-torch ~1.45 = 31.73 of 31.86 GiB => ~0.13 GiB free at boot
# measured as a CrashLoop: every attempt reached graph capture, the engine
# exited cleanly, systemd restarted it (NRestarts=3 at ~6.7 min each).
# With the default overhead the arena is 14.31 GiB and the same shape totals
# ~29.4 GiB, leaving ~2.4 GiB for capture. Re-enable the flag only with a
# pool/context reduction that pays for it.
# MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so
# 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096.
FRANK_PRIMARY_MAXLEN=262144
# --- qualified engine shape (2026-10-02, shape D) ---------------------------
# WHAT THE SLOT COUNT DOES AND DOES NOT DO. The endpoint advertises 3 slots
# (SEQS 3), but the POOL, not the slot count, decides how many full-length
# conversations are resident. fp8 measures 7,701 B/token (6,138 attention +
# 1,563 mamba/side; measured: 6,100,000,000 B -> 792,069 tokens), so one
# 200704-token request costs 1.546e9 B. KV_BYTES 3.15e9 admits 2.04 such
# requests; shorter requests use the spare blocks, and a third LONG request
# queues rather than being refused.
#
# WHY NOT --max-num-seqs 2 (measured 2026-10-02, single-variable boot probes
# against the then-deployed known-good shape, production stopped for each):
# SEQS 2 + MNBT 1024 -> Engine core initialization failed
# SEQS 2 + MNBT 2048 -> Engine core initialization failed
# SEQS 3 + MNBT 1024 -> healthy (792,069 tokens, 3.02x, arena 6,248)
# SEQS 3 + MNBT 2048 -> healthy (the deployed incumbent)
# The failure is torch.fx ConstraintViolationError: query_start_loc.size()[0]
# "marked dynamic but your code specialized it to be a constant (3)", raised
# from AOT build_guards(strict_error=True) during profile_run -> _dummy_run.
# model_runner.py sizes that input buffer max_num_reqs + 1 and _dummy_run sets
# num_reqs = min(max_num_batched_tokens, max_num_reqs), so the width is
# SEQS + 1 for ANY MNBT >= SEQS: 3 at SEQS 2, 4 at SEQS 3. MNBT only moves the
# compile RANGE ((1,1024) vs (1,2048)) and cannot dodge it — the 2x2 above is
# why MNBT is not a lever here. --enforce-eager bypasses the path but loses
# CUDA graphs. Same failure class killed the 2026-10-01 bf16 2 x 262144 shape.
#
# THE ARENA LEVER IS max_model_len, NOT KV_BYTES (measured 2026-10-02). The
# planner sizes expert cache as budget - non-expert - KV reserve - graphs -
# overhead, and the KV reserve is registry-priced at 9190 B/token x max_model_len:
# MAXLEN 262144 -> reserve 2.48 GiB -> arena 14.30 GiB / 6,248 slots
# MAXLEN 200704 -> reserve 1.91 GiB -> arena 14.87 GiB / 6,496 slots
# d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
# pool cut buys ZERO arena; it buys runtime HEADROOM (the shape it replaced
# died with 0 bytes free on a real 995-token request). To grow the arena,
# lower MAXLEN — not KV_BYTES.
#
# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
# measured +18.7% decode vs default geometry and the win grows with depth.
# Do not drop it.
# MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs
# 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5% and
# arena. 1024 keeps the workspace floor so the arena stays max. (Side note
# measured 2026-10-02: MNBT 2048 compiles the first graph in 26 s vs 172 s at
# 1024 — a startup-cost lever, not a correctness one.)
# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
# TunableOp stays OFF: CLAV_TUNABLEOP_SWEEP silently skips under expert offload
# unless CLAV_TUNABLEOP_WITH_OFFLOAD=1, and forcing it corrupts numerics.
# --expert-precise-memory stays DROPPED: it crashed the 2 x 262144 bf16 boot
# (0.13 GiB free of 31.86 GiB, NRestarts=3 at ~6.7 min each). It survives in
# the serve script as an opt-in; re-enable only with a pool/context reduction
# that pays for the 2.35 GiB it moves into the arena.
FRANK_PRIMARY_MAXLEN=200704
FRANK_PRIMARY_SEQS=3
FRANK_PRIMARY_KV_BYTES=6100000000
FRANK_PRIMARY_KV_BYTES=3150000000
FRANK_PRIMARY_KV_DTYPE=fp8
FRANK_PRIMARY_MNBT=2048
FRANK_PRIMARY_MNBT=1024
FRANK_PRIMARY_GATHER_CHUNKS=32
FRANK_PRIMARY_GATHER_LANES=64
FRANK_PRIMARY_UTIL=0.98
@@ -114,4 +104,4 @@ FRANK_PRIMARY_TOOL_PARSER=qwen3_xml
# preserve_thinking keeps prior thinking blocks in the rendered prompt across
# turns; enable_thinking makes thinking the server-side default. Verified the
# payload's tokenizer_config template consumes both keys.
FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS='{"enable_thinking": true, "preserve_thinking": true}'
scripts/frank-primary.service
+5 -4
View File
@@ -1,8 +1,9 @@
# frank-primary.service — tcclaviger RDNA4 vLLM endpoint (orcarouter
# Qwen3.8-Flash-Next-Uncensored-FP8, MTP4 spec decode, KV q8/q8, 600k total
# context over 3 sessions, TP=2, tiered experts, 131-shard payload) on the
# 2x R9700 pair, dankserver :7770. 2026-09-29 cutover from the R9V radiance
# fork (git tag frank-primary-r9v-q8-20260929 is the pre-cutover state; its
# Qwen3.8-Flash-Next-Uncensored-FP8, fp8 KV, NO spec decode, 3 slots x 200704
# context with the pool sized to ~2 full-length conversations, TP=2, tiered
# experts, 131-shard payload) on the 2x R9700 pair, dankserver :7770.
# 2026-09-29 cutover from the R9V radiance fork (git tag
# frank-primary-r9v-q8-20260929 is the pre-cutover state; its
# launcher/shim/setup-state machinery is retired and the state dirs stay on
# disk as rollback).
#
scripts/serve-frank-primary.sh
+12 -9
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# serve-frank-primary.sh — frank-primary endpoint (tcclaviger RDNA4 vLLM fork,
# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 spec decode, 2x R9700,
# orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, 2x R9700,
# dankserver :7770; 2026-09-29 cutover from the R9V radiance fork — git tag
# frank-primary-r9v-q8-20260929 is the pre-cutover state).
#
@@ -11,14 +11,17 @@
# every flag here was qualified by boot probe on 2026-09-29 — see
# frank-primary.env for the qualification record).
#
# Shape (2026-10-01 bf16 cutover): MAXLEN 262144 x 3 sessions (786,432
# tokens), KV UNQUANTIZED (--kv-cache-dtype auto = bf16; the fp8 q8/q8 shape
# is retired to git tag frank-primary-fp8-mtp-20261001), KV_BYTES
# 11,047,010,304 PER RANK (the engine measured 798,280 tokens of capacity /
# 3.05x concurrency), NO speculative decode, --max-num-batched-tokens 1024
# (Frank's lock), MoE gather geometry R4D_LRU_GATHER_CHUNKS=32 /
# R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows with depth), prefix caching,
# expert prefetch OFF (36% precision measured), TunableOp off.
# Shape (2026-10-02 shape D): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
# conversations at ~2 full-length (2.04x); shorter requests use the spare
# blocks. --max-num-seqs stays 3 because SEQS=2 CANNOT boot this model
# (torch.fx ConstraintViolationError on query_start_loc, measured at both
# MNBT 1024 and 2048 — the 2x2 lives in frank-primary.env). NO speculative
# decode, --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
# with depth), prefix caching, expert prefetch OFF (36% precision measured),
# TunableOp off. The native-262144 and fp8+MTP4 shapes are retired to git tag
# frank-primary-fp8-mtp-20261001.
#
# The root unit (scripts/frank-primary.service, User=dank,
# LimitMEMLOCK=infinity) grants the pin budget for the tiered-expert host