dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: bf16 no-spec cutover (262144x3, gather 32x64, MNBT 1024)

Browse Source 
Overnight redo campaign verdict + Frank's call: the fp8+MTP4 shape is retired
to git tag frank-primary-fp8-mtp-20261001. The new incumbent: KV cache
unquantized (auto), no speculative decode, KV_BYTES 11,047,010,304/rank
(engine-measured 798,280 tokens, 3.05x concurrency at 262,144-token
requests), MNBT 1024, MoE gather 32x64 (+18.7% decode, grows with depth).
Measured at the deployed shape: 39.38/38.79/36.42 tok/s (166/1087/4064 ctx).
Seam re-pinned: the 7.2 GiB/rank KV ceiling was the MTP4 workspace, not VRAM;
spec decode absent by measurement; bf16 trade documented (clean cache +
262k windows for ~14% decode at 4064 ctx).
This commit is contained in:
Frank committed 2026-10-01 01:54:27 -07:00
1 parent 26f6f82c98
commit c1248d4a5d
3 files changed
+118 -83
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/frank_primary_unit.bats
+70 -48
View File
@@ -2,25 +2,23 @@
# Seam: frank-primary — the Dankserver LLM endpoint on :7770.
#
# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
# 2026-10-01 rollback to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4, which
# reversed the failed GLM/DFlash2 experiment): the serve script drives
# `podman run` DIRECTLY (the image entrypoint execs `vllm serve`), with every
# engine flag qualified by boot probe and carried in the repo env file. The
# seam pins what the flags alone do not say: the KV qualification law
# (7.2 GiB/rank boots at 729,545 tokens / 3.65x concurrency; 9 GiB and 16 GiB
# both OOM the worker — 4 full-length 200k contexts do NOT fit), the fp8-KV
# qualification (e4m3 for BOTH K and V — the stock flag quantizes them
# together), the automatic-offload law (the fork plans the tiered-expert
# offload itself; no flag exists to set), the self-healing law (the run
# returns before the engine is up, so the serve script must wait on the
# container for Restart=on-failure to mean anything), the memlock law that
# makes the pinned host rows legal, the deployer's image/payload gates with
# the false-green MainPID check, and the measured laws: expert prefetch
# ABSENT (36% precision, costs ~40-51% decode — deleting the flag moved
# decode 26.8 -> 40.6 tok/s), MTP4 PRESENT (37.4-40.6 tok/s at 54% acceptance
# vs the 12.11 tok/s no-spec baseline; the earlier "MTP buys zero" reading
# carried the prefetch tax), and NO tool dialect (the GLM era added
# --enable-auto-tool-choice; the 2339fae trio does not serve tools).
# 2026-10-01 BF16 cutover, the overnight redo campaign's verdict): the serve
# script drives `podman run` DIRECTLY (the image entrypoint execs
# `vllm serve`), with every engine flag qualified by boot probe and carried
# in the repo env file. The seam pins what the flags alone do not say: the
# KV qualification law (11.05 GiB/rank boots bf16 at 798,280 tokens / 3.05x
# concurrency; the old 7.2 GiB ceiling was the MTP4 workspace, not VRAM),
# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
# clean cache + 262k windows), the automatic-offload law (the fork plans
# the tiered-expert offload itself; no flag exists to set), the self-healing
# law (the run returns before the engine is up, so the serve script must
# wait on the container for Restart=on-failure to mean anything), the
# memlock law that makes the pinned host rows legal, the deployer's
# image/payload gates with the false-green MainPID check, and the measured
# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
# no-spec family measured fastest once the prefetch tax was removed; MTP
# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
# MNBT 1024 (Frank's lock), and NO tool dialect.
load helpers
@@ -76,19 +74,22 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
}
@test "the qualified engine shape is in the env file" {
# 2026-09-29 boot probe: 200,000 x 3 seqs at fp8 KV, util 0.98, KV_BYTES
# 7.2 GiB PER RANK boots at "GPU KV cache size: 729,545 tokens" and
# reports max concurrency 3.65x for 200,000-token requests. KV_BYTES is
# PER RANK — sizing it to the global requirement OOMs the worker
# (measured twice: 16 GiB and 9 GiB both die with ~7.5 GiB free). So 4
# full-length 200k contexts do NOT fit: four clients share the ~730k
# token pool and each runs below 200k when all are deep.
grep -q '^FRANK_PRIMARY_MAXLEN=200000$' "$ENV_FILE"
# 2026-10-01 boot probe: 262,144 x 3 seqs at bf16 KV, util 0.98,
# KV_BYTES 11,047,010,304 PER RANK boots at "KV cache size: 798,280
# tokens" and reports max concurrency 3.05x for 262,144-token requests
# — all three full windows fit with an 11,848-token margin. The old
# "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 drafter's
# workspace reservation, not a VRAM wall: no-spec takes 11.05 GiB/rank
# and boots clean.
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=7730934784$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=11047010304$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
}
@@ -97,7 +98,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
grep -q -- '--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES"' "$SERVE"
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
grep -q -- '--speculative-config \"{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":\$FRANK_PRIMARY_SPEC}\"' "$SERVE"
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
grep -q -- '-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"' "$SERVE"
! grep -q -- '--speculative-config' "$SERVE"
grep -q -- '--tensor-parallel-size 2' "$SERVE"
grep -q -- '--enable-prefix-caching' "$SERVE"
grep -q -- '--trust-remote-code' "$SERVE"
@@ -125,28 +129,46 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- '--ipc=host --ulimit memlock=-1:-1' "$SERVE"
}
@test "MTP4 spec decode is PRESENT BY MEASUREMENT" {
# 2026-09-29 campaign at prefetch OFF: MTP4 measures 40.57/37.44/37.57
# tok/s (166/1087/4064 prompt tokens) at 54% acceptance vs the no-spec
# baseline 12.11 tok/s (dead flat, expert-gather bound) — a measured
# 3.1-3.4x win. MTP3 measured 24.4 (still 2x baseline), MTP6 collapses
# (17.4% acceptance). The earlier "MTP buys zero decode" reading carried
# the prefetch tax and is retired with it. ngram-mod never booted on this
# model (fork bug: PLE inputs prepared only on the MTP path).
grep -q '^FRANK_PRIMARY_SPEC=4$' "$ENV_FILE"
grep -q -- '--speculative-config' "$SERVE"
@test "speculative decode is ABSENT BY MEASUREMENT" {
# The 2026-10-01 overnight redo (prefetch OFF) re-measured the whole
# spec family against no-spec at matched geometry: no-spec measured
# 41.46/41.29/41.47 tok/s while the spec-era incumbent's 37-40 carried
# the prefetch tax, and MTP's 2.6 GiB/rank drafter reservation halves
# the pool (729,545 vs 975,342 tokens). The MTP3/4/6 untaxed re-measures
# never completed; until one shows a real win, no spec is the incumbent.
! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
! grep -q -- '--speculative-config' "$SERVE"
}
@test "fp8 KV is PRESENT — q8/q8 qualified on this arm" {
# --kv-cache-dtype fp8 is e4m3 for BOTH K and V (the stock flag quantizes
# them together; a V-only quant needs a fork patch). The GLM-era MLA
# concern (sparse-triton MLA decode path refusing fp8 KV) does not apply
# to this arm: qwen's attention path accepted the dtype and a completion
# probe qualified it 2026-09-29.
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
@test "bf16 KV is PRESENT — clean cache won the trade" {
# --kv-cache-dtype auto = bf16 attention KV. Doubles attention-KV
# bytes/token (measured 14,047 vs 7,926) and costs ~14% decode vs fp8
# at 4064 ctx (49.25 -> 36.42 with the gather win) — traded deliberately
# for clean cache and 262k windows (Frank's call). The fp8 shape is
# restorable from git tag frank-primary-fp8-mtp-20261001.
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
}
@test "gather geometry 32x64 is PRESENT BY MEASUREMENT" {
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64: the overnight
# sweep's biggest lever, +18.7% decode vs default geometry, and the win
# GROWS with depth (39.5@166 -> 54.1@1087 -> 49.3@4064 in the fp8 era).
# g64x128 is past the peak; g8x16 mild. They are container ENV (the
# fork's knobs), not engine flags — passed through with -e.
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
}
@test "MNBT is locked at 1024" {
# Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s);
# 1024-4096 are interchangeable on decode; 12288 costs 2.5% plus arena.
# 1024 keeps the workspace floor so the expert arena stays max.
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
}
@test "expert prefetch is absent BY LAW, not by omission" {
# Telemetry (2026-09-29): the prefetcher served 13,369 of 36,720
# prefetched slabs (36% precision) — roughly half the gather traffic was