From c682c6329c6096e3f1421334b13d266bccf522d8 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 3 Oct 2026 02:22:49 -0700 Subject: [PATCH] =?UTF-8?q?frank-primary:=20shape=20F=20=E2=80=94=20MNBT?= =?UTF-8?q?=202048,=20util=200.982,=20CUDA-graph=20plan=20term=20pinned=20?= =?UTF-8?q?to=20the=20measured=20capture?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The arena is budget - non-expert - KV reserve - graphs - overhead. With util capped at 0.982 (never 1.00) the graph estimate is the only plan term this repo can shrink: the fork charges an analytic 256 MiB + 52 MiB/graph = 0.55 GiB where the engine's own post-capture line reports 0.31 GiB, so pricing it at 256 + 12/graph returns ~0.24 GiB to the expert arena. MNBT 1024 -> 2048 is arena-free: the registry overhead is base + 0.22 GiB per 1024 ABOVE its nbt 4096 anchor, and the fitted 5.25 GiB was fitted AT 4096. --- AGENTS.md | 34 ++++++++++-- scripts/frank-primary.env | 53 +++++++++++++++---- scripts/serve-frank-primary.sh | 14 +++-- ...server-qwen38-fp8-expert-offload-tuning.md | 52 ++++++++++++++++-- tests/frank_primary_unit.bats | 46 +++++++++++----- 5 files changed, 165 insertions(+), 34 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9945ea9..db940fb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default → expert arena **17.22 GiB / 7,522 slots**, +16 %), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) plus the overhead term (`--expert-precise-memory` → 7,522) — util 0.98, and the client contract wired: `--reasoning-parser … +| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default → expert arena **17.22 GiB / 7,522 slots**, +16 %), **`VLLM_CUDAGRAPH_MEM_BASE_MB=256` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured 0.31 GiB capture instead of the analytic 0.55 GiB → +0.24 GiB of arena), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) plus the overhead term (`--expert-precise-memory` → 7,522) — util 0.982 (Frank's cap; the field is `le=1`, and the ~3.5 GiB of free VRAM the driver reports is estimator slack, not allocatable arena), and the client contract wired: `--reasoning-parser … | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 | | Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 | @@ -433,12 +433,13 @@ The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666 digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** (~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**, **fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since -2026-10-02 the **shape-E** geometry: `--max-model-len 200704` × 3 slots with +2026-10-03 the **shape-F** geometry (shape E + MNBT 2048 + util 0.982 + the +CUDA-graph plan term pinned): `--max-model-len 200704` × 3 slots with `--kv-cache-memory-bytes 3,150,000,000`/rank, so the pool admits **~2 full-length conversations** (measured live: `398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`) while the argv keeps the -only slot count that compiles (below), plus `--expert-precise-memory` -(`runtime overhead 5.25 GiB`, `expert cache space 17.22 GiB / 7,522 slots`), +only slot count that compiles (below), plus `--expert-precise-memory`, +`VLLM_CUDAGRAPH_MEM_BASE_MB=256` / `_PER_GRAPH_MB=12`, `GPU_MAX_HW_QUEUES=2` and a mounted `/root/.cache`. `models.yml` carries `contextWindow: 200704` for this endpoint — the served window, not the model's native 262144, so the harness compacts before vLLM answers a longer prompt @@ -455,7 +456,8 @@ sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets ((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape. -**The expert arena has TWO levers: `max_model_len` and the overhead term.** +**The expert arena has THREE reachable levers: `max_model_len`, the overhead +term, and the CUDA-graph estimate.** The fork's planner sizes expert cache as `budget − non-expert − KV reserve − graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token × `max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**; @@ -473,6 +475,28 @@ concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV headroom collapses the offload hit rate. +**Free VRAM on this endpoint is NOT arena headroom (2026-10-03 audit).** +`rocm-smi` reads ~3.5 GiB free per card after hours of real traffic, but the +planner charges `budget = total × util` against terms that overstate reality: +the KV reserve is registry-priced at 9190 B/token (`1.91 GiB` at 200704) while +the real pool bills 2.93 GiB at 7,892 B/token, and the precise overhead (5.25 +GiB) is far above the idle-measured figure. So the visible free space is the +estimator gap, not allocatable arena. `gpu_memory_utilization` is a real arena +lever (each +0.01 = +0.32 GiB) but the field is `Field(gt=0, le=1)` — **Frank +caps it at 0.982, never 1.00** (this endpoint has twice died at the ceiling). +What IS reachable from flags: `VLLM_CUDAGRAPH_MEM_BASE_MB` / +`_PER_GRAPH_MB`, read at call time by `estimate_cudagraph_memory()` — the +analytic default (256 MiB + 52 MiB/graph) charged 0.55 GiB where the engine's +own post-capture line reported 0.31 GiB, so pricing it honestly returns +~0.24 GiB to the arena. Everything else needs a registry edit inside the image +(the fork's log line says as much: *"Update the registry entry if this +drifts"*). Verify every deploy by comparing the `Estimated CUDA graph memory` +line with `Graph capturing finished … took X GiB`; if capture exceeds the +estimate, raise `_PER_GRAPH_MB` back toward 52. MNBT 1024 → 2048 costs ZERO +arena: the registry overhead is `base + 0.22 GiB × max(0,(nbt−4096)/1024)` +anchored at nbt 4096, so every rung ≤4096 charges the same 5.25 GiB — and the +fitted figure was fitted AT 4096. + **A container's compile cache does NOT survive `podman rm -f`, and on this endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no `/root/.cache` at all** (fresh container: `No such file or directory`); after a diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index 2c5fe5f..5bdb478 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -1,5 +1,15 @@ # frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770). # +# 2026-10-03 shape F: MNBT 2048 and the CUDA-graph plan term pinned to the +# measured capture cost. The expert arena is `budget - non-expert - KV reserve +# - graphs - overhead`, and with UTIL capped (see FRANK_PRIMARY_UTIL) the graph +# estimate is the only plan term this repo can reach: the fork charges an +# analytic 0.55 GiB (256 MiB base + 52 MiB/graph) where the engine's own +# post-capture line reports 0.31 GiB on the heavier rank. Pricing it honestly +# returns 0.24 GiB to the arena. The rest of the idle free VRAM (~3.3 GiB/card) +# stays unreachable from flags: it is locked inside the registry's overhead +# (5.25 GiB) and KV-price (9,190 B/token vs the measured 7,892) terms, which +# live in the image, not in our argv or env. # 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and # --expert-precise-memory ON. The window shrink from the native 262144 and the # --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model — @@ -80,13 +90,18 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" # GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 # measured +18.7% decode vs default geometry and the win grows with depth. # Do not drop it. -# MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs -# 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5% and -# arena. 1024 keeps the workspace floor so the arena stays max. (Side note -# measured 2026-10-02: MNBT 2048 compiles the first graph in 26 s vs 172 s at -# 1024 — a startup-cost lever, not a correctness one. The compile RANGE is part -# of the cache key, so FRANK_PRIMARY_CACHE_DIR pays for whichever rung -# is chosen exactly once.) +# MNBT 2048 (Frank's call, 2026-10-03; the lock was 1024). The measured law +# still holds: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s), 1024-4096 +# are interchangeable on decode, 12288 costs 2.5% plus arena. 2048 is ARENA-FREE +# by construction: the registry's overhead model is +# base + 0.22 GiB x max(0, (nbt - 4096)/1024) with the anchor at nbt 4096, so +# every rung up to and including 4096 prices the SAME 5.25 GiB overhead — and +# the fitted 5.25 was fitted at nbt 4096, i.e. our 2048 sits UNDER the anchor, +# so the activation term is covered by a figure measured at a bigger batch. +# SEQS 3 boots at both 1024 and 2048 (measured 2026-10-02; the SEQS=2 +# ConstraintViolationError is a SEQS property, not an MNBT one). Costs exactly +# one recompilation: the compile RANGE is part of the cache key, so +# FRANK_PRIMARY_CACHE_DIR pays for whichever rung is chosen once. # PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at # 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the # gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B. @@ -149,10 +164,30 @@ FRANK_PRIMARY_SEQS=3 # as the shipped value; it remains the conservative arm if free VRAM ever binds. FRANK_PRIMARY_KV_BYTES=3150000000 FRANK_PRIMARY_KV_DTYPE=fp8 -FRANK_PRIMARY_MNBT=1024 +FRANK_PRIMARY_MNBT=2048 FRANK_PRIMARY_GATHER_CHUNKS=32 FRANK_PRIMARY_GATHER_LANES=64 -FRANK_PRIMARY_UTIL=0.98 +# UTIL is Frank's cap: 0.982 maximum, never 1.00. It is an ARENA lever — the +# fork plans `budget = device_total x util` — but a weak one here: 0.980 -> 0.982 +# moves the budget 31.22 -> 31.28 GiB, i.e. +28 slots. Raising it to 1.00 would +# add 0.64 GiB and Frank vetoed it: this endpoint has twice died at the ceiling +# (the 2x262144 bf16 boot at 0.13 GiB free, the 5x262144 boot at "0 bytes free" +# on a real 995-token request), so the margin stays. +FRANK_PRIMARY_UTIL=0.982 +# CUDA-graph plan term, pinned to the measured capture cost. expert_plan.py +# charges estimate_cudagraph_memory() = BASE_MB + PER_GRAPH_MB x n_graphs with +# defaults 256/52, and the estimator's own docstring says to tune these against +# the post-capture line. MEASURED on shape E (model_runner.py:1094, same +# 6-graph plan, same shape): "Graph capturing finished in 3 secs, took 0.31 GiB" +# on the heavier rank (TP0) and 0.31/0.17 across ranks — vs the 0.55 GiB the +# plan charged. 12 MiB/graph keeps ~10 MiB of margin over the fitted marginal +# cost (~10 MiB/graph) and, unlike VLLM_CUDAGRAPH_MEM_ESTIMATE_MB (an explicit +# TOTAL), scales correctly if a future MNBT or spec-decode change grows the +# ladder. Verify every deploy against the "Estimated CUDA graph memory" and +# "Graph capturing finished" lines; if capture exceeds the estimate, raise +# FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52. +FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256 +FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12 # --- optional engine knobs (unset == the flag is absent) --------------------- # FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds diff --git a/scripts/serve-frank-primary.sh b/scripts/serve-frank-primary.sh index 69c2b30..9455a0f 100755 --- a/scripts/serve-frank-primary.sh +++ b/scripts/serve-frank-primary.sh @@ -11,7 +11,8 @@ # every flag here was qualified by boot probe on 2026-09-29 — see # frank-primary.env for the qualification record). # -# Shape (2026-10-02 shape E): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES +# Shape (2026-10-03 shape F, on the 2026-10-02 shape E base): MAXLEN 200704 x 3 +# slots, fp8 KV, KV_BYTES # 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident # conversations at ~2 full-length (measured: 398,658 tokens = 1.99x); # shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2 @@ -21,12 +22,17 @@ # is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG # / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02 # retirement of MTP compared the wrong pair — see frank-primary.env. -# --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry +# --max-num-batched-tokens 2048 (Frank's call 2026-10-03; arena-free because the +# registry prices overhead flat up to its nbt 4096 anchor), MoE gather geometry # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows # with depth), prefix caching, expert prefetch OFF (36% precision measured), # TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead # instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%) -# and GPU_MAX_HW_QUEUES=2. The native-262144 and fp8+MTP4 shapes are retired to +# and GPU_MAX_HW_QUEUES=2; shape F pins the plan's CUDA-graph term to the +# measured capture cost (VLLM_CUDAGRAPH_MEM_BASE_MB / _PER_GRAPH_MB, container +# env — the analytic default charges 0.55 GiB where capture took 0.31 GiB, so +# the 0.24 GiB difference goes to the arena) and raises util 0.98 -> 0.982, +# Frank's cap. The native-262144 and fp8+MTP4 shapes are retired to # git tag frank-primary-fp8-mtp-20261001. # # THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob: @@ -138,6 +144,8 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \ ${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \ ${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \ ${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \ + ${FRANK_PRIMARY_CUDAGRAPH_BASE_MB:+-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"} \ + ${FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB:+-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"} \ ${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \ -v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \ "$FRANK_PRIMARY_IMAGE" /models \ diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index 5f3e4e5..414118d 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -275,14 +275,16 @@ Depth caveat: every number above is at a near-empty KV. Deep-context behavior (the gather win growing with depth, bf16's decode cost at depth) is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap. -### The deployed shape (2026-10-02 shape E: 3 slots x 200704, fp8, precise-memory arena) +### The deployed shape (2026-10-03 shape F: 3 slots x 200704, fp8, precise memory, MNBT 2048, graph term pinned) ``` FRANK_PRIMARY_MAXLEN=200704 200704-token window FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below) FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x) FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized) -FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024 +FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F) +FRANK_PRIMARY_UTIL=0.982 Frank's cap; budget = total x util +FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_* FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB) FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU) FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache @@ -290,9 +292,51 @@ R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env) NO --speculative-config, NO --expert-prefetch ``` +### Shape F (2026-10-03): where the free VRAM actually is + +`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB +used of 31.86 GiB, after hours of real traffic — so it is the high-water mark, +not an idle artifact). Growing the expert arena into it is NOT a matter of +raising `--gpu-memory-utilization`, because the arena is + +``` +arena = budget - non-expert - KV reserve - CUDA graphs - runtime overhead +budget = device_total x gpu_memory_utilization # expert_plan.py:567 +``` + +and every term on the right is either fixed or a lie: + +| term | plan charges | measured reality | reachable from our side? | +|---|---|---|---| +| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | yes, but capped — **Frank: 0.982 max, never 1.00** | +| `non-expert weights` | 6.28 GiB | checkpoint census | no | +| `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) | +| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` | +| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) | + +So shape F takes the two levers that exist: **util 0.98 -> 0.982** (+0.06 GiB) +and **the graph term priced at the capture cost** (0.55 -> 0.32 GiB, +0.23 GiB). +Together **+0.30 GiB / ~+130 arena slots** — and that is the honest ceiling from +flags. The other ~2.7 GiB is invisible to the planner: it is the gap between the +registry's conservative terms and reality, and the fork's own log line says +*"Update the registry entry if this drifts"* — i.e. it is an image-side change, +not a deploy. + +**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry +overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at +nbt 4096, so any rung up to 4096 charges the identical 5.25 GiB; the fitted 5.25 +was itself fitted at nbt 4096, which means 2048 is covered by an activation +figure measured at a **larger** batch. SEQS 3 boots at both 1024 and 2048 +(measured 2026-10-02 — the `query_start_loc` `ConstraintViolationError` is a +SEQS property). The costs are a one-time recompile (the compile **range** is +part of the cache key, so `FRANK_PRIMARY_CACHE_DIR` pays once) and a wider +prefill chunk, which the 09-29 sweep found decode-interchangeable across +1024-4096 while 512 doubled TTFT at 4064 ctx (2.46 s vs 1.36 s). + **`--expert-precise-memory` is the arena lever that does not touch the window.** -The planner's overhead term is the only other place the arena can come from, and -the flag swaps the default registry figure for the fitted one. Measured +It attacks the planner's overhead term (shape F above attacks the graph term — +the third and last term this repo can reach), and the flag swaps the default +registry figure for the fitted one. Measured 2026-10-02 by a boot probe on the shape-D base (production stopped, candidate booted AS the service; numbers are the engine's own plan lines): diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index fdd0f7f..a6fae89 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -18,7 +18,8 @@ # laws: expert prefetch ABSENT (36% precision), NO speculative decode (the # no-spec family measured fastest once the prefetch tax was removed; MTP # re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode), -# MNBT 1024 (Frank's lock), and the client contract (reasoning parser qwen3, +# MNBT 2048, the CUDA-graph plan term pinned to the measured capture cost, and +# the client contract (reasoning parser qwen3, # tool parser qwen3_xml, preserve_thinking on). load helpers @@ -95,8 +96,9 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # 241664 asked 10.70 GiB against 9.18 GiB free); SEQS=5 fp8 x 262144 # BOOTED (1,350,182 / 5.15x) but left ~0 bytes free and a real 995-token # request killed the worker with `Tried to allocate 50.00 MiB ... 0 bytes - # free`. UTIL stays 0.98 (lowering it shrinks the engine's own budget - # instead of buying slack). + # free`. UTIL is a REAL arena lever (the fork plans budget = total x util) + # but a weak one, and Frank caps it at 0.982 — never 1.00 (see + # FRANK_PRIMARY_UTIL for the two ceiling deaths that justify the margin). # SEQS=2 is IMPOSSIBLE on this model, measured 2026-10-02 by # single-variable boot probe against the then-deployed shape: SEQS 2 fails # at BOTH MNBT 1024 and 2048 with torch.fx ConstraintViolationError @@ -115,9 +117,9 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # runtime headroom, not arena. grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE" # --expert-precise-memory IS set (shape E, 2026-10-02). It was refused # while the 2 x 262144 bf16 boot sat 0.13 GiB under the 31.86 GiB ceiling @@ -177,6 +179,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE" grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE" grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE" + # Shape F: the CUDA-graph plan term is carried as CONTAINER env (the fork + # reads it at call time inside estimate_cudagraph_memory), and both must be + # present — the graph term is the only plan term this repo can shrink once + # UTIL is capped, and shrinking it is what grows the arena. + grep -q -- '-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"' "$SERVE" + grep -q -- '-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"' "$SERVE" + grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12$' "$ENV_FILE" } @test "tools and reasoning are wired — the client contract is served" { @@ -254,15 +264,25 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE" } -@test "MNBT is at Frank's 1024 with the arena intact" { - # MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are - # interchangeable on decode AND identical on arena (the sweep measured - # 6,499 slots at every rung <=4096); 12288 costs 2.5% plus arena. - # 1024 is Frank's lock — it keeps the workspace floor so the expert arena - # stays max. (2026-10-02: MNBT 2048 compiles the first graph in 26 s vs - # 172 s at 1024 — a startup-cost lever, not a correctness one.) - grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE" +@test "MNBT is 2048 and the CUDA-graph plan term is priced at the measured cost" { + # MNBT: 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are + # interchangeable on decode; 12288 costs 2.5% plus arena. 2048 is arena-FREE + # because the registry prices overhead as base + 0.22 GiB per 1024 ABOVE its + # nbt 4096 anchor — every rung <=4096 charges the same 5.25 GiB, and the + # fitted 5.25 was fitted AT nbt 4096, so 2048 sits under the anchor. + # CUDA graphs: the arena is budget - non-expert - KV reserve - graphs - + # overhead, so every GiB the plan stops charging for graphs is a GiB of + # expert cache. The fork's analytic default (256 MiB + 52 MiB/graph) charged + # 0.55 GiB where the engine's post-capture line measured 0.31 GiB on the + # heavier rank; 256 + 12/graph prices the same 6-graph plan at 0.32 GiB. + # The pair must stay BELOW the analytic 563 MiB default (that is the win) + # and ABOVE the measured 317 MiB (under-pricing is what OOMs at capture). + grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE" grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE" + base=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=/{print $2}' "$ENV_FILE") + per_graph=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=/{print $2}' "$ENV_FILE") + [ $((base + 6 * per_graph)) -lt 563 ] + [ $((base + 6 * per_graph)) -gt 317 ] } @test "expert prefetch is absent BY LAW, not by omission" {