dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: shape F — MNBT 2048, util 0.982, CUDA-graph plan term pinned to the measured capture

Browse Source 
The arena is budget - non-expert - KV reserve - graphs - overhead. With util
capped at 0.982 (never 1.00) the graph estimate is the only plan term this repo
can shrink: the fork charges an analytic 256 MiB + 52 MiB/graph = 0.55 GiB where
the engine's own post-capture line reports 0.31 GiB, so pricing it at
256 + 12/graph returns ~0.24 GiB to the expert arena. MNBT 1024 -> 2048 is
arena-free: the registry overhead is base + 0.22 GiB per 1024 ABOVE its nbt 4096
anchor, and the fitted 5.25 GiB was fitted AT 4096.
This commit is contained in:
Frank
2026-10-03 02:22:49 -07:00
parent 6584c059d6
commit c682c6329c
5 changed files with 165 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+29 -5
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default → expert arena **17.22 GiB / 7,522 slots**, +16 %), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) plus the overhead term (`--expert-precise-memory` → 7,522) — util 0.98, and the client contract wired: `--reasoning-parser …
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default → expert arena **17.22 GiB / 7,522 slots**, +16 %), **`VLLM_CUDAGRAPH_MEM_BASE_MB=256` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured 0.31 GiB capture instead of the analytic 0.55 GiB → +0.24 GiB of arena), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,150,000,000/rank — the **pool, not the slot count, caps resident conversations at ~2 full-length** (measured live: `GPU KV cache size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: 1.99×`), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. The pool cut buys runtime **headroom, NOT arena**: the arena is sized by `max_model_len` via the registry KV reserve (262144 → 6,248 slots; 200704 → 6,496) plus the overhead term (`--expert-precise-memory` → 7,522) — util 0.982 (Frank's cap; the field is `le=1`, and the ~3.5 GiB of free VRAM the driver reports is estimator slack, not allocatable arena), and the client contract wired: `--reasoning-parser …
| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 |
@@ -433,12 +433,13 @@ The **tcclaviger RDNA4 vLLM fork** (`docker.io/tcclaviger/vllm@sha256:da4029666
digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`**
(~172.8 GiB, 131 shards) on **both** Radeon AI PRO R9700s (gfx1201), **TP=2**,
**fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since
2026-10-02 the **shape-E** geometry: `--max-model-len 200704` × 3 slots with
2026-10-03 the **shape-F** geometry (shape E + MNBT 2048 + util 0.982 + the
CUDA-graph plan term pinned): `--max-model-len 200704` × 3 slots with
`--kv-cache-memory-bytes 3,150,000,000`/rank, so the pool admits **~2
full-length conversations** (measured live: `398,658 tokens, Maximum
concurrency for 200,704 tokens per request: 1.99×`) while the argv keeps the
only slot count that compiles (below), plus `--expert-precise-memory`
(`runtime overhead 5.25 GiB`, `expert cache space 17.22 GiB / 7,522 slots`),
only slot count that compiles (below), plus `--expert-precise-memory`,
`VLLM_CUDAGRAPH_MEM_BASE_MB=256` / `_PER_GRAPH_MB=12`,
`GPU_MAX_HW_QUEUES=2` and a mounted `/root/.cache`. `models.yml` carries
`contextWindow: 200704` for this endpoint — the served window, not the model's
native 262144, so the harness compacts before vLLM answers a longer prompt
@@ -455,7 +456,8 @@ sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
((1,1024) vs (1,2048)). `--enforce-eager` bypasses the path but drops CUDA
graphs. The same failure class killed the 2026-10-01 bf16 2×262144 shape.
**The expert arena has TWO levers: `max_model_len` and the overhead term.**
**The expert arena has THREE reachable levers: `max_model_len`, the overhead
term, and the CUDA-graph estimate.**
The fork's planner sizes expert cache as `budget − non-expert − KV reserve −
graphs − overhead`, and the KV reserve is registry-priced at 9190 B/token ×
`max_model_len`: 262144 → 2.48 GiB reserve → arena 14.30 GiB / **6,248 slots**;
@@ -473,6 +475,28 @@ concurrency (a 2.2e9 probe reports 277,686 tokens / **1.38×**) and buy no slots
The arena (2.344 MiB per slot) is the decode-critical region — cutting it for KV
headroom collapses the offload hit rate.
**Free VRAM on this endpoint is NOT arena headroom (2026-10-03 audit).**
`rocm-smi` reads ~3.5 GiB free per card after hours of real traffic, but the
planner charges `budget = total × util` against terms that overstate reality:
the KV reserve is registry-priced at 9190 B/token (`1.91 GiB` at 200704) while
the real pool bills 2.93 GiB at 7,892 B/token, and the precise overhead (5.25
GiB) is far above the idle-measured figure. So the visible free space is the
estimator gap, not allocatable arena. `gpu_memory_utilization` is a real arena
lever (each +0.01 = +0.32 GiB) but the field is `Field(gt=0, le=1)` — **Frank
caps it at 0.982, never 1.00** (this endpoint has twice died at the ceiling).
What IS reachable from flags: `VLLM_CUDAGRAPH_MEM_BASE_MB` /
`_PER_GRAPH_MB`, read at call time by `estimate_cudagraph_memory()` — the
analytic default (256 MiB + 52 MiB/graph) charged 0.55 GiB where the engine's
own post-capture line reported 0.31 GiB, so pricing it honestly returns
~0.24 GiB to the arena. Everything else needs a registry edit inside the image
(the fork's log line says as much: *"Update the registry entry if this
drifts"*). Verify every deploy by comparing the `Estimated CUDA graph memory`
line with `Graph capturing finished … took X GiB`; if capture exceeds the
estimate, raise `_PER_GRAPH_MB` back toward 52. MNBT 1024 → 2048 costs ZERO
arena: the registry overhead is `base + 0.22 GiB × max(0,(nbt−4096)/1024)`
anchored at nbt 4096, so every rung ≤4096 charges the same 5.25 GiB — and the
fitted figure was fitted AT 4096.
**A container's compile cache does NOT survive `podman rm -f`, and on this
endpoint that is 181 s per boot.** Measured 2026-10-02: the image ships **no
`/root/.cache` at all** (fresh container: `No such file or directory`); after a
scripts/frank-primary.env
+44 -9
View File
@@ -1,5 +1,15 @@
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
#
# 2026-10-03 shape F: MNBT 2048 and the CUDA-graph plan term pinned to the
# measured capture cost. The expert arena is `budget - non-expert - KV reserve
# - graphs - overhead`, and with UTIL capped (see FRANK_PRIMARY_UTIL) the graph
# estimate is the only plan term this repo can reach: the fork charges an
# analytic 0.55 GiB (256 MiB base + 52 MiB/graph) where the engine's own
# post-capture line reports 0.31 GiB on the heavier rank. Pricing it honestly
# returns 0.24 GiB to the arena. The rest of the idle free VRAM (~3.3 GiB/card)
# stays unreachable from flags: it is locked inside the registry's overhead
# (5.25 GiB) and KV-price (9,190 B/token vs the measured 7,892) terms, which
# live in the image, not in our argv or env.
# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
# --expert-precise-memory ON. The window shrink from the native 262144 and the
# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
@@ -80,13 +90,18 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
# GATHER 32x64 stays: R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64
# measured +18.7% decode vs default geometry and the win grows with depth.
# Do not drop it.
# MNBT 1024 is Frank's lock: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs
# 1.36 s); 1024-4096 are interchangeable on decode; 12288 costs 2.5% and
# arena. 1024 keeps the workspace floor so the arena stays max. (Side note
# measured 2026-10-02: MNBT 2048 compiles the first graph in 26 s vs 172 s at
# 1024 — a startup-cost lever, not a correctness one. The compile RANGE is part
# of the cache key, so FRANK_PRIMARY_CACHE_DIR pays for whichever rung
# is chosen exactly once.)
# MNBT 2048 (Frank's call, 2026-10-03; the lock was 1024). The measured law
# still holds: MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s), 1024-4096
# are interchangeable on decode, 12288 costs 2.5% plus arena. 2048 is ARENA-FREE
# by construction: the registry's overhead model is
# base + 0.22 GiB x max(0, (nbt - 4096)/1024) with the anchor at nbt 4096, so
# every rung up to and including 4096 prices the SAME 5.25 GiB overhead — and
# the fitted 5.25 was fitted at nbt 4096, i.e. our 2048 sits UNDER the anchor,
# so the activation term is covered by a figure measured at a bigger batch.
# SEQS 3 boots at both 1024 and 2048 (measured 2026-10-02; the SEQS=2
# ConstraintViolationError is a SEQS property, not an MNBT one). Costs exactly
# one recompilation: the compile RANGE is part of the cache key, so
# FRANK_PRIMARY_CACHE_DIR pays for whichever rung is chosen once.
# PREFETCH IS ABSENT BY LAW: expert-cache telemetry measured the prefetcher at
# 36% precision (36,720 slabs prefetched, 13,369 served) — roughly half the
# gather traffic was waste. Never re-add --expert-prefetch without a fresh A/B.
@@ -149,10 +164,30 @@ FRANK_PRIMARY_SEQS=3
# as the shipped value; it remains the conservative arm if free VRAM ever binds.
FRANK_PRIMARY_KV_BYTES=3150000000
FRANK_PRIMARY_KV_DTYPE=fp8
FRANK_PRIMARY_MNBT=1024
FRANK_PRIMARY_MNBT=2048
FRANK_PRIMARY_GATHER_CHUNKS=32
FRANK_PRIMARY_GATHER_LANES=64
FRANK_PRIMARY_UTIL=0.98
# UTIL is Frank's cap: 0.982 maximum, never 1.00. It is an ARENA lever — the
# fork plans `budget = device_total x util` — but a weak one here: 0.980 -> 0.982
# moves the budget 31.22 -> 31.28 GiB, i.e. +28 slots. Raising it to 1.00 would
# add 0.64 GiB and Frank vetoed it: this endpoint has twice died at the ceiling
# (the 2x262144 bf16 boot at 0.13 GiB free, the 5x262144 boot at "0 bytes free"
# on a real 995-token request), so the margin stays.
FRANK_PRIMARY_UTIL=0.982
# CUDA-graph plan term, pinned to the measured capture cost. expert_plan.py
# charges estimate_cudagraph_memory() = BASE_MB + PER_GRAPH_MB x n_graphs with
# defaults 256/52, and the estimator's own docstring says to tune these against
# the post-capture line. MEASURED on shape E (model_runner.py:1094, same
# 6-graph plan, same shape): "Graph capturing finished in 3 secs, took 0.31 GiB"
# on the heavier rank (TP0) and 0.31/0.17 across ranks — vs the 0.55 GiB the
# plan charged. 12 MiB/graph keeps ~10 MiB of margin over the fitted marginal
# cost (~10 MiB/graph) and, unlike VLLM_CUDAGRAPH_MEM_ESTIMATE_MB (an explicit
# TOTAL), scales correctly if a future MNBT or spec-decode change grows the
# ladder. Verify every deploy against the "Estimated CUDA graph memory" and
# "Graph capturing finished" lines; if capture exceeds the estimate, raise
# FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52.
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256
FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12
# --- optional engine knobs (unset == the flag is absent) ---------------------
# FRANK_PRIMARY_SPEC holds the WHOLE --speculative-config JSON; setting it adds
scripts/serve-frank-primary.sh
+11 -3
View File
@@ -11,7 +11,8 @@
# every flag here was qualified by boot probe on 2026-09-29 — see
# frank-primary.env for the qualification record).
#
# Shape (2026-10-02 shape E): MAXLEN 200704 x 3 slots, fp8 KV, KV_BYTES
# Shape (2026-10-03 shape F, on the 2026-10-02 shape E base): MAXLEN 200704 x 3
# slots, fp8 KV, KV_BYTES
# 3,150,000,000 PER RANK — the POOL, not the slot count, caps resident
# conversations at ~2 full-length (measured: 398,658 tokens = 1.99x);
# shorter requests use the spare blocks. --max-num-seqs stays 3 because SEQS=2
@@ -21,12 +22,17 @@
# is OFF today by ENV KNOB, not by measurement: FRANK_PRIMARY_SPEC / _COMPCONFIG
# / _GEN_CONFIG hold the whole JSON value and unset means no flag. The 2026-10-02
# retirement of MTP compared the wrong pair — see frank-primary.env.
# --max-num-batched-tokens 1024 (Frank's lock), MoE gather geometry
# --max-num-batched-tokens 2048 (Frank's call 2026-10-03; arena-free because the
# registry prices overhead flat up to its nbt 4096 anchor), MoE gather geometry
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
# with depth), prefix caching, expert prefetch OFF (36% precision measured),
# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
# and GPU_MAX_HW_QUEUES=2. The native-262144 and fp8+MTP4 shapes are retired to
# and GPU_MAX_HW_QUEUES=2; shape F pins the plan's CUDA-graph term to the
# measured capture cost (VLLM_CUDAGRAPH_MEM_BASE_MB / _PER_GRAPH_MB, container
# env — the analytic default charges 0.55 GiB where capture took 0.31 GiB, so
# the 0.24 GiB difference goes to the arena) and raises util 0.98 -> 0.982,
# Frank's cap. The native-262144 and fp8+MTP4 shapes are retired to
# git tag frank-primary-fp8-mtp-20261001.
#
# THE COMPILE CACHE IS MOUNTED, and that is a boot-cost fix, not a tuning knob:
@@ -138,6 +144,8 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
${FRANK_PRIMARY_GPU_MAX_HW_QUEUES:+-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"} \
${FRANK_PRIMARY_GATHER_CHUNKS:+-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"} \
${FRANK_PRIMARY_GATHER_LANES:+-e R4D_LRU_GATHER_LANES="$FRANK_PRIMARY_GATHER_LANES"} \
${FRANK_PRIMARY_CUDAGRAPH_BASE_MB:+-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"} \
${FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB:+-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"} \
${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache} \
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
"$FRANK_PRIMARY_IMAGE" /models \
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+48 -4
View File
@@ -275,14 +275,16 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
behavior (the gather win growing with depth, bf16's decode cost at depth)
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-02 shape E: 3 slots x 200704, fp8, precise-memory arena)
### The deployed shape (2026-10-03 shape F: 3 slots x 200704, fp8, precise memory, MNBT 2048, graph term pinned)
```
FRANK_PRIMARY_MAXLEN=200704 200704-token window
FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F)
FRANK_PRIMARY_UTIL=0.982 Frank's cap; budget = total x util
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache
@@ -290,9 +292,51 @@ R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
```
### Shape F (2026-10-03): where the free VRAM actually is
`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
used of 31.86 GiB, after hours of real traffic — so it is the high-water mark,
not an idle artifact). Growing the expert arena into it is NOT a matter of
raising `--gpu-memory-utilization`, because the arena is
```
arena = budget - non-expert - KV reserve - CUDA graphs - runtime overhead
budget = device_total x gpu_memory_utilization # expert_plan.py:567
```
and every term on the right is either fixed or a lie:
| term | plan charges | measured reality | reachable from our side? |
|---|---|---|---|
| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | yes, but capped — **Frank: 0.982 max, never 1.00** |
| `non-expert weights` | 6.28 GiB | checkpoint census | no |
| `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) |
| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` |
| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
So shape F takes the two levers that exist: **util 0.98 -> 0.982** (+0.06 GiB)
and **the graph term priced at the capture cost** (0.55 -> 0.32 GiB, +0.23 GiB).
Together **+0.30 GiB / ~+130 arena slots** — and that is the honest ceiling from
flags. The other ~2.7 GiB is invisible to the planner: it is the gap between the
registry's conservative terms and reality, and the fork's own log line says
*"Update the registry entry if this drifts"* — i.e. it is an image-side change,
not a deploy.
**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
nbt 4096, so any rung up to 4096 charges the identical 5.25 GiB; the fitted 5.25
was itself fitted at nbt 4096, which means 2048 is covered by an activation
figure measured at a **larger** batch. SEQS 3 boots at both 1024 and 2048
(measured 2026-10-02 — the `query_start_loc` `ConstraintViolationError` is a
SEQS property). The costs are a one-time recompile (the compile **range** is
part of the cache key, so `FRANK_PRIMARY_CACHE_DIR` pays once) and a wider
prefill chunk, which the 09-29 sweep found decode-interchangeable across
1024-4096 while 512 doubled TTFT at 4064 ctx (2.46 s vs 1.36 s).
**`--expert-precise-memory` is the arena lever that does not touch the window.**
The planner's overhead term is the only other place the arena can come from, and
the flag swaps the default registry figure for the fitted one. Measured
It attacks the planner's overhead term (shape F above attacks the graph term —
the third and last term this repo can reach), and the flag swaps the default
registry figure for the fitted one. Measured
2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
booted AS the service; numbers are the engine's own plan lines):
tests/frank_primary_unit.bats
+33 -13
View File
@@ -18,7 +18,8 @@
# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
# no-spec family measured fastest once the prefetch tax was removed; MTP
# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
# MNBT 1024 (Frank's lock), and the client contract (reasoning parser qwen3,
# MNBT 2048, the CUDA-graph plan term pinned to the measured capture cost, and
# the client contract (reasoning parser qwen3,
# tool parser qwen3_xml, preserve_thinking on).
load helpers
@@ -95,8 +96,9 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
# 241664 asked 10.70 GiB against 9.18 GiB free); SEQS=5 fp8 x 262144
# BOOTED (1,350,182 / 5.15x) but left ~0 bytes free and a real 995-token
# request killed the worker with `Tried to allocate 50.00 MiB ... 0 bytes
# free`. UTIL stays 0.98 (lowering it shrinks the engine's own budget
# instead of buying slack).
# free`. UTIL is a REAL arena lever (the fork plans budget = total x util)
# but a weak one, and Frank caps it at 0.982 — never 1.00 (see
# FRANK_PRIMARY_UTIL for the two ceiling deaths that justify the margin).
# SEQS=2 is IMPOSSIBLE on this model, measured 2026-10-02 by
# single-variable boot probe against the then-deployed shape: SEQS 2 fails
# at BOTH MNBT 1024 and 2048 with torch.fx ConstraintViolationError
@@ -115,9 +117,9 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
# runtime headroom, not arena.
grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
# --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
# while the 2 x 262144 bf16 boot sat 0.13 GiB under the 31.86 GiB ceiling
@@ -177,6 +179,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
# Shape F: the CUDA-graph plan term is carried as CONTAINER env (the fork
# reads it at call time inside estimate_cudagraph_memory), and both must be
# present — the graph term is the only plan term this repo can shrink once
# UTIL is capped, and shrinking it is what grows the arena.
grep -q -- '-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"' "$SERVE"
grep -q -- '-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"' "$SERVE"
grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12$' "$ENV_FILE"
}
@test "tools and reasoning are wired — the client contract is served" {
@@ -254,15 +264,25 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
}
@test "MNBT is at Frank's 1024 with the arena intact" {
# MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
# interchangeable on decode AND identical on arena (the sweep measured
# 6,499 slots at every rung <=4096); 12288 costs 2.5% plus arena.
# 1024 is Frank's lock — it keeps the workspace floor so the expert arena
# stays max. (2026-10-02: MNBT 2048 compiles the first graph in 26 s vs
# 172 s at 1024 — a startup-cost lever, not a correctness one.)
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
@test "MNBT is 2048 and the CUDA-graph plan term is priced at the measured cost" {
# MNBT: 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
# interchangeable on decode; 12288 costs 2.5% plus arena. 2048 is arena-FREE
# because the registry prices overhead as base + 0.22 GiB per 1024 ABOVE its
# nbt 4096 anchor — every rung <=4096 charges the same 5.25 GiB, and the
# fitted 5.25 was fitted AT nbt 4096, so 2048 sits under the anchor.
# CUDA graphs: the arena is budget - non-expert - KV reserve - graphs -
# overhead, so every GiB the plan stops charging for graphs is a GiB of
# expert cache. The fork's analytic default (256 MiB + 52 MiB/graph) charged
# 0.55 GiB where the engine's post-capture line measured 0.31 GiB on the
# heavier rank; 256 + 12/graph prices the same 6-graph plan at 0.32 GiB.
# The pair must stay BELOW the analytic 563 MiB default (that is the win)
# and ABOVE the measured 317 MiB (under-pricing is what OOMs at capture).
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
base=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=/{print $2}' "$ENV_FILE")
per_graph=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=/{print $2}' "$ENV_FILE")
[ $((base + 6 * per_graph)) -lt 563 ]
[ $((base + 6 * per_graph)) -gt 317 ]
}
@test "expert prefetch is absent BY LAW, not by omission" {