dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: shape F — MNBT 2048, util 0.982, CUDA-graph plan term pinned to the measured capture

Browse Source 
The arena is budget - non-expert - KV reserve - graphs - overhead. With util
capped at 0.982 (never 1.00) the graph estimate is the only plan term this repo
can shrink: the fork charges an analytic 256 MiB + 52 MiB/graph = 0.55 GiB where
the engine's own post-capture line reports 0.31 GiB, so pricing it at
256 + 12/graph returns ~0.24 GiB to the expert arena. MNBT 1024 -> 2048 is
arena-free: the registry overhead is base + 0.22 GiB per 1024 ABOVE its nbt 4096
anchor, and the fitted 5.25 GiB was fitted AT 4096.
This commit is contained in:
Frank
2026-10-03 02:22:49 -07:00
parent 6584c059d6
commit c682c6329c
5 changed files with 165 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+48 -4
View File
@@ -275,14 +275,16 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
behavior (the gather win growing with depth, bf16's decode cost at depth)
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-02 shape E: 3 slots x 200704, fp8, precise-memory arena)
### The deployed shape (2026-10-03 shape F: 3 slots x 200704, fp8, precise memory, MNBT 2048, graph term pinned)
```
FRANK_PRIMARY_MAXLEN=200704 200704-token window
FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=1024 --max-num-batched-tokens 1024
FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F)
FRANK_PRIMARY_UTIL=0.982 Frank's cap; budget = total x util
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache
@@ -290,9 +292,51 @@ R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
```
### Shape F (2026-10-03): where the free VRAM actually is
`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
used of 31.86 GiB, after hours of real traffic — so it is the high-water mark,
not an idle artifact). Growing the expert arena into it is NOT a matter of
raising `--gpu-memory-utilization`, because the arena is
```
arena = budget - non-expert - KV reserve - CUDA graphs - runtime overhead
budget = device_total x gpu_memory_utilization # expert_plan.py:567
```
and every term on the right is either fixed or a lie:
| term | plan charges | measured reality | reachable from our side? |
|---|---|---|---|
| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | yes, but capped — **Frank: 0.982 max, never 1.00** |
| `non-expert weights` | 6.28 GiB | checkpoint census | no |
| `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) |
| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` |
| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
So shape F takes the two levers that exist: **util 0.98 -> 0.982** (+0.06 GiB)
and **the graph term priced at the capture cost** (0.55 -> 0.32 GiB, +0.23 GiB).
Together **+0.30 GiB / ~+130 arena slots** — and that is the honest ceiling from
flags. The other ~2.7 GiB is invisible to the planner: it is the gap between the
registry's conservative terms and reality, and the fork's own log line says
*"Update the registry entry if this drifts"* — i.e. it is an image-side change,
not a deploy.
**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
nbt 4096, so any rung up to 4096 charges the identical 5.25 GiB; the fitted 5.25
was itself fitted at nbt 4096, which means 2048 is covered by an activation
figure measured at a **larger** batch. SEQS 3 boots at both 1024 and 2048
(measured 2026-10-02 — the `query_start_loc` `ConstraintViolationError` is a
SEQS property). The costs are a one-time recompile (the compile **range** is
part of the cache key, so `FRANK_PRIMARY_CACHE_DIR` pays once) and a wider
prefill chunk, which the 09-29 sweep found decode-interchangeable across
1024-4096 while 512 doubled TTFT at 4064 ctx (2.46 s vs 1.36 s).
**`--expert-precise-memory` is the arena lever that does not touch the window.**
The planner's overhead term is the only other place the arena can come from, and
the flag swaps the default registry figure for the fitted one. Measured
It attacks the planner's overhead term (shape F above attacks the graph term —
the third and last term this repo can reach), and the flag swaps the default
registry figure for the fitted one. Measured
2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
booted AS the service; numbers are the engine's own plan lines):