dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: shape G — force the arena to the flag ceiling, KV to 400,700 tokens

Browse Source 
Two plan terms were still unspent: the CUDA-graph charge (shape F's own boot
reported a NEGATIVE capture delta, -0.06 GiB, so the 256 MiB base pool was
fiction -> charge 0 + 12 MiB/graph = 0.07 GiB) and the budget (util 0.982 -> 1.0,
the field's le=1 maximum). Together +0.82 GiB of expert cache: plan target
18.35 GiB / ~8,016 slots. KV_BYTES 3.15e9 -> 3.1665e9 = the 400,700-token ask at
the engine's measured 7,891.8 B/token. Cost of the force, recorded: driver
high-water ~30.4 GiB of 31.86, so ~1.5 GiB of real margin per card; rollback is
UTIL=0.982.
This commit is contained in:
Frank
2026-10-03 02:49:38 -07:00
parent c682c6329c
commit 0a4ff30db2
3 changed files with 108 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+36 -14
View File
@@ -275,16 +275,16 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
behavior (the gather win growing with depth, bf16's decode cost at depth)
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-03 shape F: 3 slots x 200704, fp8, precise memory, MNBT 2048, graph term pinned)
### The deployed shape (2026-10-03 shape G: 3 slots x 200704, fp8, precise memory, MNBT 2048, arena at the flag ceiling)
```
FRANK_PRIMARY_MAXLEN=200704 200704-token window
FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x)
FRANK_PRIMARY_KV_BYTES=3166500000 PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F)
FRANK_PRIMARY_UTIL=0.982 Frank's cap; budget = total x util
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
FRANK_PRIMARY_UTIL=1.0 budget = total x util, forced to the le=1 ceiling
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache
@@ -292,7 +292,7 @@ R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
```
### Shape F (2026-10-03): where the free VRAM actually is
### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
used of 31.86 GiB, after hours of real traffic — so it is the high-water mark,
@@ -308,19 +308,39 @@ and every term on the right is either fixed or a lie:
| term | plan charges | measured reality | reachable from our side? |
|---|---|---|---|
| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | yes, but capped — **Frank: 0.982 max, never 1.00** |
| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | **yes** — and shape G takes it to `1.0` |
| `non-expert weights` | 6.28 GiB | checkpoint census | no |
| `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) |
| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` |
| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
So shape F takes the two levers that exist: **util 0.98 -> 0.982** (+0.06 GiB)
and **the graph term priced at the capture cost** (0.55 -> 0.32 GiB, +0.23 GiB).
Together **+0.30 GiB / ~+130 arena slots** — and that is the honest ceiling from
flags. The other ~2.7 GiB is invisible to the planner: it is the gap between the
registry's conservative terms and reality, and the fork's own log line says
*"Update the registry entry if this drifts"* — i.e. it is an image-side change,
not a deploy.
Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982:
**arena 17.52 GiB / 7,653 slots measured**, +131 slots. That is where the
"honest from flags" line sat — until shape G forced the other two.
**Shape G forces both reachable terms.** The graph base pool is the actual
over-charge: shape F's own boot reported
`Graph capturing finished in 4 secs, took -0.06 GiB` — a **negative** capture
delta, so the 256 MiB base was fiction. Charging `0 base + 12 MiB/graph`
(0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph
spec-decode shape pays 0.21 GiB) plus **util 1.0**, the field maximum, moves
the plan to budget 31.86 / graphs 0.07 / **arena 18.35 GiB / ~8,016 slots** —
+363 slots over F, +494 over E, and the ceiling of what a deploy can ask for.
The KV ask went to `3,166,500,000` B/rank, which the engine's own 7,891.8
B/token price lands at ~400,700 tokens (verify the `GPU KV cache size` line;
the price is shape-dependent, 7,701 B/token at 262144).
**The price of the force is the margin, and it is real.** At util 1.0 the
engine's budget equals the whole card, so any under-priced registry term stops
showing up as a plan refusal and starts showing up as a runtime allocation
failure. Measured driver high-water moves 28.28 GiB (E) -> 29.53 GiB (F) ->
~30.4 GiB (G), i.e. **~1.5 GiB of true free VRAM per card** is now the safety
margin — against 0.13 GiB on the bf16 boot that died and 0 bytes on the one
that OOM'd a real 995-token request. Rollback is one number:
`FRANK_PRIMARY_UTIL=0.982` costs 0.57 GiB of arena and buys back 0.57 GiB of
margin. Everything beyond ~8,016 slots needs the registry edited inside the
image (the fork's log line says exactly that: *"Update the registry entry if
this drifts"*), not a deploy.
**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
@@ -484,7 +504,9 @@ memory environment). The night searched two concurrency contracts:
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) |
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
| **3 x 200704, fp8, + `--expert-precise-memory`** | **DEPLOYED 2026-10-02 (shape E)**: same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
| **3 x 200704, fp8, + `--expert-precise-memory`** | shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
| 3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982 | shape F (2026-10-03): arena 17.52 GiB / **7,653 slots** measured, budget 31.29, `Estimated CUDA graph memory: 0.32 GiB`, KV unchanged 398,658 |
| **3 x 200704, fp8, graph term 0.07 GiB, util 1.00, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena 18.35 GiB / ~8,016 slots (+363 over F, +494 over E), budget 31.86 = the whole card, driver high-water ~30.4 GiB -> **~1.5 GiB real free per card is the margin**; rollback is UTIL back to 0.982 |
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even