dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(frank-primary): record VLLM_CUDAGRAPH_MEM_ESTIMATE_MB, the graph-term outright pin

Browse Source 
Verified in the shipped image (v1/worker/gpu/cudagraph_mem.py):
estimate_cudagraph_memory() checks VLLM_CUDAGRAPH_MEM_ESTIMATE_MB FIRST and
returns that MiB total outright, ignoring the BASE/PER_GRAPH pair — the only
way to state the CUDA-graph plan term directly instead of deriving it. We
keep shipping the pair (0/12) so a future spec-decode shape re-prices
automatically; documented the switch condition.

Also corrected two stale shape-G claims in the runbook that predated the
1.00 revert: the deployed shape block and the lever table both still said
util 1.0 / 'le=1 ceiling', while shape G ships 0.982.
This commit is contained in:
Frank
2026-10-04 17:51:27 -07:00
parent 50459e4e31
commit 9a144f779e
2 changed files with 10 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+3 -3
View File
@@ -283,7 +283,7 @@ FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3166500000 PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F)
FRANK_PRIMARY_UTIL=1.0 budget = total x util, forced to the le=1 ceiling
FRANK_PRIMARY_UTIL=0.982 budget = total x util, the law (1.0 unbootable, see below)
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
@@ -308,10 +308,10 @@ and every term on the right is either fixed or a lie:
| term | plan charges | measured reality | reachable from our side? |
|---|---|---|---|
| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | **yes** — and shape G takes it to `1.0` |
| `budget` | 31.22 GiB (0.98) | field is `le=1` so 31.86 GiB is the field max, but 1.0 is UNBOOTABLE (below) | **yes** — shipped 0.982 |
| `non-expert weights` | 6.28 GiB | checkpoint census | no |
| `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) |
| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` |
| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB`; `VLLM_CUDAGRAPH_MEM_ESTIMATE_MB` pins the whole term outright (checked first, pair ignored when set) |
| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982: