From 9a144f779e38830d541cd54c50f7467c99b137f1 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 4 Oct 2026 17:51:27 -0700 Subject: [PATCH] docs(frank-primary): record VLLM_CUDAGRAPH_MEM_ESTIMATE_MB, the graph-term outright pin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified in the shipped image (v1/worker/gpu/cudagraph_mem.py): estimate_cudagraph_memory() checks VLLM_CUDAGRAPH_MEM_ESTIMATE_MB FIRST and returns that MiB total outright, ignoring the BASE/PER_GRAPH pair — the only way to state the CUDA-graph plan term directly instead of deriving it. We keep shipping the pair (0/12) so a future spec-decode shape re-prices automatically; documented the switch condition. Also corrected two stale shape-G claims in the runbook that predated the 1.00 revert: the deployed shape block and the lever table both still said util 1.0 / 'le=1 ceiling', while shape G ships 0.982. --- scripts/frank-primary.env | 7 +++++++ .../LLM/dankserver-qwen38-fp8-expert-offload-tuning.md | 6 +++--- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index a867e6e..8ea5bc0 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -203,6 +203,13 @@ FRANK_PRIMARY_UTIL=0.982 # Verify every deploy against the "Estimated CUDA graph memory" vs "Graph # capturing finished ... took X GiB" lines; if capture ever goes positive and # exceeds the charge, raise FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52. +# A THIRD env var exists and is the only way to state the graph term outright: +# VLLM_CUDAGRAPH_MEM_ESTIMATE_MB pins the WHOLE term to an exact MiB total in +# one knob — estimate_cudagraph_memory() checks it FIRST and returns base/ +# per-graph untouched when set (verified in the shipped image, +# v1/worker/gpu/cudagraph_mem.py). We ship the BASE/PER_GRAPH pair instead so +# a future spec-decode shape re-prices automatically; switch to ESTIMATE_MB +# only if the drift check below ever shows the pair cannot track capture. FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12 diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index 6ae201c..fb8a12e 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -283,7 +283,7 @@ FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below) FRANK_PRIMARY_KV_BYTES=3166500000 PER RANK (fp8, 7,892 B/token -> 400,700 tokens) FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized) FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F) -FRANK_PRIMARY_UTIL=1.0 budget = total x util, forced to the le=1 ceiling +FRANK_PRIMARY_UTIL=0.982 budget = total x util, the law (1.0 unbootable, see below) FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_* FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB) FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU) @@ -308,10 +308,10 @@ and every term on the right is either fixed or a lie: | term | plan charges | measured reality | reachable from our side? | |---|---|---|---| -| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | **yes** — and shape G takes it to `1.0` | +| `budget` | 31.22 GiB (0.98) | field is `le=1` so 31.86 GiB is the field max, but 1.0 is UNBOOTABLE (below) | **yes** — shipped 0.982 | | `non-expert weights` | 6.28 GiB | checkpoint census | no | | `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) | -| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` | +| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB`; `VLLM_CUDAGRAPH_MEM_ESTIMATE_MB` pins the whole term outright (checked first, pair ignored when set) | | `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) | Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982: