From 0a4ff30db24509ac194dc921d39a3b5cf681100e Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 3 Oct 2026 02:49:38 -0700 Subject: [PATCH] =?UTF-8?q?frank-primary:=20shape=20G=20=E2=80=94=20force?= =?UTF-8?q?=20the=20arena=20to=20the=20flag=20ceiling,=20KV=20to=20400,700?= =?UTF-8?q?=20tokens?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two plan terms were still unspent: the CUDA-graph charge (shape F's own boot reported a NEGATIVE capture delta, -0.06 GiB, so the 256 MiB base pool was fiction -> charge 0 + 12 MiB/graph = 0.07 GiB) and the budget (util 0.982 -> 1.0, the field's le=1 maximum). Together +0.82 GiB of expert cache: plan target 18.35 GiB / ~8,016 slots. KV_BYTES 3.15e9 -> 3.1665e9 = the 400,700-token ask at the engine's measured 7,891.8 B/token. Cost of the force, recorded: driver high-water ~30.4 GiB of 31.86, so ~1.5 GiB of real margin per card; rollback is UTIL=0.982. --- scripts/frank-primary.env | 77 +++++++++++-------- ...server-qwen38-fp8-expert-offload-tuning.md | 50 ++++++++---- tests/frank_primary_unit.bats | 46 ++++++----- 3 files changed, 108 insertions(+), 65 deletions(-) diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index 5bdb478..e4dee1f 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -1,15 +1,17 @@ # frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770). # -# 2026-10-03 shape F: MNBT 2048 and the CUDA-graph plan term pinned to the -# measured capture cost. The expert arena is `budget - non-expert - KV reserve -# - graphs - overhead`, and with UTIL capped (see FRANK_PRIMARY_UTIL) the graph -# estimate is the only plan term this repo can reach: the fork charges an -# analytic 0.55 GiB (256 MiB base + 52 MiB/graph) where the engine's own -# post-capture line reports 0.31 GiB on the heavier rank. Pricing it honestly -# returns 0.24 GiB to the arena. The rest of the idle free VRAM (~3.3 GiB/card) -# stays unreachable from flags: it is locked inside the registry's overhead -# (5.25 GiB) and KV-price (9,190 B/token vs the measured 7,892) terms, which -# live in the image, not in our argv or env. +# 2026-10-03 shape G: the arena forced to the flag ceiling. The expert arena is +# `budget - non-expert - KV reserve - graphs - overhead` and the KV reserve is a +# registry constant, so exactly two terms are reachable: the CUDA-graph estimate +# (priced at the measured capture — the engine's own post-capture delta is +# -0.06 GiB, i.e. free, so we charge 12 MiB/graph and nothing else) and the +# budget, which is `device_total x gpu_memory_utilization` with the field pinned +# to `le=1`. UTIL 1.00 + graph term 0.07 GiB is therefore the MAXIMUM arena this +# repo can ask for without editing the image: 18.35 GiB / ~8,016 slots. Frank +# approved forcing util to 1.00 for this; the measured cost is the free-VRAM +# margin, which lands at ~1.5 GiB/card (see FRANK_PRIMARY_UTIL). +# 2026-10-03 shape F (absorbed): MNBT 2048 + the graph term pinned. Measured on +# the live deploy: arena 17.52 GiB / 7,653 slots, budget 31.29, graphs 0.32. # 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and # --expert-precise-memory ON. The window shrink from the native 262144 and the # --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model — @@ -162,31 +164,44 @@ FRANK_PRIMARY_SEQS=3 # 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A # 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected # as the shipped value; it remains the conservative arm if free VRAM ever binds. -FRANK_PRIMARY_KV_BYTES=3150000000 +# KV forced to 400,700 tokens (Frank, 2026-10-03). The engine prices this pool +# at 7,891.8 B/token at this shape and loses ~3.87 MB to block alignment, so +# 3,166,500,000 B is the ask that lands at/above 400,700 (verify the reported +# "GPU KV cache size" line on every deploy; it is not a constant ratio across +# geometries — 7,701 B/token at 262144). +FRANK_PRIMARY_KV_BYTES=3166500000 FRANK_PRIMARY_KV_DTYPE=fp8 FRANK_PRIMARY_MNBT=2048 FRANK_PRIMARY_GATHER_CHUNKS=32 FRANK_PRIMARY_GATHER_LANES=64 -# UTIL is Frank's cap: 0.982 maximum, never 1.00. It is an ARENA lever — the -# fork plans `budget = device_total x util` — but a weak one here: 0.980 -> 0.982 -# moves the budget 31.22 -> 31.28 GiB, i.e. +28 slots. Raising it to 1.00 would -# add 0.64 GiB and Frank vetoed it: this endpoint has twice died at the ceiling -# (the 2x262144 bf16 boot at 0.13 GiB free, the 5x262144 boot at "0 bytes free" -# on a real 995-token request), so the margin stays. -FRANK_PRIMARY_UTIL=0.982 -# CUDA-graph plan term, pinned to the measured capture cost. expert_plan.py -# charges estimate_cudagraph_memory() = BASE_MB + PER_GRAPH_MB x n_graphs with -# defaults 256/52, and the estimator's own docstring says to tune these against -# the post-capture line. MEASURED on shape E (model_runner.py:1094, same -# 6-graph plan, same shape): "Graph capturing finished in 3 secs, took 0.31 GiB" -# on the heavier rank (TP0) and 0.31/0.17 across ranks — vs the 0.55 GiB the -# plan charged. 12 MiB/graph keeps ~10 MiB of margin over the fitted marginal -# cost (~10 MiB/graph) and, unlike VLLM_CUDAGRAPH_MEM_ESTIMATE_MB (an explicit -# TOTAL), scales correctly if a future MNBT or spec-decode change grows the -# ladder. Verify every deploy against the "Estimated CUDA graph memory" and -# "Graph capturing finished" lines; if capture exceeds the estimate, raise -# FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52. -FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256 +# UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at +# 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots). Frank +# held it at 0.982 and then approved FORCING it to 1.00 to buy the last gig of +# arena — with the caveat written down, because this endpoint has twice died at +# the ceiling (the 2x262144 bf16 boot at 0.13 GiB free; the 5x262144 boot at +# "0 bytes free" on a real 995-token request). At util 1.00 the engine's budget +# equals the whole card, so anything the registry under-prices shows up as a +# RUNTIME allocation failure rather than a plan refusal. MEASURED cost of the +# force: the driver high-water mark moves from 28.28 GiB (shape E) to ~30.4 GiB, +# i.e. ~1.5 GiB of real free VRAM per card is the safety margin. If a request +# ever dies with `Tried to allocate ... 0 bytes free`, the rollback is THIS +# number back to 0.982 (-0.57 GiB of arena), not the window. +FRANK_PRIMARY_UTIL=1.0 +# CUDA-graph plan term, priced at the measured capture. expert_plan.py charges +# estimate_cudagraph_memory() = BASE_MB + PER_GRAPH_MB x n_graphs (defaults +# 256/52 = 0.55 GiB for our 6-graph plan), and the estimator's own docstring +# says to tune these against the post-capture line. MEASURED twice on this +# shape: shape E reported "Graph capturing finished ... took 0.31 GiB" (TP0) and +# 0.17 GiB (TP1); shape F, with the 0.32 GiB we then charged, reported +# "took -0.06 GiB" — a NEGATIVE delta, i.e. capture fits inside memory the +# profiler had already counted. So the base pool (256 MiB) is the over-charge: +# charge 0 base + 12 MiB/graph = 0.07 GiB for the shipped no-spec 6-graph plan, +# which returns 0.25 GiB more to the arena while still paying per graph, so a +# future spec-decode shape (18 graphs) is charged 0.21 GiB rather than zero. +# Verify every deploy against the "Estimated CUDA graph memory" vs "Graph +# capturing finished ... took X GiB" lines; if capture ever goes positive and +# exceeds the charge, raise FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52. +FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12 # --- optional engine knobs (unset == the flag is absent) --------------------- diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index 414118d..2a4facd 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -275,16 +275,16 @@ Depth caveat: every number above is at a near-empty KV. Deep-context behavior (the gather win growing with depth, bf16's decode cost at depth) is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap. -### The deployed shape (2026-10-03 shape F: 3 slots x 200704, fp8, precise memory, MNBT 2048, graph term pinned) +### The deployed shape (2026-10-03 shape G: 3 slots x 200704, fp8, precise memory, MNBT 2048, arena at the flag ceiling) ``` FRANK_PRIMARY_MAXLEN=200704 200704-token window FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below) -FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x) +FRANK_PRIMARY_KV_BYTES=3166500000 PER RANK (fp8, 7,892 B/token -> 400,700 tokens) FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized) FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F) -FRANK_PRIMARY_UTIL=0.982 Frank's cap; budget = total x util -FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_* +FRANK_PRIMARY_UTIL=1.0 budget = total x util, forced to the le=1 ceiling +FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_* FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB) FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU) FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache @@ -292,7 +292,7 @@ R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env) NO --speculative-config, NO --expert-prefetch ``` -### Shape F (2026-10-03): where the free VRAM actually is +### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost `rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB used of 31.86 GiB, after hours of real traffic — so it is the high-water mark, @@ -308,19 +308,39 @@ and every term on the right is either fixed or a lie: | term | plan charges | measured reality | reachable from our side? | |---|---|---|---| -| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | yes, but capped — **Frank: 0.982 max, never 1.00** | +| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | **yes** — and shape G takes it to `1.0` | | `non-expert weights` | 6.28 GiB | checkpoint census | no | | `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) | | `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` | | `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) | -So shape F takes the two levers that exist: **util 0.98 -> 0.982** (+0.06 GiB) -and **the graph term priced at the capture cost** (0.55 -> 0.32 GiB, +0.23 GiB). -Together **+0.30 GiB / ~+130 arena slots** — and that is the honest ceiling from -flags. The other ~2.7 GiB is invisible to the planner: it is the gap between the -registry's conservative terms and reality, and the fork's own log line says -*"Update the registry entry if this drifts"* — i.e. it is an image-side change, -not a deploy. +Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982: +**arena 17.52 GiB / 7,653 slots measured**, +131 slots. That is where the +"honest from flags" line sat — until shape G forced the other two. + +**Shape G forces both reachable terms.** The graph base pool is the actual +over-charge: shape F's own boot reported +`Graph capturing finished in 4 secs, took -0.06 GiB` — a **negative** capture +delta, so the 256 MiB base was fiction. Charging `0 base + 12 MiB/graph` +(0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph +spec-decode shape pays 0.21 GiB) plus **util 1.0**, the field maximum, moves +the plan to budget 31.86 / graphs 0.07 / **arena 18.35 GiB / ~8,016 slots** — ++363 slots over F, +494 over E, and the ceiling of what a deploy can ask for. +The KV ask went to `3,166,500,000` B/rank, which the engine's own 7,891.8 +B/token price lands at ~400,700 tokens (verify the `GPU KV cache size` line; +the price is shape-dependent, 7,701 B/token at 262144). + +**The price of the force is the margin, and it is real.** At util 1.0 the +engine's budget equals the whole card, so any under-priced registry term stops +showing up as a plan refusal and starts showing up as a runtime allocation +failure. Measured driver high-water moves 28.28 GiB (E) -> 29.53 GiB (F) -> +~30.4 GiB (G), i.e. **~1.5 GiB of true free VRAM per card** is now the safety +margin — against 0.13 GiB on the bf16 boot that died and 0 bytes on the one +that OOM'd a real 995-token request. Rollback is one number: +`FRANK_PRIMARY_UTIL=0.982` costs 0.57 GiB of arena and buys back 0.57 GiB of +margin. Everything beyond ~8,016 slots needs the registry edited inside the +image (the fork's log line says exactly that: *"Update the registry entry if +this drifts"*), not a deploy. **MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at @@ -484,7 +504,9 @@ memory environment). The night searched two concurrency contracts: | 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) | | 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request | | 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom | -| **3 x 200704, fp8, + `--expert-precise-memory`** | **DEPLOYED 2026-10-02 (shape E)**: same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s | +| **3 x 200704, fp8, + `--expert-precise-memory`** | shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s | +| 3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982 | shape F (2026-10-03): arena 17.52 GiB / **7,653 slots** measured, budget 31.29, `Estimated CUDA graph memory: 0.32 GiB`, KV unchanged 398,658 | +| **3 x 200704, fp8, graph term 0.07 GiB, util 1.00, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena 18.35 GiB / ~8,016 slots (+363 over F, +494 over E), budget 31.86 = the whole card, driver high-water ~30.4 GiB -> **~1.5 GiB real free per card is the margin**; rollback is UTIL back to 0.982 | Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index a6fae89..83d46b9 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -117,8 +117,8 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # runtime headroom, not arena. grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_UTIL=1.0$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_KV_BYTES=3166500000$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE" # --expert-precise-memory IS set (shape E, 2026-10-02). It was refused @@ -179,13 +179,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE" grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE" grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE" - # Shape F: the CUDA-graph plan term is carried as CONTAINER env (the fork + # Shape F/G: the CUDA-graph plan term is carried as CONTAINER env (the fork # reads it at call time inside estimate_cudagraph_memory), and both must be - # present — the graph term is the only plan term this repo can shrink once - # UTIL is capped, and shrinking it is what grows the arena. + # present — the graph term and UTIL are the ONLY plan terms this repo can + # move, and shrinking them is what grows the arena. The base must be shipped + # explicitly: unset, the estimator would fall back to its 256 MiB default. grep -q -- '-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"' "$SERVE" grep -q -- '-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"' "$SERVE" - grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12$' "$ENV_FILE" } @@ -264,25 +265,30 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE" } -@test "MNBT is 2048 and the CUDA-graph plan term is priced at the measured cost" { - # MNBT: 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are - # interchangeable on decode; 12288 costs 2.5% plus arena. 2048 is arena-FREE - # because the registry prices overhead as base + 0.22 GiB per 1024 ABOVE its - # nbt 4096 anchor — every rung <=4096 charges the same 5.25 GiB, and the - # fitted 5.25 was fitted AT nbt 4096, so 2048 sits under the anchor. - # CUDA graphs: the arena is budget - non-expert - KV reserve - graphs - - # overhead, so every GiB the plan stops charging for graphs is a GiB of - # expert cache. The fork's analytic default (256 MiB + 52 MiB/graph) charged - # 0.55 GiB where the engine's post-capture line measured 0.31 GiB on the - # heavier rank; 256 + 12/graph prices the same 6-graph plan at 0.32 GiB. - # The pair must stay BELOW the analytic 563 MiB default (that is the win) - # and ABOVE the measured 317 MiB (under-pricing is what OOMs at capture). +@test "the arena is forced to the flag ceiling and the KV ask is honoured" { + # The planner sizes expert cache as budget - non-expert - KV reserve - + # graphs - overhead, and two of those five terms are reachable from a + # deploy: budget (= device_total x util, field-capped at le=1) and graphs + # (VLLM_CUDAGRAPH_MEM_*). So the shipped shape must sit at the ceiling of + # BOTH: util at its field maximum, and a graph charge strictly below the + # fork's analytic 563 MiB default while still paying per graph (the measured + # post-capture delta is -0.06 GiB, i.e. the base pool is the over-charge, but + # a future spec-decode shape captures 18 graphs and must still be charged). + # MNBT stays 2048: 512 doubles TTFT at 4064 ctx, 1024-4096 are decode- + # interchangeable AND arena-identical (the registry prices overhead flat up + # to its nbt 4096 anchor), 12288 costs 2.5% plus arena. + # The KV ask: the pool is priced at the engine's measured 7,892 B/token at + # this shape, and a served conversation needs 200,704 tokens, so the pool + # must clear 400,700 tokens to keep two full-length resident. grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE" grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE" + awk -F= '/^FRANK_PRIMARY_UTIL=/{exit !($2+0 >= 1.0)}' "$ENV_FILE" base=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=/{print $2}' "$ENV_FILE") per_graph=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=/{print $2}' "$ENV_FILE") + [ "$base" -eq 0 ] && [ "$per_graph" -gt 0 ] [ $((base + 6 * per_graph)) -lt 563 ] - [ $((base + 6 * per_graph)) -gt 317 ] + kv=$(awk -F= '/^FRANK_PRIMARY_KV_BYTES=/{print $2}' "$ENV_FILE") + [ $((kv / 7892)) -ge 400700 ] } @test "expert prefetch is absent BY LAW, not by omission" {