dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: shape G — force the arena to the flag ceiling, KV to 400,700 tokens

Browse Source 
Two plan terms were still unspent: the CUDA-graph charge (shape F's own boot
reported a NEGATIVE capture delta, -0.06 GiB, so the 256 MiB base pool was
fiction -> charge 0 + 12 MiB/graph = 0.07 GiB) and the budget (util 0.982 -> 1.0,
the field's le=1 maximum). Together +0.82 GiB of expert cache: plan target
18.35 GiB / ~8,016 slots. KV_BYTES 3.15e9 -> 3.1665e9 = the 400,700-token ask at
the engine's measured 7,891.8 B/token. Cost of the force, recorded: driver
high-water ~30.4 GiB of 31.86, so ~1.5 GiB of real margin per card; rollback is
UTIL=0.982.
This commit is contained in:
Frank
2026-10-03 02:49:38 -07:00
parent c682c6329c
commit 0a4ff30db2
3 changed files with 108 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-primary.env
+46 -31
View File
@@ -1,15 +1,17 @@
# frank-primary.env — tcclaviger RDNA4 vLLM production config (dankserver :7770).
#
# 2026-10-03 shape F: MNBT 2048 and the CUDA-graph plan term pinned to the
# measured capture cost. The expert arena is `budget - non-expert - KV reserve
# - graphs - overhead`, and with UTIL capped (see FRANK_PRIMARY_UTIL) the graph
# estimate is the only plan term this repo can reach: the fork charges an
# analytic 0.55 GiB (256 MiB base + 52 MiB/graph) where the engine's own
# post-capture line reports 0.31 GiB on the heavier rank. Pricing it honestly
# returns 0.24 GiB to the arena. The rest of the idle free VRAM (~3.3 GiB/card)
# stays unreachable from flags: it is locked inside the registry's overhead
# (5.25 GiB) and KV-price (9,190 B/token vs the measured 7,892) terms, which
# live in the image, not in our argv or env.
# 2026-10-03 shape G: the arena forced to the flag ceiling. The expert arena is
# `budget - non-expert - KV reserve - graphs - overhead` and the KV reserve is a
# registry constant, so exactly two terms are reachable: the CUDA-graph estimate
# (priced at the measured capture — the engine's own post-capture delta is
# -0.06 GiB, i.e. free, so we charge 12 MiB/graph and nothing else) and the
# budget, which is `device_total x gpu_memory_utilization` with the field pinned
# to `le=1`. UTIL 1.00 + graph term 0.07 GiB is therefore the MAXIMUM arena this
# repo can ask for without editing the image: 18.35 GiB / ~8,016 slots. Frank
# approved forcing util to 1.00 for this; the measured cost is the free-VRAM
# margin, which lands at ~1.5 GiB/card (see FRANK_PRIMARY_UTIL).
# 2026-10-03 shape F (absorbed): MNBT 2048 + the graph term pinned. Measured on
# the live deploy: arena 17.52 GiB / 7,653 slots, budget 31.29, graphs 0.32.
# 2026-10-02 shape E: fp8 KV, MAXLEN 200704 x SEQS 3, pool 3.15e9/rank, and
# --expert-precise-memory ON. The window shrink from the native 262144 and the
# --max-num-seqs 3 lock both come from shape D (SEQS=2 CANNOT boot this model —
@@ -162,31 +164,44 @@ FRANK_PRIMARY_SEQS=3
# 3.15e9 preserves the measured 1.99x while precise-memory grows the arena. A
# 2.2e9 pool was probed in the same window (277,686 tokens / 1.38x) and rejected
# as the shipped value; it remains the conservative arm if free VRAM ever binds.
FRANK_PRIMARY_KV_BYTES=3150000000
# KV forced to 400,700 tokens (Frank, 2026-10-03). The engine prices this pool
# at 7,891.8 B/token at this shape and loses ~3.87 MB to block alignment, so
# 3,166,500,000 B is the ask that lands at/above 400,700 (verify the reported
# "GPU KV cache size" line on every deploy; it is not a constant ratio across
# geometries — 7,701 B/token at 262144).
FRANK_PRIMARY_KV_BYTES=3166500000
FRANK_PRIMARY_KV_DTYPE=fp8
FRANK_PRIMARY_MNBT=2048
FRANK_PRIMARY_GATHER_CHUNKS=32
FRANK_PRIMARY_GATHER_LANES=64
# UTIL is Frank's cap: 0.982 maximum, never 1.00. It is an ARENA lever — the
# fork plans `budget = device_total x util` — but a weak one here: 0.980 -> 0.982
# moves the budget 31.22 -> 31.28 GiB, i.e. +28 slots. Raising it to 1.00 would
# add 0.64 GiB and Frank vetoed it: this endpoint has twice died at the ceiling
# (the 2x262144 bf16 boot at 0.13 GiB free, the 5x262144 boot at "0 bytes free"
# on a real 995-token request), so the margin stays.
FRANK_PRIMARY_UTIL=0.982
# CUDA-graph plan term, pinned to the measured capture cost. expert_plan.py
# charges estimate_cudagraph_memory() = BASE_MB + PER_GRAPH_MB x n_graphs with
# defaults 256/52, and the estimator's own docstring says to tune these against
# the post-capture line. MEASURED on shape E (model_runner.py:1094, same
# 6-graph plan, same shape): "Graph capturing finished in 3 secs, took 0.31 GiB"
# on the heavier rank (TP0) and 0.31/0.17 across ranks — vs the 0.55 GiB the
# plan charged. 12 MiB/graph keeps ~10 MiB of margin over the fitted marginal
# cost (~10 MiB/graph) and, unlike VLLM_CUDAGRAPH_MEM_ESTIMATE_MB (an explicit
# TOTAL), scales correctly if a future MNBT or spec-decode change grows the
# ladder. Verify every deploy against the "Estimated CUDA graph memory" and
# "Graph capturing finished" lines; if capture exceeds the estimate, raise
# FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52.
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256
# UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
# 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots). Frank
# held it at 0.982 and then approved FORCING it to 1.00 to buy the last gig of
# arena — with the caveat written down, because this endpoint has twice died at
# the ceiling (the 2x262144 bf16 boot at 0.13 GiB free; the 5x262144 boot at
# "0 bytes free" on a real 995-token request). At util 1.00 the engine's budget
# equals the whole card, so anything the registry under-prices shows up as a
# RUNTIME allocation failure rather than a plan refusal. MEASURED cost of the
# force: the driver high-water mark moves from 28.28 GiB (shape E) to ~30.4 GiB,
# i.e. ~1.5 GiB of real free VRAM per card is the safety margin. If a request
# ever dies with `Tried to allocate ... 0 bytes free`, the rollback is THIS
# number back to 0.982 (-0.57 GiB of arena), not the window.
FRANK_PRIMARY_UTIL=1.0
# CUDA-graph plan term, priced at the measured capture. expert_plan.py charges
# estimate_cudagraph_memory() = BASE_MB + PER_GRAPH_MB x n_graphs (defaults
# 256/52 = 0.55 GiB for our 6-graph plan), and the estimator's own docstring
# says to tune these against the post-capture line. MEASURED twice on this
# shape: shape E reported "Graph capturing finished ... took 0.31 GiB" (TP0) and
# 0.17 GiB (TP1); shape F, with the 0.32 GiB we then charged, reported
# "took -0.06 GiB" — a NEGATIVE delta, i.e. capture fits inside memory the
# profiler had already counted. So the base pool (256 MiB) is the over-charge:
# charge 0 base + 12 MiB/graph = 0.07 GiB for the shipped no-spec 6-graph plan,
# which returns 0.25 GiB more to the arena while still paying per graph, so a
# future spec-decode shape (18 graphs) is charged 0.21 GiB rather than zero.
# Verify every deploy against the "Estimated CUDA graph memory" vs "Graph
# capturing finished ... took X GiB" lines; if capture ever goes positive and
# exceeds the charge, raise FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB back toward 52.
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0
FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12
# --- optional engine knobs (unset == the flag is absent) ---------------------
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+36 -14
View File
@@ -275,16 +275,16 @@ Depth caveat: every number above is at a near-empty KV. Deep-context
behavior (the gather win growing with depth, bf16's decode cost at depth)
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-03 shape F: 3 slots x 200704, fp8, precise memory, MNBT 2048, graph term pinned)
### The deployed shape (2026-10-03 shape G: 3 slots x 200704, fp8, precise memory, MNBT 2048, arena at the flag ceiling)
```
FRANK_PRIMARY_MAXLEN=200704 200704-token window
FRANK_PRIMARY_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_PRIMARY_KV_BYTES=3150000000 PER RANK (fp8, 7,892 B/token -> 1.99x)
FRANK_PRIMARY_KV_BYTES=3166500000 PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
FRANK_PRIMARY_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_PRIMARY_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F)
FRANK_PRIMARY_UTIL=0.982 Frank's cap; budget = total x util
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
FRANK_PRIMARY_UTIL=1.0 budget = total x util, forced to the le=1 ceiling
FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
FRANK_PRIMARY_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache
@@ -292,7 +292,7 @@ R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
```
### Shape F (2026-10-03): where the free VRAM actually is
### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
used of 31.86 GiB, after hours of real traffic — so it is the high-water mark,
@@ -308,19 +308,39 @@ and every term on the right is either fixed or a lie:
| term | plan charges | measured reality | reachable from our side? |
|---|---|---|---|
| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | yes, but capped — **Frank: 0.982 max, never 1.00** |
| `budget` | 31.22 GiB (0.98) | field is `le=1`, so 31.86 GiB max | **yes** — and shape G takes it to `1.0` |
| `non-expert weights` | 6.28 GiB | checkpoint census | no |
| `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) |
| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB` |
| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
So shape F takes the two levers that exist: **util 0.98 -> 0.982** (+0.06 GiB)
and **the graph term priced at the capture cost** (0.55 -> 0.32 GiB, +0.23 GiB).
Together **+0.30 GiB / ~+130 arena slots** — and that is the honest ceiling from
flags. The other ~2.7 GiB is invisible to the planner: it is the gap between the
registry's conservative terms and reality, and the fork's own log line says
*"Update the registry entry if this drifts"* — i.e. it is an image-side change,
not a deploy.
Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982:
**arena 17.52 GiB / 7,653 slots measured**, +131 slots. That is where the
"honest from flags" line sat — until shape G forced the other two.
**Shape G forces both reachable terms.** The graph base pool is the actual
over-charge: shape F's own boot reported
`Graph capturing finished in 4 secs, took -0.06 GiB` — a **negative** capture
delta, so the 256 MiB base was fiction. Charging `0 base + 12 MiB/graph`
(0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph
spec-decode shape pays 0.21 GiB) plus **util 1.0**, the field maximum, moves
the plan to budget 31.86 / graphs 0.07 / **arena 18.35 GiB / ~8,016 slots** —
+363 slots over F, +494 over E, and the ceiling of what a deploy can ask for.
The KV ask went to `3,166,500,000` B/rank, which the engine's own 7,891.8
B/token price lands at ~400,700 tokens (verify the `GPU KV cache size` line;
the price is shape-dependent, 7,701 B/token at 262144).
**The price of the force is the margin, and it is real.** At util 1.0 the
engine's budget equals the whole card, so any under-priced registry term stops
showing up as a plan refusal and starts showing up as a runtime allocation
failure. Measured driver high-water moves 28.28 GiB (E) -> 29.53 GiB (F) ->
~30.4 GiB (G), i.e. **~1.5 GiB of true free VRAM per card** is now the safety
margin — against 0.13 GiB on the bf16 boot that died and 0 bytes on the one
that OOM'd a real 995-token request. Rollback is one number:
`FRANK_PRIMARY_UTIL=0.982` costs 0.57 GiB of arena and buys back 0.57 GiB of
margin. Everything beyond ~8,016 slots needs the registry edited inside the
image (the fork's log line says exactly that: *"Update the registry entry if
this drifts"*), not a deploy.
**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
@@ -484,7 +504,9 @@ memory environment). The night searched two concurrency contracts:
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) |
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
| **3 x 200704, fp8, + `--expert-precise-memory`** | **DEPLOYED 2026-10-02 (shape E)**: same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
| **3 x 200704, fp8, + `--expert-precise-memory`** | shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
| 3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982 | shape F (2026-10-03): arena 17.52 GiB / **7,653 slots** measured, budget 31.29, `Estimated CUDA graph memory: 0.32 GiB`, KV unchanged 398,658 |
| **3 x 200704, fp8, graph term 0.07 GiB, util 1.00, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena 18.35 GiB / ~8,016 slots (+363 over F, +494 over E), budget 31.86 = the whole card, driver high-water ~30.4 GiB -> **~1.5 GiB real free per card is the margin**; rollback is UTIL back to 0.982 |
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
tests/frank_primary_unit.bats
+26 -20
View File
@@ -117,8 +117,8 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
# runtime headroom, not arena.
grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=1.0$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=3166500000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
# --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
@@ -179,13 +179,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
# Shape F: the CUDA-graph plan term is carried as CONTAINER env (the fork
# Shape F/G: the CUDA-graph plan term is carried as CONTAINER env (the fork
# reads it at call time inside estimate_cudagraph_memory), and both must be
# present — the graph term is the only plan term this repo can shrink once
# UTIL is capped, and shrinking it is what grows the arena.
# present — the graph term and UTIL are the ONLY plan terms this repo can
# move, and shrinking them is what grows the arena. The base must be shipped
# explicitly: unset, the estimator would fall back to its 256 MiB default.
grep -q -- '-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"' "$SERVE"
grep -q -- '-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"' "$SERVE"
grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12$' "$ENV_FILE"
}
@@ -264,25 +265,30 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
}
@test "MNBT is 2048 and the CUDA-graph plan term is priced at the measured cost" {
# MNBT: 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
# interchangeable on decode; 12288 costs 2.5% plus arena. 2048 is arena-FREE
# because the registry prices overhead as base + 0.22 GiB per 1024 ABOVE its
# nbt 4096 anchor — every rung <=4096 charges the same 5.25 GiB, and the
# fitted 5.25 was fitted AT nbt 4096, so 2048 sits under the anchor.
# CUDA graphs: the arena is budget - non-expert - KV reserve - graphs -
# overhead, so every GiB the plan stops charging for graphs is a GiB of
# expert cache. The fork's analytic default (256 MiB + 52 MiB/graph) charged
# 0.55 GiB where the engine's post-capture line measured 0.31 GiB on the
# heavier rank; 256 + 12/graph prices the same 6-graph plan at 0.32 GiB.
# The pair must stay BELOW the analytic 563 MiB default (that is the win)
# and ABOVE the measured 317 MiB (under-pricing is what OOMs at capture).
@test "the arena is forced to the flag ceiling and the KV ask is honoured" {
# The planner sizes expert cache as budget - non-expert - KV reserve -
# graphs - overhead, and two of those five terms are reachable from a
# deploy: budget (= device_total x util, field-capped at le=1) and graphs
# (VLLM_CUDAGRAPH_MEM_*). So the shipped shape must sit at the ceiling of
# BOTH: util at its field maximum, and a graph charge strictly below the
# fork's analytic 563 MiB default while still paying per graph (the measured
# post-capture delta is -0.06 GiB, i.e. the base pool is the over-charge, but
# a future spec-decode shape captures 18 graphs and must still be charged).
# MNBT stays 2048: 512 doubles TTFT at 4064 ctx, 1024-4096 are decode-
# interchangeable AND arena-identical (the registry prices overhead flat up
# to its nbt 4096 anchor), 12288 costs 2.5% plus arena.
# The KV ask: the pool is priced at the engine's measured 7,892 B/token at
# this shape, and a served conversation needs 200,704 tokens, so the pool
# must clear 400,700 tokens to keep two full-length resident.
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
awk -F= '/^FRANK_PRIMARY_UTIL=/{exit !($2+0 >= 1.0)}' "$ENV_FILE"
base=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=/{print $2}' "$ENV_FILE")
per_graph=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=/{print $2}' "$ENV_FILE")
[ "$base" -eq 0 ] && [ "$per_graph" -gt 0 ]
[ $((base + 6 * per_graph)) -lt 563 ]
[ $((base + 6 * per_graph)) -gt 317 ]
kv=$(awk -F= '/^FRANK_PRIMARY_KV_BYTES=/{print $2}' "$ENV_FILE")
[ $((kv / 7892)) -ge 400700 ]
}
@test "expert prefetch is absent BY LAW, not by omission" {