dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: shape F — MNBT 2048, util 0.982, CUDA-graph plan term pinned to the measured capture

Browse Source 
The arena is budget - non-expert - KV reserve - graphs - overhead. With util
capped at 0.982 (never 1.00) the graph estimate is the only plan term this repo
can shrink: the fork charges an analytic 256 MiB + 52 MiB/graph = 0.55 GiB where
the engine's own post-capture line reports 0.31 GiB, so pricing it at
256 + 12/graph returns ~0.24 GiB to the expert arena. MNBT 1024 -> 2048 is
arena-free: the registry overhead is base + 0.22 GiB per 1024 ABOVE its nbt 4096
anchor, and the fitted 5.25 GiB was fitted AT 4096.
This commit is contained in:
Frank
2026-10-03 02:22:49 -07:00
parent 6584c059d6
commit c682c6329c
5 changed files with 165 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_primary_unit.bats
+33 -13
View File
@@ -18,7 +18,8 @@
# laws: expert prefetch ABSENT (36% precision), NO speculative decode (the
# no-spec family measured fastest once the prefetch tax was removed; MTP
# re-measures pending), gather geometry 32x64 PRESENT (+18.7% decode),
# MNBT 1024 (Frank's lock), and the client contract (reasoning parser qwen3,
# MNBT 2048, the CUDA-graph plan term pinned to the measured capture cost, and
# the client contract (reasoning parser qwen3,
# tool parser qwen3_xml, preserve_thinking on).
load helpers
@@ -95,8 +96,9 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
# 241664 asked 10.70 GiB against 9.18 GiB free); SEQS=5 fp8 x 262144
# BOOTED (1,350,182 / 5.15x) but left ~0 bytes free and a real 995-token
# request killed the worker with `Tried to allocate 50.00 MiB ... 0 bytes
# free`. UTIL stays 0.98 (lowering it shrinks the engine's own budget
# instead of buying slack).
# free`. UTIL is a REAL arena lever (the fork plans budget = total x util)
# but a weak one, and Frank caps it at 0.982 — never 1.00 (see
# FRANK_PRIMARY_UTIL for the two ceiling deaths that justify the margin).
# SEQS=2 is IMPOSSIBLE on this model, measured 2026-10-02 by
# single-variable boot probe against the then-deployed shape: SEQS 2 fails
# at BOTH MNBT 1024 and 2048 with torch.fx ConstraintViolationError
@@ -115,9 +117,9 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
# runtime headroom, not arena.
grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
# --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
# while the 2 x 262144 bf16 boot sat 0.13 GiB under the 31.86 GiB ceiling
@@ -177,6 +179,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
# Shape F: the CUDA-graph plan term is carried as CONTAINER env (the fork
# reads it at call time inside estimate_cudagraph_memory), and both must be
# present — the graph term is the only plan term this repo can shrink once
# UTIL is capped, and shrinking it is what grows the arena.
grep -q -- '-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"' "$SERVE"
grep -q -- '-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"' "$SERVE"
grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12$' "$ENV_FILE"
}
@test "tools and reasoning are wired — the client contract is served" {
@@ -254,15 +264,25 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
}
@test "MNBT is at Frank's 1024 with the arena intact" {
# MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
# interchangeable on decode AND identical on arena (the sweep measured
# 6,499 slots at every rung <=4096); 12288 costs 2.5% plus arena.
# 1024 is Frank's lock — it keeps the workspace floor so the expert arena
# stays max. (2026-10-02: MNBT 2048 compiles the first graph in 26 s vs
# 172 s at 1024 — a startup-cost lever, not a correctness one.)
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
@test "MNBT is 2048 and the CUDA-graph plan term is priced at the measured cost" {
# MNBT: 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
# interchangeable on decode; 12288 costs 2.5% plus arena. 2048 is arena-FREE
# because the registry prices overhead as base + 0.22 GiB per 1024 ABOVE its
# nbt 4096 anchor — every rung <=4096 charges the same 5.25 GiB, and the
# fitted 5.25 was fitted AT nbt 4096, so 2048 sits under the anchor.
# CUDA graphs: the arena is budget - non-expert - KV reserve - graphs -
# overhead, so every GiB the plan stops charging for graphs is a GiB of
# expert cache. The fork's analytic default (256 MiB + 52 MiB/graph) charged
# 0.55 GiB where the engine's post-capture line measured 0.31 GiB on the
# heavier rank; 256 + 12/graph prices the same 6-graph plan at 0.32 GiB.
# The pair must stay BELOW the analytic 563 MiB default (that is the win)
# and ABOVE the measured 317 MiB (under-pricing is what OOMs at capture).
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
base=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=/{print $2}' "$ENV_FILE")
per_graph=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=/{print $2}' "$ENV_FILE")
[ $((base + 6 * per_graph)) -lt 563 ]
[ $((base + 6 * per_graph)) -gt 317 ]
}
@test "expert prefetch is absent BY LAW, not by omission" {