frank-primary: shape G — force the arena to the flag ceiling, KV to 400,700 tokens

Two plan terms were still unspent: the CUDA-graph charge (shape F's own boot
reported a NEGATIVE capture delta, -0.06 GiB, so the 256 MiB base pool was
fiction -> charge 0 + 12 MiB/graph = 0.07 GiB) and the budget (util 0.982 -> 1.0,
the field's le=1 maximum). Together +0.82 GiB of expert cache: plan target
18.35 GiB / ~8,016 slots. KV_BYTES 3.15e9 -> 3.1665e9 = the 400,700-token ask at
the engine's measured 7,891.8 B/token. Cost of the force, recorded: driver
high-water ~30.4 GiB of 31.86, so ~1.5 GiB of real margin per card; rollback is
UTIL=0.982.
This commit is contained in:
Frank committed 2026-10-03 02:49:38 -07:00
1 parent c682c6329c
commit 0a4ff30db2
3 files changed
+108 -65

No files matched your search

+26 -20
View File
@@ -117,8 +117,8 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
# runtime headroom, not arena.
grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_UTIL=1.0$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_BYTES=3166500000$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
# --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
@@ -179,13 +179,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
# Shape F: the CUDA-graph plan term is carried as CONTAINER env (the fork
# Shape F/G: the CUDA-graph plan term is carried as CONTAINER env (the fork
# reads it at call time inside estimate_cudagraph_memory), and both must be
# present — the graph term is the only plan term this repo can shrink once
# UTIL is capped, and shrinking it is what grows the arena.
# present — the graph term and UTIL are the ONLY plan terms this repo can
# move, and shrinking them is what grows the arena. The base must be shipped
# explicitly: unset, the estimator would fall back to its 256 MiB default.
grep -q -- '-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"' "$SERVE"
grep -q -- '-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"' "$SERVE"
grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12$' "$ENV_FILE"
}
@@ -264,25 +265,30 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
}
@test "MNBT is 2048 and the CUDA-graph plan term is priced at the measured cost" {
# MNBT: 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
# interchangeable on decode; 12288 costs 2.5% plus arena. 2048 is arena-FREE
# because the registry prices overhead as base + 0.22 GiB per 1024 ABOVE its
# nbt 4096 anchor — every rung <=4096 charges the same 5.25 GiB, and the
# fitted 5.25 was fitted AT nbt 4096, so 2048 sits under the anchor.
# CUDA graphs: the arena is budget - non-expert - KV reserve - graphs -
# overhead, so every GiB the plan stops charging for graphs is a GiB of
# expert cache. The fork's analytic default (256 MiB + 52 MiB/graph) charged
# 0.55 GiB where the engine's post-capture line measured 0.31 GiB on the
# heavier rank; 256 + 12/graph prices the same 6-graph plan at 0.32 GiB.
# The pair must stay BELOW the analytic 563 MiB default (that is the win)
# and ABOVE the measured 317 MiB (under-pricing is what OOMs at capture).
@test "the arena is forced to the flag ceiling and the KV ask is honoured" {
# The planner sizes expert cache as budget - non-expert - KV reserve -
# graphs - overhead, and two of those five terms are reachable from a
# deploy: budget (= device_total x util, field-capped at le=1) and graphs
# (VLLM_CUDAGRAPH_MEM_*). So the shipped shape must sit at the ceiling of
# BOTH: util at its field maximum, and a graph charge strictly below the
# fork's analytic 563 MiB default while still paying per graph (the measured
# post-capture delta is -0.06 GiB, i.e. the base pool is the over-charge, but
# a future spec-decode shape captures 18 graphs and must still be charged).
# MNBT stays 2048: 512 doubles TTFT at 4064 ctx, 1024-4096 are decode-
# interchangeable AND arena-identical (the registry prices overhead flat up
# to its nbt 4096 anchor), 12288 costs 2.5% plus arena.
# The KV ask: the pool is priced at the engine's measured 7,892 B/token at
# this shape, and a served conversation needs 200,704 tokens, so the pool
# must clear 400,700 tokens to keep two full-length resident.
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
awk -F= '/^FRANK_PRIMARY_UTIL=/{exit !($2+0 >= 1.0)}' "$ENV_FILE"
base=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=/{print $2}' "$ENV_FILE")
per_graph=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=/{print $2}' "$ENV_FILE")
[ "$base" -eq 0 ] && [ "$per_graph" -gt 0 ]
[ $((base + 6 * per_graph)) -lt 563 ]
[ $((base + 6 * per_graph)) -gt 317 ]
kv=$(awk -F= '/^FRANK_PRIMARY_KV_BYTES=/{print $2}' "$ENV_FILE")
[ $((kv / 7892)) -ge 400700 ]
}
@test "expert prefetch is absent BY LAW, not by omission" {