frank-primary: shape G — force the arena to the flag ceiling, KV to 400,700 tokens
Two plan terms were still unspent: the CUDA-graph charge (shape F's own boot reported a NEGATIVE capture delta, -0.06 GiB, so the 256 MiB base pool was fiction -> charge 0 + 12 MiB/graph = 0.07 GiB) and the budget (util 0.982 -> 1.0, the field's le=1 maximum). Together +0.82 GiB of expert cache: plan target 18.35 GiB / ~8,016 slots. KV_BYTES 3.15e9 -> 3.1665e9 = the 400,700-token ask at the engine's measured 7,891.8 B/token. Cost of the force, recorded: driver high-water ~30.4 GiB of 31.86, so ~1.5 GiB of real margin per card; rollback is UTIL=0.982.
This commit is contained in:
1 parent
c682c6329c
commit
0a4ff30db2
3 files changed
+108
-65
No files matched your search
@@ -117,8 +117,8 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# runtime headroom, not arena.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.982$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=1.0$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=3166500000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
# --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
|
||||
@@ -179,13 +179,14 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
|
||||
grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
|
||||
grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
|
||||
# Shape F: the CUDA-graph plan term is carried as CONTAINER env (the fork
|
||||
# Shape F/G: the CUDA-graph plan term is carried as CONTAINER env (the fork
|
||||
# reads it at call time inside estimate_cudagraph_memory), and both must be
|
||||
# present — the graph term is the only plan term this repo can shrink once
|
||||
# UTIL is capped, and shrinking it is what grows the arena.
|
||||
# present — the graph term and UTIL are the ONLY plan terms this repo can
|
||||
# move, and shrinking them is what grows the arena. The base must be shipped
|
||||
# explicitly: unset, the estimator would fall back to its 256 MiB default.
|
||||
grep -q -- '-e VLLM_CUDAGRAPH_MEM_BASE_MB="$FRANK_PRIMARY_CUDAGRAPH_BASE_MB"' "$SERVE"
|
||||
grep -q -- '-e VLLM_CUDAGRAPH_MEM_PER_GRAPH_MB="$FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB"' "$SERVE"
|
||||
grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=256$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=0$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=12$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@@ -264,25 +265,30 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "MNBT is 2048 and the CUDA-graph plan term is priced at the measured cost" {
|
||||
# MNBT: 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
|
||||
# interchangeable on decode; 12288 costs 2.5% plus arena. 2048 is arena-FREE
|
||||
# because the registry prices overhead as base + 0.22 GiB per 1024 ABOVE its
|
||||
# nbt 4096 anchor — every rung <=4096 charges the same 5.25 GiB, and the
|
||||
# fitted 5.25 was fitted AT nbt 4096, so 2048 sits under the anchor.
|
||||
# CUDA graphs: the arena is budget - non-expert - KV reserve - graphs -
|
||||
# overhead, so every GiB the plan stops charging for graphs is a GiB of
|
||||
# expert cache. The fork's analytic default (256 MiB + 52 MiB/graph) charged
|
||||
# 0.55 GiB where the engine's post-capture line measured 0.31 GiB on the
|
||||
# heavier rank; 256 + 12/graph prices the same 6-graph plan at 0.32 GiB.
|
||||
# The pair must stay BELOW the analytic 563 MiB default (that is the win)
|
||||
# and ABOVE the measured 317 MiB (under-pricing is what OOMs at capture).
|
||||
@test "the arena is forced to the flag ceiling and the KV ask is honoured" {
|
||||
# The planner sizes expert cache as budget - non-expert - KV reserve -
|
||||
# graphs - overhead, and two of those five terms are reachable from a
|
||||
# deploy: budget (= device_total x util, field-capped at le=1) and graphs
|
||||
# (VLLM_CUDAGRAPH_MEM_*). So the shipped shape must sit at the ceiling of
|
||||
# BOTH: util at its field maximum, and a graph charge strictly below the
|
||||
# fork's analytic 563 MiB default while still paying per graph (the measured
|
||||
# post-capture delta is -0.06 GiB, i.e. the base pool is the over-charge, but
|
||||
# a future spec-decode shape captures 18 graphs and must still be charged).
|
||||
# MNBT stays 2048: 512 doubles TTFT at 4064 ctx, 1024-4096 are decode-
|
||||
# interchangeable AND arena-identical (the registry prices overhead flat up
|
||||
# to its nbt 4096 anchor), 12288 costs 2.5% plus arena.
|
||||
# The KV ask: the pool is priced at the engine's measured 7,892 B/token at
|
||||
# this shape, and a served conversation needs 200,704 tokens, so the pool
|
||||
# must clear 400,700 tokens to keep two full-length resident.
|
||||
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
|
||||
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
|
||||
awk -F= '/^FRANK_PRIMARY_UTIL=/{exit !($2+0 >= 1.0)}' "$ENV_FILE"
|
||||
base=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_BASE_MB=/{print $2}' "$ENV_FILE")
|
||||
per_graph=$(awk -F= '/^FRANK_PRIMARY_CUDAGRAPH_PER_GRAPH_MB=/{print $2}' "$ENV_FILE")
|
||||
[ "$base" -eq 0 ] && [ "$per_graph" -gt 0 ]
|
||||
[ $((base + 6 * per_graph)) -lt 563 ]
|
||||
[ $((base + 6 * per_graph)) -gt 317 ]
|
||||
kv=$(awk -F= '/^FRANK_PRIMARY_KV_BYTES=/{print $2}' "$ENV_FILE")
|
||||
[ $((kv / 7892)) -ge 400700 ]
|
||||
}
|
||||
|
||||
@test "expert prefetch is absent BY LAW, not by omission" {
|
||||
|
||||
Reference in new issue
Block a user