frank-primary: revert to known-good fp8 3x262144
The 2 x 200704 fp8 / MNBT 1024 shape (30904ba) crash-loops at boot: the
engine dies in determine_available_memory -> profile_run -> _dummy_run with
the SAME torch.compile dynamo constraint that killed the bf16 2 x 262144
shape on 2026-10-01:
RuntimeError: Worker failed with error 'Constraints violated
(L['query_start_loc'].size()[0])! You marked it as dynamic but your code
specialized it to be a constant (3)'
Not an OOM. Restoring the last shape measured serving (fp8 3 x 262144 /
KV 6.1e9 / MNBT 2048 / arena 6,248) so the endpoint serves again; the
desired edits are re-applied one at a time from this base.
This commit is contained in:
@@ -2,11 +2,10 @@
|
||||
# Seam: frank-primary — the Dankserver LLM endpoint on :7770.
|
||||
#
|
||||
# This file asserts the tcclaviger RDNA4 vLLM ROOT-UNIT shape (the
|
||||
# 2026-10-02 correction of the 2026-10-01 BF16 cutover): the serve
|
||||
# 2026-10-01 BF16 cutover, the overnight redo campaign's verdict): the serve
|
||||
# script drives `podman run` DIRECTLY (the image entrypoint execs
|
||||
# `vllm serve`), with every engine flag qualified by boot probe and carried
|
||||
# in the repo env file.
|
||||
# The seam pins what the flags alone do not say: the
|
||||
# in the repo env file. The seam pins what the flags alone do not say: the
|
||||
# KV qualification law (the night ladder: 262144/245760 OOM at graph
|
||||
# capture; 241664 boots at 732,359 tokens / 3.03x, util 0.98;
|
||||
# the bf16-KV trade (double the attention-KV bytes/token, ~14% decode cost,
|
||||
@@ -92,31 +91,29 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
|
||||
@test "the qualified engine shape is in the env file" {
|
||||
# 2026-10-01 night ladder (Frank's method), then a runtime-survivability
|
||||
# correction, then Frank's 2026-10-02 decision to give the pool back to
|
||||
# the expert arena. SEQS=3 bf16 topped out at 241664 (KV 732,359 / 3.03x);
|
||||
# correction. SEQS=3 bf16 topped out at 241664 (KV 732,359 / 3.03x);
|
||||
# SEQS=6 fp8 does NOT fit (6 x 241664 asked 10.70 GiB against 9.18 GiB
|
||||
# free); SEQS=5 fp8 x 262144 BOOTED (KV 1,350,182 / 5.15x) but left ~0
|
||||
# bytes free at runtime and a real client request (995-token prompt with
|
||||
# max_tokens=261149) killed the worker with `Tried to allocate 50.00 MiB
|
||||
# ... 0 bytes free`. 200704 sits below the 241664 rung while leaving a
|
||||
# full 2.95 GiB/rank to cut loose for the arena. UTIL stays 0.98
|
||||
# (lowering it shrinks the engine's own budget instead of buying slack).
|
||||
# FINAL (2026-10-02, Frank): TWO windows at 200704 with FP8 KV and a pool
|
||||
# cut for the arena. fp8 costs 7,701 B/token (measured 6,138 attention +
|
||||
# 1,563 mamba/side), so 2 x 200704 = 401,408 tokens needs 3.091e9 B;
|
||||
# pinned at 3.15e9 (~405k tokens, ~44 MB margin). The arena is what
|
||||
# remains — the planner sizes it AFTER the KV override — so the 6,248
|
||||
# slots of the 6.1e9 shape become roughly 7,500 here.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=200704$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE"
|
||||
# ... 0 bytes free`. FOUR slots at the full native 262144 window is the
|
||||
# deployed shape: ~1.93 GiB more headroom per rank for
|
||||
# activations/graphs. UTIL stays 0.98 (lowering it shrinks the engine's
|
||||
# own budget instead of buying slack).
|
||||
# FINAL (2026-10-01, Frank): TWO full-size windows with BF16 KV plus
|
||||
# --expert-precise-memory. bf16 costs 13,839 B/token (measured), so
|
||||
# 2 x 262144 = 524,288 tokens needs 7.256e9 B; pinned at 7.3e9 (~527.5k
|
||||
# tokens, 3,206 margin). The precise flag drops the plan's overhead
|
||||
# 7.60 -> 5.25 GiB, moving the arena from 6,248 to ~7,278 slots.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=6100000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
# --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the
|
||||
# boot 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident)
|
||||
# and crash-looped at graph capture. The serve script keeps the opt-in
|
||||
# flag; the smaller pool here is the arena it would have been paying for.
|
||||
# --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the boot
|
||||
# 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and
|
||||
# crash-looped at graph capture. The serve script keeps the opt-in flag.
|
||||
! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
@@ -124,22 +121,6 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the pool is sized as the arena floor, not the context floor" {
|
||||
# THE LAW this seam pins: the fork's expert planner sizes the arena
|
||||
# AFTER the KV override, so every byte cut out of KV_BYTES becomes
|
||||
# arena. A context-size change must move the pool with it, or the
|
||||
# reduction buys nothing. 2 x 200704 = 401,408 tokens at fp8's 7,701
|
||||
# B/token is 3.091e9 B; the pin is 3.15e9, ~44 MB of margin above the
|
||||
# floor — the same convention the 6.1e9 shape used.
|
||||
floor=$(awk -v n="401408" -v b="7701" 'BEGIN{printf "%.0f", n*b}')
|
||||
[ "$floor" = "3091243008" ]
|
||||
kv=$(grep '^FRANK_PRIMARY_KV_BYTES=' "$ENV_FILE" | cut -d= -f2)
|
||||
[ "$kv" = "3150000000" ]
|
||||
margin=$(( kv - floor ))
|
||||
[ "$margin" -gt 40000000 ]
|
||||
[ "$margin" -lt 100000000 ]
|
||||
}
|
||||
|
||||
@test "the serve script wires the engine flags from the env" {
|
||||
grep -q -- '--max-num-seqs "$FRANK_PRIMARY_SEQS"' "$SERVE"
|
||||
grep -q -- '--max-model-len "$FRANK_PRIMARY_MAXLEN"' "$SERVE"
|
||||
@@ -227,14 +208,13 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
grep -q -- '-e R4D_LRU_GATHER_CHUNKS="$FRANK_PRIMARY_GATHER_CHUNKS"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "MNBT is at Frank's 1024 with the arena intact" {
|
||||
@test "MNBT is at Frank's 2048 with the arena intact" {
|
||||
# MNBT 512 doubles TTFT at 4064 ctx (2.46 s vs 1.36 s); 1024-4096 are
|
||||
# interchangeable on decode AND identical on arena (the sweep measured
|
||||
# 6,499 slots at every rung <=4096); 12288 costs 2.5% plus arena.
|
||||
# 1024 is the 2026-10-02 setting — the pool was cut to feed the expert
|
||||
# arena, so the workspace floor matters more than it did at 2048: the
|
||||
# arena budget is what is left, and 1024-4096 pay nothing for it.
|
||||
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
|
||||
# 2048 is Frank's setting as of 2026-10-01 — it keeps the workspace floor
|
||||
# so the expert arena stays max.
|
||||
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
|
||||
grep -q -- '--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"' "$SERVE"
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user