frank-primary: restore fp8 3x262144 (bf16 boot fails on a dynamo constraint)
bf16 at 2 x 262144 fails with and without --expert-precise-memory. The real error (captured from the journal this time, container fixed via journald rather than podman logs): RuntimeError: Worker failed with error 'Constraints violated (L['query_start_loc'].size()[0])! You marked it as dynamic but your code specialized it to be a constant (3)' - a torch.compile/dynamo dynamic-shape violation, NOT an OOM. Ruled out a stale compile cache: ~/.cache/vllm holds only modelinfos (16K) and each podman run gets a fresh container filesystem. The deltas from the last shape measured serving (fp8 3 x 262144 / MNBT 2048 / 792,069 tokens / 3.02x / arena 6,248) are KV dtype (fp8 -> bf16) and SEQS (3 -> 2). Restoring the known-good shape; bisect bf16 vs SEQS in a window, with --enforce-eager as the bypass.
This commit is contained in:
1 parent
d3d180fd3f
commit
e703b0c15c
3 files changed
+42
-8
No files matched your search
@@ -106,11 +106,11 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# tokens, 3,206 margin). The precise flag drops the plan's overhead
|
||||
# 7.60 -> 5.25 GiB, moving the arena from 6,248 to ~7,278 slots.
|
||||
grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=7300000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=6100000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
# --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the boot
|
||||
# 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and
|
||||
# crash-looped at graph capture. The serve script keeps the opt-in flag.
|
||||
@@ -193,7 +193,7 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# is two full-size windows instead of three; in exchange the KV is exact
|
||||
# (no quantization error) and measured +4% decode at shallow depth, where
|
||||
# fp8's dequant cost dominates over the extra bytes read.
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE"
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user