frank-primary: restore fp8 3x262144 (bf16 boot fails on a dynamo constraint)
bf16 at 2 x 262144 fails with and without --expert-precise-memory. The real error (captured from the journal this time, container fixed via journald rather than podman logs): RuntimeError: Worker failed with error 'Constraints violated (L['query_start_loc'].size()[0])! You marked it as dynamic but your code specialized it to be a constant (3)' - a torch.compile/dynamo dynamic-shape violation, NOT an OOM. Ruled out a stale compile cache: ~/.cache/vllm holds only modelinfos (16K) and each podman run gets a fresh container filesystem. The deltas from the last shape measured serving (fp8 3 x 262144 / MNBT 2048 / 792,069 tokens / 3.02x / arena 6,248) are KV dtype (fp8 -> bf16) and SEQS (3 -> 2). Restoring the known-good shape; bisect bf16 vs SEQS in a window, with --enforce-eager as the bypass.
This commit is contained in:
1 parent
d3d180fd3f
commit
e703b0c15c
3 files changed
+42
-8
No files matched your search
@@ -59,7 +59,14 @@ FRANK_PRIMARY_PUBLISH=7770:7770
|
||||
# hoc experiment identity (both hold the GPUs / name registrations).
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
|
||||
# --- qualified engine shape (2026-10-01, final: 2 bf16 windows + precise mem) --
|
||||
# --- qualified engine shape (2026-10-01, RESTORED KNOWN-GOOD: fp8 3 windows) ---
|
||||
# bf16 at 2 x 262144 did NOT stay up: with and without --expert-precise-memory
|
||||
# the boot failed (`Engine core initialization failed`, NRestarts climbing).
|
||||
# The last shape measured serving was fp8 3 x 262144 / MNBT 2048 / KV 6.1e9
|
||||
# (KV 792,069 tokens, 3.02x, arena 6,248, ~3.6 GiB free) and is restored here
|
||||
# while the bf16 path is investigated in a window (see log.md 2026-10-01).
|
||||
# Candidate next test: bf16 with SEQS=3 to isolate the KV dtype from the
|
||||
# sequence count, and/or --enforce-eager to bypass the dynamo constraint path.
|
||||
# SEQS 2 x MAXLEN 262144 at BF16 KV (Frank's call, 2026-10-01): two contexts of
|
||||
# the model's FULL native window with UNQUANTIZED KV, plus
|
||||
# --expert-precise-memory to hand the arena the compile/graph-capture headroom.
|
||||
@@ -85,9 +92,9 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8"
|
||||
# MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so
|
||||
# 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096.
|
||||
FRANK_PRIMARY_MAXLEN=262144
|
||||
FRANK_PRIMARY_SEQS=2
|
||||
FRANK_PRIMARY_KV_BYTES=7300000000
|
||||
FRANK_PRIMARY_KV_DTYPE=auto
|
||||
FRANK_PRIMARY_SEQS=3
|
||||
FRANK_PRIMARY_KV_BYTES=6100000000
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8
|
||||
FRANK_PRIMARY_MNBT=2048
|
||||
FRANK_PRIMARY_GATHER_CHUNKS=32
|
||||
FRANK_PRIMARY_GATHER_LANES=64
|
||||
|
||||
Reference in new issue
Block a user