-
frank-primary: restore fp8 3x262144 (bf16 boot fails on a dynamo constraint)
dank released this
2026-10-01 14:40:17 -07:00 | 483 commits to main since this releasebf16 at 2 x 262144 fails with and without --expert-precise-memory. The real
error (captured from the journal this time, container fixed via journald
rather than podman logs):RuntimeError: Worker failed with error 'Constraints violated
(L['query_start_loc'].size()[0])! You marked it as dynamic but your code
specialized it to be a constant (3)'- a torch.compile/dynamo dynamic-shape violation, NOT an OOM. Ruled out a
stale compile cache: ~/.cache/vllm holds only modelinfos (16K) and each
podman run gets a fresh container filesystem.
The deltas from the last shape measured serving (fp8 3 x 262144 / MNBT 2048 /
792,069 tokens / 3.02x / arena 6,248) are KV dtype (fp8 -> bf16) and SEQS
(3 -> 2). Restoring the known-good shape; bisect bf16 vs SEQS in a window,
with --enforce-eager as the bypass.Downloads
- a torch.compile/dynamo dynamic-shape violation, NOT an OOM. Ruled out a