frank-primary: restore fp8 3x262144 (bf16 boot fails on a dynamo constraint)
bf16 at 2 x 262144 fails with and without --expert-precise-memory. The real error (captured from the journal this time, container fixed via journald rather than podman logs): RuntimeError: Worker failed with error 'Constraints violated (L['query_start_loc'].size()[0])! You marked it as dynamic but your code specialized it to be a constant (3)' - a torch.compile/dynamo dynamic-shape violation, NOT an OOM. Ruled out a stale compile cache: ~/.cache/vllm holds only modelinfos (16K) and each podman run gets a fresh container filesystem. The deltas from the last shape measured serving (fp8 3 x 262144 / MNBT 2048 / 792,069 tokens / 3.02x / arena 6,248) are KV dtype (fp8 -> bf16) and SEQS (3 -> 2). Restoring the known-good shape; bisect bf16 vs SEQS in a window, with --enforce-eager as the bypass.
This commit is contained in:
1 parent
d3d180fd3f
commit
e703b0c15c
3 files changed
+42
-8
No files matched your search
@@ -663,6 +663,33 @@
|
||||
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
|
||||
|
||||
## 2026-10-01 — frank-primary RESTORED to fp8 3 x 262144 (bf16 boot fails)
|
||||
|
||||
bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without
|
||||
--expert-precise-memory:
|
||||
|
||||
- with precise: NRestarts 1->3, each attempt ~6.7 min then
|
||||
`Deactivated successfully` (API server drains after an EngineCore fatal
|
||||
error) — cause not captured, the serve script rm'd the container.
|
||||
- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts
|
||||
climbing, container `Exited (1)`, and the journal finally showed the real
|
||||
error:
|
||||
`RuntimeError: Worker failed with error 'Constraints violated
|
||||
(L['query_start_loc'].size()[0])! You marked L['query_start_loc'].size()[0]
|
||||
as dynamic but your code specialized it to be a constant (3)'`
|
||||
— a torch.compile/dynamo dynamic-shape constraint violation, NOT an OOM.
|
||||
`~/.cache/vllm` holds only `modelinfos` (16K) and each `podman run` gets a
|
||||
fresh container filesystem, so a stale compile cache is ruled out: the
|
||||
conflict is produced by this configuration's own compile path.
|
||||
|
||||
Differences from the last shape measured serving (fp8 3 x 262144, MNBT 2048,
|
||||
KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype
|
||||
(fp8 -> bf16) and SEQS (3 -> 2). Both are candidates for the dynamo trigger;
|
||||
`--enforce-eager` is the bypass that would sidestep the constraint path.
|
||||
|
||||
Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag.
|
||||
Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype).
|
||||
|
||||
## 2026-10-01 — frank-primary: --expert-precise-memory + bf16 = boot crash loop
|
||||
|
||||
**Reverted.** The two changes Frank asked for were applied together and the
|
||||
|
||||
Reference in new issue
Block a user