From e703b0c15c60767968869d4fad0a1c67af9a5616 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 1 Oct 2026 14:40:17 -0700 Subject: [PATCH] frank-primary: restore fp8 3x262144 (bf16 boot fails on a dynamo constraint) bf16 at 2 x 262144 fails with and without --expert-precise-memory. The real error (captured from the journal this time, container fixed via journald rather than podman logs): RuntimeError: Worker failed with error 'Constraints violated (L['query_start_loc'].size()[0])! You marked it as dynamic but your code specialized it to be a constant (3)' - a torch.compile/dynamo dynamic-shape violation, NOT an OOM. Ruled out a stale compile cache: ~/.cache/vllm holds only modelinfos (16K) and each podman run gets a fresh container filesystem. The deltas from the last shape measured serving (fp8 3 x 262144 / MNBT 2048 / 792,069 tokens / 3.02x / arena 6,248) are KV dtype (fp8 -> bf16) and SEQS (3 -> 2). Restoring the known-good shape; bisect bf16 vs SEQS in a window, with --enforce-eager as the bypass. --- scripts/frank-primary.env | 15 +++++++++++---- serverAppConfigs-docs/log.md | 27 +++++++++++++++++++++++++++ tests/frank_primary_unit.bats | 8 ++++---- 3 files changed, 42 insertions(+), 8 deletions(-) diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index 372497e..c3107a3 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -59,7 +59,14 @@ FRANK_PRIMARY_PUBLISH=7770:7770 # hoc experiment identity (both hold the GPUs / name registrations). FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" -# --- qualified engine shape (2026-10-01, final: 2 bf16 windows + precise mem) -- +# --- qualified engine shape (2026-10-01, RESTORED KNOWN-GOOD: fp8 3 windows) --- +# bf16 at 2 x 262144 did NOT stay up: with and without --expert-precise-memory +# the boot failed (`Engine core initialization failed`, NRestarts climbing). +# The last shape measured serving was fp8 3 x 262144 / MNBT 2048 / KV 6.1e9 +# (KV 792,069 tokens, 3.02x, arena 6,248, ~3.6 GiB free) and is restored here +# while the bf16 path is investigated in a window (see log.md 2026-10-01). +# Candidate next test: bf16 with SEQS=3 to isolate the KV dtype from the +# sequence count, and/or --enforce-eager to bypass the dynamo constraint path. # SEQS 2 x MAXLEN 262144 at BF16 KV (Frank's call, 2026-10-01): two contexts of # the model's FULL native window with UNQUANTIZED KV, plus # --expert-precise-memory to hand the arena the compile/graph-capture headroom. @@ -85,9 +92,9 @@ FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-r9v qwen-fp8" # MNBT 2048 (Frank): the overhead formula only adds bytes above nbt 4096, so # 2048 pays nothing, and the arena sweep measured the same ceiling <= 4096. FRANK_PRIMARY_MAXLEN=262144 -FRANK_PRIMARY_SEQS=2 -FRANK_PRIMARY_KV_BYTES=7300000000 -FRANK_PRIMARY_KV_DTYPE=auto +FRANK_PRIMARY_SEQS=3 +FRANK_PRIMARY_KV_BYTES=6100000000 +FRANK_PRIMARY_KV_DTYPE=fp8 FRANK_PRIMARY_MNBT=2048 FRANK_PRIMARY_GATHER_CHUNKS=32 FRANK_PRIMARY_GATHER_LANES=64 diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index da38cbe..455efd7 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -663,6 +663,33 @@ - `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage. - `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership. +## 2026-10-01 — frank-primary RESTORED to fp8 3 x 262144 (bf16 boot fails) + +bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without +--expert-precise-memory: + +- with precise: NRestarts 1->3, each attempt ~6.7 min then + `Deactivated successfully` (API server drains after an EngineCore fatal + error) — cause not captured, the serve script rm'd the container. +- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts + climbing, container `Exited (1)`, and the journal finally showed the real + error: + `RuntimeError: Worker failed with error 'Constraints violated + (L['query_start_loc'].size()[0])! You marked L['query_start_loc'].size()[0] + as dynamic but your code specialized it to be a constant (3)'` + — a torch.compile/dynamo dynamic-shape constraint violation, NOT an OOM. + `~/.cache/vllm` holds only `modelinfos` (16K) and each `podman run` gets a + fresh container filesystem, so a stale compile cache is ruled out: the + conflict is produced by this configuration's own compile path. + +Differences from the last shape measured serving (fp8 3 x 262144, MNBT 2048, +KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype +(fp8 -> bf16) and SEQS (3 -> 2). Both are candidates for the dynamo trigger; +`--enforce-eager` is the bypass that would sidestep the constraint path. + +Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag. +Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype). + ## 2026-10-01 — frank-primary: --expert-precise-memory + bf16 = boot crash loop **Reverted.** The two changes Frank asked for were applied together and the diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats index 35cc95d..9822ec1 100755 --- a/tests/frank_primary_unit.bats +++ b/tests/frank_primary_unit.bats @@ -106,11 +106,11 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # tokens, 3,206 margin). The precise flag drops the plan's overhead # 7.60 -> 5.25 GiB, moving the arena from 6,248 to ~7,278 slots. grep -q '^FRANK_PRIMARY_MAXLEN=262144$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_SEQS=2$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_SEQS=3$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_UTIL=0.98$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_KV_BYTES=7300000000$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_KV_BYTES=6100000000$' "$ENV_FILE" grep -q '^FRANK_PRIMARY_MNBT=2048$' "$ENV_FILE" - grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE" # --expert-precise-memory is NOT set: with bf16 2 x 262144 it put the boot # 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and # crash-looped at graph capture. The serve script keeps the opt-in flag. @@ -193,7 +193,7 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh" # is two full-size windows instead of three; in exchange the KV is exact # (no quantization error) and measured +4% decode at shallow depth, where # fp8's dequant cost dominates over the extra bytes read. - grep -q '^FRANK_PRIMARY_KV_DTYPE=auto$' "$ENV_FILE" + grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE" grep -q -- '--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE"' "$SERVE" }