From 9d8b18553d26d17561f01fc349bf0878b23f5fb4 Mon Sep 17 00:00:00 2001
From: Frank <frank@dankware>
Date: Mon, 5 Oct 2026 03:45:26 -0700
Subject: [PATCH] =?UTF-8?q?frank-primary:=20prefix=20caching=20OFF=20?=
 =?UTF-8?q?=E2=80=94=20the=20fork's=20QFN=20align-mode=20GDN=20state=20fau?=
 =?UTF-8?q?lt?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Measured 2026-10-04 18:46:07 PDT (live podman logs): 48 gdn_hip state
faults in one 0.6 s burst — 'state out of ssm-cache range: max|S|=0 with
non-finite values (slot=42 head=3|8)', one per TP rank, 23 GDN layers
each — then the process-lifetime report cap. The guard's own text in
this build names it: 'known Qwen3.8-Flash-Next (QFN) flaw under prefix
caching … Mitigation: --no-enable-prefix-caching'.

Mechanism: prefix caching puts QFN on mamba_cache_mode 'align', which
checkpoints each sequence's GDN recurrent state into a mamba page per
3136-token block and re-seeds it from that page when a request resumes
from a cached prefix. The NaN state landed at the 13th boundary
(13 x 3136 = 40,768 — the reported ~40k mark) and persisted in the page:
every later turn of that conversation re-seeded the same non-finite
state for two heads, silently (fault-report cap already spent).

RAM ruled out the same day: zero MCE/EDAC records since the boot under
test, 8 GiB in-band pattern stable under live load, 4800 MT/s is the
JEDEC floor on 5600-rated DIMMs, and the two-head single-slot NaN
signature is a state-path fault, not scattered bit corruption.

Fix: FRANK_PRIMARY_NO_PREFIX_CACHING=1 -> --no-enable-prefix-caching
(flag renders in both directions from the knob). Cost: every turn
re-prefills the whole history instead of at most 3135 tail tokens.

Seam updated: the prefix-cache pin now asserts the conditional render
and the shipped OFF value. Runbook carries the evidence + acceptance
criteria; all 368 tests pass.
---
 scripts/frank-primary.env                     | 41 ++++++++++++
 scripts/serve-frank-primary.sh                | 11 +++-
 ...server-qwen38-fp8-expert-offload-tuning.md | 65 +++++++++++++++++++
 tests/frank_primary_unit.bats                 | 16 ++++-
 4 files changed, 130 insertions(+), 3 deletions(-)

diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env
index deb2475..2572bb7 100644
--- a/scripts/frank-primary.env
+++ b/scripts/frank-primary.env
@@ -175,6 +175,47 @@ FRANK_PRIMARY_KV_DTYPE=fp8
 FRANK_PRIMARY_MNBT=2048
 FRANK_PRIMARY_GATHER_CHUNKS=32
 FRANK_PRIMARY_GATHER_LANES=64
+# PREFIX CACHING IS OFF — THE FORK'S OWN NAMED MITIGATION (2026-10-05).
+# MEASURED 2026-10-04, 18:46:07 PDT (5.6 h into the live boot, one running
+# request, prefix hit rate 48.1%): 48 gdn_hip state faults in one 0.6 s
+# burst, then the process-lifetime silence cap:
+#   [gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
+#   with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
+#   [gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
+#   [gdn_hip] ERROR: state fault report limit (32) reached; further faults
+#   are counted silently
+# One physical SSM slot, two heads, BOTH ranks (heads 3 and 8), 23 layers
+# each. The guard's own message in this build: "known Qwen3.8-Flash-Next
+# (QFN) flaw under prefix caching and is being actively worked; output
+# quality for this request may degrade. Mitigation:
+# --no-enable-prefix-caching".
+# MECHANISM: with prefix caching on, this fork puts QFN on
+# mamba_cache_mode 'align' (the default-for-this-arch branch in
+# config/vllm.py; 'all' is deprecated and 'none' is the no-prefix-cache
+# mode, so align is the only sound mode WITH the flag). Align mode
+# checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
+# mamba page every block, where block = 3136 tokens (the attention block
+# size is raised to 3136 so the page sizes match exactly — the boot log's
+# "Padding mamba page size by 0.13% ..."), and RE-SEEDS the state from that
+# page whenever a request resumes from a cached prefix. The corrupt state
+# landed at the 13th boundary: 13 x 3136 = 40,768 tokens — the "~40k" mark
+# where the request's output turned repetitive/corrupted. The NaN stayed
+# in the mamba page, so every later turn of that conversation re-seeded the
+# same non-finite state for those two heads (silent: the fault-report cap
+# was already spent), while fresh short sessions looked fine.
+# RAM RULED OUT THE SAME DAY: zero MCE/EDAC kernel records since the boot
+# under test, an 8 GiB in-band pattern check stable over 60 s under live
+# load, 4800 MT/s is the JEDEC DDR5 floor on 5600-rated DIMMs (not an
+# overclock), and the signature — a two-head, single-slot NaN state in one
+# burst — is a state-path fault, not scattered finite bit corruption.
+# COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole
+# history instead of at most 3135 tail tokens (a full 40k context costs
+# tens of seconds of TTFT per turn at this shape's prefill rate). A silently
+# corrupted conversation is worse than a slower first token.
+# RE-ENABLE PATH: if the fork ships a fixed align path, set this to empty,
+# redeploy, and re-verify by taking one conversation past 40,768 tokens
+# (decode must stay coherent and podman logs must show zero gdn_hip lines).
+FRANK_PRIMARY_NO_PREFIX_CACHING=1
 # UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
 # 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and
 # 0.982 is Frank's standing law: NEVER 1.00. The 1.00 attempt (2026-10-03,
diff --git a/scripts/serve-frank-primary.sh b/scripts/serve-frank-primary.sh
index 8c4d1b6..bbcf615 100755
--- a/scripts/serve-frank-primary.sh
+++ b/scripts/serve-frank-primary.sh
@@ -25,7 +25,9 @@
 # --max-num-batched-tokens 2048 (Frank's call 2026-10-03; arena-free because the
 # registry prices overhead flat up to its nbt 4096 anchor), MoE gather geometry
 # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
-# with depth), prefix caching, expert prefetch OFF (36% precision measured),
+# with depth), prefix caching OFF 2026-10-05 (the fork's QFN align-mode GDN
+# state fault — the evidence block is in frank-primary.env), expert prefetch
+# OFF (36% precision measured),
 # TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
 # instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
 # and GPU_MAX_HW_QUEUES=2; shape F pins the plan's CUDA-graph term to the
@@ -160,7 +162,12 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
   --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
   --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
   --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
-  --enable-prefix-caching \
+  # Prefix caching: the fork's QFN mamba align path corrupts GDN SSM state at
+  # ~40k tokens (2026-10-04, 48 gdn_hip state faults, one request). FRANK_
+  # PRIMARY_NO_PREFIX_CACHING flips to the fork-named mitigation
+  # (--no-enable-prefix-caching); empty keeps the historical flag. The
+  # evidence block is in frank-primary.env.
+  ${FRANK_PRIMARY_NO_PREFIX_CACHING:+--no-enable-prefix-caching}${FRANK_PRIMARY_NO_PREFIX_CACHING:---enable-prefix-caching} \
   --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
   --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
   --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
index c4ca2f1..b3954b3 100644
--- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
@@ -291,8 +291,73 @@ FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2   -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
 FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary   mounted at /root/.cache
 R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
 NO --speculative-config, NO --expert-prefetch
+FRANK_PRIMARY_NO_PREFIX_CACHING=1  --no-enable-prefix-caching (2026-10-05, the QFN align-mode GDN state fault — below)
 ```
 
+### 2026-10-05 — prefix caching OFF: the fork's QFN align-mode GDN state fault
+
+**Symptom (Frank, 2026-10-04):** output turned repetitive/corrupted at
+~40k tokens into a long conversation; short sessions stayed clean.
+
+**Evidence (live `podman logs frank-primary-qwen`, the 2026-10-04 18:19
+boot):** one 0.6 s burst at 18:46:07 PDT — 5.6 h into service, one
+running request, and the engine's own 18:46:08 log line reading
+"Prefix cache hit rate: 48.1%". 48 `gdn_hip` faults, then the
+process-lifetime silence cap:
+
+```
+[gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
+  with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
+[gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
+[gdn_hip] ERROR: state fault report limit (32) reached; further faults
+  are counted silently
+```
+
+One physical SSM slot (42), two heads (3 and 8 — one per TP rank), 23
+GDN layers each, the state non-finite (NaN). The guard's own text in
+this build: *"known Qwen3.8-Flash-Next (QFN) flaw under prefix caching
+and is being actively worked; output quality for this request may
+degrade. Mitigation: --no-enable-prefix-caching."*
+
+**Mechanism.** With prefix caching on, the fork defaults QFN to
+`mamba_cache_mode=align` (boot log: "Mamba cache mode is set to 'align'
+for Qwen4ExpForConditionalGeneration … when prefix caching is enabled";
+`config/cache.py`: "none" is the no-prefix-cache mode, "all" is
+deprecated — align is the only sound mode WITH the flag). Align mode
+checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
+mamba page per block and RE-SEEDS the state from that page when a
+request resumes from a cached prefix. The block size is raised to
+**3136 tokens** for the page-size equality (boot log: "Setting
+attention block size to 3136 tokens to ensure that attention page size
+is >= mamba page size" + "Padding mamba page size by 0.13% … exactly
+equal"). The corrupt state landed at the **13th boundary: 13 × 3136 =
+40,768 tokens** — the reported "~40k" mark. The NaN persisted in the
+mamba page, so every later turn of that conversation re-seeded the same
+non-finite state for those two heads (silently — the fault-report cap
+was already spent) while fresh short sessions looked fine.
+
+**RAM ruled out the same day** (the 2026-10-04 17:17 host reboot is when
+the current memory speed took effect): zero MCE/EDAC kernel records
+since boot; an 8 GiB in-band pattern check stable over 60 s under live
+load; 4800 MT/s is the JEDEC DDR5 floor on 5600 MT/s-rated DIMMs (not an
+overclock); and a two-head, single-slot NaN state is a state-path fault,
+not scattered finite bit corruption.
+
+**Fix.** `FRANK_PRIMARY_NO_PREFIX_CACHING=1` → `--no-enable-prefix-caching`
+(the fork-named mitigation; the flag renders in both directions from the
+knob so a fork default change cannot silently flip the contract). Cost:
+no prefix reuse — every turn re-prefills the whole history instead of at
+most 3135 tail tokens. A silently corrupted conversation is worse than a
+slower first token.
+
+**Acceptance (2026-10-05 deploy):** boot healthy with the expert plan
+unchanged (arena 17.77 GiB / 7,762 slots — the flag touches no plan
+term), "GPU KV cache size" still clears 400,700 tokens (if the
+align-mode block-size raise disappears the mamba per-token price moves
+and the pool may need `FRANK_PRIMARY_KV_BYTES` up a notch), zero
+`gdn_hip` lines, and one request whose context crosses 40,768 tokens
+in-session decodes coherently.
+
 ### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
 
 `rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats
index b0f42a0..59397d3 100755
--- a/tests/frank_primary_unit.bats
+++ b/tests/frank_primary_unit.bats
@@ -142,6 +142,16 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
+    # Prefix caching is OFF (2026-10-05): with the flag on, the fork runs
+    # QFN on mamba_cache_mode 'align', and its GDN state path went
+    # non-finite mid-conversation (48 gdn_hip state faults, one request,
+    # slot 42 heads 3/8, at the 13th 3136-token boundary = 40,768 — the
+    # "~40k corruption" report). The fork's own guard names the mitigation:
+    # --no-enable-prefix-caching. RAM was ruled out the same day (zero MCE,
+    # in-band pattern stable, 4800 MT/s is the JEDEC floor). The evidence
+    # block is FRANK_PRIMARY_NO_PREFIX_CACHING in the env file; re-enable
+    # only with a fixed align path and a re-verification past 40,768 tokens.
+    grep -q '^FRANK_PRIMARY_NO_PREFIX_CACHING=1$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
 }
 
@@ -161,7 +171,11 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     grep -q -- '${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"}' "$SERVE"
     grep -q -- '${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"}' "$SERVE"
     grep -q -- '--tensor-parallel-size 2' "$SERVE"
-    grep -q -- '--enable-prefix-caching' "$SERVE"
+    # Prefix caching renders exactly ONE flag from the knob: set ->
+    # --no-enable-prefix-caching (the fork-named mitigation), unset -> the
+    # historical --enable-prefix-caching. Both directions are explicit so a
+    # fork default change cannot silently flip the contract.
+    grep -qF '${FRANK_PRIMARY_NO_PREFIX_CACHING:+--no-enable-prefix-caching}${FRANK_PRIMARY_NO_PREFIX_CACHING:---enable-prefix-caching}' "$SERVE"
     grep -q -- '--trust-remote-code' "$SERVE"
     # DCP is opt-in: the flag is passed only when FRANK_PRIMARY_DCP is set
     # (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).