frank-primary: prefix caching OFF — the fork's QFN align-mode GDN state fault

Measured 2026-10-04 18:46:07 PDT (live podman logs): 48 gdn_hip state
faults in one 0.6 s burst — 'state out of ssm-cache range: max|S|=0 with
non-finite values (slot=42 head=3|8)', one per TP rank, 23 GDN layers
each — then the process-lifetime report cap. The guard's own text in
this build names it: 'known Qwen3.8-Flash-Next (QFN) flaw under prefix
caching … Mitigation: --no-enable-prefix-caching'.

Mechanism: prefix caching puts QFN on mamba_cache_mode 'align', which
checkpoints each sequence's GDN recurrent state into a mamba page per
3136-token block and re-seeds it from that page when a request resumes
from a cached prefix. The NaN state landed at the 13th boundary
(13 x 3136 = 40,768 — the reported ~40k mark) and persisted in the page:
every later turn of that conversation re-seeded the same non-finite
state for two heads, silently (fault-report cap already spent).

RAM ruled out the same day: zero MCE/EDAC records since the boot under
test, 8 GiB in-band pattern stable under live load, 4800 MT/s is the
JEDEC floor on 5600-rated DIMMs, and the two-head single-slot NaN
signature is a state-path fault, not scattered bit corruption.

Fix: FRANK_PRIMARY_NO_PREFIX_CACHING=1 -> --no-enable-prefix-caching
(flag renders in both directions from the knob). Cost: every turn
re-prefills the whole history instead of at most 3135 tail tokens.

Seam updated: the prefix-cache pin now asserts the conditional render
and the shipped OFF value. Runbook carries the evidence + acceptance
criteria; all 368 tests pass.

scripts/frank-primary.env

@@ -175,6 +175,47 @@ FRANK_PRIMARY_KV_DTYPE=fp8
 FRANK_PRIMARY_MNBT=2048
 FRANK_PRIMARY_GATHER_CHUNKS=32
 FRANK_PRIMARY_GATHER_LANES=64
+# PREFIX CACHING IS OFF — THE FORK'S OWN NAMED MITIGATION (2026-10-05).
+# MEASURED 2026-10-04, 18:46:07 PDT (5.6 h into the live boot, one running
+# request, prefix hit rate 48.1%): 48 gdn_hip state faults in one 0.6 s
+# burst, then the process-lifetime silence cap:
+#   [gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
+#   with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
+#   [gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
+#   [gdn_hip] ERROR: state fault report limit (32) reached; further faults
+#   are counted silently
+# One physical SSM slot, two heads, BOTH ranks (heads 3 and 8), 23 layers
+# each. The guard's own message in this build: "known Qwen3.8-Flash-Next
+# (QFN) flaw under prefix caching and is being actively worked; output
+# quality for this request may degrade. Mitigation:
+# --no-enable-prefix-caching".
+# MECHANISM: with prefix caching on, this fork puts QFN on
+# mamba_cache_mode 'align' (the default-for-this-arch branch in
+# config/vllm.py; 'all' is deprecated and 'none' is the no-prefix-cache
+# mode, so align is the only sound mode WITH the flag). Align mode
+# checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
+# mamba page every block, where block = 3136 tokens (the attention block
+# size is raised to 3136 so the page sizes match exactly — the boot log's
+# "Padding mamba page size by 0.13% ..."), and RE-SEEDS the state from that
+# page whenever a request resumes from a cached prefix. The corrupt state
+# landed at the 13th boundary: 13 x 3136 = 40,768 tokens — the "~40k" mark
+# where the request's output turned repetitive/corrupted. The NaN stayed
+# in the mamba page, so every later turn of that conversation re-seeded the
+# same non-finite state for those two heads (silent: the fault-report cap
+# was already spent), while fresh short sessions looked fine.
+# RAM RULED OUT THE SAME DAY: zero MCE/EDAC kernel records since the boot
+# under test, an 8 GiB in-band pattern check stable over 60 s under live
+# load, 4800 MT/s is the JEDEC DDR5 floor on 5600-rated DIMMs (not an
+# overclock), and the signature — a two-head, single-slot NaN state in one
+# burst — is a state-path fault, not scattered finite bit corruption.
+# COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole
+# history instead of at most 3135 tail tokens (a full 40k context costs
+# tens of seconds of TTFT per turn at this shape's prefill rate). A silently
+# corrupted conversation is worse than a slower first token.
+# RE-ENABLE PATH: if the fork ships a fixed align path, set this to empty,
+# redeploy, and re-verify by taking one conversation past 40,768 tokens
+# (decode must stay coherent and podman logs must show zero gdn_hip lines).
+FRANK_PRIMARY_NO_PREFIX_CACHING=1
 # UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
 # 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and
 # 0.982 is Frank's standing law: NEVER 1.00. The 1.00 attempt (2026-10-03,

scripts/serve-frank-primary.sh

@@ -25,7 +25,9 @@
 # --max-num-batched-tokens 2048 (Frank's call 2026-10-03; arena-free because the
 # registry prices overhead flat up to its nbt 4096 anchor), MoE gather geometry
 # R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
-# with depth), prefix caching, expert prefetch OFF (36% precision measured),
+# with depth), prefix caching OFF 2026-10-05 (the fork's QFN align-mode GDN
+# state fault — the evidence block is in frank-primary.env), expert prefetch
+# OFF (36% precision measured),
 # TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
 # instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
 # and GPU_MAX_HW_QUEUES=2; shape F pins the plan's CUDA-graph term to the
@@ -160,7 +162,12 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
   --gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
   --kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
   --kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
-  --enable-prefix-caching \
+  # Prefix caching: the fork's QFN mamba align path corrupts GDN SSM state at
+  # ~40k tokens (2026-10-04, 48 gdn_hip state faults, one request). FRANK_
+  # PRIMARY_NO_PREFIX_CACHING flips to the fork-named mitigation
+  # (--no-enable-prefix-caching); empty keeps the historical flag. The
+  # evidence block is in frank-primary.env.
+  ${FRANK_PRIMARY_NO_PREFIX_CACHING:+--no-enable-prefix-caching}${FRANK_PRIMARY_NO_PREFIX_CACHING:---enable-prefix-caching} \
   --enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
   --reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
   --default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \