frank-primary: prefix caching OFF — the fork's QFN align-mode GDN state fault

Measured 2026-10-04 18:46:07 PDT (live podman logs): 48 gdn_hip state
faults in one 0.6 s burst — 'state out of ssm-cache range: max|S|=0 with
non-finite values (slot=42 head=3|8)', one per TP rank, 23 GDN layers
each — then the process-lifetime report cap. The guard's own text in
this build names it: 'known Qwen3.8-Flash-Next (QFN) flaw under prefix
caching … Mitigation: --no-enable-prefix-caching'.

Mechanism: prefix caching puts QFN on mamba_cache_mode 'align', which
checkpoints each sequence's GDN recurrent state into a mamba page per
3136-token block and re-seeds it from that page when a request resumes
from a cached prefix. The NaN state landed at the 13th boundary
(13 x 3136 = 40,768 — the reported ~40k mark) and persisted in the page:
every later turn of that conversation re-seeded the same non-finite
state for two heads, silently (fault-report cap already spent).

RAM ruled out the same day: zero MCE/EDAC records since the boot under
test, 8 GiB in-band pattern stable under live load, 4800 MT/s is the
JEDEC floor on 5600-rated DIMMs, and the two-head single-slot NaN
signature is a state-path fault, not scattered bit corruption.

Fix: FRANK_PRIMARY_NO_PREFIX_CACHING=1 -> --no-enable-prefix-caching
(flag renders in both directions from the knob). Cost: every turn
re-prefills the whole history instead of at most 3135 tail tokens.

Seam updated: the prefix-cache pin now asserts the conditional render
and the shipped OFF value. Runbook carries the evidence + acceptance
criteria; all 368 tests pass.

scripts/frank-primary.env

@@ -175,6 +175,47 @@ FRANK_PRIMARY_KV_DTYPE=fp8
 FRANK_PRIMARY_MNBT=2048
 FRANK_PRIMARY_GATHER_CHUNKS=32
 FRANK_PRIMARY_GATHER_LANES=64
+# PREFIX CACHING IS OFF — THE FORK'S OWN NAMED MITIGATION (2026-10-05).
+# MEASURED 2026-10-04, 18:46:07 PDT (5.6 h into the live boot, one running
+# request, prefix hit rate 48.1%): 48 gdn_hip state faults in one 0.6 s
+# burst, then the process-lifetime silence cap:
+#   [gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
+#   with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
+#   [gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
+#   [gdn_hip] ERROR: state fault report limit (32) reached; further faults
+#   are counted silently
+# One physical SSM slot, two heads, BOTH ranks (heads 3 and 8), 23 layers
+# each. The guard's own message in this build: "known Qwen3.8-Flash-Next
+# (QFN) flaw under prefix caching and is being actively worked; output
+# quality for this request may degrade. Mitigation:
+# --no-enable-prefix-caching".
+# MECHANISM: with prefix caching on, this fork puts QFN on
+# mamba_cache_mode 'align' (the default-for-this-arch branch in
+# config/vllm.py; 'all' is deprecated and 'none' is the no-prefix-cache
+# mode, so align is the only sound mode WITH the flag). Align mode
+# checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
+# mamba page every block, where block = 3136 tokens (the attention block
+# size is raised to 3136 so the page sizes match exactly — the boot log's
+# "Padding mamba page size by 0.13% ..."), and RE-SEEDS the state from that
+# page whenever a request resumes from a cached prefix. The corrupt state
+# landed at the 13th boundary: 13 x 3136 = 40,768 tokens — the "~40k" mark
+# where the request's output turned repetitive/corrupted. The NaN stayed
+# in the mamba page, so every later turn of that conversation re-seeded the
+# same non-finite state for those two heads (silent: the fault-report cap
+# was already spent), while fresh short sessions looked fine.
+# RAM RULED OUT THE SAME DAY: zero MCE/EDAC kernel records since the boot
+# under test, an 8 GiB in-band pattern check stable over 60 s under live
+# load, 4800 MT/s is the JEDEC DDR5 floor on 5600-rated DIMMs (not an
+# overclock), and the signature — a two-head, single-slot NaN state in one
+# burst — is a state-path fault, not scattered finite bit corruption.
+# COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole
+# history instead of at most 3135 tail tokens (a full 40k context costs
+# tens of seconds of TTFT per turn at this shape's prefill rate). A silently
+# corrupted conversation is worse than a slower first token.
+# RE-ENABLE PATH: if the fork ships a fixed align path, set this to empty,
+# redeploy, and re-verify by taking one conversation past 40,768 tokens
+# (decode must stay coherent and podman logs must show zero gdn_hip lines).
+FRANK_PRIMARY_NO_PREFIX_CACHING=1
 # UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
 # 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and
 # 0.982 is Frank's standing law: NEVER 1.00. The 1.00 attempt (2026-10-03,