From 35a45b8a66c8b9fabfde7fceb441a6437c3f8ecc Mon Sep 17 00:00:00 2001
From: Frank <frank@dankware>
Date: Mon, 5 Oct 2026 13:13:26 -0700
Subject: [PATCH] =?UTF-8?q?frank-primary:=20prefix=20caching=20back=20ON?=
 =?UTF-8?q?=20=E2=80=94=20the=20OFF=20experiment=20did=20not=20fix=20the?=
 =?UTF-8?q?=20corruption?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Frank: still corrupted at ~40k with --no-enable-prefix-caching deployed
and mechanically verified (zero gdn_hip faults in the container serving
that window, coherent 40,771-token boundary probe). The recurrence is
FAULT-FREE, so the align-mode page re-seed is not the (only) mechanism
and the mitigation bought whole-history re-prefill per turn for nothing.

Revert: knob FRANK_PRIMARY_NO_PREFIX_CACHING commented out -> the serve
branch renders --enable-prefix-caching (the shipped contract). The knob,
the branch and the seam stay documented for a future fixed fork build.

Open hypotheses recorded in the runbook: re-fed poisoned transcript (the
same conversation cannot distinguish fixed from unfixed), or a second
fault-free long-context path — fp8 KV at depth is the next suspect. Next
diagnostic: fresh-conversation reproduction, then a KV-dtype A/B.
---
 scripts/frank-primary.env                     | 20 +++++++++----
 scripts/serve-frank-primary.sh                | 15 +++++-----
 ...server-qwen38-fp8-expert-offload-tuning.md | 19 ++++++++++++
 tests/frank_primary_unit.bats                 | 29 ++++++++++---------
 4 files changed, 57 insertions(+), 26 deletions(-)

diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env
index 2572bb7..47cfe79 100644
--- a/scripts/frank-primary.env
+++ b/scripts/frank-primary.env
@@ -175,7 +175,8 @@ FRANK_PRIMARY_KV_DTYPE=fp8
 FRANK_PRIMARY_MNBT=2048
 FRANK_PRIMARY_GATHER_CHUNKS=32
 FRANK_PRIMARY_GATHER_LANES=64
-# PREFIX CACHING IS OFF — THE FORK'S OWN NAMED MITIGATION (2026-10-05).
+# PREFIX CACHING: ON. An OFF experiment ran 2026-10-05 (below) and FAILED
+# to stop the corruption, so the historical flag is back.
 # MEASURED 2026-10-04, 18:46:07 PDT (5.6 h into the live boot, one running
 # request, prefix hit rate 48.1%): 48 gdn_hip state faults in one 0.6 s
 # burst, then the process-lifetime silence cap:
@@ -212,10 +213,19 @@ FRANK_PRIMARY_GATHER_LANES=64
 # history instead of at most 3135 tail tokens (a full 40k context costs
 # tens of seconds of TTFT per turn at this shape's prefill rate). A silently
 # corrupted conversation is worse than a slower first token.
-# RE-ENABLE PATH: if the fork ships a fixed align path, set this to empty,
-# redeploy, and re-verify by taking one conversation past 40,768 tokens
-# (decode must stay coherent and podman logs must show zero gdn_hip lines).
-FRANK_PRIMARY_NO_PREFIX_CACHING=1
+# RESULT (2026-10-05, later that day): THE MITIGATION DID NOT WORK. With
+# --no-enable-prefix-caching deployed and mechanically verified (zero
+# gdn_hip faults, a 40,771-token single-request boundary probe coherent),
+# live long conversations STILL degraded at ~40k on the running endpoint
+# (Frank, direct). The container serving that window logged ZERO faults
+# while the corruption reproduced, so the align-mode page re-seed is not
+# the (only) mechanism — see the runbook's RESULT note for the open
+# hypotheses (re-fed poisoned transcript vs a fault-free long-context
+# path such as fp8 KV at depth). Cost bought nothing: caching is back ON.
+# The knob stays commented for a future fork build with a fixed GDN state
+# path: set 1, redeploy, re-verify past 40,768 tokens (one conversation
+# coherent AND zero gdn_hip lines).
+# FRANK_PRIMARY_NO_PREFIX_CACHING=1
 # UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
 # 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and
 # 0.982 is Frank's standing law: NEVER 1.00. The 1.00 attempt (2026-10-03,
diff --git a/scripts/serve-frank-primary.sh b/scripts/serve-frank-primary.sh
index 0803e5d..2d8387a 100755
--- a/scripts/serve-frank-primary.sh
+++ b/scripts/serve-frank-primary.sh
@@ -137,13 +137,14 @@ fi
 # available" while device_count() says 2). --ipc=host for the engine's shared
 # memory; memlock unbounded for the pinned host slabs.
 # The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
-# Prefix caching: the fork's QFN mamba align path corrupts GDN SSM state
-# at ~40k tokens (2026-10-04, 48 gdn_hip state faults, one request — the
-# evidence block is in frank-primary.env). FRANK_PRIMARY_NO_PREFIX_CACHING
-# flips to the fork-named mitigation (--no-enable-prefix-caching); empty
-# keeps the historical flag. NO comment lines inside the backslash chain
-# below — a comment there terminates the continuation and the next line
-# runs as its own command (the 2026-10-05 exit-127 crash loop).
+# Prefix caching: ON — the shipped contract. FRANK_PRIMARY_NO_PREFIX_-
+# CACHING=1 flips to the fork-named mitigation (--no-enable-prefix-caching)
+# for the QFN align-mode GDN state fault; it was tried 2026-10-05 and
+# REVERTED the same day (live conversations still degraded at ~40k with
+# caching off and zero faults — see the evidence block in
+# frank-primary.env). NO comment lines inside the backslash chain below —
+# a comment there terminates the continuation and the next line runs as
+# its own command (the 2026-10-05 exit-127 crash loop).
 # The ${VAR:-word} / ${VAR:+word} double-expansion for the two directions is
 # AMBIGUOUS bash (${VAR:--flag} parses as a substring offset when set —
 # measured 2026-10-05: rendered "--no-enable-prefix-caching1"), so the flag
diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
index 02a06a5..c6c6721 100644
--- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
@@ -374,6 +374,25 @@ slower first token.
   192.168.0.112:8989 leg with FrankBook-Pro asleep (environmental,
   unrelated to this change; re-run `tests/e2e/run.sh` once the mini is up).
 
+**RESULT — REVERTED THE SAME DAY.** With every mechanical check green
+(zero faults, coherent 40,771-token probe), live long conversations
+STILL degraded at ~40k on the running endpoint (Frank, direct
+observation). The container serving that window (`--no-enable-prefix-
+caching`, booted 2026-10-05 12:47 PDT) logged ZERO `gdn_hip` faults
+while the corruption reproduced — so this recurrence is **fault-free**:
+the NaN-state guard did not fire, which the re-seed theory requires.
+Open hypotheses, in order: (1) the poisoned transcript is being re-fed —
+corrupted assistant turns already in the history keep producing
+corrupted continuations even with perfect state, so the same
+conversation cannot distinguish fixed from unfixed; (2) a second
+long-context mechanism with no guard coverage — fp8 KV degradation at
+depth is the next suspect (a bf16-KV boot costs ~2× the pool per token
+and would settle it). Prefix caching is **back ON**
+(`FRANK_PRIMARY_NO_PREFIX_CACHING` commented out — the cost bought no
+fix). The knob and the serve branch stay for a future fork build with a
+fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if
+it still breaks at ~40k with zero faults, A/B the KV dtype.
+
 ### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
 
 `rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
diff --git a/tests/frank_primary_unit.bats b/tests/frank_primary_unit.bats
index 007c7d0..771fd24 100755
--- a/tests/frank_primary_unit.bats
+++ b/tests/frank_primary_unit.bats
@@ -142,16 +142,16 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
-    # Prefix caching is OFF (2026-10-05): with the flag on, the fork runs
-    # QFN on mamba_cache_mode 'align', and its GDN state path went
-    # non-finite mid-conversation (48 gdn_hip state faults, one request,
-    # slot 42 heads 3/8, at the 13th 3136-token boundary = 40,768 — the
-    # "~40k corruption" report). The fork's own guard names the mitigation:
-    # --no-enable-prefix-caching. RAM was ruled out the same day (zero MCE,
-    # in-band pattern stable, 4800 MT/s is the JEDEC floor). The evidence
-    # block is FRANK_PRIMARY_NO_PREFIX_CACHING in the env file; re-enable
-    # only with a fixed align path and a re-verification past 40,768 tokens.
-    grep -q '^FRANK_PRIMARY_NO_PREFIX_CACHING=1$' "$ENV_FILE"
+    # Prefix caching is ON (reverted 2026-10-05 the same day it was
+    # disabled): the fork's QFN align-mode GDN state fault (48 gdn_hip
+    # state faults, slot 42 heads 3/8, at the 13th 3136-token boundary =
+    # 40,768 — the "~40k corruption" report) was mitigated with
+    # --no-enable-prefix-caching, but live long conversations STILL
+    # degraded at ~40k with caching off and zero faults logged — cost
+    # with no fix, so the historical flag returned. The knob stays
+    # commented in the env file for a future fixed fork build; flip it to
+    # 1 only together with a re-verification past 40,768 tokens.
+    grep -qE '^# FRANK_PRIMARY_NO_PREFIX_CACHING=1$' "$ENV_FILE"
     grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
 }
 
@@ -175,9 +175,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
     # explicitly in the serve script (the ${VAR:--flag} double-expansion is
     # ambiguous bash — when set it parsed as a substring offset, measured
     # 2026-10-05): set -> --no-enable-prefix-caching (the fork-named
-    # mitigation), unset -> the historical --enable-prefix-caching. Both
-    # directions are spelled out so a fork default change cannot silently
-    # flip the contract.
+    # mitigation, tried and REVERTED the same day — live corruption
+    # persisted at ~40k with zero faults), unset -> the shipped
+    # --enable-prefix-caching. Both directions are spelled out so a fork
+    # default change cannot silently flip the contract.
     grep -qF 'FRANK_PRIMARY_PREFIX_CACHE_FLAG=--no-enable-prefix-caching' "$SERVE"
     grep -qF 'FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching' "$SERVE"
     grep -qF '"$FRANK_PRIMARY_PREFIX_CACHE_FLAG"' "$SERVE"
@@ -192,7 +193,7 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
             FRANK_PRIMARY_PREFIX_CACHE_FLAG=--enable-prefix-caching
         fi
         printf %s "$FRANK_PRIMARY_PREFIX_CACHE_FLAG"' _ "$ENV_FILE"
-    [ "$output" = "--no-enable-prefix-caching" ]
+    [ "$output" = "--enable-prefix-caching" ]
     grep -q -- '--trust-remote-code' "$SERVE"
     # DCP is opt-in: the flag is passed only when FRANK_PRIMARY_DCP is set
     # (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).