docs(frank-primary): RESOLVED - the ~40k corruption was marginal DDR5 at 4800, fixed at 4600

The prefix-caching OFF/ON experiment was confounded (both arms ran on
the marginal speed); the align-mode re-seed theory is unproven, not
disproven. Caching ships ON with zero gdn_hip faults under heavy
re-send traffic on stable RAM; knob stays commented. Corrects the
env's 'RAM RULED OUT' claim and adds the AGENTS.md lesson.

AGENTS.md

@@ -729,6 +729,17 @@ linked runbooks.
   the KV pool is worse than the OOM it prevents.
 - Host swap state is invisible to `deploy.sh` (no repo file) — check
   `swapon --show` before reasoning about headroom.
+- **Silent RAM corruption outranks every software theory.** frank-primary's
+  ~40k-token output corruption (2026-10-04/05, repetition/garbage in long
+  agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
+  by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
+  a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability
+  proof — combined, those two instruments said "ruled out" while the RAM
+  was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating,
+  not a margin guarantee. When the fault is data-shaped (NaN states,
+  garbage tokens) and a software revert doesn't move it, price in the RAM
+  before the runtime — and remember a revert tested under the same marginal
+  hardware is itself confounded. See the runbook's 2026-10-05 section.
 
 **Runtime-specific guards**
 

scripts/frank-primary.env

@@ -204,11 +204,14 @@ FRANK_PRIMARY_GATHER_LANES=64
 # in the mamba page, so every later turn of that conversation re-seeded the
 # same non-finite state for those two heads (silent: the fault-report cap
 # was already spent), while fresh short sessions looked fine.
-# RAM RULED OUT THE SAME DAY: zero MCE/EDAC kernel records since the boot
+# RAM WAS "RULED OUT" THE SAME DAY — AND THAT WAS WRONG (RESOLVED below).
-# under test, an 8 GiB in-band pattern check stable over 60 s under live
+# What the evidence actually supported: zero MCE/EDAC records is the
-# load, 4800 MT/s is the JEDEC DDR5 floor on 5600-rated DIMMs (not an
+# EXPECTED silence of consumer non-ECC memory (a silent flip raises no
-# overclock), and the signature — a two-head, single-slot NaN state in one
+# machine check); an 8 GiB pattern check over 60 s covers ~0.5 % of
-# burst — is a state-path fault, not scattered finite bit corruption.
+# 246 GiB — no verdict on a speed setting; "4800 is the JEDEC floor on
+# 5600-rated DIMMs" is a rating, not a stability proof at four-DIMM
+# loading. Only the two-head NaN signature argument was load-bearing,
+# and the revert test outvoted it.
 # COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole
 # history instead of at most 3135 tail tokens (a full 40k context costs
 # tens of seconds of TTFT per turn at this shape's prefill rate). A silently
@@ -225,6 +228,19 @@ FRANK_PRIMARY_GATHER_LANES=64
 # The knob stays commented for a future fork build with a fixed GDN state
 # path: set 1, redeploy, re-verify past 40,768 tokens (one conversation
 # coherent AND zero gdn_hip lines).
+# RESOLVED 2026-10-05 EVENING (Frank): IT WAS THE RAM. 4800 MT/s (in
+# effect since the 2026-10-04 17:17 boot) was marginal; 4600 MT/s (since
+# the 2026-10-05 12:21 boot) and the endpoint is clean — zero gdn_hip
+# ERROR lines, zero MCE, and the live agent session on it past 140k
+# context with no degradation. Both failure eras sit under the RAM: the
+# 2026-10-04 NaN burst (corrupted weight/activation traffic poisoning a
+# GDN state write — the guard-visible path) and the fault-free ~40k
+# degradation (silent flips in KV/expert data, which no guard covers on
+# non-ECC). CONSEQUENCE FOR THE KNOB: the OFF experiment was CONFOUNDED
+# — both arms ran on the marginal speed — so it neither proved nor
+# disproved the fork's align-mode flaw. Prefix caching stays ON (zero
+# gdn_hip lines under heavy re-send traffic since the speed drop); flip
+# the knob only if a fault recurs on stable RAM or the fork ships a fix.
 # FRANK_PRIMARY_NO_PREFIX_CACHING=1
 # UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
 # 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and

serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

@@ -393,6 +393,25 @@ fix). The knob and the serve branch stay for a future fork build with a
 fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if
 it still breaks at ~40k with zero faults, A/B the KV dtype.
 
+**RESOLVED (2026-10-05 evening): it was the RAM.** Frank dropped the
+memory from 4800 to **4600 MT/s** (in effect since the 2026-10-05 12:21
+PDT host boot) and the corruption stopped: zero `gdn_hip` ERROR lines in
+the container since, zero MCE records, and the live agent session on
+this endpoint past 140k context, "comfy cozy" (Frank). Marginal DDR5 on
+four DIMMs is a common cause for BOTH eras: the 2026-10-04 NaN burst
+(corrupted weight/activation traffic poisoning a GDN state write — the
+guard-visible path) and the fault-free ~40k degradation (silent flips in
+KV/expert data, which no guard covers on non-ECC memory). The re-seed
+theory is therefore UNPROVEN, not disproven — the entire caching
+experiment ran on marginal RAM and was confounded on both arms.
+Post-mortem on the rule-out: consumer non-ECC RAM raises no MCE for a
+silent flip; a 60 s / 8 GiB pattern sample covers ~0.5 % of 246 GiB;
+"4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, not a
+stability proof at four-DIMM loading. The revert test was the decisive
+instrument and it pointed at the RAM all along. Prefix caching stays ON
+(zero faults under heavy re-send traffic since the speed drop); the knob
+stays for a future fork fix. The fp8-KV A/B is NOT needed.
+
 ### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
 
 `rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB