docs(frank-primary): RESOLVED - the ~40k corruption was marginal DDR5 at 4800, fixed at 4600

The prefix-caching OFF/ON experiment was confounded (both arms ran on
the marginal speed); the align-mode re-seed theory is unproven, not
disproven. Caching ships ON with zero gdn_hip faults under heavy
re-send traffic on stable RAM; knob stays commented. Corrects the
env's 'RAM RULED OUT' claim and adds the AGENTS.md lesson.
This commit is contained in:
Frank
2026-10-05 13:54:06 -07:00
parent 35a45b8a66
commit 2e64f66b82
3 changed files with 51 additions and 5 deletions
@@ -393,6 +393,25 @@ fix). The knob and the serve branch stay for a future fork build with a
fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if
it still breaks at ~40k with zero faults, A/B the KV dtype.
**RESOLVED (2026-10-05 evening): it was the RAM.** Frank dropped the
memory from 4800 to **4600 MT/s** (in effect since the 2026-10-05 12:21
PDT host boot) and the corruption stopped: zero `gdn_hip` ERROR lines in
the container since, zero MCE records, and the live agent session on
this endpoint past 140k context, "comfy cozy" (Frank). Marginal DDR5 on
four DIMMs is a common cause for BOTH eras: the 2026-10-04 NaN burst
(corrupted weight/activation traffic poisoning a GDN state write — the
guard-visible path) and the fault-free ~40k degradation (silent flips in
KV/expert data, which no guard covers on non-ECC memory). The re-seed
theory is therefore UNPROVEN, not disproven — the entire caching
experiment ran on marginal RAM and was confounded on both arms.
Post-mortem on the rule-out: consumer non-ECC RAM raises no MCE for a
silent flip; a 60 s / 8 GiB pattern sample covers ~0.5 % of 246 GiB;
"4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, not a
stability proof at four-DIMM loading. The revert test was the decisive
instrument and it pointed at the RAM all along. Prefix caching stays ON
(zero faults under heavy re-send traffic since the speed drop); the knob
stays for a future fork fix. The fp8-KV A/B is NOT needed.
### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB