docs(frank-primary): RESOLVED - the ~40k corruption was marginal DDR5 at 4800, fixed at 4600
The prefix-caching OFF/ON experiment was confounded (both arms ran on the marginal speed); the align-mode re-seed theory is unproven, not disproven. Caching ships ON with zero gdn_hip faults under heavy re-send traffic on stable RAM; knob stays commented. Corrects the env's 'RAM RULED OUT' claim and adds the AGENTS.md lesson.
This commit is contained in:
@@ -393,6 +393,25 @@ fix). The knob and the serve branch stay for a future fork build with a
|
||||
fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if
|
||||
it still breaks at ~40k with zero faults, A/B the KV dtype.
|
||||
|
||||
**RESOLVED (2026-10-05 evening): it was the RAM.** Frank dropped the
|
||||
memory from 4800 to **4600 MT/s** (in effect since the 2026-10-05 12:21
|
||||
PDT host boot) and the corruption stopped: zero `gdn_hip` ERROR lines in
|
||||
the container since, zero MCE records, and the live agent session on
|
||||
this endpoint past 140k context, "comfy cozy" (Frank). Marginal DDR5 on
|
||||
four DIMMs is a common cause for BOTH eras: the 2026-10-04 NaN burst
|
||||
(corrupted weight/activation traffic poisoning a GDN state write — the
|
||||
guard-visible path) and the fault-free ~40k degradation (silent flips in
|
||||
KV/expert data, which no guard covers on non-ECC memory). The re-seed
|
||||
theory is therefore UNPROVEN, not disproven — the entire caching
|
||||
experiment ran on marginal RAM and was confounded on both arms.
|
||||
Post-mortem on the rule-out: consumer non-ECC RAM raises no MCE for a
|
||||
silent flip; a 60 s / 8 GiB pattern sample covers ~0.5 % of 246 GiB;
|
||||
"4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, not a
|
||||
stability proof at four-DIMM loading. The revert test was the decisive
|
||||
instrument and it pointed at the RAM all along. Prefix caching stays ON
|
||||
(zero faults under heavy re-send traffic since the speed drop); the knob
|
||||
stays for a future fork fix. The fp8-KV A/B is NOT needed.
|
||||
|
||||
### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
|
||||
|
||||
`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
|
||||
|
||||
Reference in New Issue
Block a user