docs(frank-primary): RESOLVED - the ~40k corruption was marginal DDR5 at 4800, fixed at 4600

The prefix-caching OFF/ON experiment was confounded (both arms ran on
the marginal speed); the align-mode re-seed theory is unproven, not
disproven. Caching ships ON with zero gdn_hip faults under heavy
re-send traffic on stable RAM; knob stays commented. Corrects the
env's 'RAM RULED OUT' claim and adds the AGENTS.md lesson.
This commit is contained in:
Frank
2026-10-05 13:54:06 -07:00
parent 35a45b8a66
commit 2e64f66b82
3 changed files with 51 additions and 5 deletions
+21 -5
View File
@@ -204,11 +204,14 @@ FRANK_PRIMARY_GATHER_LANES=64
# in the mamba page, so every later turn of that conversation re-seeded the
# same non-finite state for those two heads (silent: the fault-report cap
# was already spent), while fresh short sessions looked fine.
# RAM RULED OUT THE SAME DAY: zero MCE/EDAC kernel records since the boot
# under test, an 8 GiB in-band pattern check stable over 60 s under live
# load, 4800 MT/s is the JEDEC DDR5 floor on 5600-rated DIMMs (not an
# overclock), and the signature — a two-head, single-slot NaN state in one
# burst — is a state-path fault, not scattered finite bit corruption.
# RAM WAS "RULED OUT" THE SAME DAY — AND THAT WAS WRONG (RESOLVED below).
# What the evidence actually supported: zero MCE/EDAC records is the
# EXPECTED silence of consumer non-ECC memory (a silent flip raises no
# machine check); an 8 GiB pattern check over 60 s covers ~0.5 % of
# 246 GiB — no verdict on a speed setting; "4800 is the JEDEC floor on
# 5600-rated DIMMs" is a rating, not a stability proof at four-DIMM
# loading. Only the two-head NaN signature argument was load-bearing,
# and the revert test outvoted it.
# COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole
# history instead of at most 3135 tail tokens (a full 40k context costs
# tens of seconds of TTFT per turn at this shape's prefill rate). A silently
@@ -225,6 +228,19 @@ FRANK_PRIMARY_GATHER_LANES=64
# The knob stays commented for a future fork build with a fixed GDN state
# path: set 1, redeploy, re-verify past 40,768 tokens (one conversation
# coherent AND zero gdn_hip lines).
# RESOLVED 2026-10-05 EVENING (Frank): IT WAS THE RAM. 4800 MT/s (in
# effect since the 2026-10-04 17:17 boot) was marginal; 4600 MT/s (since
# the 2026-10-05 12:21 boot) and the endpoint is clean — zero gdn_hip
# ERROR lines, zero MCE, and the live agent session on it past 140k
# context with no degradation. Both failure eras sit under the RAM: the
# 2026-10-04 NaN burst (corrupted weight/activation traffic poisoning a
# GDN state write — the guard-visible path) and the fault-free ~40k
# degradation (silent flips in KV/expert data, which no guard covers on
# non-ECC). CONSEQUENCE FOR THE KNOB: the OFF experiment was CONFOUNDED
# — both arms ran on the marginal speed — so it neither proved nor
# disproved the fork's align-mode flaw. Prefix caching stays ON (zero
# gdn_hip lines under heavy re-send traffic since the speed drop); flip
# the knob only if a fault recurs on stable RAM or the fork ships a fix.
# FRANK_PRIMARY_NO_PREFIX_CACHING=1
# UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
# 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and