diff --git a/AGENTS.md b/AGENTS.md index 05e9f04..c446c46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -729,6 +729,17 @@ linked runbooks. the KV pool is worse than the OOM it prevents. - Host swap state is invisible to `deploy.sh` (no repo file) — check `swapon --show` before reasoning about headroom. +- **Silent RAM corruption outranks every software theory.** frank-primary's + ~40k-token output corruption (2026-10-04/05, repetition/garbage in long + agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed + by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and + a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability + proof — combined, those two instruments said "ruled out" while the RAM + was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, + not a margin guarantee. When the fault is data-shaped (NaN states, + garbage tokens) and a software revert doesn't move it, price in the RAM + before the runtime — and remember a revert tested under the same marginal + hardware is itself confounded. See the runbook's 2026-10-05 section. **Runtime-specific guards** diff --git a/scripts/frank-primary.env b/scripts/frank-primary.env index 47cfe79..ce160b2 100644 --- a/scripts/frank-primary.env +++ b/scripts/frank-primary.env @@ -204,11 +204,14 @@ FRANK_PRIMARY_GATHER_LANES=64 # in the mamba page, so every later turn of that conversation re-seeded the # same non-finite state for those two heads (silent: the fault-report cap # was already spent), while fresh short sessions looked fine. -# RAM RULED OUT THE SAME DAY: zero MCE/EDAC kernel records since the boot -# under test, an 8 GiB in-band pattern check stable over 60 s under live -# load, 4800 MT/s is the JEDEC DDR5 floor on 5600-rated DIMMs (not an -# overclock), and the signature — a two-head, single-slot NaN state in one -# burst — is a state-path fault, not scattered finite bit corruption. +# RAM WAS "RULED OUT" THE SAME DAY — AND THAT WAS WRONG (RESOLVED below). +# What the evidence actually supported: zero MCE/EDAC records is the +# EXPECTED silence of consumer non-ECC memory (a silent flip raises no +# machine check); an 8 GiB pattern check over 60 s covers ~0.5 % of +# 246 GiB — no verdict on a speed setting; "4800 is the JEDEC floor on +# 5600-rated DIMMs" is a rating, not a stability proof at four-DIMM +# loading. Only the two-head NaN signature argument was load-bearing, +# and the revert test outvoted it. # COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole # history instead of at most 3135 tail tokens (a full 40k context costs # tens of seconds of TTFT per turn at this shape's prefill rate). A silently @@ -225,6 +228,19 @@ FRANK_PRIMARY_GATHER_LANES=64 # The knob stays commented for a future fork build with a fixed GDN state # path: set 1, redeploy, re-verify past 40,768 tokens (one conversation # coherent AND zero gdn_hip lines). +# RESOLVED 2026-10-05 EVENING (Frank): IT WAS THE RAM. 4800 MT/s (in +# effect since the 2026-10-04 17:17 boot) was marginal; 4600 MT/s (since +# the 2026-10-05 12:21 boot) and the endpoint is clean — zero gdn_hip +# ERROR lines, zero MCE, and the live agent session on it past 140k +# context with no degradation. Both failure eras sit under the RAM: the +# 2026-10-04 NaN burst (corrupted weight/activation traffic poisoning a +# GDN state write — the guard-visible path) and the fault-free ~40k +# degradation (silent flips in KV/expert data, which no guard covers on +# non-ECC). CONSEQUENCE FOR THE KNOB: the OFF experiment was CONFOUNDED +# — both arms ran on the marginal speed — so it neither proved nor +# disproved the fork's align-mode flaw. Prefix caching stays ON (zero +# gdn_hip lines under heavy re-send traffic since the speed drop); flip +# the knob only if a fault recurs on stable RAM or the fork ships a fix. # FRANK_PRIMARY_NO_PREFIX_CACHING=1 # UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at # 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and diff --git a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md index c6c6721..5177c89 100644 --- a/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md +++ b/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md @@ -393,6 +393,25 @@ fix). The knob and the serve branch stay for a future fork build with a fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if it still breaks at ~40k with zero faults, A/B the KV dtype. +**RESOLVED (2026-10-05 evening): it was the RAM.** Frank dropped the +memory from 4800 to **4600 MT/s** (in effect since the 2026-10-05 12:21 +PDT host boot) and the corruption stopped: zero `gdn_hip` ERROR lines in +the container since, zero MCE records, and the live agent session on +this endpoint past 140k context, "comfy cozy" (Frank). Marginal DDR5 on +four DIMMs is a common cause for BOTH eras: the 2026-10-04 NaN burst +(corrupted weight/activation traffic poisoning a GDN state write — the +guard-visible path) and the fault-free ~40k degradation (silent flips in +KV/expert data, which no guard covers on non-ECC memory). The re-seed +theory is therefore UNPROVEN, not disproven — the entire caching +experiment ran on marginal RAM and was confounded on both arms. +Post-mortem on the rule-out: consumer non-ECC RAM raises no MCE for a +silent flip; a 60 s / 8 GiB pattern sample covers ~0.5 % of 246 GiB; +"4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, not a +stability proof at four-DIMM loading. The revert test was the decisive +instrument and it pointed at the RAM all along. Prefix caching stays ON +(zero faults under heavy re-send traffic since the speed drop); the knob +stays for a future fork fix. The fp8-KV A/B is NOT needed. + ### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost `rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB