dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-primary: prefix caching back ON — the OFF experiment did not fix the corruption

Browse Source 
Frank: still corrupted at ~40k with --no-enable-prefix-caching deployed
and mechanically verified (zero gdn_hip faults in the container serving
that window, coherent 40,771-token boundary probe). The recurrence is
FAULT-FREE, so the align-mode page re-seed is not the (only) mechanism
and the mitigation bought whole-history re-prefill per turn for nothing.

Revert: knob FRANK_PRIMARY_NO_PREFIX_CACHING commented out -> the serve
branch renders --enable-prefix-caching (the shipped contract). The knob,
the branch and the seam stay documented for a future fixed fork build.

Open hypotheses recorded in the runbook: re-fed poisoned transcript (the
same conversation cannot distinguish fixed from unfixed), or a second
fault-free long-context path — fp8 KV at depth is the next suspect. Next
diagnostic: fresh-conversation reproduction, then a KV-dtype A/B.
This commit is contained in:
Frank
2026-10-05 13:13:26 -07:00
parent 1fee99c8ce
commit 35a45b8a66
4 changed files with 57 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+19
View File
@@ -374,6 +374,25 @@ slower first token.
192.168.0.112:8989 leg with FrankBook-Pro asleep (environmental,
unrelated to this change; re-run `tests/e2e/run.sh` once the mini is up).
**RESULT — REVERTED THE SAME DAY.** With every mechanical check green
(zero faults, coherent 40,771-token probe), live long conversations
STILL degraded at ~40k on the running endpoint (Frank, direct
observation). The container serving that window (`--no-enable-prefix-
caching`, booted 2026-10-05 12:47 PDT) logged ZERO `gdn_hip` faults
while the corruption reproduced — so this recurrence is **fault-free**:
the NaN-state guard did not fire, which the re-seed theory requires.
Open hypotheses, in order: (1) the poisoned transcript is being re-fed —
corrupted assistant turns already in the history keep producing
corrupted continuations even with perfect state, so the same
conversation cannot distinguish fixed from unfixed; (2) a second
long-context mechanism with no guard coverage — fp8 KV degradation at
depth is the next suspect (a bf16-KV boot costs ~2× the pool per token
and would settle it). Prefix caching is **back ON**
(`FRANK_PRIMARY_NO_PREFIX_CACHING` commented out — the cost bought no
fix). The knob and the serve branch stay for a future fork build with a
fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if
it still breaks at ~40k with zero faults, A/B the KV dtype.
### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB