dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: prefix caching OFF — the fork's QFN align-mode GDN state fault

Browse Source 
Measured 2026-10-04 18:46:07 PDT (live podman logs): 48 gdn_hip state
faults in one 0.6 s burst — 'state out of ssm-cache range: max|S|=0 with
non-finite values (slot=42 head=3|8)', one per TP rank, 23 GDN layers
each — then the process-lifetime report cap. The guard's own text in
this build names it: 'known Qwen3.8-Flash-Next (QFN) flaw under prefix
caching … Mitigation: --no-enable-prefix-caching'.

Mechanism: prefix caching puts QFN on mamba_cache_mode 'align', which
checkpoints each sequence's GDN recurrent state into a mamba page per
3136-token block and re-seeds it from that page when a request resumes
from a cached prefix. The NaN state landed at the 13th boundary
(13 x 3136 = 40,768 — the reported ~40k mark) and persisted in the page:
every later turn of that conversation re-seeded the same non-finite
state for two heads, silently (fault-report cap already spent).

RAM ruled out the same day: zero MCE/EDAC records since the boot under
test, 8 GiB in-band pattern stable under live load, 4800 MT/s is the
JEDEC floor on 5600-rated DIMMs, and the two-head single-slot NaN
signature is a state-path fault, not scattered bit corruption.

Fix: FRANK_PRIMARY_NO_PREFIX_CACHING=1 -> --no-enable-prefix-caching
(flag renders in both directions from the knob). Cost: every turn
re-prefills the whole history instead of at most 3135 tail tokens.

Seam updated: the prefix-cache pin now asserts the conditional render
and the shipped OFF value. Runbook carries the evidence + acceptance
criteria; all 368 tests pass.
This commit is contained in:
Frank
2026-10-05 03:45:26 -07:00
parent a1c4701d7e
commit 9d8b18553d
4 changed files with 130 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-primary.env
+41
View File
@@ -175,6 +175,47 @@ FRANK_PRIMARY_KV_DTYPE=fp8
FRANK_PRIMARY_MNBT=2048
FRANK_PRIMARY_GATHER_CHUNKS=32
FRANK_PRIMARY_GATHER_LANES=64
# PREFIX CACHING IS OFF — THE FORK'S OWN NAMED MITIGATION (2026-10-05).
# MEASURED 2026-10-04, 18:46:07 PDT (5.6 h into the live boot, one running
# request, prefix hit rate 48.1%): 48 gdn_hip state faults in one 0.6 s
# burst, then the process-lifetime silence cap:
# [gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
# with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
# [gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
# [gdn_hip] ERROR: state fault report limit (32) reached; further faults
# are counted silently
# One physical SSM slot, two heads, BOTH ranks (heads 3 and 8), 23 layers
# each. The guard's own message in this build: "known Qwen3.8-Flash-Next
# (QFN) flaw under prefix caching and is being actively worked; output
# quality for this request may degrade. Mitigation:
# --no-enable-prefix-caching".
# MECHANISM: with prefix caching on, this fork puts QFN on
# mamba_cache_mode 'align' (the default-for-this-arch branch in
# config/vllm.py; 'all' is deprecated and 'none' is the no-prefix-cache
# mode, so align is the only sound mode WITH the flag). Align mode
# checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
# mamba page every block, where block = 3136 tokens (the attention block
# size is raised to 3136 so the page sizes match exactly — the boot log's
# "Padding mamba page size by 0.13% ..."), and RE-SEEDS the state from that
# page whenever a request resumes from a cached prefix. The corrupt state
# landed at the 13th boundary: 13 x 3136 = 40,768 tokens — the "~40k" mark
# where the request's output turned repetitive/corrupted. The NaN stayed
# in the mamba page, so every later turn of that conversation re-seeded the
# same non-finite state for those two heads (silent: the fault-report cap
# was already spent), while fresh short sessions looked fine.
# RAM RULED OUT THE SAME DAY: zero MCE/EDAC kernel records since the boot
# under test, an 8 GiB in-band pattern check stable over 60 s under live
# load, 4800 MT/s is the JEDEC DDR5 floor on 5600-rated DIMMs (not an
# overclock), and the signature — a two-head, single-slot NaN state in one
# burst — is a state-path fault, not scattered finite bit corruption.
# COST ACCEPTED: no prefix reuse, so each turn re-prefills the whole
# history instead of at most 3135 tail tokens (a full 40k context costs
# tens of seconds of TTFT per turn at this shape's prefill rate). A silently
# corrupted conversation is worse than a slower first token.
# RE-ENABLE PATH: if the fork ships a fixed align path, set this to empty,
# redeploy, and re-verify by taking one conversation past 40,768 tokens
# (decode must stay coherent and podman logs must show zero gdn_hip lines).
FRANK_PRIMARY_NO_PREFIX_CACHING=1
# UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
# 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and
# 0.982 is Frank's standing law: NEVER 1.00. The 1.00 attempt (2026-10-03,
scripts/serve-frank-primary.sh
+9 -2
View File
@@ -25,7 +25,9 @@
# --max-num-batched-tokens 2048 (Frank's call 2026-10-03; arena-free because the
# registry prices overhead flat up to its nbt 4096 anchor), MoE gather geometry
# R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (+18.7% decode, grows
# with depth), prefix caching, expert prefetch OFF (36% precision measured),
# with depth), prefix caching OFF 2026-10-05 (the fork's QFN align-mode GDN
# state fault — the evidence block is in frank-primary.env), expert prefetch
# OFF (36% precision measured),
# TunableOp off. Shape E adds --expert-precise-memory (fitted 5.25 GiB overhead
# instead of the 7.60 GiB default -> expert arena 17.22 GiB / 7,522 slots, +16%)
# and GPU_MAX_HW_QUEUES=2; shape F pins the plan's CUDA-graph term to the
@@ -160,7 +162,12 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
--gpu-memory-utilization "$FRANK_PRIMARY_UTIL" \
--kv-cache-memory-bytes "$FRANK_PRIMARY_KV_BYTES" \
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
--enable-prefix-caching \
# Prefix caching: the fork's QFN mamba align path corrupts GDN SSM state at
# ~40k tokens (2026-10-04, 48 gdn_hip state faults, one request). FRANK_
# PRIMARY_NO_PREFIX_CACHING flips to the fork-named mitigation
# (--no-enable-prefix-caching); empty keeps the historical flag. The
# evidence block is in frank-primary.env.
${FRANK_PRIMARY_NO_PREFIX_CACHING:+--no-enable-prefix-caching}${FRANK_PRIMARY_NO_PREFIX_CACHING:---enable-prefix-caching} \
--enable-auto-tool-choice --tool-call-parser "$FRANK_PRIMARY_TOOL_PARSER" \
--reasoning-parser "$FRANK_PRIMARY_REASONING_PARSER" \
--default-chat-template-kwargs "$FRANK_PRIMARY_CHAT_TEMPLATE_KWARGS" \
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
+65
View File
@@ -291,8 +291,73 @@ FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary mounted at /root/.cache
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
FRANK_PRIMARY_NO_PREFIX_CACHING=1 --no-enable-prefix-caching (2026-10-05, the QFN align-mode GDN state fault — below)
```
### 2026-10-05 — prefix caching OFF: the fork's QFN align-mode GDN state fault
**Symptom (Frank, 2026-10-04):** output turned repetitive/corrupted at
~40k tokens into a long conversation; short sessions stayed clean.
**Evidence (live `podman logs frank-primary-qwen`, the 2026-10-04 18:19
boot):** one 0.6 s burst at 18:46:07 PDT — 5.6 h into service, one
running request, and the engine's own 18:46:08 log line reading
"Prefix cache hit rate: 48.1%". 48 `gdn_hip` faults, then the
process-lifetime silence cap:
```
[gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
[gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
[gdn_hip] ERROR: state fault report limit (32) reached; further faults
are counted silently
```
One physical SSM slot (42), two heads (3 and 8 — one per TP rank), 23
GDN layers each, the state non-finite (NaN). The guard's own text in
this build: *"known Qwen3.8-Flash-Next (QFN) flaw under prefix caching
and is being actively worked; output quality for this request may
degrade. Mitigation: --no-enable-prefix-caching."*
**Mechanism.** With prefix caching on, the fork defaults QFN to
`mamba_cache_mode=align` (boot log: "Mamba cache mode is set to 'align'
for Qwen4ExpForConditionalGeneration … when prefix caching is enabled";
`config/cache.py`: "none" is the no-prefix-cache mode, "all" is
deprecated — align is the only sound mode WITH the flag). Align mode
checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
mamba page per block and RE-SEEDS the state from that page when a
request resumes from a cached prefix. The block size is raised to
**3136 tokens** for the page-size equality (boot log: "Setting
attention block size to 3136 tokens to ensure that attention page size
is >= mamba page size" + "Padding mamba page size by 0.13% … exactly
equal"). The corrupt state landed at the **13th boundary: 13 × 3136 =
40,768 tokens** — the reported "~40k" mark. The NaN persisted in the
mamba page, so every later turn of that conversation re-seeded the same
non-finite state for those two heads (silently — the fault-report cap
was already spent) while fresh short sessions looked fine.
**RAM ruled out the same day** (the 2026-10-04 17:17 host reboot is when
the current memory speed took effect): zero MCE/EDAC kernel records
since boot; an 8 GiB in-band pattern check stable over 60 s under live
load; 4800 MT/s is the JEDEC DDR5 floor on 5600 MT/s-rated DIMMs (not an
overclock); and a two-head, single-slot NaN state is a state-path fault,
not scattered finite bit corruption.
**Fix.** `FRANK_PRIMARY_NO_PREFIX_CACHING=1` → `--no-enable-prefix-caching`
(the fork-named mitigation; the flag renders in both directions from the
knob so a fork default change cannot silently flip the contract). Cost:
no prefix reuse — every turn re-prefills the whole history instead of at
most 3135 tail tokens. A silently corrupted conversation is worse than a
slower first token.
**Acceptance (2026-10-05 deploy):** boot healthy with the expert plan
unchanged (arena 17.77 GiB / 7,762 slots — the flag touches no plan
term), "GPU KV cache size" still clears 400,700 tokens (if the
align-mode block-size raise disappears the mamba per-token price moves
and the pool may need `FRANK_PRIMARY_KV_BYTES` up a notch), zero
`gdn_hip` lines, and one request whose context crosses 40,768 tokens
in-session decodes coherently.
### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
tests/frank_primary_unit.bats
+15 -1
View File
@@ -142,6 +142,16 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
# Prefix caching is OFF (2026-10-05): with the flag on, the fork runs
# QFN on mamba_cache_mode 'align', and its GDN state path went
# non-finite mid-conversation (48 gdn_hip state faults, one request,
# slot 42 heads 3/8, at the 13th 3136-token boundary = 40,768 — the
# "~40k corruption" report). The fork's own guard names the mitigation:
# --no-enable-prefix-caching. RAM was ruled out the same day (zero MCE,
# in-band pattern stable, 4800 MT/s is the JEDEC floor). The evidence
# block is FRANK_PRIMARY_NO_PREFIX_CACHING in the env file; re-enable
# only with a fixed align path and a re-verification past 40,768 tokens.
grep -q '^FRANK_PRIMARY_NO_PREFIX_CACHING=1$' "$ENV_FILE"
grep -q '^FRANK_PRIMARY_MODEL_SHARDS=131$' "$ENV_FILE"
}
@@ -161,7 +171,11 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
grep -q -- '${FRANK_PRIMARY_COMPCONFIG:+--compilation-config "$FRANK_PRIMARY_COMPCONFIG"}' "$SERVE"
grep -q -- '${FRANK_PRIMARY_GEN_CONFIG:+--override-generation-config "$FRANK_PRIMARY_GEN_CONFIG"}' "$SERVE"
grep -q -- '--tensor-parallel-size 2' "$SERVE"
grep -q -- '--enable-prefix-caching' "$SERVE"
# Prefix caching renders exactly ONE flag from the knob: set ->
# --no-enable-prefix-caching (the fork-named mitigation), unset -> the
# historical --enable-prefix-caching. Both directions are explicit so a
# fork default change cannot silently flip the contract.
grep -qF '${FRANK_PRIMARY_NO_PREFIX_CACHING:+--no-enable-prefix-caching}${FRANK_PRIMARY_NO_PREFIX_CACHING:---enable-prefix-caching}' "$SERVE"
grep -q -- '--trust-remote-code' "$SERVE"
# DCP is opt-in: the flag is passed only when FRANK_PRIMARY_DCP is set
# (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).