frank-primary: prefix caching back ON — the OFF experiment did not fix the corruption

Frank: still corrupted at ~40k with --no-enable-prefix-caching deployed
and mechanically verified (zero gdn_hip faults in the container serving
that window, coherent 40,771-token boundary probe). The recurrence is
FAULT-FREE, so the align-mode page re-seed is not the (only) mechanism
and the mitigation bought whole-history re-prefill per turn for nothing.

Revert: knob FRANK_PRIMARY_NO_PREFIX_CACHING commented out -> the serve
branch renders --enable-prefix-caching (the shipped contract). The knob,
the branch and the seam stay documented for a future fixed fork build.

Open hypotheses recorded in the runbook: re-fed poisoned transcript (the
same conversation cannot distinguish fixed from unfixed), or a second
fault-free long-context path — fp8 KV at depth is the next suspect. Next
diagnostic: fresh-conversation reproduction, then a KV-dtype A/B.
This commit is contained in:
Frank
2026-10-05 13:13:26 -07:00
parent 1fee99c8ce
commit 35a45b8a66
4 changed files with 57 additions and 26 deletions
+15 -5
View File
@@ -175,7 +175,8 @@ FRANK_PRIMARY_KV_DTYPE=fp8
FRANK_PRIMARY_MNBT=2048
FRANK_PRIMARY_GATHER_CHUNKS=32
FRANK_PRIMARY_GATHER_LANES=64
# PREFIX CACHING IS OFF — THE FORK'S OWN NAMED MITIGATION (2026-10-05).
# PREFIX CACHING: ON. An OFF experiment ran 2026-10-05 (below) and FAILED
# to stop the corruption, so the historical flag is back.
# MEASURED 2026-10-04, 18:46:07 PDT (5.6 h into the live boot, one running
# request, prefix hit rate 48.1%): 48 gdn_hip state faults in one 0.6 s
# burst, then the process-lifetime silence cap:
@@ -212,10 +213,19 @@ FRANK_PRIMARY_GATHER_LANES=64
# history instead of at most 3135 tail tokens (a full 40k context costs
# tens of seconds of TTFT per turn at this shape's prefill rate). A silently
# corrupted conversation is worse than a slower first token.
# RE-ENABLE PATH: if the fork ships a fixed align path, set this to empty,
# redeploy, and re-verify by taking one conversation past 40,768 tokens
# (decode must stay coherent and podman logs must show zero gdn_hip lines).
FRANK_PRIMARY_NO_PREFIX_CACHING=1
# RESULT (2026-10-05, later that day): THE MITIGATION DID NOT WORK. With
# --no-enable-prefix-caching deployed and mechanically verified (zero
# gdn_hip faults, a 40,771-token single-request boundary probe coherent),
# live long conversations STILL degraded at ~40k on the running endpoint
# (Frank, direct). The container serving that window logged ZERO faults
# while the corruption reproduced, so the align-mode page re-seed is not
# the (only) mechanism — see the runbook's RESULT note for the open
# hypotheses (re-fed poisoned transcript vs a fault-free long-context
# path such as fp8 KV at depth). Cost bought nothing: caching is back ON.
# The knob stays commented for a future fork build with a fixed GDN state
# path: set 1, redeploy, re-verify past 40,768 tokens (one conversation
# coherent AND zero gdn_hip lines).
# FRANK_PRIMARY_NO_PREFIX_CACHING=1
# UTIL is an ARENA lever: the fork plans `budget = device_total x util`, and at
# 31.86 GiB/card every +0.01 is +0.32 GiB of expert cache (~+139 slots), and
# 0.982 is Frank's standing law: NEVER 1.00. The 1.00 attempt (2026-10-03,
+8 -7
View File
@@ -137,13 +137,14 @@ fi
# available" while device_count() says 2). --ipc=host for the engine's shared
# memory; memlock unbounded for the pinned host slabs.
# The entrypoint already execs `vllm serve "$@"`, so no `serve` here.
# Prefix caching: the fork's QFN mamba align path corrupts GDN SSM state
# at ~40k tokens (2026-10-04, 48 gdn_hip state faults, one request — the
# evidence block is in frank-primary.env). FRANK_PRIMARY_NO_PREFIX_CACHING
# flips to the fork-named mitigation (--no-enable-prefix-caching); empty
# keeps the historical flag. NO comment lines inside the backslash chain
# below — a comment there terminates the continuation and the next line
# runs as its own command (the 2026-10-05 exit-127 crash loop).
# Prefix caching: ON — the shipped contract. FRANK_PRIMARY_NO_PREFIX_-
# CACHING=1 flips to the fork-named mitigation (--no-enable-prefix-caching)
# for the QFN align-mode GDN state fault; it was tried 2026-10-05 and
# REVERTED the same day (live conversations still degraded at ~40k with
# caching off and zero faults — see the evidence block in
# frank-primary.env). NO comment lines inside the backslash chain below —
# a comment there terminates the continuation and the next line runs as
# its own command (the 2026-10-05 exit-127 crash loop).
# The ${VAR:-word} / ${VAR:+word} double-expansion for the two directions is
# AMBIGUOUS bash (${VAR:--flag} parses as a substring offset when set —
# measured 2026-10-05: rendered "--no-enable-prefix-caching1"), so the flag