dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: prefix caching deployed on frank-primary (E8-qualified, 9x/63x reuse TTFT)

Browse Source
This commit is contained in:
Frank
2026-09-28 12:33:28 -07:00
parent 41e918929f
commit ba2100e22b
2 changed files with 17 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/log.md
+7
View File
@@ -1,6 +1,13 @@
# Bundle Change Log
## 2026-09-28
- `scripts/frank-primary.env`, runbook — **prefix caching DEPLOYED on
frank-primary** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero
VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200,
completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is
in dank's ROOTLESS podman store (`sudo podman` inspects an empty store
and reports nothing); probe via the shim's `docker` or the unit MainPID.
- `/scripts/frank-primary.env`, `/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`, `/serverAppConfigs-docs/index.md` — **frank-primary prefill/decode bench matrix** (scratch :7773, prod stopped 00:06 for the window per fleet bench law): baseline E0 (chunk 4096 / 80-slot sync / MTP4) = PP8K 954.6, PP131K 883.0 tok/s, decode 141.9 ms/step ≈ 23.8 t/s effective. Chunk 8192 (E1R): PP8K +25.9% (1201.4) but decode −8.5% and PP131K −4.4% — short-prompt win, decode tax, not the fleet's dominant shape. Chunk 16384 boots but OOMs at request time (0.93 GiB free) — 4096 is the ceiling (engine spec-config cap + memory). Expert cache 80 slots is the policy max (doctor: capacities 48/64/80, synchronous LRU only — async unsupported, 96/112 rejected); MTP depth 6 is 3.7x SLOWER than depth 4. Prefix caching (E8/E9): zero VRAM cost, cold path unchanged, reuse hits 32k 44.2→3.6–4.3 s (9x), 131k 439→6.9 s (63x); hit pattern rounds 1–2 miss / 3+ hit — RECOMMENDED, pending Frank's go-ahead. Two failure laws recorded: unqualified shapes fail mid-request (health stays 200), and PLE registration can blow the engine's 600 s `VLLM_PLE_OFFLOAD_READY_TIMEOUT` under page-cache churn (measured 740 s) — launch.sh gained `R9V_PLE_READY_TIMEOUT` forwarding (fork `2de0357`, local-only) and prod now sets 1800.
- **Incident — frank-primary down 00:06 → ~11:00 (11h)**: the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait). Restored by the `R9V_PLE_READY_TIMEOUT=1800` fix deploy (f187165, from main — the deploy gate refuses non-main): health 200 + MainPID = the serve script's own `podman wait` + live completion verified ~11:00. Deploy e2e gate 32/33 — the one failure (11-admin) needs the FrankBook-Pro Mac mini, which was asleep and ignored two WoL bursts (240 packets); recurring pattern per the wol-wake skill. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.