frank-primary: PLE ready timeout 1800s + bench matrix runbook; index/doc-link fixes

- frank-primary.env: R9V_PLE_READY_TIMEOUT=1800 (PLE registration measured
  740s under page-cache churn; engine default 600s killed a prod boot
  2026-09-28; launch.sh 2de0357 forwards it)
- runbook: prefill/decode bench matrix section (chunk ceiling 4096,
  80-slot policy max, MTP6 3.7x worse, prefix-caching evidence +
  recommendation), fork commit list, residency wording
- index: q8 runbook ACTIVE, uncensored RETIRED (rollback); log entry
  incl. the 9h downtime incident

scripts/frank-primary.env

@@ -19,6 +19,12 @@ R9V_HOST_PORT=7770
 R9V_SERVED_MODEL_NAME=frank-primary
 R9V_CONTAINER_NAME=frank-primary-r9v
 R9V_EXPECTED_GPU_BDFS=0000:03:00.0,0000:07:00.0
+# PLE registration (54.4 GB mmap -> HIP) can exceed the engine's 600 s
+# VLLM_PLE_OFFLOAD_READY_TIMEOUT default under page-cache churn (measured
+# 740 s on 2026-09-28, right after a bench teardown; a production boot
+# failed on it the same morning). launch.sh 2de0357 forwards this to the
+# engine; 1800 s covers the slowest observed registration with margin.
+R9V_PLE_READY_TIMEOUT=1800
 # The Q8_0 payload quantizes BOTH fused hyper-connection shards (the IQ4_XS
 # package ships inject as F32), so the shard-uniformity check rejects the
 # fused projection at load. This dev-overlay adapter dequantizes a quantized

serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md

@@ -29,9 +29,9 @@ on disk; a rollback is an env-file `R9V_STATE_FILE` flip plus a redeploy.
 The uncensored IQ4_XS payload measured 59–96 tok/s decode with
 draft-acceptance swings; the Q8_0 payload trades VRAM-resident expert
 headroom for precision — quantization to Q8_0 doubles expert bytes per
-block, so the tiered-expert residency budget halves (see [[live-llm-benchmark-methodology]]
+block, so the tiered-expert residency budget halves — the probe-era
-caveats on decode-vs-residency). The probe-era numbers are **not** the
+11 tok/s decode was residency-bound (expert streaming, not compute).
-cutover numbers; see "Known costs" below.
+The "Known costs" section below carries the accepted trade.
 
 ## Production shape (repo-tracked)
 
@@ -98,7 +98,8 @@ edited in place — mutating it drifts the qualification.
 
 - Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp` (fork commits
   `363d606` dev-overlay, `2fe6dbb`, `50a79ec` qualification doc,
-  `30d4a32` chat-template passthrough — local-only, remote is GitHub)
+  `30d4a32` chat-template passthrough, `2de0357` PLE-timeout forwarding —
+  local-only, remote is GitHub)
 - State: `/home/dank/r9v-state/q8xl/setup.json` (0600, `ready: true`)
 - Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V/` — 175.30 GiB
   payload as six `Q8_0/Qwen3.8-Flash-Next-Q8_0-0000N-of-00006.gguf` shards
@@ -143,6 +144,49 @@ edited in place — mutating it drifts the qualification.
 - Prefix caching OFF (`R9V_ENABLE_PREFIX_CACHING=0`), CACHE192 off — the
   KV envelope at 131,072 is smaller than the uncensored 196,608 shape.
 
+## Prefill / decode bench matrix (2026-09-28, scratch port 7773)
+
+Production stopped for the window (fleet bench law); one matrix plus three
+follow-up runs, fork's own bench-pp + decode_speed harness, corpus-sliced
+prompts with the fork's canonical decode rounds. Baseline E0 = production
+shape (chunk 4096, 80-slot sync cache, MTP4):
+
+| Config | Override | PP8K tok/s | PP131K tok/s | Decode | Verdict |
+|---|---|---|---|---|---|
+| E0 | — | 954.6 | 883.0 | 141.9 ms/step ≈ 23.8 t/s eff | baseline |
+| E1R | chunk 8192 | **1201.4 (+25.9%)** | 844.6 (−4.4%) | 158.5 ms (−8.5% t/s) | short-prompt win, decode tax |
+| E2 | chunk 16384 | — | — | — | boots; 33.3/34.2 GiB VRAM (0.93 free) → requests hang/OOM |
+| E1/E3 | chunk 8192/32768 | — | — | — | PLE registration blew the 600 s engine timeout (see law below) |
+| E4 | cache async=1 | — | — | — | doctor: cache-prepare backend requires synchronous single-fill LRU |
+| E5/E6 | 96/112 slots | — | — | — | doctor: capacities (48, 64, 80) only; expert-budget [108, 198] rank maxima — **80 is the policy max, prod is at it** |
+| E7 | MTP depth 6 | 843.1 | 819.6 | 454–489 ms/step (3.7x worse) | depth 4 stays |
+
+Laws from the matrix:
+
+- **4096 is the prefill-chunk ceiling twice over**: the engine caps
+  scheduled tokens by the speculative config, and E2's VRAM (0.93 GiB free
+  at 16384) shows bigger chunks OOM at request time. 883–955 tok/s is what
+  this payload shape delivers at long context.
+- **Unqualified shapes fail mid-request, not at health** — E2 and E7 both
+  answered /health 200 and then died streaming.
+- **PLE registration can exceed the 600 s engine timeout**
+  (`VLLM_PLE_OFFLOAD_READY_TIMEOUT`) under page-cache churn — measured 740 s
+  after a bench teardown; a production boot failed on it the same morning
+  (unit "Deactivated successfully", no error in journal — the serve script
+  exits 0 on the wait). Fixed with `R9V_PLE_READY_TIMEOUT=1800` in
+  `frank-primary.env` (forwarded by launch.sh `2de0357`).
+
+Prefix caching (E8/E9, `R9V_ENABLE_PREFIX_CACHING=1`, same port): **zero
+VRAM cost** (hits reuse freed KV blocks; load VRAM identical to E0),
+cold path unchanged (first request ≈ cold bench), reuse hits large — 32k
+prompt TTFT 44.2 s → **3.6–4.3 s** (9x), 131k 439 s → **6.9 s** (63x).
+Hit pattern: rounds 1–2 miss, round 3+ hit (identical-prompt and
+conversation-growth probes both). For the harness's resend-every-turn
+traffic that is a ~40–440 s TTFT win per steady-state turn; the one-turn
+miss costs nothing extra. **RECOMMENDED for production — pending Frank's
+go-ahead** (one env line; the launcher default is already ON, the q8 state
+opted out).
+
 ## Ops
 
 - **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve

serverAppConfigs-docs/index.md

@@ -124,13 +124,20 @@ The LLM inference fleet on the two hosts. Full index:
 
 - [[deployment]] — unified deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
-- [[frank-primary-r9v-radiance-qwen38-uncensored]] — the ACTIVE
+- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE
   frank-primary runbook: the R9V radiance-vLLM fork (docker-pinned bundle
-  `2dac17a2`, docker→podman shim) serving the abliterated
+  `2dac17a2`, docker→podman shim) serving the unsloth
+  Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) + blockfp8 MTP4 +
+  BF16 vision on Dankserver's both R9700s — TP=2 tiered experts (80-slot
+  LRU cache), PLE pinned 54.4 GB in host RAM, CED off, 131,072 ctx single
+  stream, Qwen-Sharp template, root unit with LimitMEMLOCK, the on-host
+  qualified state, the cutover, and the rollback
+- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
+  (rollback target): the R9V radiance-vLLM fork serving the abliterated
   Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS (92.39 GiB) + MTP4 + F16
-  vision on Dankserver's both R9700s — TP=2 tiered experts with a UVA
+  vision — TP=2 tiered experts with a UVA pinned host copy, PLE on
-  pinned host copy, PLE on ssd-mmap, CED on, 196608 ctx, root unit with
+  ssd-mmap, CED on, 196,608 ctx; state dir + frozen config + payload
-  LimitMEMLOCK, the on-host 104-key tuned state, cutover and rollback
+  intact on disk
 - [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
   (rollback target): unsloth llama.cpp (b11030-mix prebuilt, ROCm
   gfx120X, root unit) serving unsloth Qwen3.8-Flash-Next at Q8_0 (175.3

serverAppConfigs-docs/log.md

@@ -1,5 +1,9 @@
 # Bundle Change Log
 
+## 2026-09-28
+- `/scripts/frank-primary.env`, `/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`, `/serverAppConfigs-docs/index.md` — **frank-primary prefill/decode bench matrix** (scratch :7773, prod stopped 00:06 for the window per fleet bench law): baseline E0 (chunk 4096 / 80-slot sync / MTP4) = PP8K 954.6, PP131K 883.0 tok/s, decode 141.9 ms/step ≈ 23.8 t/s effective. Chunk 8192 (E1R): PP8K +25.9% (1201.4) but decode −8.5% and PP131K −4.4% — short-prompt win, decode tax, not the fleet's dominant shape. Chunk 16384 boots but OOMs at request time (0.93 GiB free) — 4096 is the ceiling (engine spec-config cap + memory). Expert cache 80 slots is the policy max (doctor: capacities 48/64/80, synchronous LRU only — async unsupported, 96/112 rejected); MTP depth 6 is 3.7x SLOWER than depth 4. Prefix caching (E8/E9): zero VRAM cost, cold path unchanged, reuse hits 32k 44.2→3.6–4.3 s (9x), 131k 439→6.9 s (63x); hit pattern rounds 1–2 miss / 3+ hit — RECOMMENDED, pending Frank's go-ahead. Two failure laws recorded: unqualified shapes fail mid-request (health stays 200), and PLE registration can blow the engine's 600 s `VLLM_PLE_OFFLOAD_READY_TIMEOUT` under page-cache churn (measured 740 s) — launch.sh gained `R9V_PLE_READY_TIMEOUT` forwarding (fork `2de0357`, local-only) and prod now sets 1800.
+- **Incident — frank-primary down 00:06 → ~10:10 (9h)**: the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait); the `R9V_PLE_READY_TIMEOUT=1800` env fix + redeploy restores it. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.
+
 ## 2026-09-27
 - `/scripts/frank-primary.env`, `/scripts/frank-primary.service`, `/scripts/serve-frank-primary.sh`, `/scripts/deploy-remote.sh`, `/tests/frank_primary_unit.bats`, `/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` (new), AGENTS.md + LLM index — **frank-primary cut over to the Q8_0 payload** (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in `R9V_CHAT_TEMPLATE_FILE`/`R9V_CHAT_TEMPLATE_KWARGS` passthrough (fork commit `30d4a32`, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-large. Env points at `/home/dank/r9v-state/q8xl/setup.json` + the quantized-inject dev-overlay adapter (fork `363d606`) + `R9V_FULL_MUTABLE_CACHE=0`; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's `speculative_config` is a single dict, no chaining, so llama.cpp's `draft-mtp,ngram-mod` pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green.
 - Dankserver frank-primary slot (experiment window) — the `radiance-vllm-qwen4exp` fork qualified the unsloth `Qwen3.8-Flash-Next-GGUF` **Q8_0** payload (175.30 GiB, 6 split parts) on the dual R9700. Two launch blockers fixed loader-side in the fork: `runtimes/dev-overrides/qwen4_exp.py` dequantizes quantized `block_inject` shards in-stream (the Q8_0 payload quantizes BOTH fused-HC shards; IQ4_XS ships inject as F32 — the uniformity check rejected the mixed projection) and `R9V_FULL_MUTABLE_CACHE=0` passthrough for the 80-slot tiered cache (fork commits 363d606, 2fe6dbb, 50a79ec — GitHub remote, not pushed). Battery on :7773: text + reasoning channel, vision ("Red" on a red PNG), prose ≈11.0 tok/s wall, MTP4 mean acceptance length 3.19–5.00; PLE pinned residency proven (`mapping_bytes=54400261120`, `mapping_rss_bytes` == mapping size, both ranks required 108.8 GiB vs 140.9 available). Relaunch recipe saved at `dankserver:/home/dank/r9v-state/q8xl/START-COMMAND.txt`; record at fork `docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md`.