frank-primary: PLE ready timeout 1800s + bench matrix runbook; index/doc-link fixes
- frank-primary.env: R9V_PLE_READY_TIMEOUT=1800 (PLE registration measured 740s under page-cache churn; engine default 600s killed a prod boot 2026-09-28; launch.sh 2de0357 forwards it) - runbook: prefill/decode bench matrix section (chunk ceiling 4096, 80-slot policy max, MTP6 3.7x worse, prefix-caching evidence + recommendation), fork commit list, residency wording - index: q8 runbook ACTIVE, uncensored RETIRED (rollback); log entry incl. the 9h downtime incident
This commit is contained in:
@@ -29,9 +29,9 @@ on disk; a rollback is an env-file `R9V_STATE_FILE` flip plus a redeploy.
|
||||
The uncensored IQ4_XS payload measured 59–96 tok/s decode with
|
||||
draft-acceptance swings; the Q8_0 payload trades VRAM-resident expert
|
||||
headroom for precision — quantization to Q8_0 doubles expert bytes per
|
||||
block, so the tiered-expert residency budget halves (see [[live-llm-benchmark-methodology]]
|
||||
caveats on decode-vs-residency). The probe-era numbers are **not** the
|
||||
cutover numbers; see "Known costs" below.
|
||||
block, so the tiered-expert residency budget halves — the probe-era
|
||||
11 tok/s decode was residency-bound (expert streaming, not compute).
|
||||
The "Known costs" section below carries the accepted trade.
|
||||
|
||||
## Production shape (repo-tracked)
|
||||
|
||||
@@ -98,7 +98,8 @@ edited in place — mutating it drifts the qualification.
|
||||
|
||||
- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp` (fork commits
|
||||
`363d606` dev-overlay, `2fe6dbb`, `50a79ec` qualification doc,
|
||||
`30d4a32` chat-template passthrough — local-only, remote is GitHub)
|
||||
`30d4a32` chat-template passthrough, `2de0357` PLE-timeout forwarding —
|
||||
local-only, remote is GitHub)
|
||||
- State: `/home/dank/r9v-state/q8xl/setup.json` (0600, `ready: true`)
|
||||
- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V/` — 175.30 GiB
|
||||
payload as six `Q8_0/Qwen3.8-Flash-Next-Q8_0-0000N-of-00006.gguf` shards
|
||||
@@ -143,6 +144,49 @@ edited in place — mutating it drifts the qualification.
|
||||
- Prefix caching OFF (`R9V_ENABLE_PREFIX_CACHING=0`), CACHE192 off — the
|
||||
KV envelope at 131,072 is smaller than the uncensored 196,608 shape.
|
||||
|
||||
## Prefill / decode bench matrix (2026-09-28, scratch port 7773)
|
||||
|
||||
Production stopped for the window (fleet bench law); one matrix plus three
|
||||
follow-up runs, fork's own bench-pp + decode_speed harness, corpus-sliced
|
||||
prompts with the fork's canonical decode rounds. Baseline E0 = production
|
||||
shape (chunk 4096, 80-slot sync cache, MTP4):
|
||||
|
||||
| Config | Override | PP8K tok/s | PP131K tok/s | Decode | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| E0 | — | 954.6 | 883.0 | 141.9 ms/step ≈ 23.8 t/s eff | baseline |
|
||||
| E1R | chunk 8192 | **1201.4 (+25.9%)** | 844.6 (−4.4%) | 158.5 ms (−8.5% t/s) | short-prompt win, decode tax |
|
||||
| E2 | chunk 16384 | — | — | — | boots; 33.3/34.2 GiB VRAM (0.93 free) → requests hang/OOM |
|
||||
| E1/E3 | chunk 8192/32768 | — | — | — | PLE registration blew the 600 s engine timeout (see law below) |
|
||||
| E4 | cache async=1 | — | — | — | doctor: cache-prepare backend requires synchronous single-fill LRU |
|
||||
| E5/E6 | 96/112 slots | — | — | — | doctor: capacities (48, 64, 80) only; expert-budget [108, 198] rank maxima — **80 is the policy max, prod is at it** |
|
||||
| E7 | MTP depth 6 | 843.1 | 819.6 | 454–489 ms/step (3.7x worse) | depth 4 stays |
|
||||
|
||||
Laws from the matrix:
|
||||
|
||||
- **4096 is the prefill-chunk ceiling twice over**: the engine caps
|
||||
scheduled tokens by the speculative config, and E2's VRAM (0.93 GiB free
|
||||
at 16384) shows bigger chunks OOM at request time. 883–955 tok/s is what
|
||||
this payload shape delivers at long context.
|
||||
- **Unqualified shapes fail mid-request, not at health** — E2 and E7 both
|
||||
answered /health 200 and then died streaming.
|
||||
- **PLE registration can exceed the 600 s engine timeout**
|
||||
(`VLLM_PLE_OFFLOAD_READY_TIMEOUT`) under page-cache churn — measured 740 s
|
||||
after a bench teardown; a production boot failed on it the same morning
|
||||
(unit "Deactivated successfully", no error in journal — the serve script
|
||||
exits 0 on the wait). Fixed with `R9V_PLE_READY_TIMEOUT=1800` in
|
||||
`frank-primary.env` (forwarded by launch.sh `2de0357`).
|
||||
|
||||
Prefix caching (E8/E9, `R9V_ENABLE_PREFIX_CACHING=1`, same port): **zero
|
||||
VRAM cost** (hits reuse freed KV blocks; load VRAM identical to E0),
|
||||
cold path unchanged (first request ≈ cold bench), reuse hits large — 32k
|
||||
prompt TTFT 44.2 s → **3.6–4.3 s** (9x), 131k 439 s → **6.9 s** (63x).
|
||||
Hit pattern: rounds 1–2 miss, round 3+ hit (identical-prompt and
|
||||
conversation-growth probes both). For the harness's resend-every-turn
|
||||
traffic that is a ~40–440 s TTFT win per steady-state turn; the one-turn
|
||||
miss costs nothing extra. **RECOMMENDED for production — pending Frank's
|
||||
go-ahead** (one env line; the launcher default is already ON, the q8 state
|
||||
opted out).
|
||||
|
||||
## Ops
|
||||
|
||||
- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
|
||||
|
||||
Reference in New Issue
Block a user