dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: prefix caching deployed on frank-primary (E8-qualified, 9x/63x reuse TTFT)

Browse Source
This commit is contained in:
Frank
2026-09-28 12:33:28 -07:00
parent 41e918929f
commit ba2100e22b
2 changed files with 17 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md
+10 -7
View File
@@ -40,7 +40,7 @@ The repo carries a **thin** wiring; the engine shape lives on-host:
| Piece | File | Law |
|---|---|---|
| Unit | `scripts/frank-primary.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-primary-r9v`, `Restart=on-failure` |
| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, sharp chat template |
| Env | `scripts/frank-primary.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary`, container `frank-primary-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, `R9V_PLE_READY_TIMEOUT=1800`, prefix caching ON, sharp chat template |
| Serve | `scripts/serve-frank-primary.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_primary` | Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check |
@@ -140,9 +140,11 @@ edited in place — mutating it drifts the qualification.
ssd-mmap residency).
- Context envelope: `R9V_MAX_MODEL_LEN=131072`; harness `models.yml`
carries `contextWindow: 131072`, `maxTokens: 36864` (compaction-gap
math: 70% × 131,072 = 91,750 + 36,864 = 128,614 ≤ 131,072).
- Prefix caching OFF (`R9V_ENABLE_PREFIX_CACHING=0`), CACHE192 off — the
KV envelope at 131,072 is smaller than the uncensored 196,608 shape.
math: 70% × 131,072 = 91,750 + 36,864 = 128,614 ≤ 131,072). The KV
envelope at 131,072 is smaller than the uncensored 196,608 shape.
- Prefix caching OFF at cutover (`R9V_ENABLE_PREFIX_CACHING=0`), CACHE192
off — superseded 2026-09-28: prefix caching is ON (see bench matrix
below).
## Prefill / decode bench matrix (2026-09-28, scratch port 7773)
@@ -183,9 +185,10 @@ prompt TTFT 44.2 s → **3.6–4.3 s** (9x), 131k 439 s → **6.9 s** (63x).
Hit pattern: rounds 1–2 miss, round 3+ hit (identical-prompt and
conversation-growth probes both). For the harness's resend-every-turn
traffic that is a ~40–440 s TTFT win per steady-state turn; the one-turn
miss costs nothing extra. **RECOMMENDED for production — pending Frank's
go-ahead** (one env line; the launcher default is already ON, the q8 state
opted out).
miss costs nothing extra. **DEPLOYED 2026-09-28** (Frank's go-ahead): the
repo env sets `R9V_ENABLE_PREFIX_CACHING=1` (commit `41e9189`), verified
on :7770 (`--enable-prefix-caching` in the container argv, health 200,
live completion answering).
## Ops