dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: double the host-RAM prompt cache to 16 GiB

Browse Source 
--cache-ram caps the total bytes of saved prompt states in HOST RAM, and
alloc() refuses any single state larger than the whole cap. That refusal was
firing in production: 137 times in the two days from 2026-09-16, at 11.0-11.4
GiB per state, so every conversation past ~170k tokens was uncacheable and
re-prefilled from scratch on its next turn.

The state law at this shape is 68 KiB/token (16 full-attention layers x 4 kv
heads x 256 dim x 2 B x 2 for K/V, plus the MTP draft context's 4096 B), so
8192 MiB held ~120k tokens and 16384 holds ~240k: a FULL 196608-token slot
state (~12.8 GiB) now fits with ~3 GiB spare.

Host RAM only, allocated on demand and never pinned, so the VRAM fit is
untouched. New seam: the cap must clear one full slot's state and stay under
half the host's RAM (it fails on the old 8192 and passes on 16384). Also
corrects the adjacent claim that 0 is the build's default for
--ctx-checkpoints -- this binary's --help says 32.
This commit is contained in:
Frank
2026-09-18 15:25:55 -07:00
parent cfabe07959
commit 1acca430a7
3 changed files with 100 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+37 -1
View File
@@ -149,7 +149,7 @@ Deployed flags:
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
| `--ctx-size` / `--parallel` | `393216` / `2` | the pool over two slots of 196608; `--parallel` **divides** the context. 1.5x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section |
| `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] |
| `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB |
| `--ctx-checkpoints` / `--cache-ram` | `0` / `16384` | no per-slot KV snapshots (the build's default is **32** — this is a deliberate deviation, not an inherited default); the RAM-side prompt cache is capped at 16 GiB, which is what lets a full 196608-token slot state be cached at all (2026-09-18, see below) |
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
| `--spec-type` / `--spec-draft-n-max` | `draft-mtp` / `3` | the model's own head, `shares_model` — no sidecar file, no second model copy; n_max 3 is the measured MTP winner (32.4 tok/s vs 26.7 at n_max 7). The DFlash2 sidecar stays on disk: `SPEC_TYPE=draft-dflash` brings the 43.5 tok/s chain back at ~1.1 GiB on ROCm1 |
| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
@@ -380,6 +380,42 @@ to give. A chat completion answers in 1.4 s end-to-end; MTP acceptance
runs 0.51–0.91 (mean len 2.7–3.7) on the live traffic.
## 2026-09-18: the prompt cache doubles to 16 GiB
`--cache-ram` is the one cache tunable on this service that is **not VRAM**: it
caps the total bytes of saved prompt states in host RAM (`server_prompt_cache`,
`tools/server/server-task.cpp`). `update()` evicts oldest-first past the cap,
and `alloc()` **refuses outright** any single state larger than the whole cap —
with one `WRN` line and nothing else:
```
- prompt state size 11388.990 MiB exceeds cache size limit 8192.000 MiB, skipping
```
That line appeared **137 times** in the two days from 2026-09-16. At 8192 MiB
every conversation past ~170k tokens was uncacheable — the state was refused,
the slot's KV went with it, and the next turn re-prefilled the whole thing. The
cap is a token budget in disguise, because a state is the serialized KV of a
whole sequence plus its constant-size DeltaNet state plus the MTP draft
context's own KV:
| Component | Per token |
|---|---|
| 16 full-attention layers x 4 kv heads x 256 dim x 2 B x 2 (K,V) | 64 KiB |
| MTP draft context (one nextn layer, same arithmetic) | 4096 B |
| **state law** | **68 KiB** |
| Cap | Tokens storable | Fits one 196608-token slot (~12.8 GiB)? |
|---|---|---|
| 8192 MiB (until 2026-09-18) | ~120k | no — refused, then re-prefilled |
| **16384 MiB (deployed)** | ~240k | **yes**, with ~3 GiB of budget spare |
The cost is host RAM, allocated on demand and never pinned, so no VRAM figure
above moves. dankserver has 62 GiB beside the media-stack co-tenants, which is
why the cache is bounded rather than set to `-1` (unlimited): the seam test in
`tests/frank_medium_unit.bats` gates both ends — the cap must clear one full
slot's state, and it may not claim more than half the host's RAM.
## Verified serving behaviour (2026-09-16)
- Load: `model loaded` ~25 s after start with a warm page cache, then