dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-medium prompt cache -- why the cap is load-bearing, and the vision correction

Browse Source
This commit is contained in:
Frank
2026-09-18 15:57:25 -07:00
parent 3364e9afad
commit cf06c4eab8
1 changed files with 15 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+15 -7
View File
@@ -392,12 +392,17 @@ with one `WRN` line and nothing else:
- prompt state size 11388.990 MiB exceeds cache size limit 8192.000 MiB, skipping - prompt state size 11388.990 MiB exceeds cache size limit 8192.000 MiB, skipping
``` ```
That line appeared **137 times** in the two days from 2026-09-16. At 8192 MiB That line appeared **137 times** in the two days from 2026-09-16: at 8192 MiB
every conversation past ~170k tokens was uncacheable — the state was refused, every conversation past ~170k tokens lost its state on the next slot
the slot's KV went with it, and the next turn re-prefilled the whole thing. The reassignment and had to be re-prefilled in full. It matters more here than the
cap is a token budget in disguise, because a state is the serialized KV of a cap alone suggests — this is a hybrid model (16 x 3 DeltaNet + 1 gated
whole sequence plus its constant-size DeltaNet state plus the MTP draft attention), so a slot's recurrent cell sits at the end of its own prompt and
context's own KV: `--ctx-checkpoints 0` leaves no checkpoint to roll back to; llama.cpp then
takes the "forcing full prompt re-processing due to lack of cache data" path.
The RAM-side cache is the only mechanism that can carry a state *across* slots,
and a refused state is simply gone. The cap is a token budget in disguise,
because a state is the serialized KV of a whole sequence plus its
constant-size DeltaNet state plus the MTP draft context's own KV:
| Component | Per token | | Component | Per token |
|---|---| |---|---|
@@ -670,7 +675,10 @@ failed to load.
OpenAI-compatible `/v1` on **7770**, alias **`frank-medium`**, streaming and OpenAI-compatible `/v1` on **7770**, alias **`frank-medium`**, streaming and
`/health`, `/props`, `/metrics` as llama.cpp provides them. Reasoning traces `/health`, `/props`, `/metrics` as llama.cpp provides them. Reasoning traces
are projected into `reasoning_content`. A request that sends no sampler fields are projected into `reasoning_content`. A request that sends no sampler fields
gets the server defaults above; a client that sends its own wins. Text only. gets the server defaults above; a client that sends its own wins. Images ride
the BF16 projector — `/props` reports `modalities: {vision: true, video: true,
audio: false}` (read live 2026-09-18), and there is no audio anywhere in the
fleet.
## Troubleshooting ## Troubleshooting