dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: note prefix caching enabled on frank-default-small (opt-in for hybrid; ~4x TTFT on repeated prefix)

Browse Source
This commit is contained in:
Frank
2026-08-17 18:45:49 -07:00
parent b854ec78ad
commit 044dd49011
2 changed files with 18 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment.md
+7 -2
View File
@@ -120,9 +120,14 @@ The active service on Dankclaw is vLLM (ROCm) serving
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name
`frank-default-small` on port 52625. Full 262144 context, unquantized KV
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram
speculative decoding, `--reasoning-parser qwen3`, and tool calling
speculative decoding, `--reasoning-parser qwen3`, tool calling
(`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom
chat template `/home/dank/models/qwen36-chat_template-tools.jinja`). The
chat template `/home/dank/models/qwen36-chat_template-tools.jinja`), and
prefix caching (`--enable-prefix-caching`). The model is hybrid (GatedDeltaNet
+ full attention); vLLM keeps prefix caching opt-in for hybrid models, so the
flag is required. It reuses the KV of repeated prompt prefixes (pi re-sends
the same system prompt + tool schema every turn), cutting follow-up TTFT
roughly 4x (2023-token prefix: 4.6s cold -> 1.1s cached, 2026-08-17). The
unit file is `scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
`scripts/frank-default-small.env`.
docs/frank-redo-2026-08-17.md
+11
View File
@@ -189,3 +189,14 @@ cache for rollback.
`MAX_NUM_BATCHED_TOKENS` lowered from 16384 to 8192 (user request).
vLLM has no separate ubatch knob; the scheduler issues one batch per
step up to this token cap.
### 4. Prefix caching enabled
Added `--enable-prefix-caching`. The model is hybrid (GatedDeltaNet +
full attention); vLLM keeps prefix caching **opt-in** for hybrid models
(`default_prefix_caching = is_prefix_caching_supported and not
is_hybrid`), so it was off by default. Enabling it reuses the KV of
repeated prompt prefixes. Measured on a 2023-token prefix (2026-08-17
evening): 4.6s cold -> 1.1s cached (~4x TTFT), output unchanged. This is
the main TTFT lever for pi, which re-sends the same system prompt + tool
schema every turn. Toggled via `ENABLE_PREFIX_CACHING` in the env file.