dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: note prefix caching enabled on frank-default-small (opt-in for hybrid; ~4x TTFT on repeated prefix)

Browse Source
This commit is contained in:
Frank
2026-08-17 18:45:49 -07:00
parent b854ec78ad
commit 044dd49011
2 changed files with 18 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment.md
+7 -2
View File
@@ -120,9 +120,14 @@ The active service on Dankclaw is vLLM (ROCm) serving
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name
`frank-default-small` on port 52625. Full 262144 context, unquantized KV
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram
speculative decoding, `--reasoning-parser qwen3`, and tool calling
speculative decoding, `--reasoning-parser qwen3`, tool calling
(`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom
chat template `/home/dank/models/qwen36-chat_template-tools.jinja`). The
chat template `/home/dank/models/qwen36-chat_template-tools.jinja`), and
prefix caching (`--enable-prefix-caching`). The model is hybrid (GatedDeltaNet
+ full attention); vLLM keeps prefix caching opt-in for hybrid models, so the
flag is required. It reuses the KV of repeated prompt prefixes (pi re-sends
the same system prompt + tool schema every turn), cutting follow-up TTFT
roughly 4x (2023-token prefix: 4.6s cold -> 1.1s cached, 2026-08-17). The
unit file is `scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
`scripts/frank-default-small.env`.