diff --git a/docs/deployment.md b/docs/deployment.md index bd286a5..9d8d0cf 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -120,9 +120,14 @@ The active service on Dankclaw is vLLM (ROCm) serving `nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name `frank-default-small` on port 52625. Full 262144 context, unquantized KV cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram -speculative decoding, `--reasoning-parser qwen3`, and tool calling +speculative decoding, `--reasoning-parser qwen3`, tool calling (`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom -chat template `/home/dank/models/qwen36-chat_template-tools.jinja`). The +chat template `/home/dank/models/qwen36-chat_template-tools.jinja`), and +prefix caching (`--enable-prefix-caching`). The model is hybrid (GatedDeltaNet ++ full attention); vLLM keeps prefix caching opt-in for hybrid models, so the +flag is required. It reuses the KV of repeated prompt prefixes (pi re-sends +the same system prompt + tool schema every turn), cutting follow-up TTFT +roughly 4x (2023-token prefix: 4.6s cold -> 1.1s cached, 2026-08-17). The unit file is `scripts/frank-default-small.service`. It launches through `scripts/serve-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`. diff --git a/docs/frank-redo-2026-08-17.md b/docs/frank-redo-2026-08-17.md index cd748b7..d2e2c06 100644 --- a/docs/frank-redo-2026-08-17.md +++ b/docs/frank-redo-2026-08-17.md @@ -189,3 +189,14 @@ cache for rollback. `MAX_NUM_BATCHED_TOKENS` lowered from 16384 to 8192 (user request). vLLM has no separate ubatch knob; the scheduler issues one batch per step up to this token cap. + +### 4. Prefix caching enabled + +Added `--enable-prefix-caching`. The model is hybrid (GatedDeltaNet + +full attention); vLLM keeps prefix caching **opt-in** for hybrid models +(`default_prefix_caching = is_prefix_caching_supported and not +is_hybrid`), so it was off by default. Enabling it reuses the KV of +repeated prompt prefixes. Measured on a 2023-token prefix (2026-08-17 +evening): 4.6s cold -> 1.1s cached (~4x TTFT), output unchanged. This is +the main TTFT lever for pi, which re-sends the same system prompt + tool +schema every turn. Toggled via `ENABLE_PREFIX_CACHING` in the env file.