frank-small: two wide unquantized contexts (2 x 200,704) on the 3090
Frank's order, two messages: "extend the max context of frank-small to 200700 per individual stream without further quantization", then "reduce it to 2 streams for a total of 401400-ish tokens, set ubatch to 1k", then "unquantize the KV if we have extra space". CTX_SIZE 540672 -> 401408 with PARALLEL 3 -> 2 gives n_ctx_slot 200,704: above the 200,700 the harness compacts at, so a full-length request lands in one stream instead of a 400 exceed_context_size_error. The pool must divide evenly across slots and stay 256-aligned for the FA path, which is why it is 401,408 and not 401,400. CACHE_TYPE_V q8_0 -> f16: the freed third slot is exactly what pays for unquantized values (24.5 -> 32 KiB/token), and the budget lands back in the 2-3 GiB band Frank set on 2026-10-02 instead of leaving ~6 GiB of the card idle. The asymmetric-KV contract that created this endpoint is spent with it; both --cache-type flags stay explicit. --batch-size/--ubatch-size are now passed (2048/1024). The endpoint had never passed either, so it ran on engine defaults nobody wrote down. Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md
This commit is contained in:
1 parent
7d3dd33de0
commit
6584c059d6
4 files changed
+112
-72
No files matched your search
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 3x180224, f16 K + q8_0 V, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
|
||||
Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 2x200704, unquantized f16 KV, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
|
||||
Reference in new issue
Block a user