Frank
6584c059d6
Frank's order, two messages: "extend the max context of frank-small to 200700 per individual stream without further quantization", then "reduce it to 2 streams for a total of 401400-ish tokens, set ubatch to 1k", then "unquantize the KV if we have extra space". CTX_SIZE 540672 -> 401408 with PARALLEL 3 -> 2 gives n_ctx_slot 200,704: above the 200,700 the harness compacts at, so a full-length request lands in one stream instead of a 400 exceed_context_size_error. The pool must divide evenly across slots and stay 256-aligned for the FA path, which is why it is 401,408 and not 401,400. CACHE_TYPE_V q8_0 -> f16: the freed third slot is exactly what pays for unquantized values (24.5 -> 32 KiB/token), and the budget lands back in the 2-3 GiB band Frank set on 2026-10-02 instead of leaving ~6 GiB of the card idle. The asymmetric-KV contract that created this endpoint is spent with it; both --cache-type flags stay explicit. --batch-size/--ubatch-size are now passed (2048/1024). The endpoint had never passed either, so it ran on engine defaults nobody wrote down. Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md
20 lines
647 B
Desktop File
20 lines
647 B
Desktop File
[Unit]
|
|
Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 2x200704, unquantized f16 KV, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=dank
|
|
Group=dank
|
|
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
|
# though the weights themselves are copied into VRAM.
|
|
LimitMEMLOCK=infinity
|
|
Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
|
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|