dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-small: two 128k slots, no spec, ubatch 512 (Frank's tight-fit order, 2026-10-06)

Browse Source 
CTX 262144 / PARALLEL 2 = two 131,072 conversations; SPEC_TYPE=none strips both speculative flags in the serve script (ngram-mod skipped on purpose — on a qwen35 hybrid it disables recurrent-state rollback); ubatch 1024->512 claws back ~0.3 GiB. Budget: non-KV ~15.5 GiB + two q8_0 slots 8,704 MiB lands AT the card — the 2-3 GiB floor is knowingly violated by explicit order; nvidia-smi free at boot is the measured truth. Seam retargeted: shape (2 slots, 131072/slot), budget (two fit, three don't), spec-off (absence of both flags), unit Description.
This commit is contained in:
Frank committed 2026-10-06 00:07:03 -07:00
1 parent fb1a4fd150
commit 812d9ebff5
4 files changed
+113 -97
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/frank-small.service
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=llama.cpp Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S-mtp GGUF (qwen35 hybrid, 1x131072, q8_0 K+V, in-file MTP head, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
Description=llama.cpp Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S-mtp GGUF (qwen35 hybrid, 2x131072 slots, q8_0 K+V, no speculative decoding, mmproj vision) on the RTX 3090 (dankclaw, frank-small)
After=network-online.target
Wants=network-online.target