dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-secondary: swap Gemma4-26B-A4B Q4_K_M -> Qwen3.5-9B-Uncensored Q6_K 2x200,704

Browse Source 
Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention
+ 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less,
natively multimodal) costs ~7 GiB of weights where the 26B MoE cost
~16.6 GiB. The freed room buys back the second conversation slot and
exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K +
f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable
is impossible), vision via the publisher's BF16 mmproj, ubatch 1024
(the CUDA decode-optimal value), verbosity 4, publisher sampler recipe
(0.6/0.95/20/0) in models.yml.

Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB
used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch
2048. Compaction stays routed here; window math is unchanged (maxTokens
16,384 keeps the worst-case harness request inside the slot).

Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the
rollback; its runbook is retired in place, keeping the attention_k_eq_v
pair-billing law.

Co-Authored-By: omp <omp@franksdank.cloud>
This commit is contained in:
Frank
2026-10-07 14:47:32 -07:00
co-authored by omp
parent b51e078e36
commit 1d062f06a3
10 changed files with 360 additions and 213 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/index.md
+13 -6
View File
@@ -102,13 +102,20 @@ frank-secondary, port 7771]
(`slow`/`advisor` → frank-primary) carried forward. Superseded by the
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
engine moved with the model
- [[frank-secondary-llama-gemma4-q4km-200704]] — **ACTIVE since 2026-10-07**
- [[frank-secondary-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07**
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP **Q4_K_M**
on the RTX 3090 — ONE 200,704-token conversation, q8_0 K + q8_0 V,
vision via the publisher's mmproj. MTP is OFF (the gemma4-assistant
drafter costs ~1.9 GiB: 23,806 used/319 free with it, 21,890/2,235
without). Supersedes the Swift-1.5 era (below).
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090
— TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), exact
f16 K + f16 V, vision via the publisher's BF16 mmproj. MTP is OFF because
the file has no MTP tensors at all. Priced from the MiMo Q6_K boots (same
qwen35 geometry, same card, same shape). Supersedes the Gemma4 era (below).
- [[frank-secondary-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
HauhauCS-Balanced-MTP **Q4_K_M** on the RTX 3090 — ONE 200,704-token
conversation, f16 K + q8_0 V, vision via the publisher's mmproj. MTP was
OFF (the gemma4-assistant drafter costs ~1.9 GiB: 23,806 used/319 free with
it, 21,890/2,235 without).
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback);
ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with