frank-secondary: swap Gemma4-26B-A4B Q4_K_M -> Qwen3.5-9B-Uncensored Q6_K 2x200,704

Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention
+ 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less,
natively multimodal) costs ~7 GiB of weights where the 26B MoE cost
~16.6 GiB. The freed room buys back the second conversation slot and
exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K +
f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable
is impossible), vision via the publisher's BF16 mmproj, ubatch 1024
(the CUDA decode-optimal value), verbosity 4, publisher sampler recipe
(0.6/0.95/20/0) in models.yml.

Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB
used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch
2048. Compaction stays routed here; window math is unchanged (maxTokens
16,384 keeps the worst-case harness request inside the slot).

Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the
rollback; its runbook is retired in place, keeping the attention_k_eq_v
pair-billing law.

Co-Authored-By: omp <omp@franksdank.cloud>
This commit is contained in:
Frankandomp committed 2026-10-07 14:47:32 -07:00
1 parent b51e078e36
commit 1d062f06a3
10 files changed
+360 -213

No files matched your search

+9 -3
View File
@@ -372,10 +372,16 @@ The LLM inference fleet on the two hosts. Full index:
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-secondary-llama-gemma4-q4km-200704]] — the **ACTIVE** frank-secondary
runbook since 2026-10-07: llama.cpp master (CUDA sm_86) serving
- [[frank-secondary-llama-qwen35-9b-q6k-2x200704]] — the **ACTIVE** frank-secondary
runbook since 2026-10-07 (late): llama.cpp master (CUDA sm_86) serving
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090 —
TWO 200,704-token conversations, exact f16 K + f16 V, MTP **impossible**
(the file carries no `nextn`/`mtp` tensors), vision through the publisher's
BF16 mmproj; priced from the MiMo Q6_K boots (same qwen35 geometry)
- [[frank-secondary-llama-gemma4-q4km-200704]] — RETIRED 2026-10-07 (same day;
payload on disk as rollback): llama.cpp master (CUDA sm_86) serving
HauhauCS Gemma4-26B-A4B QAT Uncensored **Q4_K_M** on the RTX 3090 — ONE
200,704-token conversation, q8_0 K + q8_0 V, MTP **disabled** (the
200,704-token conversation, f16 K + q8_0 V, MTP **disabled** (the
`gemma4-assistant` drafter costs ~1.9 GiB: boot was 23,806 used/319 free
with it, 21,890/2,235 without), vision through the publisher's mmproj
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — RETIRED 2026-10-07