Frankandomp
1d062f06a3
Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention + 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less, natively multimodal) costs ~7 GiB of weights where the 26B MoE cost ~16.6 GiB. The freed room buys back the second conversation slot and exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K + f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable is impossible), vision via the publisher's BF16 mmproj, ubatch 1024 (the CUDA decode-optimal value), verbosity 4, publisher sampler recipe (0.6/0.95/20/0) in models.yml. Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch 2048. Compaction stays routed here; window math is unchanged (maxTokens 16,384 keeps the worst-case harness request inside the slot). Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the rollback; its runbook is retired in place, keeping the attention_k_eq_v pair-billing law. Co-Authored-By: omp <omp@franksdank.cloud>
57 lines
2.9 KiB
Bash
57 lines
2.9 KiB
Bash
# frank-secondary — llama.cpp (master, CUDA sm_86) serving
|
||
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
|
||
# :7771.
|
||
#
|
||
# 2026-10-07 (Frank): swap from Gemma4-26B-A4B QAT Q4_K_M (one 200,704 slot)
|
||
# to Qwen3.5-9B Q6_K at TWO 200,704-token conversations, EXACT (f16) KV,
|
||
# MTP disabled (SPEC_TYPE=none), log verbosity 4.
|
||
#
|
||
# Architecture — read from the GGUF's own header (HTTP range read, not the
|
||
# model card): arch qwen35, 32 layers, dense 9B, full_attention_interval 4
|
||
# ⇒ 8 full-attention layers (3,7,11,...,31) + 24 Gated DeltaNet linear layers,
|
||
# 16 attention heads / 4 KV heads × 256 dim, ssm conv_kernel 4 / state_size
|
||
# 128 / group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144,
|
||
# vocab 248,320, file_type 18 (Q6_K). Tensor-type histogram: Q6_K×250 +
|
||
# F32×177 — standard types only, so the stock master build loads it (no
|
||
# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
|
||
# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
|
||
#
|
||
# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
|
||
# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
|
||
# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
|
||
# recurrent state per SLOT (not per token) — that is why two 200k slots fit
|
||
# where the 26B MoE fit only one.
|
||
#
|
||
# Feasibility comes from the predecessor's MEASURED boots, not a formula:
|
||
# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at
|
||
# 2×200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024
|
||
# and 21,846 / 2,279 at ubatch 2048. This file is 415 MiB lighter in weights,
|
||
# its BF16 mmproj is 3 MiB heavier ⇒ projected ~21,430 used / ~2,700 free.
|
||
# The verbosity-4 boot ledger is the referee.
|
||
#
|
||
# Ubatch 2048 is the measured CUDA optimum (measured on the MiMo predecessor
|
||
# at this exact arch + shape: +4.6 % prefill at 8.9k depth, decode flat, cost
|
||
# 616 MiB). 1024 is the fallback rung if the boot lands under the 2 GiB floor.
|
||
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
||
MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf
|
||
MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
|
||
ALIAS=frank-secondary
|
||
BIND_ADDR=0.0.0.0
|
||
LISTEN_PORT=7771
|
||
# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
|
||
# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
|
||
# the harness contextWindow quotes the PER-SLOT number — a longer prompt is
|
||
# answered with a 400, never truncated.
|
||
CTX_SIZE=401408
|
||
PARALLEL=2
|
||
BATCH_SIZE=4096
|
||
UBATCH_SIZE=2048
|
||
CACHE_TYPE_K=f16
|
||
CACHE_TYPE_V=f16
|
||
SPEC_TYPE=none
|
||
NGL=99
|
||
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
|
||
# and graph capture land in the journal, which is what the VRAM-fit forensics
|
||
# read. Levels above 4 flood the journal.
|
||
VERBOSITY=4
|