serverAppConfigs/serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-secondary on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (ACTIVE)	The ACTIVE frank-secondary runbook (since 2026-10-07) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape.	llm frank-secondary dankclaw llama-cpp gguf rtx3090	2026-10-07

frank-secondary — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV

Live since 2026-10-07, replacing the frank-secondary-llama-gemma4-q4km-200704 era (Q4_K_M on ONE 200,704 slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second conversation slot and unquantized KV back.

Identity

Host / device	Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB
Endpoint	http://192.168.0.143:7771 — alias frank-secondary, /v1 OpenAI surface
Runtime	llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build …/llama-master/build-cuda/bin/llama-server (GGML_CUDA_FA_ALL_QUANTS=ON); qwen35 is registered in the build's own src/llama-arch.cpp and src/models/qwen35.cpp exists at that commit
Unit	root systemd frank-secondary.service (scripts/frank-secondary.service), User=dank, Restart=on-failure, LimitMEMLOCK=infinity
Serve script	scripts/serve-frank-secondary.sh, tunables scripts/frank-secondary.env
Weights	/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf — HauhauCS Q6_K, 7,359,259,008 B, sha256 c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986
Projector	mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf, 921,704,448 B, sha256 05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934 — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability
Fetch	scripts/download-frank-secondary.sh — -f, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, --verify-only
Seam	tests/frank_secondary_unit.bats (argv assertions against a stub engine)

Payload geometry (read from the GGUF header, not the model card)

Range-read of the file's own metadata + tensor-info section:

Key	Value
general.architecture	qwen35 (dense hybrid: Gated DeltaNet + softmax attention)
qwen35.block_count	32, full_attention_interval 4 ⇒ 8 full-attention layers (3, 7, 11, …, 31) + 24 DeltaNet layers
attention	16 heads, 4 KV heads, key_length = value_length = 256
qwen35.ssm.*	conv_kernel 4, state_size 128, group_count 16, time_step_rank 32, inner_size 4096
context / vocab	262,144 native / 248,320
tensor types	Q6_K × 250 + F32 × 177 — standard types only, so the stock master build loads it (the ROCmFPX-style custom-quant lock-in does not apply)
MTP	no nextn/mtp tensors exist — in-file MTP is impossible; the card's "MTP support" line describes the base architecture, not this quantized export
chat template	supports enable_thinking, tools/tool_call, image/video content parts; it has no reasoning_effort variable (unused Jinja variables are ignored, so the harness's effort map is inert here, not an error source)

Shape

--ctx-size 401408 --parallel 2 — llama.cpp divides the pool across slots, so each conversation gets n_ctx_slot = 200,704 (Frank's cap; the harness contextWindow quotes the per-slot number). --flash-attn on, --cache-type-k f16 --cache-type-v f16 (exact KV), batch 4096 / ubatch 2048, --log-verbosity 4, --jinja --reasoning auto --reasoning-format deepseek, --mmproj, speculative decoding OFF (SPEC_TYPE=none strips both spec flags — the file has no MTP tensors).

KV rate law. The pool scales with the 8 full-attention layers: 8 × 4 KV heads × 256 dim = 8,192 elements/token per cache, so f16 K + f16 V bills 32,768 B/token = 32 KiB/token → 12.25 GiB at the 401,408 pool. The 24 DeltaNet layers bill a fixed recurrent state per slot, not per token — that is what makes two 200k slots affordable where the 26B MoE fit only one.

Priced from boots, not formulas. MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B, same mmproj class) at 2 × 200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024 and 21,846 / 2,279 at ubatch 2048 — see frank-secondary-llama-mimo-q6k-2x200704. This file is 415 MiB lighter in weights and its projector is 3 MiB heavier, so the projection is ~21,430 used / ~2,700 free.

Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 % prefill at 8.9k depth, decode flat, cost 616 MiB); 1024 is the documented fallback rung if a future shape lands under the 2 GiB floor.

Measured boot (2026-10-07, deploy ec6184d)

nvidia-smi: 21,436 MiB used / 2,689 MiB free — the MiMo-priced projection (~21,430 / ~2,700) was within 11 MiB. Engine ledger (--log-verbosity 4, journalctl -u frank-secondary):

Line	Value
llama_context	n_ctx = 401408, n_ctx_seq = 200704, kv_unified = false
llama_kv_cache	CUDA0 KV buffer size = 12544.00 MiB — size = 12544.00 MiB (200704 cells, 8 layers, 2/2 seqs), K (f16): 6272.00 MiB, V (f16): 6272.00 MiB
sched_reserve	CUDA0 compute buffer size = 1136.06 MiB, CUDA_Host compute buffer size = 848.07 MiB
clip_model_loader (mmproj)	model size: 878.99 MiB + reserve_compute_meta: CUDA0 248.10 MiB, CPU 24.93 MiB
srv load_model	n_slots = 2, n_ctx_slot = 200704
print_info	n_ctx_train = 262144, n_head_kv = 4

The 32 KiB/token law is confirmed exactly: 401,408 × 32 KiB = 12,544 MiB, which is what the engine printed. The n_ctx_seq (200704) < n_ctx_train (262144) line is informational (a context below the trained window is safe; the Gemma4 era's warning was the opposite case, a drafter running 53 % past its window). Weights load in ~6 s from page cache; /health answers {"status":"ok"} and /v1/models serves the alias frank-secondary.

The deploy's Playwright gate passed (33 tests) and the fleet e2e set is unchanged.

Client contract

llama.cpp surface: model id frank-secondary, native tools via --jinja, thinking in reasoning_content, vision via OpenAI image_url content parts (probe with a ≥ 256 px image, or the Qwen-VL tower answers "video"). The publisher's recipe ships in models.yml extraBody: thinking mode temperature 0.6 / top_p 0.95 / top_k 20 / min_p 0 (non-thinking mode is 0.7/0.8/20/0). The card's advice "maintain at least 128K context to preserve thinking" is satisfied by the 200,704 slot.

Compaction hub: models.yml routes compactionModel for all three endpoints here. With --parallel 2 a compaction request no longer serializes behind a live generation (the Gemma4 era's --parallel 1 law), and the 200,704 slot still fits the worst-case compaction input (~133k tokens).

Rollback

Target	How
Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V	revert the swap commit — payload stays at /home/dank/models/Gemma4-26B-A4B/; see frank-secondary-llama-gemma4-q4km-200704
MiMo-V2.6 Q6_K, 2 × 200,704, f16 KV	MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf + its mmproj — measured 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048
Swift-1.5-27B IQ3_XXS 2 × 114,688 + MTP	payload on disk at /home/dank/models/Swift15-UncIQ3XXS/

When it misbehaves

Same decision tree as the predecessors: No CUDA GPUs are available ⇒ nvidia-modprobe -u -c=0; a 400 naming 200,704 ⇒ the conversation outgrew one slot, not the pool; queue behind two long generations ⇒ the slot ceiling is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in journalctl -u frank-secondary (KV MiB, compute buffers) before touching flags — the engine prints the exact figures the shape needs.

status=203/EXEC boot loop: the rewrite of serve-frank-secondary.sh shipped without its #!/usr/bin/env bash line, and systemd failed the ExecStart before the script ran — 10 s restarts, Mem peak: 1.9M, and nvidia-smi flat at 1 MiB used (the GPU is never touched). Journal signature: Failed at step EXEC spawning /home/dank/…/serve-frank-secondary.sh. A shebang is not decoration on this host: every LLM serve script is a direct ExecStart, so head -1 is part of the deploy check (commit ec6184d).