serverAppConfigs/serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-zealot on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (RETIRED, rollback target)	The frank-zealot runbook of 2026-10-07 → 2026-10-17 (RETIRED — rollback target) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape.	llm frank-zealot dankclaw llama-cpp gguf rtx3090	2026-10-07

frank-zealot — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV

RETIRED 2026-10-07 → 2026-10-17 (AGENTS.md Runbook Index), superseded by frank-zealot-llama-prism-bonsai2-pq20 (27B PQ2_0, 2 × 200,704 q8_0 KV, build 10754), itself retired 2026-10-17 → 10-18 and now the rollback target: :7771 today is llama.cpp PrismML build 10777 on Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf, 3 × 262,144 q8_0 KV, MTP off by measurement (AGENTS.md Runbook Index active row; scripts/frank-zealot.env). Transient 2026-10-18: :7771 is inside a bench window — the unit is failed since 03:14 and a hand-booted process answers the port; restore with ./scripts/deploy.sh dankclaw --service frank-zealot. This era is the rollback target: the payload is still on disk (stat -c %s /home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf → 7359259008, read 2026-10-18) with its master CUDA build. Same census: the 10754 rollback payload Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf is present at 7206168928 with its Q8_0 mmproj at 629246976, and Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf at 2675410720 — nothing this era or its successors need has been purged. It replaced the frank-zealot-llama-gemma4-q4km-200704 era (Q4_K_M on ONE 200,704 slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second conversation slot and unquantized KV back.

Identity

Host / device	Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB
Endpoint	http://192.168.0.143:7771 — served model id frank-zealot (ALIAS in the env; the SERVICE keeps the frank-zealot name), /v1 OpenAI surface
Runtime	llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build …/llama-master/build-cuda/bin/llama-server — --version on the disk build reads 0.5.0-dev (build 1, commit 5fc4f3c) (built 2026-10-01); its CMakeCache.txt says GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF and that tree carries no CUDA-side guard for the option (ggml/src/ggml-cuda/fattn.cu:438-449 dispatches Q4/Q5 KV unconditionally), so quantized KV never required it; qwen35 is registered in the build's own src/llama-arch.cpp and src/models/qwen35.cpp exists at that commit
Unit	root systemd frank-zealot.service (scripts/frank-zealot.service), User=dank, Restart=on-failure, LimitMEMLOCK=infinity
Serve script	scripts/serve-frank-zealot.sh, tunables scripts/frank-zealot.env
Weights	/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf — HauhauCS Q6_K, 7,359,259,008 B, sha256 c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986
Projector	mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf, 921,704,448 B, sha256 05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934 — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability
Fetch	scripts/download-frank-zealot.sh — -f, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, --verify-only
Seam	tests/frank_zealot_unit.bats (argv assertions against a stub engine)

Payload geometry (read from the GGUF header, not the model card)

Range-read of the file's own metadata + tensor-info section:

Key	Value
general.architecture	qwen35 (dense hybrid: Gated DeltaNet + softmax attention)
qwen35.block_count	32, full_attention_interval 4 ⇒ 8 full-attention layers (3, 7, 11, …, 31) + 24 DeltaNet layers
attention	16 heads, 4 KV heads, key_length = value_length = 256
qwen35.ssm.*	conv_kernel 4, state_size 128, group_count 16, time_step_rank 32, inner_size 4096
context / vocab	262,144 native / 248,320
tensor types	Q6_K × 250 + F32 × 177 — standard types only, so the stock master build loads it (the ROCmFPX-style custom-quant lock-in does not apply)
MTP	no nextn/mtp tensors exist — in-file MTP is impossible; the card's "MTP support" line describes the base architecture, not this quantized export
chat template	supports enable_thinking, tools/tool_call, image/video content parts; it has no reasoning_effort variable (unused Jinja variables are ignored, so the harness's effort map is inert here, not an error source)

Shape

--ctx-size 401408 --parallel 2 — llama.cpp divides the pool across slots, so each conversation gets n_ctx_slot = 200,704 (Frank's cap; the harness contextWindow quotes the per-slot number). --flash-attn on, --cache-type-k f16 --cache-type-v f16 (exact KV), batch 4096 / ubatch 2048, --log-verbosity 4, --jinja --reasoning auto --reasoning-format deepseek, --mmproj, speculative decoding OFF (SPEC_TYPE=none strips both spec flags — the file has no MTP tensors).

KV rate law. The pool scales with the 8 full-attention layers: 8 × 4 KV heads × 256 dim = 8,192 elements/token per cache, so f16 K + f16 V bills 32,768 B/token = 32 KiB/token → 12.25 GiB at the 401,408 pool. The 24 DeltaNet layers bill a fixed recurrent state per slot, not per token — that is what makes two 200k slots affordable where the 26B MoE fit only one.

Priced from boots, not formulas. MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B, same mmproj class) at 2 × 200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024 and 21,846 / 2,279 at ubatch 2048 — see frank-zealot-llama-mimo-q6k-2x200704. This file is 415 MiB lighter in weights and its projector is 3 MiB heavier, so the projection is ~21,430 used / ~2,700 free.

Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 % prefill at 8.9k depth, decode flat, cost 616 MiB); 1024 is the documented fallback rung if a future shape lands under the 2 GiB floor.

Measured boot (2026-10-07, deploy ec6184d)

nvidia-smi: 21,436 MiB used / 2,689 MiB free — the MiMo-priced projection (~21,430 / ~2,700) was within 11 MiB. Engine ledger (--log-verbosity 4, journalctl -u frank-zealot):

Line	Value
llama_context	n_ctx = 401408, n_ctx_seq = 200704, kv_unified = false
llama_kv_cache	CUDA0 KV buffer size = 12544.00 MiB — size = 12544.00 MiB (200704 cells, 8 layers, 2/2 seqs), K (f16): 6272.00 MiB, V (f16): 6272.00 MiB
sched_reserve	CUDA0 compute buffer size = 1136.06 MiB, CUDA_Host compute buffer size = 848.07 MiB
clip_model_loader (mmproj)	model size: 878.99 MiB + reserve_compute_meta: CUDA0 248.10 MiB, CPU 24.93 MiB
srv load_model	n_slots = 2, n_ctx_slot = 200704
print_info	n_ctx_train = 262144, n_head_kv = 4

The 32 KiB/token law is confirmed exactly: 401,408 × 32 KiB = 12,544 MiB, which is what the engine printed. The n_ctx_seq (200704) < n_ctx_train (262144) line is informational (a context below the trained window is safe; the Gemma4 era's warning was the opposite case, a drafter running 53 % past its window). Weights load in ~6 s from page cache; /health answers {"status":"ok"} and /v1/models serves the alias frank-zealot.

Wire rename (2026-10-09): ALIAS moved frank-secondary → frank-zealot (commit 7c77326; the service-file rename to the frank-zealot.* names followed 2026-10-10, commit 55f87fc), so the served id matches the harness model id and the family-facing name. Consumers re-keyed in the same change: OpenWebUI (model row, access_grant row, openai.api_configs index-2 model_ids) and models.yml (already renamed 2026-10-08). llama.cpp does NOT validate the requested model name — it answers and echoes the served alias — so a stale client keeps working; the picker and any id-keyed grant is what breaks. See OpenWebUI.

The deploy's Playwright gate passed (33 tests) and the fleet e2e set is unchanged.

Client contract

llama.cpp surface: model id frank-zealot, native tools via --jinja, thinking in reasoning_content, vision via OpenAI image_url content parts (probe with a ≥ 256 px image, or the Qwen-VL tower answers "video"). The publisher's recipe ships in models.yml extraBody: thinking mode temperature 0.6 / top_p 0.95 / top_k 20 / min_p 0 (non-thinking mode is 0.7/0.8/20/0). The card's advice "maintain at least 128K context to preserve thinking" is satisfied by the 200,704 slot.

Compaction hub: models.yml routes compactionModel for all three endpoints here — still true 2026-10-18 (~/.omp/agent/models.yml:25,107,185 all name llamacpp (dankclaw-3090)/frank-zealot). With --parallel 2 a compaction request no longer serializes behind a live generation (the Gemma4 era's --parallel 1 law), and the 200,704 slot still fits the worst-case compaction input (~133k tokens).

Rollback

Target	How
Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V	revert the swap commit — payload stays at /home/dank/models/Gemma4-26B-A4B/; see frank-zealot-llama-gemma4-q4km-200704
MiMo-V2.6 Q6_K, 2 × 200,704, f16 KV	MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf + its mmproj — measured 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048
Swift-1.5-27B IQ3_XXS 2 × 114,688 + MTP	payload on disk at /home/dank/models/Swift15-UncIQ3XXS/

When it misbehaves

Same decision tree as the predecessors: No CUDA GPUs are available ⇒ nvidia-modprobe -u -c=0; a 400 naming 200,704 ⇒ the conversation outgrew one slot, not the pool; queue behind two long generations ⇒ the slot ceiling is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in journalctl -u frank-zealot (KV MiB, compute buffers) before touching flags — the engine prints the exact figures the shape needs.

status=203/EXEC boot loop: the rewrite of serve-frank-zealot.sh shipped without its #!/usr/bin/env bash line, and systemd failed the ExecStart before the script ran — 10 s restarts, Mem peak: 1.9M, and nvidia-smi flat at 1 MiB used (the GPU is never touched). Journal signature: Failed at step EXEC spawning /home/dank/…/serve-frank-zealot.sh. A shebang is not decoration on this host: every LLM serve script is a direct ExecStart, so head -1 is part of the deploy check (commit ec6184d).