Frankandomp
1d062f06a3
Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention + 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less, natively multimodal) costs ~7 GiB of weights where the 26B MoE cost ~16.6 GiB. The freed room buys back the second conversation slot and exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K + f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable is impossible), vision via the publisher's BF16 mmproj, ubatch 1024 (the CUDA decode-optimal value), verbosity 4, publisher sampler recipe (0.6/0.95/20/0) in models.yml. Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch 2048. Compaction stays routed here; window math is unchanged (maxTokens 16,384 keeps the worst-case harness request inside the slot). Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the rollback; its runbook is retired in place, keeping the attention_k_eq_v pair-billing law. Co-Authored-By: omp <omp@franksdank.cloud>
114 lines
7.0 KiB
Markdown
114 lines
7.0 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-secondary on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (ACTIVE)"
|
||
description: "The ACTIVE frank-secondary runbook (since 2026-10-07) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
|
||
tags:
|
||
- llm
|
||
- frank-secondary
|
||
- dankclaw
|
||
- llama-cpp
|
||
- gguf
|
||
- rtx3090
|
||
timestamp: 2026-10-07
|
||
---
|
||
|
||
# frank-secondary — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV
|
||
|
||
Live since 2026-10-07, replacing the
|
||
[[frank-secondary-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704
|
||
slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of
|
||
weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second
|
||
conversation slot and unquantized KV back.
|
||
|
||
## Identity
|
||
|
||
| | |
|
||
|---|---|
|
||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface |
|
||
| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`); `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit |
|
||
| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||
| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
|
||
| Weights | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf` — HauhauCS **Q6_K**, 7,359,259,008 B, sha256 `c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986` |
|
||
| Projector | `mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf`, 921,704,448 B, sha256 `05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934` — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability |
|
||
| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, `--verify-only` |
|
||
| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine) |
|
||
|
||
## Payload geometry (read from the GGUF header, not the model card)
|
||
|
||
Range-read of the file's own metadata + tensor-info section:
|
||
|
||
|Key|Value|
|
||
|---|---|
|
||
|`general.architecture`|`qwen35` (dense hybrid: Gated DeltaNet + softmax attention)|
|
||
|`qwen35.block_count`|32, `full_attention_interval` 4 ⇒ **8 full-attention layers (3, 7, 11, …, 31) + 24 DeltaNet layers**|
|
||
|attention|16 heads, **4 KV heads**, `key_length` = `value_length` = 256|
|
||
|`qwen35.ssm.*`|`conv_kernel` 4, `state_size` 128, `group_count` 16, `time_step_rank` 32, `inner_size` 4096|
|
||
|context / vocab|262,144 native / 248,320|
|
||
|tensor types|Q6_K × 250 + F32 × 177 — standard types only, so the stock master build loads it (the ROCmFPX-style custom-quant lock-in does not apply)|
|
||
|MTP|**no `nextn`/`mtp` tensors exist** — in-file MTP is impossible; the card's "MTP support" line describes the base architecture, not this quantized export|
|
||
|chat template|supports `enable_thinking`, `tools`/`tool_call`, image/video content parts; it has **no `reasoning_effort` variable** (unused Jinja variables are ignored, so the harness's effort map is inert here, not an error source)|
|
||
|
||
## Shape
|
||
|
||
`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
|
||
so each conversation gets `n_ctx_slot = 200,704` (Frank's cap; the harness
|
||
`contextWindow` quotes the per-slot number). `--flash-attn on`,
|
||
**`--cache-type-k f16 --cache-type-v f16`** (exact KV), batch 4096 /
|
||
**ubatch 2048**, `--log-verbosity 4`, `--jinja --reasoning auto
|
||
--reasoning-format deepseek`, `--mmproj`, **speculative decoding OFF**
|
||
(`SPEC_TYPE=none` strips both spec flags — the file has no MTP tensors).
|
||
|
||
**KV rate law.** The pool scales with the 8 full-attention layers: 8 × 4 KV
|
||
heads × 256 dim = 8,192 elements/token per cache, so f16 K + f16 V bills
|
||
32,768 B/token = **32 KiB/token** → 12.25 GiB at the 401,408 pool. The 24
|
||
DeltaNet layers bill a fixed recurrent state **per slot**, not per token —
|
||
that is what makes two 200k slots affordable where the 26B MoE fit only one.
|
||
|
||
**Priced from boots, not formulas.** MiMo-V2.6-Distill-Qwen-9B (identical
|
||
qwen35 geometry, 7,793,710,624 B, same mmproj class) at 2 × 200,704 f16 K +
|
||
f16 V measured **21,276 used / 2,849 free** at ubatch 1024 and **21,846 /
|
||
2,279** at ubatch 2048 — see [[frank-secondary-llama-mimo-q6k-2x200704]].
|
||
This file is 415 MiB lighter in weights and its projector is 3 MiB heavier,
|
||
so the projection is ~21,430 used / ~2,700 free.
|
||
|
||
Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 %
|
||
prefill at 8.9k depth, decode flat, cost 616 MiB); `1024` is the documented
|
||
fallback rung if a future shape lands under the 2 GiB floor.
|
||
|
||
## Measured boot (2026-10-07)
|
||
|
||
TO BE FILLED BY THE DEPLOY VERIFICATION.
|
||
|
||
## Client contract
|
||
|
||
llama.cpp surface: model id `frank-secondary`, native tools via `--jinja`,
|
||
thinking in **`reasoning_content`**, vision via OpenAI `image_url` content
|
||
parts (probe with a ≥ 256 px image, or the Qwen-VL tower answers "video").
|
||
The publisher's recipe ships in `models.yml` `extraBody`: thinking mode
|
||
`temperature 0.6 / top_p 0.95 / top_k 20 / min_p 0` (non-thinking mode is
|
||
0.7/0.8/20/0). The card's advice "maintain at least 128K context to preserve
|
||
thinking" is satisfied by the 200,704 slot.
|
||
|
||
**Compaction hub**: `models.yml` routes `compactionModel` for all three
|
||
endpoints here. With `--parallel 2` a compaction request no longer serializes
|
||
behind a live generation (the Gemma4 era's `--parallel 1` law), and the
|
||
200,704 slot still fits the worst-case compaction input (~133k tokens).
|
||
|
||
## Rollback
|
||
|
||
|Target|How|
|
||
|---|---|
|
||
|Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V|revert the swap commit — payload stays at `/home/dank/models/Gemma4-26B-A4B/`; see [[frank-secondary-llama-gemma4-q4km-200704]]|
|
||
|MiMo-V2.6 Q6_K, 2 × 200,704, f16 KV|`MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + its mmproj — measured 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048|
|
||
|Swift-1.5-27B IQ3_XXS 2 × 114,688 + MTP|payload on disk at `/home/dank/models/Swift15-UncIQ3XXS/`|
|
||
|
||
## When it misbehaves
|
||
|
||
Same decision tree as the predecessors: `No CUDA GPUs are available` ⇒
|
||
`nvidia-modprobe -u -c=0`; a 400 naming 200,704 ⇒ the conversation outgrew
|
||
one slot, not the pool; queue behind two long generations ⇒ the slot ceiling
|
||
is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in
|
||
`journalctl -u frank-secondary` (KV MiB, compute buffers) before touching
|
||
flags — the engine prints the exact figures the shape needs.
|