frank-secondary: swap Gemma4-26B-A4B Q4_K_M -> Qwen3.5-9B-Uncensored Q6_K 2x200,704
Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention + 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less, natively multimodal) costs ~7 GiB of weights where the 26B MoE cost ~16.6 GiB. The freed room buys back the second conversation slot and exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K + f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable is impossible), vision via the publisher's BF16 mmproj, ubatch 1024 (the CUDA decode-optimal value), verbosity 4, publisher sampler recipe (0.6/0.95/20/0) in models.yml. Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch 2048. Compaction stays routed here; window math is unchanged (maxTokens 16,384 keeps the worst-case harness request inside the slot). Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the rollback; its runbook is retired in place, keeping the attention_k_eq_v pair-billing law. Co-Authored-By: omp <omp@franksdank.cloud>
This commit is contained in:
Frank
omp
@@ -1,10 +1,11 @@
|
||||
# frank-secondary — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)
|
||||
|
||||
**ACTIVE since 2026-10-07.** RTX 3090, port 7771, root systemd unit
|
||||
`frank-secondary.service`, served by `scripts/serve-frank-secondary.sh` +
|
||||
`scripts/frank-secondary.env`. Supersedes
|
||||
[[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] (Swift-1.5-Qwen3.8-27B
|
||||
RCO-IQ3_XXS-mtp, 2×114,688, in-file MTP).
|
||||
**RETIRED 2026-10-07** — same day it went live. Superseded by
|
||||
[[frank-secondary-llama-qwen35-9b-q6k-2x200704]] (Qwen3.5-9B Q6_K at
|
||||
2 × 200,704 with exact KV); the payload stays at
|
||||
`/home/dank/models/Gemma4-26B-A4B/` as the rollback. This page keeps the
|
||||
Gemma4 KV-billing law (the `attention_k_eq_v` pair-billing measurement) and
|
||||
the drafter-cost boots, both of which outlive the model.
|
||||
|
||||
## Payload
|
||||
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (ACTIVE)"
|
||||
description: "The ACTIVE frank-secondary runbook (since 2026-10-07) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
- rtx3090
|
||||
timestamp: 2026-10-07
|
||||
---
|
||||
|
||||
# frank-secondary — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV
|
||||
|
||||
Live since 2026-10-07, replacing the
|
||||
[[frank-secondary-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704
|
||||
slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of
|
||||
weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second
|
||||
conversation slot and unquantized KV back.
|
||||
|
||||
## Identity
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary`, `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`); `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit |
|
||||
| Unit | root systemd `frank-secondary.service` ([`scripts/frank-secondary.service`](../../scripts/frank-secondary.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-secondary.sh`](../../scripts/serve-frank-secondary.sh), tunables [`scripts/frank-secondary.env`](../../scripts/frank-secondary.env) |
|
||||
| Weights | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf` — HauhauCS **Q6_K**, 7,359,259,008 B, sha256 `c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986` |
|
||||
| Projector | `mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf`, 921,704,448 B, sha256 `05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934` — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability |
|
||||
| Fetch | [`scripts/download-frank-secondary.sh`](../../scripts/download-frank-secondary.sh) — `-f`, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, `--verify-only` |
|
||||
| Seam | `tests/frank_secondary_unit.bats` (argv assertions against a stub engine) |
|
||||
|
||||
## Payload geometry (read from the GGUF header, not the model card)
|
||||
|
||||
Range-read of the file's own metadata + tensor-info section:
|
||||
|
||||
|Key|Value|
|
||||
|---|---|
|
||||
|`general.architecture`|`qwen35` (dense hybrid: Gated DeltaNet + softmax attention)|
|
||||
|`qwen35.block_count`|32, `full_attention_interval` 4 ⇒ **8 full-attention layers (3, 7, 11, …, 31) + 24 DeltaNet layers**|
|
||||
|attention|16 heads, **4 KV heads**, `key_length` = `value_length` = 256|
|
||||
|`qwen35.ssm.*`|`conv_kernel` 4, `state_size` 128, `group_count` 16, `time_step_rank` 32, `inner_size` 4096|
|
||||
|context / vocab|262,144 native / 248,320|
|
||||
|tensor types|Q6_K × 250 + F32 × 177 — standard types only, so the stock master build loads it (the ROCmFPX-style custom-quant lock-in does not apply)|
|
||||
|MTP|**no `nextn`/`mtp` tensors exist** — in-file MTP is impossible; the card's "MTP support" line describes the base architecture, not this quantized export|
|
||||
|chat template|supports `enable_thinking`, `tools`/`tool_call`, image/video content parts; it has **no `reasoning_effort` variable** (unused Jinja variables are ignored, so the harness's effort map is inert here, not an error source)|
|
||||
|
||||
## Shape
|
||||
|
||||
`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
|
||||
so each conversation gets `n_ctx_slot = 200,704` (Frank's cap; the harness
|
||||
`contextWindow` quotes the per-slot number). `--flash-attn on`,
|
||||
**`--cache-type-k f16 --cache-type-v f16`** (exact KV), batch 4096 /
|
||||
**ubatch 2048**, `--log-verbosity 4`, `--jinja --reasoning auto
|
||||
--reasoning-format deepseek`, `--mmproj`, **speculative decoding OFF**
|
||||
(`SPEC_TYPE=none` strips both spec flags — the file has no MTP tensors).
|
||||
|
||||
**KV rate law.** The pool scales with the 8 full-attention layers: 8 × 4 KV
|
||||
heads × 256 dim = 8,192 elements/token per cache, so f16 K + f16 V bills
|
||||
32,768 B/token = **32 KiB/token** → 12.25 GiB at the 401,408 pool. The 24
|
||||
DeltaNet layers bill a fixed recurrent state **per slot**, not per token —
|
||||
that is what makes two 200k slots affordable where the 26B MoE fit only one.
|
||||
|
||||
**Priced from boots, not formulas.** MiMo-V2.6-Distill-Qwen-9B (identical
|
||||
qwen35 geometry, 7,793,710,624 B, same mmproj class) at 2 × 200,704 f16 K +
|
||||
f16 V measured **21,276 used / 2,849 free** at ubatch 1024 and **21,846 /
|
||||
2,279** at ubatch 2048 — see [[frank-secondary-llama-mimo-q6k-2x200704]].
|
||||
This file is 415 MiB lighter in weights and its projector is 3 MiB heavier,
|
||||
so the projection is ~21,430 used / ~2,700 free.
|
||||
|
||||
Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 %
|
||||
prefill at 8.9k depth, decode flat, cost 616 MiB); `1024` is the documented
|
||||
fallback rung if a future shape lands under the 2 GiB floor.
|
||||
|
||||
## Measured boot (2026-10-07)
|
||||
|
||||
TO BE FILLED BY THE DEPLOY VERIFICATION.
|
||||
|
||||
## Client contract
|
||||
|
||||
llama.cpp surface: model id `frank-secondary`, native tools via `--jinja`,
|
||||
thinking in **`reasoning_content`**, vision via OpenAI `image_url` content
|
||||
parts (probe with a ≥ 256 px image, or the Qwen-VL tower answers "video").
|
||||
The publisher's recipe ships in `models.yml` `extraBody`: thinking mode
|
||||
`temperature 0.6 / top_p 0.95 / top_k 20 / min_p 0` (non-thinking mode is
|
||||
0.7/0.8/20/0). The card's advice "maintain at least 128K context to preserve
|
||||
thinking" is satisfied by the 200,704 slot.
|
||||
|
||||
**Compaction hub**: `models.yml` routes `compactionModel` for all three
|
||||
endpoints here. With `--parallel 2` a compaction request no longer serializes
|
||||
behind a live generation (the Gemma4 era's `--parallel 1` law), and the
|
||||
200,704 slot still fits the worst-case compaction input (~133k tokens).
|
||||
|
||||
## Rollback
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V|revert the swap commit — payload stays at `/home/dank/models/Gemma4-26B-A4B/`; see [[frank-secondary-llama-gemma4-q4km-200704]]|
|
||||
|MiMo-V2.6 Q6_K, 2 × 200,704, f16 KV|`MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + its mmproj — measured 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048|
|
||||
|Swift-1.5-27B IQ3_XXS 2 × 114,688 + MTP|payload on disk at `/home/dank/models/Swift15-UncIQ3XXS/`|
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
Same decision tree as the predecessors: `No CUDA GPUs are available` ⇒
|
||||
`nvidia-modprobe -u -c=0`; a 400 naming 200,704 ⇒ the conversation outgrew
|
||||
one slot, not the pool; queue behind two long generations ⇒ the slot ceiling
|
||||
is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in
|
||||
`journalctl -u frank-secondary` (KV MiB, compute buffers) before touching
|
||||
flags — the engine prints the exact figures the shape needs.
|
||||
@@ -102,13 +102,20 @@ frank-secondary, port 7771]
|
||||
(`slow`/`advisor` → frank-primary) carried forward. Superseded by the
|
||||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||||
engine moved with the model
|
||||
- [[frank-secondary-llama-gemma4-q4km-200704]] — **ACTIVE since 2026-10-07**
|
||||
- [[frank-secondary-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07**
|
||||
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP **Q4_K_M**
|
||||
on the RTX 3090 — ONE 200,704-token conversation, q8_0 K + q8_0 V,
|
||||
vision via the publisher's mmproj. MTP is OFF (the gemma4-assistant
|
||||
drafter costs ~1.9 GiB: 23,806 used/319 free with it, 21,890/2,235
|
||||
without). Supersedes the Swift-1.5 era (below).
|
||||
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090
|
||||
— TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), exact
|
||||
f16 K + f16 V, vision via the publisher's BF16 mmproj. MTP is OFF because
|
||||
the file has no MTP tensors at all. Priced from the MiMo Q6_K boots (same
|
||||
qwen35 geometry, same card, same shape). Supersedes the Gemma4 era (below).
|
||||
- [[frank-secondary-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
|
||||
day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
|
||||
master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
|
||||
HauhauCS-Balanced-MTP **Q4_K_M** on the RTX 3090 — ONE 200,704-token
|
||||
conversation, f16 K + q8_0 V, vision via the publisher's mmproj. MTP was
|
||||
OFF (the gemma4-assistant drafter costs ~1.9 GiB: 23,806 used/319 free with
|
||||
it, 21,890/2,235 without).
|
||||
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
|
||||
2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback);
|
||||
ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with
|
||||
|
||||
@@ -372,10 +372,16 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-secondary-llama-gemma4-q4km-200704]] — the **ACTIVE** frank-secondary
|
||||
runbook since 2026-10-07: llama.cpp master (CUDA sm_86) serving
|
||||
- [[frank-secondary-llama-qwen35-9b-q6k-2x200704]] — the **ACTIVE** frank-secondary
|
||||
runbook since 2026-10-07 (late): llama.cpp master (CUDA sm_86) serving
|
||||
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090 —
|
||||
TWO 200,704-token conversations, exact f16 K + f16 V, MTP **impossible**
|
||||
(the file carries no `nextn`/`mtp` tensors), vision through the publisher's
|
||||
BF16 mmproj; priced from the MiMo Q6_K boots (same qwen35 geometry)
|
||||
- [[frank-secondary-llama-gemma4-q4km-200704]] — RETIRED 2026-10-07 (same day;
|
||||
payload on disk as rollback): llama.cpp master (CUDA sm_86) serving
|
||||
HauhauCS Gemma4-26B-A4B QAT Uncensored **Q4_K_M** on the RTX 3090 — ONE
|
||||
200,704-token conversation, q8_0 K + q8_0 V, MTP **disabled** (the
|
||||
200,704-token conversation, f16 K + q8_0 V, MTP **disabled** (the
|
||||
`gemma4-assistant` drafter costs ~1.9 GiB: boot was 23,806 used/319 free
|
||||
with it, 21,890/2,235 without), vision through the publisher's mmproj
|
||||
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — RETIRED 2026-10-07
|
||||
|
||||
Reference in New Issue
Block a user