Frank
e7f0d93c86
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
168 lines
11 KiB
Markdown
168 lines
11 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (RETIRED, rollback target)"
|
||
description: "The frank-zealot runbook of 2026-10-07 → 2026-10-17 (RETIRED — rollback target) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
|
||
tags:
|
||
- llm
|
||
- frank-zealot
|
||
- dankclaw
|
||
- llama-cpp
|
||
- gguf
|
||
- rtx3090
|
||
timestamp: 2026-10-07
|
||
---
|
||
|
||
# frank-zealot — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV
|
||
|
||
**RETIRED 2026-10-07 → 2026-10-17** (AGENTS.md Runbook Index), superseded by
|
||
[[frank-zealot-llama-prism-bonsai2-pq20]] (27B PQ2_0, 2 × 200,704 q8_0 KV, build
|
||
10754), itself retired 2026-10-17 → 10-18 and now the rollback target: :7771
|
||
today is llama.cpp PrismML build **10777** on
|
||
`Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 3 × 262,144 q8_0 KV, MTP off by
|
||
measurement (AGENTS.md Runbook Index active row; `scripts/frank-zealot.env`).
|
||
Transient 2026-10-18: :7771 is inside a bench window — the unit is `failed`
|
||
since 03:14 and a hand-booted process answers the port; restore with
|
||
`./scripts/deploy.sh dankclaw --service frank-zealot`.
|
||
This era is the rollback target: the payload is still on disk
|
||
(`stat -c %s /home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf`
|
||
→ 7359259008, read 2026-10-18) with its master CUDA build. Same census: the
|
||
10754 rollback payload `Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf` is
|
||
present at 7206168928 with its Q8_0 mmproj at 629246976, and
|
||
`Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` at 2675410720 —
|
||
nothing this era or its successors need has been purged. It replaced the
|
||
[[frank-zealot-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704
|
||
slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of
|
||
weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second
|
||
conversation slot and unquantized KV back.
|
||
|
||
## Identity
|
||
|
||
| | |
|
||
|---|---|
|
||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS` in the env; the SERVICE keeps the `frank-zealot` name), `/v1` OpenAI surface |
|
||
| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` — `--version` on the disk build reads `0.5.0-dev (build 1, commit 5fc4f3c)` (built 2026-10-01); its `CMakeCache.txt` says `GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF` and that tree carries no CUDA-side guard for the option (`ggml/src/ggml-cuda/fattn.cu:438-449` dispatches Q4/Q5 KV unconditionally), so quantized KV never required it; `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit |
|
||
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
|
||
| Weights | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf` — HauhauCS **Q6_K**, 7,359,259,008 B, sha256 `c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986` |
|
||
| Projector | `mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf`, 921,704,448 B, sha256 `05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934` — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability |
|
||
| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, `--verify-only` |
|
||
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine) |
|
||
|
||
## Payload geometry (read from the GGUF header, not the model card)
|
||
|
||
Range-read of the file's own metadata + tensor-info section:
|
||
|
||
|Key|Value|
|
||
|---|---|
|
||
|`general.architecture`|`qwen35` (dense hybrid: Gated DeltaNet + softmax attention)|
|
||
|`qwen35.block_count`|32, `full_attention_interval` 4 ⇒ **8 full-attention layers (3, 7, 11, …, 31) + 24 DeltaNet layers**|
|
||
|attention|16 heads, **4 KV heads**, `key_length` = `value_length` = 256|
|
||
|`qwen35.ssm.*`|`conv_kernel` 4, `state_size` 128, `group_count` 16, `time_step_rank` 32, `inner_size` 4096|
|
||
|context / vocab|262,144 native / 248,320|
|
||
|tensor types|Q6_K × 250 + F32 × 177 — standard types only, so the stock master build loads it (the ROCmFPX-style custom-quant lock-in does not apply)|
|
||
|MTP|**no `nextn`/`mtp` tensors exist** — in-file MTP is impossible; the card's "MTP support" line describes the base architecture, not this quantized export|
|
||
|chat template|supports `enable_thinking`, `tools`/`tool_call`, image/video content parts; it has **no `reasoning_effort` variable** (unused Jinja variables are ignored, so the harness's effort map is inert here, not an error source)|
|
||
|
||
## Shape
|
||
|
||
`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
|
||
so each conversation gets `n_ctx_slot = 200,704` (Frank's cap; the harness
|
||
`contextWindow` quotes the per-slot number). `--flash-attn on`,
|
||
**`--cache-type-k f16 --cache-type-v f16`** (exact KV), batch 4096 /
|
||
**ubatch 2048**, `--log-verbosity 4`, `--jinja --reasoning auto
|
||
--reasoning-format deepseek`, `--mmproj`, **speculative decoding OFF**
|
||
(`SPEC_TYPE=none` strips both spec flags — the file has no MTP tensors).
|
||
|
||
**KV rate law.** The pool scales with the 8 full-attention layers: 8 × 4 KV
|
||
heads × 256 dim = 8,192 elements/token per cache, so f16 K + f16 V bills
|
||
32,768 B/token = **32 KiB/token** → 12.25 GiB at the 401,408 pool. The 24
|
||
DeltaNet layers bill a fixed recurrent state **per slot**, not per token —
|
||
that is what makes two 200k slots affordable where the 26B MoE fit only one.
|
||
|
||
**Priced from boots, not formulas.** MiMo-V2.6-Distill-Qwen-9B (identical
|
||
qwen35 geometry, 7,793,710,624 B, same mmproj class) at 2 × 200,704 f16 K +
|
||
f16 V measured **21,276 used / 2,849 free** at ubatch 1024 and **21,846 /
|
||
2,279** at ubatch 2048 — see [[frank-zealot-llama-mimo-q6k-2x200704]].
|
||
This file is 415 MiB lighter in weights and its projector is 3 MiB heavier,
|
||
so the projection is ~21,430 used / ~2,700 free.
|
||
|
||
Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 %
|
||
prefill at 8.9k depth, decode flat, cost 616 MiB); `1024` is the documented
|
||
fallback rung if a future shape lands under the 2 GiB floor.
|
||
|
||
## Measured boot (2026-10-07, deploy `ec6184d`)
|
||
|
||
`nvidia-smi`: **21,436 MiB used / 2,689 MiB free** — the MiMo-priced
|
||
projection (~21,430 / ~2,700) was within 11 MiB. Engine ledger
|
||
(`--log-verbosity 4`, `journalctl -u frank-zealot`):
|
||
|
||
|Line|Value|
|
||
|---|---|
|
||
|`llama_context`|`n_ctx = 401408`, `n_ctx_seq = 200704`, `kv_unified = false`|
|
||
|`llama_kv_cache`|`CUDA0 KV buffer size = 12544.00 MiB` — `size = 12544.00 MiB (200704 cells, 8 layers, 2/2 seqs), K (f16): 6272.00 MiB, V (f16): 6272.00 MiB`|
|
||
|`sched_reserve`|`CUDA0 compute buffer size = 1136.06 MiB`, `CUDA_Host compute buffer size = 848.07 MiB`|
|
||
|`clip_model_loader` (mmproj)|`model size: 878.99 MiB` + `reserve_compute_meta: CUDA0 248.10 MiB, CPU 24.93 MiB`|
|
||
|`srv load_model`|`n_slots = 2, n_ctx_slot = 200704`|
|
||
|`print_info`|`n_ctx_train = 262144`, `n_head_kv = 4`|
|
||
|
||
The `32 KiB/token` law is confirmed exactly: 401,408 × 32 KiB = 12,544 MiB,
|
||
which is what the engine printed. The `n_ctx_seq (200704) < n_ctx_train
|
||
(262144)` line is informational (a context **below** the trained window is
|
||
safe; the Gemma4 era's warning was the opposite case, a drafter running 53 %
|
||
**past** its window). Weights load in ~6 s from page cache; `/health` answers
|
||
`{"status":"ok"}` and `/v1/models` serves the alias `frank-zealot`.
|
||
|
||
**Wire rename (2026-10-09):** `ALIAS` moved `frank-secondary` → `frank-zealot`
|
||
(commit `7c77326`; the service-file rename to the `frank-zealot.*` names
|
||
followed 2026-10-10, commit `55f87fc`),
|
||
so the served id matches the harness model id and the family-facing name.
|
||
Consumers re-keyed in the same change: OpenWebUI (`model` row, `access_grant`
|
||
row, `openai.api_configs` index-2 `model_ids`) and `models.yml` (already
|
||
renamed 2026-10-08). llama.cpp does NOT validate the requested model name —
|
||
it answers and echoes the served alias — so a stale client keeps working;
|
||
the picker and any id-keyed grant is what breaks. See [[OpenWebUI]].
|
||
|
||
The deploy's Playwright gate passed (33 tests) and the fleet e2e set is
|
||
unchanged.
|
||
|
||
## Client contract
|
||
|
||
llama.cpp surface: model id `frank-zealot`, native tools via `--jinja`,
|
||
thinking in **`reasoning_content`**, vision via OpenAI `image_url` content
|
||
parts (probe with a ≥ 256 px image, or the Qwen-VL tower answers "video").
|
||
The publisher's recipe ships in `models.yml` `extraBody`: thinking mode
|
||
`temperature 0.6 / top_p 0.95 / top_k 20 / min_p 0` (non-thinking mode is
|
||
0.7/0.8/20/0). The card's advice "maintain at least 128K context to preserve
|
||
thinking" is satisfied by the 200,704 slot.
|
||
|
||
**Compaction hub**: `models.yml` routes `compactionModel` for all three
|
||
endpoints here — still true 2026-10-18 (`~/.omp/agent/models.yml:25,107,185` all name `llamacpp (dankclaw-3090)/frank-zealot`). With `--parallel 2` a compaction request no longer serializes
|
||
behind a live generation (the Gemma4 era's `--parallel 1` law), and the
|
||
200,704 slot still fits the worst-case compaction input (~133k tokens).
|
||
|
||
## Rollback
|
||
|
||
|Target|How|
|
||
|---|---|
|
||
|Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V|revert the swap commit — payload stays at `/home/dank/models/Gemma4-26B-A4B/`; see [[frank-zealot-llama-gemma4-q4km-200704]]|
|
||
|MiMo-V2.6 Q6_K, 2 × 200,704, f16 KV|`MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + its mmproj — measured 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048|
|
||
|Swift-1.5-27B IQ3_XXS 2 × 114,688 + MTP|payload on disk at `/home/dank/models/Swift15-UncIQ3XXS/`|
|
||
|
||
## When it misbehaves
|
||
|
||
Same decision tree as the predecessors: `No CUDA GPUs are available` ⇒
|
||
`nvidia-modprobe -u -c=0`; a 400 naming 200,704 ⇒ the conversation outgrew
|
||
one slot, not the pool; queue behind two long generations ⇒ the slot ceiling
|
||
is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in
|
||
`journalctl -u frank-zealot` (KV MiB, compute buffers) before touching
|
||
flags — the engine prints the exact figures the shape needs.
|
||
|
||
**`status=203/EXEC` boot loop**: the rewrite of `serve-frank-zealot.sh`
|
||
shipped without its `#!/usr/bin/env bash` line, and systemd failed the
|
||
`ExecStart` before the script ran — 10 s restarts, `Mem peak: 1.9M`, and
|
||
`nvidia-smi` flat at 1 MiB used (the GPU is never touched). Journal signature:
|
||
`Failed at step EXEC spawning /home/dank/…/serve-frank-zealot.sh`. A
|
||
shebang is not decoration on this host: every LLM serve script is a direct
|
||
`ExecStart`, so `head -1` is part of the deploy check (commit `ec6184d`).
|