dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(frank-zealot): re-key the exact-KV shape across the vault

Browse Source 
AGENTS.md: topology row, the frank-zealot runbook section (shape, KV
accounting, MTP verdict, compaction-hub concurrency), and two Key
Lessons bullets -- the draft-context bill restated with the real knob
names, plus a new lesson that a quant's scale overhead belongs in its
byte rate (the 6.25 % invented by doubling 13,056.00 MiB).

LLM/frank-zealot-llama-prism-bonsai9-mtp.md: retitled to the live shape;
the 3-slot q8_0 boot ledger and its measurements are relabelled as the
arm this replaced (byte-identical numbers, honest attribution), the
corrected f16 rate derivation is added, the MTP census is labelled
q8_0-era, the fallback rungs are re-derived for the 524,288 pool, and the
boot-ledger section carries an explicit PROJECTION that must be replaced
with the measured ledger from the first f16 boot.

Pointers re-pointed (live-shape sentences only; every historical ledger
left byte-identical): LLM/index.md, vault index.md, Dankclaw/OpenWebUI.md,
and the retired-runbook "today" pointers in mimo-q5km, mimo-q6k,
qwen35-9b-q6k, swift15, sglang-mimo-v26. log.md gains the dated entry.
This commit is contained in:
Frank committed 2026-10-18 21:02:42 -07:00
1 parent 68afc6dbaf
commit 71bc1735d3
11 files changed
+300 -137
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md
+126 -50
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 3 × 262144, q8_0 KV (ACTIVE)"
description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: THREE conversations of 262,144 tokens (every slot at the model's trained window) over q8_0 K + q8_0 V, vision through the publisher's Q8_0 Qwen3-VL projector, and the five-arm boot census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head."
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 2 × 262144, exact f16 KV (ACTIVE)"
description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: TWO conversations of 262,144 tokens (every slot at the model's trained window) over an UNQUANTIZED f16 K + V cache, vision through the publisher's Q8_0 Qwen3-VL projector, and the boot-arm census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head."
tags:
- llm
- frank-zealot
@@ -14,15 +14,22 @@ tags:
timestamp: 2026-10-18
---
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 3 × 262,144, q8_0 KV
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 2 × 262,144, exact f16 KV
Live since 2026-10-18, replacing the
[[frank-zealot-llama-prism-bonsai2-pq20]] era (Ternary-Bonsai-2-**27B** PQ2_0 at
2 × 200,704). The trade in one sentence: the 9B bills **half** the KV per token
of the 27B, and that is exactly what funds **three** conversations at the
model's **native 262,144** window where the 27B afforded only two shortened
ones (Frank's ask: *"if you can possibly fit a third context, do it"*). Every
axis improved — decode, prefill, three-stream aggregate, and free VRAM.
2 × 200,704). The original trade: the 9B bills **half** the KV per token of the
27B, and that is what funded **three** conversations at the model's **native
262,144** window where the 27B afforded only two shortened ones (Frank's ask:
*"if you can possibly fit a third context, do it"*).
The shape moved again the same evening (Frank: *"shift frank-zealot to 2
parallels of unquantized kv"*). The KV cache went **exact** — f16 K + f16 V, no
quantization error anywhere in the attention state — and the **slot count paid
for it: 3 → 2**. The per-slot contract never moved; every conversation is still
262,144, still the trained window. What the dtype bought is fidelity, and what
it costs is one concurrent conversation plus the bytes attention reads per
generated token.
## Payload identity
@@ -46,9 +53,12 @@ axis improved — decode, prefill, three-stream aggregate, and free VRAM.
refuses the custom type at load; a loader that ignored the Hadamard metadata
would dequant **wrong**, silently. Same lock class as the ROCmFPX era.
2. **The build floor.** The fork's card gates **MTP beside a quantized V cache**
on `prism-b10776-d00e252` or newer, so the prebuilt moved **10754 → 10777**
for this swap. The 10754 tree stays on disk (`~/GitPackages/llama-prism/`) as
the 27B rollback runtime. `tests/frank_zealot_unit.bats` pins both paths.
on `prism-b10776-d00e252` or newer, which is why the prebuilt moved
**10754 → 10777** for the swap. Exact f16 KV makes that gate moot — there is
no quantized V to pair with the drafter — but tumbler 1 pins the runtime on
its own: stock llama.cpp cannot load this file at any flag value. The 10754
tree stays on disk (`~/GitPackages/llama-prism/`) as the 27B rollback
runtime. `tests/frank_zealot_unit.bats` pins both paths.
## Header census (read from the GGUF, not the card)
@@ -71,41 +81,66 @@ axis improved — decode, prefill, three-stream aggregate, and free VRAM.
| `general.sampling.*` | temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / repeat 1.0 | the card recipe; llama.cpp reads it itself |
**KV accounting.** Only the 8 full-attention layers scale with context:
8 × 4 KV heads × 256 dim = **8,192 elements/token per cache**, so
**q8_0 K + q8_0 V bills 17,408 B/token** — the 786,432-position pool is
**13,056.00 MiB**, ledger-exact. That is exactly HALF the 27B's 34,816 B/token.
**f16 KV would bill 26,112 MiB against a 24,123 MiB card** and dies at
`cudaMalloc`. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
instead: the PRODUCTION ledger prints `RS buffer size = 150.75 MiB (3 cells,
32 layers, 3 seqs)` ≈ 50 MiB a cell, while the SPEC-ARMED arms printed
452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples
even the recurrent state. Pool arithmetic alone under-prices the shape.
8 × 4 KV heads × 256 dim = **8,192 elements/token per cache** (the engine prints
`n_embd_k_gqa = n_embd_v_gqa = 1024`). **The deployed pair is exact: f16 K +
f16 V bills 32,768 B/token** (2 B/element × 8,192 elements × 2 caches), so the
524,288-position pool is **16,384 MiB**. The q8_0 pair this endpoint shipped
first billed **17,408 B/token** — 1.0625 B/element, 32 codes plus one 2-byte
scale per 32-block — the **13,056.00 MiB** its ledger printed at the 786,432
pool, exactly HALF the 27B's 34,816 B/token.
**Corrected arithmetic, and it matters.** The note that lived here and in
`frank-zealot.env` — "f16 would bill 26,112 MiB" — doubled the q8_0 figure
*including* its scale overhead, overstating exact KV by 6.25 %. Exact f16 at the
786,432 pool is **24,576 MiB** against the **24,123 MiB** the driver leaves
addressable (`nvidia-smi`'s total column reads 24,576; used + free reads 24,123
— the difference is driver reservation). The wall stands in both readings; the
honest rate is what makes the fit calculable instead of superstitious. **Halving
the POOL is what funds exact KV** — the window is already at the native ceiling
and there is no YaRN in this path.
The 24 DeltaNet layers bill a fixed recurrent state per SLOT instead: the 3-slot
ledger printed `RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)` ≈
**50.25 MiB a cell**, so two slots bill 100.50. The SPEC-ARMED arms printed
452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples even
the recurrent state. Pool arithmetic alone under-prices the shape.
## The shape (`scripts/frank-zealot.env`)
`--ctx-size 786432 --parallel 3` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`,
`--cache-type-k q8_0 --cache-type-v q8_0`, `--batch-size 4096`,
`--ctx-size 524288 --parallel 2` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`,
`--cache-type-k f16 --cache-type-v f16`, `--batch-size 4096`,
`--ubatch-size 2048`, `--n-gpu-layers 99`, `--jinja`, `--reasoning auto`,
`--reasoning-format deepseek`, `--metrics`, `--mmproj …`, `--log-verbosity 4`,
`--presence-penalty 1.0`,
**`SPEC_TYPE=none`** (which strips `--spec-type` AND every `--spec-draft-*`).
Per-slot window is the **contract** the harness quotes: `models.yml` carries
`contextWindow: 262144`. llama.cpp refuses a prompt longer than the SLOT with a
400 naming the limit — it never truncates — and since this swap the slot window
equals the native window, so no YaRN and no cap is in play.
`contextWindow: 262144`, and it did **not** move with the slot count because
llama.cpp divides `--ctx-size` across slots. It refuses a prompt longer than the
SLOT with a 400 naming the limit — never a truncation — and since this swap the
slot window equals the native window, so no YaRN and no cap is in play.
`tests/frank_zealot_unit.bats` now gates the arithmetic: the shipped dtype ×
pool plus the measured non-KV ledger terms must fit 24,123 MiB with ≥1 GiB of
slack, and the gate is proven to REFUSE the impossible arm (exact KV at the old
786,432 pool = 24,576 MiB of cache alone).
## The MTP verdict: the head is in the file and it is still OFF
The drafter is **good**: acceptance 0.798 / 0.617 per position, mean accepted
length 2.41, and the card recommends depth 2. It loses **at this shape**
because the draft context bills its own **f16** KV over the WHOLE pool —
`blk.32` carries real attention (4 KV heads × 256 × 2 caches × 2 B =
**4,096 B/position**) × 786,432 positions = **3,072.00 MiB**, 4,830 MiB with its
compute buffers — and `--ubatch-size 2048` refuses to give that up.
length 2.41, and the card recommends depth 2. It is OFF because the draft
context bills its own **f16** KV over the WHOLE pool — `blk.32` carries real
attention (4 KV heads × 256 × 2 caches × 2 B = **4,096 B/position** at the
default dtype). The draft dtype is itself a knob — `--spec-draft-type-k` /
`--spec-draft-type-v`, both default f16, verified against this build's `--help`
(so the "no flag exists" claim that lived here is FALSE) — and quantizing the
draft to q8_0 would bill 2,176 B/position, **1,088 MiB** at the shipped pool.
**No measured arm runs a drafter beside exact KV**; every row below is f16 draft.
Booted arms, same card, same traffic, 2026-10-18 (`/completion` `timings`,
greedy 512 tokens; never the first request after a boot):
The census below is the **q8_0 era**: every row is q8_0 K + q8_0 V at the
786,432 pool, where the draft bill was 3,072.00 MiB and 4,830 MiB with its
compute buffers. Booted arms, same card, same traffic, 2026-10-18
(`/completion` `timings`, greedy 512 tokens; never the first request after a
boot):
| slots | spec | ubatch | used / free | decode | 3-stream | prefill @106k |
|---|---|---|---|---|---|---|
@@ -117,15 +152,26 @@ greedy 512 tokens; never the first request after a boot):
**MTP is worth +28 % single-stream decode whenever the card has room** (rows
4–5 prove the drafter itself is a win; row 2 proves the squeeze is what halves
it). The arm that runs MTP at three windows is `SPEC_TYPE=draft-mtp` +
`UBATCH_SIZE=1024` (row 3) and pays **prefill −17 %** and **2,725 MiB of
it). The arm that ran it at three windows was `SPEC_TYPE=draft-mtp` +
`UBATCH_SIZE=1024` (row 3), paying **prefill −17 %** and **2,725 MiB of
headroom** on a card whose mmproj worst-case lands AFTER the engine's own fit
check. This endpoint is the fleet's **compaction hub** — prefill-bound ~170k
inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 %
prefill and +12 % three-stream aggregate over single-stream speed, and keeps
6.2 GiB free. Frank confirmed this arm on 2026-10-18.
check.
## Boot ledger (production boot 2026-10-18 13:19, under the unit, verbosity 4)
**Under exact KV an f16-draft arm is priced out of this card.** The draft
context is built at the TARGET's pool, so at 524,288 positions it bills
4,096 × 524,288 = **2,048.00 MiB** plus its own buffers, on a trunk that already
projects **~21,200 MiB** of the 24,123 MiB addressable. The doors back to
speculation are **one window per conversation** (row 5's class of shape), **KV
back to q8_0** with `UBATCH_SIZE=1024`, or the **unmeasured**
`--spec-draft-type-k q8_0 --spec-draft-type-v q8_0` arm — 1,088 MiB of draft,
legal on b10777, never booted beside exact KV. Frank chose exact KV over the
drafter on 2026-10-18, and the endpoint's actual job is what made that
affordable: the fleet's **compaction hub** is prefill-bound on ~170k inputs,
prefill is weight-bandwidth bound and does not re-read the cache per generated
token — so the dtype is paid in decode and concurrency, not in the work this
service exists to do.
## Boot ledger — the q8_0 3-slot production boot (2026-10-18 13:19, under the unit, verbosity 4)
```
CUDA0 model buffer size = 2036.71 MiB # + 257.66 MiB CPU_Mapped (mmap)
@@ -141,7 +187,16 @@ No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class
numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`,
never the projection**.
## Measured (production boot, `/completion` `timings`, greedy, `ignore_eos`)
**The exact-KV arm's projection, recorded before its own boot.** Every term of
that ledger survives the reslice except the KV pool (13,056 → 16,384 MiB) and
one recurrent-state cell (−50.25 MiB), so the pre-flight prediction is
`17,922 − 13,056 + 16,384 − 50 ≈ 21,200 MiB used / ~2,920 free`, with ~2 GiB
still standing through the mmproj worst case. **Replace this paragraph with the
MEASURED ledger and settled `nvidia-smi` from the first f16 boot** — a
projection is a pre-flight check, not a record, and the whole reason the shape
changed dtype is that projections drift from ledgers.
## Measured — the q8_0 3-slot boot (`/completion` `timings`, greedy, `ignore_eos`)
- **168.7 t/s** decode — three runs of 512 tokens: 168.4 / 168.7 / 168.7
(the trial arm printed 167.2; same shape, same noise band).
@@ -156,6 +211,13 @@ never the projection**.
stops on its own: a 24-token countdown measured 72 t/s across three streams
while the same shape with `ignore_eos` gave 300.
**These are the numbers the exact-KV arm is judged against.** Exact KV makes
attention read 1.882× the bytes per generated token, so a decode drop is
EXPECTED and has to be measured, not assumed; prefill should hold. Re-run the
same instrument on the first f16 boot — greedy, `ignore_eos`, 512 tokens, the
second request or later, and the two-stream aggregate in place of the
three-stream one — and record it here.
## Contract traps
- **`--metrics` is mandatory.** Without it llama.cpp answers **501** on
@@ -164,7 +226,7 @@ never the projection**.
(`up{job="frank-zealot"}` oldest retained sample, 2026-10-11, is already 0).
- **Q8_0 mmproj, not BF16.** The publisher's card: the BF16 tower "can crash
llama-server when several image requests run at once; Q8_0 is not affected" —
and `--parallel 3` is precisely that shape.
and `--parallel 2` is still precisely that shape.
- **Only the `image_url` content part reaches the model.** The legacy top-level
`images` field bumps `usage.prompt_tokens` and the model still invents an
answer. Probe with a solid color and expect the exact word.
@@ -216,9 +278,11 @@ never the projection**.
`frank-zealot` here — 262,144 is the widest window in the fleet and the only one
that comfortably fits a worst-case compaction input (~170k tokens; the clamp is
`min(maxTokens 16384, contextWindow − est − 4096)`, compaction fires at 85 % of
the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With `--parallel 3`
a compaction request no longer serializes behind a live generation (the Gemma4
era's `--parallel 1` law).
the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). What the reslice to
**two** slots does cost is here: a compaction request still runs beside ONE live
generation, but with two conversations already busy a third waits — the Gemma4
era's `--parallel 1` serialization law at half strength. The window itself is
untouched, and the window is the term the compaction contract depends on.
## Deploy ledger (2026-10-18)
@@ -263,12 +327,24 @@ era's `--parallel 1` law).
## Fallback rungs
1. `UBATCH_SIZE=1024` — frees **1,306 MiB** measured on this shape (prefill
−17 %).
2. `CACHE_TYPE_K/V=q4_0` — frees ~6.5 GiB at three windows (the 27B era measured
the q4_0 rung at 16,392 used / 7,733 free at 2 × 200,704).
3. Cut to 2 slots — but that breaks the third concurrent subagent, so it is a
last resort, not a knob.
Cheapest first; every one is a single line of `scripts/frank-zealot.env`, and
the KV figures are for the 524,288-position pool unless stated otherwise:
1. `UBATCH_SIZE=1024` — hands back **1,306 MiB** of compute buffers measured on
this card (prefill −17 %).
2. `CACHE_TYPE_V=q8_0` — exact K, quantized V: the pool goes 16,384 →
**12,544 MiB**, freeing 3,840, and it is the halfway door back.
`--flash-attn on` is what keeps quantized V legal.
3. `CTX_SIZE=401408` — two 200,704 conversations: exact KV at
**12,544.00 MiB**, same 3,840 freed, but `models.yml` `contextWindow` MUST
follow the per-slot number or long turns start 400-ing.
4. Revert to `q8_0` K + `q8_0` V at `CTX_SIZE=786432 PARALLEL=3` — the shape
this one replaced: 13,056.00 MiB of cache and the THIRD conversation back.
5. `CACHE_TYPE_K/V=q4_0` — the far rung: 10,240 B/token ⇒ **5,120 MiB** here
(the 27B era measured q4_0 at 16,392 used / 7,733 free at 2 × 200,704).
Any retune has to keep the seam's fit gate green — it is the only instrument
that can see a boot that would die at `cudaMalloc`.
## Rollback