docs(frank-zealot): re-key the exact-KV shape across the vault
AGENTS.md: topology row, the frank-zealot runbook section (shape, KV accounting, MTP verdict, compaction-hub concurrency), and two Key Lessons bullets -- the draft-context bill restated with the real knob names, plus a new lesson that a quant's scale overhead belongs in its byte rate (the 6.25 % invented by doubling 13,056.00 MiB). LLM/frank-zealot-llama-prism-bonsai9-mtp.md: retitled to the live shape; the 3-slot q8_0 boot ledger and its measurements are relabelled as the arm this replaced (byte-identical numbers, honest attribution), the corrected f16 rate derivation is added, the MTP census is labelled q8_0-era, the fallback rungs are re-derived for the 524,288 pool, and the boot-ledger section carries an explicit PROJECTION that must be replaced with the measured ledger from the first f16 boot. Pointers re-pointed (live-shape sentences only; every historical ledger left byte-identical): LLM/index.md, vault index.md, Dankclaw/OpenWebUI.md, and the retired-runbook "today" pointers in mimo-q5km, mimo-q6k, qwen35-9b-q6k, swift15, sglang-mimo-v26. log.md gains the dated entry.
This commit is contained in:
Frank
@@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
|
||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 and DEPLOYED 2026-10-18 so BOTH slots can hold a maximum-length request at once; the parity play was frank-zealot's then-401,408 total in-flight context — zealot moved to 786,432 the same evening and dragoon's pool was NOT raised again, because every position is page cache the n-gram table loses; **11.0 GiB reserved, measured on that deploy** (the preflight formula bills 11.3) where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
|
||||
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10777**, commit `e3de71522`; root systemd unit, relinked — the ONLY loader for this payload, and since 2026-10-18 also the minimum build that pairs MTP with a quantized V cache) | `prism-ml/Ternary-Bonsai-2-9B-gguf` **PQ2_0 carrying the MTP head in the file** (2,675,410,720 B, Hub-LFS sha256 `c79fcdfa…` verified at fetch by `scripts/download-frank-zealot-bonsai9.sh`; `qwen35` hybrid: `block_count 33` = **32 trunk layers** (8 full-attention + 24 Gated DeltaNet at `full_attention_interval 4`) + the `blk.32` MTP block, 4 KV heads × 256 dim, native 262,144; `general.file_type 141` with **202 custom-type PQ2_0 tensors of 442** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (624,229,664 B, sha256 `e0cb6f69…`; **Q8_0 is the publisher's requirement** for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs three slots) for **vision**, on the RTX 3090, **THREE 262,144-token slots** (`--ctx-size 786432 --parallel 3` = every conversation at the trained window), **q8_0 K + q8_0 V** under flash attention (17 KiB/token over the 8 full layers = 13,056 MiB pool, ledger-exact; **f16 KV needs 26,112 MiB and cannot boot this card**; +151 MiB of recurrent state for the three DeltaNet cells — the SPEC-ARMED arms print 452 MiB, because MTP depth 2 bills three seqs per slot), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF — measured, not because the head is absent**: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented MTP arm is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s. Production boot 17,922 used / 6,203 free; **168.7 t/s** single-stream greedy decode (three runs, 168.4–168.7) and **~300 t/s** aggregate across three concurrent streams (102.8 each); 2,877 t/s prefill at 106,418 was the trial arm. Cutover 2026-10-18 from Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback). | 7771 |
|
||||
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10777**, commit `e3de71522`; root systemd unit, relinked — the ONLY loader for this payload; the b10776 floor that gates MTP beside a quantized V cache is moot under exact KV, but the custom tensor type alone keeps the build pinned) | `prism-ml/Ternary-Bonsai-2-9B-gguf` **PQ2_0 carrying the MTP head in the file** (2,675,410,720 B, Hub-LFS sha256 `c79fcdfa…` verified at fetch by `scripts/download-frank-zealot-bonsai9.sh`; `qwen35` hybrid: `block_count 33` = **32 trunk layers** (8 full-attention + 24 Gated DeltaNet at `full_attention_interval 4`) + the `blk.32` MTP block, 4 KV heads × 256 dim, native 262,144; `general.file_type 141` with **202 custom-type PQ2_0 tensors of 442** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (624,229,664 B, sha256 `e0cb6f69…`; **Q8_0 is the publisher's requirement** for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs several slots) for **vision**, on the RTX 3090, **TWO 262,144-token slots** (`--ctx-size 524288 --parallel 2` = every conversation STILL at the trained window — the per-slot contract never moved) over **unquantized f16 K + f16 V** (32 KiB/token over the 8 full layers = **16,384 MiB** pool; the POOL halved because exact KV at the old 786,432 pool is 24,576 MiB against the 24,123 MiB the driver leaves addressable — the repo's older "26,112 MiB" figure had doubled the q8_0 number including its 1/32 scale overhead; +100.5 MiB of recurrent state for the two DeltaNet cells, ~50.25 MiB a cell), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF — measured, and now doubly so**: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB at the old pool, 2,048 MiB at this one) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented q8_0-era MTP arm is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s. Last MEASURED boot (the 3 × 262,144 q8_0 arm this replaces): 17,922 used / 6,203 free, **168.7 t/s** single-stream greedy decode (three runs, 168.4–168.7), **~300 t/s** aggregate across three concurrent streams (102.8 each), 2,877 t/s prefill at 106,418 — the f16 boot's ledger and rates land in the runbook once measured. Cutover 2026-10-18 from Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback). | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -350,9 +350,9 @@ and `frank-zealot` through
|
||||
the publisher's own **Q8_0** mmproj passed as `--mmproj` — Q8_0 is the
|
||||
publisher's requirement for concurrent image requests (the card states the
|
||||
BF16 tower can crash `llama-server` when several image requests run at once,
|
||||
and this endpoint runs three slots). `frank-zealot`'s `contextWindow` is
|
||||
**262144** — the PER-SLOT window (`--ctx-size 786432 --parallel 3` since the
|
||||
2026-10-18 Bonsai-2-**9B** swap), which since that swap EQUALS the model's
|
||||
and this endpoint runs several slots). `frank-zealot`'s `contextWindow` is
|
||||
**262144** — the PER-SLOT window (`--ctx-size 524288 --parallel 2` since the
|
||||
2026-10-18 reslice to exact f16 KV), which EQUALS the model's
|
||||
native window; llama.cpp answers a longer prompt with a 400 naming the limit,
|
||||
never a truncation, so the harness must compact before it.
|
||||
Note also
|
||||
@@ -510,7 +510,8 @@ llama.cpp refuses the custom type at load, and a loader that
|
||||
ignored the Hadamard metadata would dequant WRONG — same lock class as the
|
||||
ROCmFPX era. On top of that lock, **the fork gates MTP beside a quantized V
|
||||
cache on build `prism-b10776-d00e252` or newer**, which is why the prebuilt
|
||||
moved 10754 → 10777; the 10754 tree stays on disk as the 27B rollback runtime.
|
||||
moved 10754 → 10777 (exact f16 KV makes that gate moot — the custom type alone
|
||||
pins the runtime); the 10754 tree stays on disk as the 27B rollback runtime.
|
||||
Retargeting the payload without the fork ships an endpoint that cannot load,
|
||||
so the seam pins both paths.
|
||||
|
||||
@@ -520,36 +521,47 @@ attention + FFN in q8_0, which is why the draft context bills real KV), and
|
||||
`qwen35.full_attention_interval = 4` ⇒ **8 full-attention + 24 Gated
|
||||
DeltaNet**, 4 KV heads × 256 dim, native 262,144. **KV accounting:** the pool
|
||||
scales with the 8 full layers — 8 × 4 × 256 = 8,192 elements/token per cache,
|
||||
so **q8_0 K + q8_0 V bills 17,408 B/token ≈ 17 KiB** = **13,056.00 MiB** at the
|
||||
786,432 pool (ledger-exact) — exactly HALF the 27B's 34,816 B/token, which is
|
||||
the whole reason three native windows fit where the 27B afforded only two
|
||||
shortened ones. **f16 K would bill 26,112 MiB and cannot boot on a 24,123 MiB
|
||||
card.** The 24 DeltaNet layers bill a fixed recurrent state per SLOT — the
|
||||
production ledger prints `RS buffer size = 150.75 MiB (3 cells, 32 layers,
|
||||
3 seqs)`, while the SPEC-ARMED arms print 452.25 MiB (MTP depth 2 bills three
|
||||
seqs per slot, so 9 cells) — so pool arithmetic alone under-prices the shape.
|
||||
so **the deployed exact pair f16 K + f16 V bills 32,768 B/token = 32 KiB** =
|
||||
**16,384 MiB** at the 524,288 pool. The q8_0 pair this endpoint shipped first
|
||||
billed **17,408 B/token** (1.0625 B/element: 32 codes plus one 2-byte scale per
|
||||
32-block) = **13,056.00 MiB** at the old 786,432 pool, ledger-exact — exactly
|
||||
HALF the 27B's 34,816 B/token. **Exact KV at that 786,432 pool is 24,576 MiB,
|
||||
more than the 24,123 MiB the driver leaves addressable, so the POOL is what
|
||||
pays for f16 — the window never moved.** (The "26,112 MiB" figure this repo
|
||||
carried doubled the q8_0 number *including* its scale overhead; the true ratio
|
||||
is 1.882×, not 2×.) The 24 DeltaNet layers bill a fixed recurrent state per
|
||||
SLOT — ~50.25 MiB a cell, so the 3-slot ledger printed `RS buffer size =
|
||||
150.75 MiB (3 cells, 32 layers, 3 seqs)` and two slots bill 100.50, while the
|
||||
SPEC-ARMED arms print 452.25 MiB (MTP depth 2 bills three seqs per slot, so 9
|
||||
cells) — so pool arithmetic alone under-prices the shape.
|
||||
|
||||
**The shape (2026-10-18):** `--ctx-size 786432 --parallel 3` = THREE
|
||||
conversations of 262,144, every one at the model's TRAINED window (Frank: "if
|
||||
you can possibly fit a third context, do it"), `--flash-attn on`, **q8_0 K +
|
||||
q8_0 V**, batch 4096 / **ubatch 2048** (the measured CUDA optimum carried from
|
||||
the MiMo/Swift A/Bs on this card; it also sets the compute-buffer size, which
|
||||
is what decides whether MTP fits), `--log-verbosity 4`, `--metrics`,
|
||||
`--jinja`, `--mmproj` for vision, **speculative decoding OFF**
|
||||
(`SPEC_TYPE=none`). Boot: **17,876 MiB used / 6,249 free**; compute 1,409 MiB
|
||||
CUDA + 1,201 MiB CUDA_Host, projector 595 + 248 MiB. Measured on the same
|
||||
boot: **167.2 t/s** greedy decode, 158.9 t/s single-stream chat, **280.2 t/s
|
||||
aggregate across three concurrent streams**, **2,877 t/s prefill at 106,418**
|
||||
tokens — the 27B record (76.6 decode / 109.4 aggregate / 691.7 prefill at 166k)
|
||||
beaten on every axis. The **first request after boot is a warmup artifact**
|
||||
(37.7 t/s class) — never report it.
|
||||
**The shape (2026-10-18, resliced the same evening):** `--ctx-size 524288
|
||||
--parallel 2` = TWO conversations of 262,144, every one still at the model's
|
||||
TRAINED window (the first arm was THREE — Frank: "if you can possibly fit a
|
||||
third context, do it" — and the exact-KV reslice handed one back),
|
||||
`--flash-attn on`, **unquantized f16 K + f16 V**, batch 4096 / **ubatch 2048**
|
||||
(the measured CUDA optimum carried from the MiMo/Swift A/Bs on this card; it
|
||||
also sets the compute-buffer size, which is what decides whether MTP fits),
|
||||
`--log-verbosity 4`, `--metrics`, `--jinja`, `--mmproj` for vision,
|
||||
**speculative decoding OFF** (`SPEC_TYPE=none`). The MEASURED record still
|
||||
belongs to the arm this replaced (3 × 262,144, q8_0 K + q8_0 V): boot **17,876
|
||||
MiB used / 6,249 free**; compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host, projector
|
||||
595 + 248 MiB; **167.2 t/s** greedy decode, 158.9 t/s single-stream chat,
|
||||
**280.2 t/s aggregate across three concurrent streams**, **2,877 t/s prefill at
|
||||
106,418** tokens — the 27B record (76.6 decode / 109.4 aggregate / 691.7 prefill
|
||||
at 166k) beaten on every axis. The exact-KV arm's ledger and rates go in the
|
||||
runbook once booted; its projection is ~21,200 MiB used / ~2,900 free, since
|
||||
only the KV pool (+3,328) and one recurrent-state cell (−50) move. The **first
|
||||
request after boot is a warmup artifact** (37.7 t/s class) — never report it.
|
||||
|
||||
**Why MTP is off even though the head is in the file.** The drafter is GOOD —
|
||||
acceptance 0.798 / 0.617 per position, mean accepted length 2.41 — and the card
|
||||
recommends depth 2. It still loses AT THIS SHAPE because the draft context
|
||||
bills its OWN **f16** KV over the WHOLE pool (4,096 B/position × 786,432 =
|
||||
**3,072.00 MiB**, 4,830 MiB with its compute buffers) and ubatch 2048 refuses to
|
||||
give it up. Booted arms, same card, same traffic (2026-10-18):
|
||||
recommends depth 2. It still loses because the draft context bills its OWN
|
||||
**f16** KV over the WHOLE
|
||||
pool (4,096 B/position — **3,072.00 MiB** at the 786,432 pool, 4,830 MiB with
|
||||
its compute buffers; **2,048.00 MiB** at the shipped 524,288 pool) and ubatch
|
||||
2048 refuses to give it up. Booted arms, same card, same traffic (2026-10-18,
|
||||
every row q8_0 K + q8_0 V at the 786,432 pool):
|
||||
|
||||
| slots | spec | ubatch | used / free | decode |
|
||||
|---|---|---|---|---|
|
||||
@@ -563,13 +575,17 @@ give it up. Booted arms, same card, same traffic (2026-10-18):
|
||||
drafter, is what halves it at three windows. The arm that WOULD run it at three
|
||||
windows is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` (213.5 t/s single-stream,
|
||||
250.8 three-stream aggregate) and pays prefill −17 % and 2.7 GiB of headroom on
|
||||
a card whose projector worst-case lands AFTER the engine's own fit check. This
|
||||
endpoint is the fleet's compaction hub — prefill-bound, 2–3 concurrent
|
||||
subagents — so the shipped arm takes the +17 % prefill (2,877 vs 2,395 t/s) and
|
||||
the +12 % three-stream aggregate (280.2 vs 250.8) over single-stream speed, and
|
||||
keeps 6.2 GiB free. `SPEC_TYPE=none` strips the WHOLE spec set (`--spec-type`
|
||||
and every `--spec-draft-*`) — flags move as a set or the arm measures a crash
|
||||
instead of an absence.
|
||||
a card whose projector worst-case lands AFTER the engine's own fit check. Under
|
||||
**exact KV the drafter is arithmetically out**: the trunk projects ~21,200 MiB
|
||||
of the 24,123 MiB addressable before the draft context's 2,048 MiB and its
|
||||
buffers are even asked for, so the doors back to speculation are ONE window per
|
||||
conversation, or KV back to q8_0.
|
||||
The compaction-hub load is what makes exact KV affordable: prefill is
|
||||
weight-bandwidth bound and does not re-read the cache per generated token, so
|
||||
the dtype is paid in decode and concurrency (two conversations now, not three),
|
||||
not in the work this endpoint exists to do. `SPEC_TYPE=none` strips the WHOLE
|
||||
spec set (`--spec-type` and every `--spec-draft-*`) — flags move as a set or the
|
||||
arm measures a crash instead of an absence.
|
||||
|
||||
**Sampler**: the recipe lives in the checkpoint (`general.sampling.*` =
|
||||
temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0,
|
||||
@@ -581,7 +597,7 @@ would silently re-add the flag on an explicit empty value).
|
||||
|
||||
**Vision**: the publisher's own **Q8_0** mmproj via `--mmproj` — Q8_0 is
|
||||
MANDATORY, not a preference: the card says the BF16 tower "can crash
|
||||
llama-server when several image requests run at once", and `--parallel 3` is
|
||||
llama-server when several image requests run at once", and `--parallel 2` is
|
||||
exactly that shape. Probe traps carry over: ≥256 px probe images or the model
|
||||
answers "video", only the OpenAI `image_url` content part reaches the model,
|
||||
and thinking tokens count against `max_tokens` — budget ≥512 or the probe
|
||||
@@ -596,10 +612,12 @@ have to be fetched first.
|
||||
|
||||
**Compaction hub**: `models.yml` routes `compactionModel` for
|
||||
`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here —
|
||||
262,144 (since the 2026-10-18 reslice) is the widest window in the fleet and
|
||||
the only one that comfortably fits a worst-case compaction input (~170k
|
||||
tokens). With `--parallel 3` a compaction request no longer serializes behind a
|
||||
live generation (the Gemma4 era's `--parallel 1` law). Retrieval through a
|
||||
262,144 (still the PER-SLOT window after the exact-KV reslice) is the widest
|
||||
window in the fleet and the only one that comfortably fits a worst-case
|
||||
compaction input (~170k tokens). With `--parallel 2` a compaction request still
|
||||
runs beside ONE live generation, and with two already busy it waits — the Gemma4
|
||||
era's `--parallel 1` law at half strength, which is the concurrency price of
|
||||
exact KV. Retrieval through a
|
||||
25,960-token prompt was verified exact after the swap.
|
||||
|
||||
**Contract law learned at verbosity 4:** a raw `/completion` request prints
|
||||
@@ -1040,18 +1058,34 @@ linked runbooks.
|
||||
- **A draft context bills its own f16 KV over the WHOLE POOL, and on a
|
||||
single-card endpoint that squeeze can cost more than the drafter wins.** The
|
||||
draft context is built at the target's `n_ctx`, so with `--parallel N` the
|
||||
drafter pays the POOL, not the slot: on frank-zealot's 786,432-position
|
||||
qwen35 pool, `--spec-type draft-mtp` added **3,072.00 MiB of f16 draft KV**
|
||||
(4,096 B/position — the draft cache is never quantized) plus its own compute
|
||||
buffers, 4,830 MiB all told. Booted arms, same card, same traffic
|
||||
(2026-10-18): 3 slots + ubatch 2048 → **167.2 t/s decode with spec OFF,
|
||||
109.2 with spec ON**; dropping `UBATCH_SIZE` to 1024 hands the buffers back
|
||||
and spec ON gives **213.5** (+28 % over the spec-OFF arm), and at 1–2 slots
|
||||
the SAME card runs MTP at ubatch 2048 and still gains ~28 %. So the MTP
|
||||
question is never "does the file carry the head" — it is "what do draft KV
|
||||
+ buffers cost against whatever the batch knobs keep for compute buffers",
|
||||
and that answer is per-shape. Measure the arm, never the flag. See
|
||||
drafter pays the POOL, not the slot: on frank-zealot's qwen35 pool
|
||||
`--spec-type draft-mtp` adds **4,096 B/position of draft KV at the default
|
||||
f16 dtype** (`--spec-draft-type-k/-v` can quantize the draft — untested here),
|
||||
**3,072.00 MiB** at the 786,432 pool it shipped with and **2,048.00 MiB** at
|
||||
the 524,288 pool it runs now, plus its own compute buffers (4,830 MiB all told
|
||||
at the old pool). Booted arms, same card, same traffic (2026-10-18, every row
|
||||
q8_0 KV): 3 slots + ubatch 2048 → **167.2 t/s decode with spec OFF, 109.2 with
|
||||
spec ON**; dropping `UBATCH_SIZE` to 1024 hands the buffers back and spec ON
|
||||
gives **213.5** (+28 % over the spec-OFF arm), and at 1–2 slots the SAME card
|
||||
runs MTP at ubatch 2048 and still gains ~28 %. So the MTP question is never
|
||||
"does the file carry the head" — it is "what do draft KV + buffers cost
|
||||
against whatever the batch knobs keep for compute buffers", and that answer is
|
||||
per-shape. Measure the arm, never the flag. Production corollary: beside
|
||||
exact KV an **f16-draft** arm has no room — the trunk projects ~21,200 MiB of
|
||||
the 24,123 addressable before the drafter is asked for anything; the untested
|
||||
door is `--spec-draft-type-k/-v q8_0` (1,088 MiB). See
|
||||
[[frank-zealot-llama-prism-bonsai9-mtp]].
|
||||
- **A quant's scale overhead belongs in its byte rate, and doubling a measured
|
||||
pool figure to "get the f16 number" invents 6.25 %.** This repo carried "f16
|
||||
KV would bill 26,112 MiB" for weeks by doubling the q8_0 ledger figure
|
||||
(13,056.00 MiB). q8_0 is 1.0625 B/element (32 codes plus one 2-byte scale per
|
||||
32-block), f16 is exactly 2.0 ⇒ the true ratio is **1.882×**, so exact KV over
|
||||
that pool is **24,576 MiB**. The conclusion (three f16 windows cannot boot a
|
||||
card that leaves 24,123 MiB addressable) survived; the number did not — and
|
||||
the honest rate is what made the resize calculable, because it shows 524,288
|
||||
positions is the largest exact-KV pool that fits with room for the projector.
|
||||
Derive byte rates from element count × block layout, then check them against
|
||||
the ledger.
|
||||
- **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau —
|
||||
and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its
|
||||
compute buffers land about TWICE as heavy on the card that holds the output
|
||||
|
||||
@@ -43,14 +43,16 @@ As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unve
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
|
||||
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver |
|
||||
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090; production shape 3×262,144 per `scripts/frank-zealot.env`) | Dankclaw |
|
||||
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090; production shape 2×262,144 exact f16 KV per `scripts/frank-zealot.env`) | Dankclaw |
|
||||
|
||||
The URLs in the first column are the DB's recorded values and predate the 2026-08-22
|
||||
bridge move; on this network they are wrong — see the reachability gotcha below, which
|
||||
is the corrected law. Note also that the 2026-10-18 **bench window** on `:7771`
|
||||
closed the same day (production unit stopped 03:38 per the repo Benchmarking
|
||||
Policy while candidate arms held the port): the candidate — Ternary-Bonsai-2-**9B**
|
||||
PQ2_0 with the MTP head, 3 × 262,144 — was **deployed as production**, so the
|
||||
PQ2_0 with the MTP head, deployed that afternoon at 3 × 262,144 q8_0 KV and
|
||||
resliced the same evening to **2 × 262,144 over exact f16 KV** — was **deployed
|
||||
as production**, so the
|
||||
picker row below was re-keyed to it (see the next paragraph).
|
||||
|
||||
The `model` table carries exactly two decoration rows, `frank-zealot` and
|
||||
|
||||
@@ -26,7 +26,7 @@ timestamp: 2026-10-03
|
||||
> the PrismML 27B era of [[frank-zealot-llama-prism-bonsai2-pq20]] (build 10754,
|
||||
> 2 × 200,704 q8_0 KV, MTP off) — retired 2026-10-17 → 10-18 as a rollback
|
||||
> target; the active row today is build **10777** on
|
||||
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` at 3 × 262,144 q8_0 KV, MTP off by
|
||||
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` at 2 × 262,144 exact f16 KV, MTP off by
|
||||
> measurement (AGENTS.md Runbook Index). The Q5_K_M
|
||||
> payload is still on disk (`stat -c %s` 2026-10-18 → 6876124704).
|
||||
|
||||
|
||||
@@ -20,13 +20,16 @@ timestamp: 2026-10-01
|
||||
> [[frank-zealot-llama-mimo-q5km]], the same runtime and shape one quant rung
|
||||
> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
|
||||
> target (on disk 2026-10-18, `stat -c %s` → 7793710624). :7771 today runs
|
||||
> [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
|
||||
> `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704, q8_0 K + q8_0 V, MTP off).
|
||||
> [[frank-zealot-llama-prism-bonsai9-mtp]] (llama.cpp PrismML fork build
|
||||
> **10777**, `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144,
|
||||
> exact f16 K + f16 V, MTP off by measurement).
|
||||
>
|
||||
> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF
|
||||
> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp
|
||||
> exposes per-tensor. The per-tensor lever stayed; this pairing did not — the
|
||||
> live endpoint bills q8_0 K + q8_0 V (`scripts/frank-zealot.env:55-56`).
|
||||
> live endpoint bills **exact f16 K + f16 V** — 32,768 B/token = 16,384 MiB
|
||||
> over the 524,288-position pool of two 262,144 windows
|
||||
> (`scripts/frank-zealot.env:57-58,80-81`).
|
||||
|
||||
## Identity
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 3 × 262144, q8_0 KV (ACTIVE)"
|
||||
description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: THREE conversations of 262,144 tokens (every slot at the model's trained window) over q8_0 K + q8_0 V, vision through the publisher's Q8_0 Qwen3-VL projector, and the five-arm boot census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head."
|
||||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 2 × 262144, exact f16 KV (ACTIVE)"
|
||||
description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: TWO conversations of 262,144 tokens (every slot at the model's trained window) over an UNQUANTIZED f16 K + V cache, vision through the publisher's Q8_0 Qwen3-VL projector, and the boot-arm census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head."
|
||||
tags:
|
||||
- llm
|
||||
- frank-zealot
|
||||
@@ -14,15 +14,22 @@ tags:
|
||||
timestamp: 2026-10-18
|
||||
---
|
||||
|
||||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 3 × 262,144, q8_0 KV
|
||||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 2 × 262,144, exact f16 KV
|
||||
|
||||
Live since 2026-10-18, replacing the
|
||||
[[frank-zealot-llama-prism-bonsai2-pq20]] era (Ternary-Bonsai-2-**27B** PQ2_0 at
|
||||
2 × 200,704). The trade in one sentence: the 9B bills **half** the KV per token
|
||||
of the 27B, and that is exactly what funds **three** conversations at the
|
||||
model's **native 262,144** window where the 27B afforded only two shortened
|
||||
ones (Frank's ask: *"if you can possibly fit a third context, do it"*). Every
|
||||
axis improved — decode, prefill, three-stream aggregate, and free VRAM.
|
||||
2 × 200,704). The original trade: the 9B bills **half** the KV per token of the
|
||||
27B, and that is what funded **three** conversations at the model's **native
|
||||
262,144** window where the 27B afforded only two shortened ones (Frank's ask:
|
||||
*"if you can possibly fit a third context, do it"*).
|
||||
|
||||
The shape moved again the same evening (Frank: *"shift frank-zealot to 2
|
||||
parallels of unquantized kv"*). The KV cache went **exact** — f16 K + f16 V, no
|
||||
quantization error anywhere in the attention state — and the **slot count paid
|
||||
for it: 3 → 2**. The per-slot contract never moved; every conversation is still
|
||||
262,144, still the trained window. What the dtype bought is fidelity, and what
|
||||
it costs is one concurrent conversation plus the bytes attention reads per
|
||||
generated token.
|
||||
|
||||
## Payload identity
|
||||
|
||||
@@ -46,9 +53,12 @@ axis improved — decode, prefill, three-stream aggregate, and free VRAM.
|
||||
refuses the custom type at load; a loader that ignored the Hadamard metadata
|
||||
would dequant **wrong**, silently. Same lock class as the ROCmFPX era.
|
||||
2. **The build floor.** The fork's card gates **MTP beside a quantized V cache**
|
||||
on `prism-b10776-d00e252` or newer, so the prebuilt moved **10754 → 10777**
|
||||
for this swap. The 10754 tree stays on disk (`~/GitPackages/llama-prism/`) as
|
||||
the 27B rollback runtime. `tests/frank_zealot_unit.bats` pins both paths.
|
||||
on `prism-b10776-d00e252` or newer, which is why the prebuilt moved
|
||||
**10754 → 10777** for the swap. Exact f16 KV makes that gate moot — there is
|
||||
no quantized V to pair with the drafter — but tumbler 1 pins the runtime on
|
||||
its own: stock llama.cpp cannot load this file at any flag value. The 10754
|
||||
tree stays on disk (`~/GitPackages/llama-prism/`) as the 27B rollback
|
||||
runtime. `tests/frank_zealot_unit.bats` pins both paths.
|
||||
|
||||
## Header census (read from the GGUF, not the card)
|
||||
|
||||
@@ -71,41 +81,66 @@ axis improved — decode, prefill, three-stream aggregate, and free VRAM.
|
||||
| `general.sampling.*` | temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / repeat 1.0 | the card recipe; llama.cpp reads it itself |
|
||||
|
||||
**KV accounting.** Only the 8 full-attention layers scale with context:
|
||||
8 × 4 KV heads × 256 dim = **8,192 elements/token per cache**, so
|
||||
**q8_0 K + q8_0 V bills 17,408 B/token** — the 786,432-position pool is
|
||||
**13,056.00 MiB**, ledger-exact. That is exactly HALF the 27B's 34,816 B/token.
|
||||
**f16 KV would bill 26,112 MiB against a 24,123 MiB card** and dies at
|
||||
`cudaMalloc`. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
|
||||
instead: the PRODUCTION ledger prints `RS buffer size = 150.75 MiB (3 cells,
|
||||
32 layers, 3 seqs)` ≈ 50 MiB a cell, while the SPEC-ARMED arms printed
|
||||
452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples
|
||||
even the recurrent state. Pool arithmetic alone under-prices the shape.
|
||||
8 × 4 KV heads × 256 dim = **8,192 elements/token per cache** (the engine prints
|
||||
`n_embd_k_gqa = n_embd_v_gqa = 1024`). **The deployed pair is exact: f16 K +
|
||||
f16 V bills 32,768 B/token** (2 B/element × 8,192 elements × 2 caches), so the
|
||||
524,288-position pool is **16,384 MiB**. The q8_0 pair this endpoint shipped
|
||||
first billed **17,408 B/token** — 1.0625 B/element, 32 codes plus one 2-byte
|
||||
scale per 32-block — the **13,056.00 MiB** its ledger printed at the 786,432
|
||||
pool, exactly HALF the 27B's 34,816 B/token.
|
||||
|
||||
**Corrected arithmetic, and it matters.** The note that lived here and in
|
||||
`frank-zealot.env` — "f16 would bill 26,112 MiB" — doubled the q8_0 figure
|
||||
*including* its scale overhead, overstating exact KV by 6.25 %. Exact f16 at the
|
||||
786,432 pool is **24,576 MiB** against the **24,123 MiB** the driver leaves
|
||||
addressable (`nvidia-smi`'s total column reads 24,576; used + free reads 24,123
|
||||
— the difference is driver reservation). The wall stands in both readings; the
|
||||
honest rate is what makes the fit calculable instead of superstitious. **Halving
|
||||
the POOL is what funds exact KV** — the window is already at the native ceiling
|
||||
and there is no YaRN in this path.
|
||||
|
||||
The 24 DeltaNet layers bill a fixed recurrent state per SLOT instead: the 3-slot
|
||||
ledger printed `RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)` ≈
|
||||
**50.25 MiB a cell**, so two slots bill 100.50. The SPEC-ARMED arms printed
|
||||
452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples even
|
||||
the recurrent state. Pool arithmetic alone under-prices the shape.
|
||||
|
||||
## The shape (`scripts/frank-zealot.env`)
|
||||
|
||||
`--ctx-size 786432 --parallel 3` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`,
|
||||
`--cache-type-k q8_0 --cache-type-v q8_0`, `--batch-size 4096`,
|
||||
`--ctx-size 524288 --parallel 2` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`,
|
||||
`--cache-type-k f16 --cache-type-v f16`, `--batch-size 4096`,
|
||||
`--ubatch-size 2048`, `--n-gpu-layers 99`, `--jinja`, `--reasoning auto`,
|
||||
`--reasoning-format deepseek`, `--metrics`, `--mmproj …`, `--log-verbosity 4`,
|
||||
`--presence-penalty 1.0`,
|
||||
**`SPEC_TYPE=none`** (which strips `--spec-type` AND every `--spec-draft-*`).
|
||||
|
||||
Per-slot window is the **contract** the harness quotes: `models.yml` carries
|
||||
`contextWindow: 262144`. llama.cpp refuses a prompt longer than the SLOT with a
|
||||
400 naming the limit — it never truncates — and since this swap the slot window
|
||||
equals the native window, so no YaRN and no cap is in play.
|
||||
`contextWindow: 262144`, and it did **not** move with the slot count because
|
||||
llama.cpp divides `--ctx-size` across slots. It refuses a prompt longer than the
|
||||
SLOT with a 400 naming the limit — never a truncation — and since this swap the
|
||||
slot window equals the native window, so no YaRN and no cap is in play.
|
||||
`tests/frank_zealot_unit.bats` now gates the arithmetic: the shipped dtype ×
|
||||
pool plus the measured non-KV ledger terms must fit 24,123 MiB with ≥1 GiB of
|
||||
slack, and the gate is proven to REFUSE the impossible arm (exact KV at the old
|
||||
786,432 pool = 24,576 MiB of cache alone).
|
||||
|
||||
## The MTP verdict: the head is in the file and it is still OFF
|
||||
|
||||
The drafter is **good**: acceptance 0.798 / 0.617 per position, mean accepted
|
||||
length 2.41, and the card recommends depth 2. It loses **at this shape**
|
||||
because the draft context bills its own **f16** KV over the WHOLE pool —
|
||||
`blk.32` carries real attention (4 KV heads × 256 × 2 caches × 2 B =
|
||||
**4,096 B/position**) × 786,432 positions = **3,072.00 MiB**, 4,830 MiB with its
|
||||
compute buffers — and `--ubatch-size 2048` refuses to give that up.
|
||||
length 2.41, and the card recommends depth 2. It is OFF because the draft
|
||||
context bills its own **f16** KV over the WHOLE pool — `blk.32` carries real
|
||||
attention (4 KV heads × 256 × 2 caches × 2 B = **4,096 B/position** at the
|
||||
default dtype). The draft dtype is itself a knob — `--spec-draft-type-k` /
|
||||
`--spec-draft-type-v`, both default f16, verified against this build's `--help`
|
||||
(so the "no flag exists" claim that lived here is FALSE) — and quantizing the
|
||||
draft to q8_0 would bill 2,176 B/position, **1,088 MiB** at the shipped pool.
|
||||
**No measured arm runs a drafter beside exact KV**; every row below is f16 draft.
|
||||
|
||||
Booted arms, same card, same traffic, 2026-10-18 (`/completion` `timings`,
|
||||
greedy 512 tokens; never the first request after a boot):
|
||||
The census below is the **q8_0 era**: every row is q8_0 K + q8_0 V at the
|
||||
786,432 pool, where the draft bill was 3,072.00 MiB and 4,830 MiB with its
|
||||
compute buffers. Booted arms, same card, same traffic, 2026-10-18
|
||||
(`/completion` `timings`, greedy 512 tokens; never the first request after a
|
||||
boot):
|
||||
|
||||
| slots | spec | ubatch | used / free | decode | 3-stream | prefill @106k |
|
||||
|---|---|---|---|---|---|---|
|
||||
@@ -117,15 +152,26 @@ greedy 512 tokens; never the first request after a boot):
|
||||
|
||||
**MTP is worth +28 % single-stream decode whenever the card has room** (rows
|
||||
4–5 prove the drafter itself is a win; row 2 proves the squeeze is what halves
|
||||
it). The arm that runs MTP at three windows is `SPEC_TYPE=draft-mtp` +
|
||||
`UBATCH_SIZE=1024` (row 3) and pays **prefill −17 %** and **2,725 MiB of
|
||||
it). The arm that ran it at three windows was `SPEC_TYPE=draft-mtp` +
|
||||
`UBATCH_SIZE=1024` (row 3), paying **prefill −17 %** and **2,725 MiB of
|
||||
headroom** on a card whose mmproj worst-case lands AFTER the engine's own fit
|
||||
check. This endpoint is the fleet's **compaction hub** — prefill-bound ~170k
|
||||
inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 %
|
||||
prefill and +12 % three-stream aggregate over single-stream speed, and keeps
|
||||
6.2 GiB free. Frank confirmed this arm on 2026-10-18.
|
||||
check.
|
||||
|
||||
## Boot ledger (production boot 2026-10-18 13:19, under the unit, verbosity 4)
|
||||
**Under exact KV an f16-draft arm is priced out of this card.** The draft
|
||||
context is built at the TARGET's pool, so at 524,288 positions it bills
|
||||
4,096 × 524,288 = **2,048.00 MiB** plus its own buffers, on a trunk that already
|
||||
projects **~21,200 MiB** of the 24,123 MiB addressable. The doors back to
|
||||
speculation are **one window per conversation** (row 5's class of shape), **KV
|
||||
back to q8_0** with `UBATCH_SIZE=1024`, or the **unmeasured**
|
||||
`--spec-draft-type-k q8_0 --spec-draft-type-v q8_0` arm — 1,088 MiB of draft,
|
||||
legal on b10777, never booted beside exact KV. Frank chose exact KV over the
|
||||
drafter on 2026-10-18, and the endpoint's actual job is what made that
|
||||
affordable: the fleet's **compaction hub** is prefill-bound on ~170k inputs,
|
||||
prefill is weight-bandwidth bound and does not re-read the cache per generated
|
||||
token — so the dtype is paid in decode and concurrency, not in the work this
|
||||
service exists to do.
|
||||
|
||||
## Boot ledger — the q8_0 3-slot production boot (2026-10-18 13:19, under the unit, verbosity 4)
|
||||
|
||||
```
|
||||
CUDA0 model buffer size = 2036.71 MiB # + 257.66 MiB CPU_Mapped (mmap)
|
||||
@@ -141,7 +187,16 @@ No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class
|
||||
numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`,
|
||||
never the projection**.
|
||||
|
||||
## Measured (production boot, `/completion` `timings`, greedy, `ignore_eos`)
|
||||
**The exact-KV arm's projection, recorded before its own boot.** Every term of
|
||||
that ledger survives the reslice except the KV pool (13,056 → 16,384 MiB) and
|
||||
one recurrent-state cell (−50.25 MiB), so the pre-flight prediction is
|
||||
`17,922 − 13,056 + 16,384 − 50 ≈ 21,200 MiB used / ~2,920 free`, with ~2 GiB
|
||||
still standing through the mmproj worst case. **Replace this paragraph with the
|
||||
MEASURED ledger and settled `nvidia-smi` from the first f16 boot** — a
|
||||
projection is a pre-flight check, not a record, and the whole reason the shape
|
||||
changed dtype is that projections drift from ledgers.
|
||||
|
||||
## Measured — the q8_0 3-slot boot (`/completion` `timings`, greedy, `ignore_eos`)
|
||||
|
||||
- **168.7 t/s** decode — three runs of 512 tokens: 168.4 / 168.7 / 168.7
|
||||
(the trial arm printed 167.2; same shape, same noise band).
|
||||
@@ -156,6 +211,13 @@ never the projection**.
|
||||
stops on its own: a 24-token countdown measured 72 t/s across three streams
|
||||
while the same shape with `ignore_eos` gave 300.
|
||||
|
||||
**These are the numbers the exact-KV arm is judged against.** Exact KV makes
|
||||
attention read 1.882× the bytes per generated token, so a decode drop is
|
||||
EXPECTED and has to be measured, not assumed; prefill should hold. Re-run the
|
||||
same instrument on the first f16 boot — greedy, `ignore_eos`, 512 tokens, the
|
||||
second request or later, and the two-stream aggregate in place of the
|
||||
three-stream one — and record it here.
|
||||
|
||||
## Contract traps
|
||||
|
||||
- **`--metrics` is mandatory.** Without it llama.cpp answers **501** on
|
||||
@@ -164,7 +226,7 @@ never the projection**.
|
||||
(`up{job="frank-zealot"}` oldest retained sample, 2026-10-11, is already 0).
|
||||
- **Q8_0 mmproj, not BF16.** The publisher's card: the BF16 tower "can crash
|
||||
llama-server when several image requests run at once; Q8_0 is not affected" —
|
||||
and `--parallel 3` is precisely that shape.
|
||||
and `--parallel 2` is still precisely that shape.
|
||||
- **Only the `image_url` content part reaches the model.** The legacy top-level
|
||||
`images` field bumps `usage.prompt_tokens` and the model still invents an
|
||||
answer. Probe with a solid color and expect the exact word.
|
||||
@@ -216,9 +278,11 @@ never the projection**.
|
||||
`frank-zealot` here — 262,144 is the widest window in the fleet and the only one
|
||||
that comfortably fits a worst-case compaction input (~170k tokens; the clamp is
|
||||
`min(maxTokens 16384, contextWindow − est − 4096)`, compaction fires at 85 % of
|
||||
the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With `--parallel 3`
|
||||
a compaction request no longer serializes behind a live generation (the Gemma4
|
||||
era's `--parallel 1` law).
|
||||
the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). What the reslice to
|
||||
**two** slots does cost is here: a compaction request still runs beside ONE live
|
||||
generation, but with two conversations already busy a third waits — the Gemma4
|
||||
era's `--parallel 1` serialization law at half strength. The window itself is
|
||||
untouched, and the window is the term the compaction contract depends on.
|
||||
|
||||
## Deploy ledger (2026-10-18)
|
||||
|
||||
@@ -263,12 +327,24 @@ era's `--parallel 1` law).
|
||||
|
||||
## Fallback rungs
|
||||
|
||||
1. `UBATCH_SIZE=1024` — frees **1,306 MiB** measured on this shape (prefill
|
||||
−17 %).
|
||||
2. `CACHE_TYPE_K/V=q4_0` — frees ~6.5 GiB at three windows (the 27B era measured
|
||||
the q4_0 rung at 16,392 used / 7,733 free at 2 × 200,704).
|
||||
3. Cut to 2 slots — but that breaks the third concurrent subagent, so it is a
|
||||
last resort, not a knob.
|
||||
Cheapest first; every one is a single line of `scripts/frank-zealot.env`, and
|
||||
the KV figures are for the 524,288-position pool unless stated otherwise:
|
||||
|
||||
1. `UBATCH_SIZE=1024` — hands back **1,306 MiB** of compute buffers measured on
|
||||
this card (prefill −17 %).
|
||||
2. `CACHE_TYPE_V=q8_0` — exact K, quantized V: the pool goes 16,384 →
|
||||
**12,544 MiB**, freeing 3,840, and it is the halfway door back.
|
||||
`--flash-attn on` is what keeps quantized V legal.
|
||||
3. `CTX_SIZE=401408` — two 200,704 conversations: exact KV at
|
||||
**12,544.00 MiB**, same 3,840 freed, but `models.yml` `contextWindow` MUST
|
||||
follow the per-slot number or long turns start 400-ing.
|
||||
4. Revert to `q8_0` K + `q8_0` V at `CTX_SIZE=786432 PARALLEL=3` — the shape
|
||||
this one replaced: 13,056.00 MiB of cache and the THIRD conversation back.
|
||||
5. `CACHE_TYPE_K/V=q4_0` — the far rung: 10,240 B/token ⇒ **5,120 MiB** here
|
||||
(the 27B era measured q4_0 at 16,392 used / 7,733 free at 2 × 200,704).
|
||||
|
||||
Any retune has to keep the seam's fit gate green — it is the only instrument
|
||||
that can see a boot that would die at `cudaMalloc`.
|
||||
|
||||
## Rollback
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ timestamp: 2026-10-07
|
||||
[[frank-zealot-llama-prism-bonsai2-pq20]] (27B PQ2_0, 2 × 200,704 q8_0 KV, build
|
||||
10754), itself retired 2026-10-17 → 10-18 and now the rollback target: :7771
|
||||
today is llama.cpp PrismML build **10777** on
|
||||
`Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 3 × 262,144 q8_0 KV, MTP off by
|
||||
`Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144 exact f16 KV, MTP off by
|
||||
measurement (AGENTS.md Runbook Index active row; `scripts/frank-zealot.env`).
|
||||
Transient 2026-10-18: :7771 is inside a bench window — the unit is `failed`
|
||||
since 03:14 and a hand-booted process answers the port; restore with
|
||||
|
||||
@@ -22,7 +22,7 @@ timestamp: 2026-10-06
|
||||
> frank-small the same night). Superseded by
|
||||
> [[frank-zealot-llama-qwen35-9b-q6k-2x200704]]; the production shape on :7771
|
||||
> today is llama.cpp PrismML build **10777** on
|
||||
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 3 × 262,144 q8_0 KV, MTP off by
|
||||
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144 exact f16 KV, MTP off by
|
||||
> measurement (AGENTS.md Runbook Index active row); the 27B PQ2_0 / 2 × 200,704
|
||||
> shape of [[frank-zealot-llama-prism-bonsai2-pq20]] is that era's rollback
|
||||
> target. Transient 2026-10-18 bench window on :7771 — the unit is `failed`
|
||||
|
||||
@@ -20,8 +20,9 @@ timestamp: 2026-10-01
|
||||
> **RETIRED 2026-10-01, hours after cutover** (AGENTS.md Runbook Index). Replaced by
|
||||
> [[frank-zealot-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
|
||||
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express. :7771
|
||||
> today runs [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork
|
||||
> build 10754, `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off).
|
||||
> today runs [[frank-zealot-llama-prism-bonsai9-mtp]] (llama.cpp PrismML fork
|
||||
> build **10777**, `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144,
|
||||
> exact f16 K + f16 V, MTP off by measurement).
|
||||
> The venv (`/home/dank/GitPackages/sglang-cu128/`) is still on disk; the INT8
|
||||
> payload was staged to
|
||||
> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 and
|
||||
@@ -59,8 +60,8 @@ timestamp: 2026-10-01
|
||||
the two-way concurrency contract. **That compaction arithmetic is
|
||||
specific to a 262,144-token slot**: the shape that followed on 2026-10-02
|
||||
ran three slots of 180,224, which 183,501 tokens already exceeds — see *What
|
||||
three slots cost* in [[frank-zealot-llama-mimo-q5km]]; :7771 runs two 200,704
|
||||
slots today.
|
||||
three slots cost* in [[frank-zealot-llama-mimo-q5km]]; :7771 today runs two
|
||||
262,144-token conversations over one 524,288-position pool with exact f16 KV.
|
||||
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
|
||||
dim x 1 B), pool 9.31 GiB.
|
||||
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
|
||||
|
||||
@@ -26,9 +26,10 @@ MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
|
||||
200704, vision ON)** and
|
||||
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
|
||||
Ternary-Bonsai-2-9B at **PQ2_0** — 9B qwen35 hybrid (8 full-attention + 24
|
||||
Gated DeltaNet layers + the blk.32 MTP block), THREE 262,144-token
|
||||
conversations, MTP present in the file but OFF by measurement,
|
||||
q8_0 K + q8_0 V (f16 KV does not fit the RTX 3090 at this window), vision via
|
||||
Gated DeltaNet layers + the blk.32 MTP block), TWO slots over ONE
|
||||
524,288-position pool (2 × the 262,144 window), MTP present in the file but
|
||||
OFF by measurement, **exact f16 K + f16 V** — 32,768 B/token = **16,384 MiB**
|
||||
of KV (three 262,144 windows at exact KV do not fit the RTX 3090), vision via
|
||||
the publisher's own Q8_0 mmproj, on the RTX 3090 — and the custom
|
||||
tensor type 141 + `prism.hadamard.*` metadata mean no other loader can open the
|
||||
file)**.
|
||||
@@ -63,7 +64,7 @@ ONE 401408-position KV pool at ctx 200704 = 2 slots x ctx, MTP always-on, image
|
||||
G[llama.cpp PrismML fork CUDA build
|
||||
frank-zealot, port 7771]
|
||||
H[Ternary-Bonsai-2-9B PQ2_0 GGUF + Q8_0 mmproj
|
||||
3 x 262144 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision]
|
||||
2 slots over ONE 524288-position pool at ctx 262144, exact f16 K + f16 V, MTP off, mmproj vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -132,19 +133,26 @@ frank-zealot, port 7771]
|
||||
custom PQ2_0 tensor type + `prism.hadamard.*` transform, and since this swap
|
||||
also the minimum build that pairs MTP with a quantized V cache) serving
|
||||
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head IN the file** on
|
||||
the RTX 3090 — **THREE 262,144-token conversations**
|
||||
(`--ctx-size 786432 --parallel 3`, every slot at the window the model was
|
||||
trained at), **q8_0 K + q8_0 V** (17 KiB/token over the 8 full layers =
|
||||
13,056 MiB ledger-exact; the f16 pair needs 26,112 MiB and dies at
|
||||
`cudaMalloc`), settled **17,876 used / 6,249 free**, vision via the
|
||||
publisher's **Q8_0** mmproj (Q8_0 is the publisher's requirement for
|
||||
concurrent image requests, and this runs three slots), **MTP OFF by
|
||||
measurement** — the draft context bills its own f16 KV over the WHOLE pool
|
||||
(3,072 MiB) and at `UBATCH_SIZE=2048` that squeeze halves decode
|
||||
(167.2 → 109.2 t/s); `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the
|
||||
documented 213.5 t/s arm, declined for +17 % prefill and 6.2 GiB of headroom.
|
||||
Measured: 167.2 t/s single-stream decode, 2,877 t/s prefill at 106,418,
|
||||
280.2 t/s aggregate across three streams. `--metrics` rides (`METRICS=1`):
|
||||
the RTX 3090 — **TWO 262,144-token conversations**
|
||||
(`--ctx-size 524288 --parallel 2`, every slot at the window the model was
|
||||
trained at), **unquantized f16 K + f16 V** (32,768 B/token over the 8 full
|
||||
layers = **16,384 MiB** exact; the POOL halved because that rate over the
|
||||
OLD 786,432 pool bills 24,576 MiB against the 24,123 the driver leaves
|
||||
addressable, and the old "26,112 MiB" note was wrong — it doubled the
|
||||
q8_0 figure (17,408 B/token = 13,056 MiB) *including* its 1/32 scale
|
||||
overhead, so f16/q8_0 is **1.882×**, not 2×), the q8_0-era boot settled
|
||||
**17,876 used / 6,249 free**, vision via the publisher's **Q8_0** mmproj
|
||||
(Q8_0 is the publisher's requirement for concurrent image requests, and this
|
||||
runs two slots), **MTP OFF by measurement** — the draft context bills its own
|
||||
f16 KV over the WHOLE pool (4,096 B/position × 524,288 = 2,048 MiB) and at
|
||||
`UBATCH_SIZE=2048` that squeeze halves decode (167.2 → 109.2 t/s);
|
||||
`SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the documented 213.5 t/s arm,
|
||||
declined for +17 % prefill and 6.2 GiB of headroom — and exact KV leaves an
|
||||
f16-draft arm no room at all (a quantized draft, `--spec-draft-type-* q8_0`,
|
||||
is the unmeasured door).
|
||||
Measured at the q8_0-era shape: 167.2 t/s single-stream decode, 2,877 t/s
|
||||
prefill at 106,418, 280.2 t/s aggregate across three streams. `--metrics`
|
||||
rides (`METRICS=1`):
|
||||
llama.cpp answers 501 on `/metrics` without it, which is why Prometheus
|
||||
`up{job="frank-zealot"}` read 0 for its whole retained history until
|
||||
2026-10-18. `contextWindow: 262144` is the **widest in the fleet**, so this
|
||||
|
||||
@@ -409,10 +409,15 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
loader for the custom PQ2_0 tensor type / `prism.hadamard.*`, and the floor
|
||||
for MTP beside a quantized V cache) serving
|
||||
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head** on the RTX 3090
|
||||
— THREE 262,144-token conversations, q8_0 K + q8_0 V (f16 OOMs at this pool:
|
||||
26,112 vs 13,056 MiB over 8 full-attention layers), MTP **off by
|
||||
measurement** (the draft context bills 3,072 MiB of f16 KV over the whole
|
||||
pool), vision through the Q8_0 mmproj; boot 17,876 used / 6,249 free
|
||||
— TWO 262,144-token conversations (`--ctx-size 524288 --parallel 2`), exact
|
||||
f16 K + f16 V = **16,384 MiB** over 8 full-attention layers (32,768 B/token;
|
||||
the per-slot window is UNCHANGED at 262,144 — what halved is the POOL,
|
||||
because exact f16 over the OLD 786,432 pool bills **24,576 MiB** against the
|
||||
**24,123** the driver leaves addressable; the old "26,112 vs 13,056 MiB" note
|
||||
doubled the q8_0 figure with its 1/32 scale overhead included, so f16/q8_0 is
|
||||
1.882×, not 2×), MTP **off by measurement** (the draft context bills 2,048
|
||||
MiB of f16 KV over the whole pool, and exact KV leaves no room for one),
|
||||
vision through the Q8_0 mmproj; boot 17,876 used / 6,249 free (the q8_0 era)
|
||||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — RETIRED 2026-10-17 → 10-18
|
||||
(rollback target, payload + build 10754 on disk): Ternary-Bonsai-2-**27B**
|
||||
PQ2_0 at TWO 200,704-token conversations, q8_0 K + q8_0 V, MTP **impossible**
|
||||
|
||||
@@ -2787,3 +2787,37 @@ every restart since the cutover.
|
||||
to the player, temp dir cleaned; the audible pass through the default
|
||||
`pw-play` detection confirmed by Frank. Sessions open before the merge need
|
||||
a restart to pick the command up.
|
||||
|
||||
## 2026-10-18 — frank-zealot: 2 × 262,144 over unquantized f16 KV
|
||||
|
||||
- Frank's instruction: **"shift frank-zealot to 2 parallels of unquantized
|
||||
kv"**. The env delta is four lines: `CTX_SIZE=524288`, `PARALLEL=2`,
|
||||
`CACHE_TYPE_K=f16`, `CACHE_TYPE_V=f16`. Everything else holds — batch
|
||||
**4096** / ubatch **2048**, `--flash-attn on`, `--jinja`, `--metrics`,
|
||||
`--presence-penalty 1.0`, `SPEC_TYPE=none`, the Q8_0 mmproj, port 7771, and
|
||||
the per-slot window **262,144** — `models.yml` `contextWindow` untouched.
|
||||
- **Why the pool had to halve.** Exact KV bills **32,768 B/token** (8
|
||||
full-attention layers × 4 KV heads × 256 dim × 2 B × 2 caches), so the old
|
||||
786,432 pool asks **24,576 MiB** against the **24,123 MiB** the driver leaves
|
||||
addressable — impossible, not tight. 524,288 positions ⇒ **16,384 MiB** and
|
||||
exactly two windows of 262,144.
|
||||
- Arithmetic correction to the repo's long-standing figure: "the f16 pair would
|
||||
need **26,112 MiB**" doubled the q8_0 number *including* its 1/32 scale
|
||||
overhead — q8_0 is 17,408 B/token (1.0625 B/element), so f16/q8_0 is
|
||||
**1.882×**, not 2×, and exact f16 over 786,432 is **24,576 MiB**. The
|
||||
conclusion stands (three f16 windows cannot boot this card); the number was
|
||||
wrong.
|
||||
- The cost: **concurrency 3 → 2 conversations**, and MTP is now **doubly** out.
|
||||
The 13:16 measurement still says off, and exact KV adds a second reason — the
|
||||
draft context bills its own f16 KV over the WHOLE pool (**4,096 B/position**
|
||||
× 524,288 = **2,048 MiB**) on a trunk projecting ~21,200 used.
|
||||
- **Projection, explicitly not a measurement:** ~**21,200 MiB** used /
|
||||
~**2,920 free**, derived from the 13:19 ledger's 17,922 used because only two
|
||||
terms move — the KV pool (**+3,328**) and one recurrent-state cell
|
||||
(**−50.25**, 3 cells → 2). The measured boot ledger and the decode/prefill
|
||||
re-measure land in the follow-up commit, after the first f16 boot.
|
||||
- New gate in `tests/frank_zealot_unit.bats`: dtype × pool plus the measured
|
||||
non-KV terms must fit **24,123 MiB** with **≥ 1 GiB** of slack — and the same
|
||||
gate proves it refuses the impossible arm (**f16 at 786,432**), not merely
|
||||
passes the shipped one. Runbook re-keyed to the new shape:
|
||||
`LLM/frank-zealot-llama-prism-bonsai9-mtp.md`.
|
||||
|
||||
Reference in New Issue
Block a user