dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(frank-zealot): Bonsai-2-9B PQ2_0 with the MTP head at 3 x 262,144

Browse Source 
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.

MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.

Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.

contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
This commit is contained in:
Frank
2026-10-18 13:15:18 -07:00
parent 2b127163c8
commit 466fd477d8
14 changed files with 1001 additions and 314 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+132 -70
View File
@@ -26,8 +26,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 on 2026-10-18 so BOTH slots can hold a maximum-length request at once — the parity with frank-zealot's 401,408 pool on total in-flight context; ~10.8 GiB reserved where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`; every pool position is page cache the n-gram table loses), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10754**, commit `2459f68b5`; root systemd unit, relinked — the ONLY loader for this payload) | `prism-ml/Ternary-Bonsai-2-27B-gguf` **PQ2_0** (7,206,168,928 B, sha256 `3907dc16…`; `qwen35` hybrid: **64 blocks, 16 full-attention + 48 Gated DeltaNet**, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP head in the file**; `general.file_type 141` = custom tensor type **142** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (629,246,976 B; Qwen3-VL tower, 27 blocks, 768 px, `qwen3vl_merger`) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **q8_0 K + q8_0 V** under flash attention (34 KiB/token over the 16 full layers = 13,328 MiB pool; **f16 KV needs 25,088 MiB and cannot boot this card**), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (no head in the file). Boot 22,664 used / 1,461 free; 76.6 t/s single-stream decode, 691.7 t/s prefill at 166k. Cutover 2026-10-17 from Qwen3.5-9B-Uncensored Q6_K (same shape; payload + master CUDA build on disk as rollback). |
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 on 2026-10-18 so BOTH slots can hold a maximum-length request at once; the parity play was frank-zealot's then-401,408 total in-flight context — zealot moved to 786,432 the same evening and dragoon's pool was NOT raised again, because every position is page cache the n-gram table loses; ~10.8 GiB reserved where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10777**, commit `e3de71522`; root systemd unit, relinked — the ONLY loader for this payload, and since 2026-10-18 also the minimum build that pairs MTP with a quantized V cache) | `prism-ml/Ternary-Bonsai-2-9B-gguf` **PQ2_0 carrying the MTP head in the file** (2,675,410,720 B, Hub-LFS sha256 `c79fcdfa…` verified at fetch by `scripts/download-frank-zealot-bonsai9.sh`; `qwen35` hybrid: `block_count 33` = **32 trunk layers** (8 full-attention + 24 Gated DeltaNet at `full_attention_interval 4`) + the `blk.32` MTP block, 4 KV heads × 256 dim, native 262,144; `general.file_type 141` with **202 custom-type PQ2_0 tensors of 442** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (624,229,664 B, sha256 `e0cb6f69…`; **Q8_0 is the publisher's requirement** for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs three slots) for **vision**, on the RTX 3090, **THREE 262,144-token slots** (`--ctx-size 786432 --parallel 3` = every conversation at the trained window), **q8_0 K + q8_0 V** under flash attention (17 KiB/token over the 8 full layers = 13,056 MiB pool, ledger-exact; **f16 KV needs 26,112 MiB and cannot boot this card**; +452 MiB of recurrent state for the three DeltaNet cells), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF — measured, not because the head is absent**: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented MTP arm is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s. Boot 17,876 used / 6,249 free; 167.2 t/s single-stream decode, 280.2 t/s aggregate across three concurrent streams, 2,877 t/s prefill at 106,418. Cutover 2026-10-18 from Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback). | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in
the [runbook index](#runbook-index). The full swap history lives in
@@ -319,12 +319,14 @@ image]`) — `frank-dragoon` returned to vision with the flash-server v2 swap
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
are the engine core; `--mm-max-patches` available),
and `frank-zealot` through
the publisher's own BF16 mmproj passed as `--mmproj` — the model is natively
multimodal but the Q6_K GGUF carries no vision tensors (same arrangement the
Gemma4 era used). `frank-zealot`'s `contextWindow` is
**200704** — the PER-SLOT window (`--ctx-size 401408 --parallel 2` since the
2026-10-07 Qwen3.5-9B swap), not the pool and not the model's native 262144:
the harness must compact before llama.cpp answers a longer prompt with a 400.
the publisher's own **Q8_0** mmproj passed as `--mmproj` — Q8_0 is the
publisher's requirement for concurrent image requests (the card states the
BF16 tower can crash `llama-server` when several image requests run at once,
and this endpoint runs three slots). `frank-zealot`'s `contextWindow` is
**262144** — the PER-SLOT window (`--ctx-size 786432 --parallel 3` since the
2026-10-18 Bonsai-2-**9B** swap), which since that swap EQUALS the model's
native window; llama.cpp answers a longer prompt with a 400 naming the limit,
never a truncation, so the harness must compact before it.
Note also
that
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
@@ -460,71 +462,113 @@ digest all remain on disk; the ROCmFPX door behind that
### frank-zealot — llama.cpp (PrismML fork) on the RTX 3090 (Dankclaw, ACTIVE)
**The PrismML fork build of llama.cpp** (vendor release, `--version` =
`0.2.0-dev (build 10754, commit 2459f68b5)`;
`/home/dank/GitPackages/llama-prism/llama-server`) as a **root systemd unit**
(`scripts/frank-zealot.service`, `relink_unit`ed into
`/etc/systemd/system/`), serving `prism-ml/Ternary-Bonsai-2-27B-gguf` at
**PQ2_0** (7,206,168,928 B, sha256 `3907dc16…` = the Hub LFS oid) **plus the
publisher's own Q8_0 mmproj** (629,246,976 B), both
`/home/dank/models/Bonsai2-PQ2_0/`, on port 7771. Cutover 2026-10-17 from the
Qwen3.5-9B-Uncensored Q6_K era (same shape).
`0.2.0-dev (build 10777, commit e3de71522)`;
`/home/dank/GitPackages/llama-prism-b10777/llama-server`) as a **root systemd
unit** (`scripts/frank-zealot.service`, `relink_unit`ed into
`/etc/systemd/system/`), serving `prism-ml/Ternary-Bonsai-2-9B-gguf` at
**PQ2_0 with the MTP head IN the file** (2,675,410,720 B, Hub-LFS sha256
`c79fcdfa…` verified at fetch) **plus the publisher's own Q8_0 mmproj**
(624,229,664 B, sha256 `e0cb6f69…`), both `/home/dank/models/Bonsai2-9B-MTP/`,
on port 7771. Cutover 2026-10-18 from the Bonsai-2-**27B** PQ2_0 era
(2 × 200,704, same card, same fork family).
**The runtime is part of the payload identity.** The file is
`general.file_type 141` (PQ2_0, 2.13 bpw, group 128) with **402 tensors of the
custom GGML type 142** plus `prism.hadamard.*` metadata (normalized
Sylvester–Walsh transform, block 1024, 401 named weights, inverse on
`token_embd`, `gdn_v_grouped`). Stock llama.cpp refuses type 142 at load, and a
loader that ignored the Hadamard metadata would dequant wrongly — same lock class
as the ROCmFPX era. Retargeting the payload without the fork ships an endpoint
that cannot load, so the seam pins both paths.
**The runtime is part of the payload identity — twice over.** The file is
`general.file_type 141` — **202 of the 442 tensors carry the fork's PQ2_0 type**
(the rest are f32 norms/scalars plus 8 q8_0 in the MTP block) — beside
`prism.hadamard.*` metadata (normalized Sylvester–Walsh transform): stock
llama.cpp refuses the custom type at load, and a loader that
ignored the Hadamard metadata would dequant WRONG — same lock class as the
ROCmFPX era. On top of that lock, **the fork gates MTP beside a quantized V
cache on build `prism-b10776-d00e252` or newer**, which is why the prebuilt
moved 10754 → 10777; the 10754 tree stays on disk as the 27B rollback runtime.
Retargeting the payload without the fork ships an endpoint that cannot load,
so the seam pins both paths.
**Architecture: qwen35 hybrid** — read from the GGUF's own header: **64
blocks**, `full_attention_interval = 4` ⇒ **16 full-attention + 48 Gated
DeltaNet layers**, 24 attention heads / **4 KV heads** × 256 dim, embedding
5,120, ffn 17,408, `ssm` conv_kernel 4 / state_size 128 / group_count 16 /
time_step_rank 48 / inner_size 6,144, native 262,144, vocab 248,320. **KV
accounting:** the pool scales with the 16 full layers — 16 × 4 × 256 = 16,384
elements/token per cache, so **q8_0 K + q8_0 V bills 34 KiB/token** = 13,328
MiB at the 401,408 pool (ledger-confirmed to the MiB), while **f16 KV needs
25,088 MiB and cannot boot on a 24,123 MiB card** (measured `cudaMalloc` OOM).
The 48 DeltaNet layers bill a fixed recurrent state per SLOT — ~150 MiB each —
so pool arithmetic alone under-prices the shape.
**Architecture: qwen35 hybrid** — read from the GGUF's own header: `block_count`
**33 = 32 trunk layers + the `blk.32` MTP block** (the MTP block has its own
attention + FFN in q8_0, which is why the draft context bills real KV), and
`qwen35.full_attention_interval = 4` ⇒ **8 full-attention + 24 Gated
DeltaNet**, 4 KV heads × 256 dim, native 262,144. **KV accounting:** the pool
scales with the 8 full layers — 8 × 4 × 256 = 8,192 elements/token per cache,
so **q8_0 K + q8_0 V bills 17,408 B/token ≈ 17 KiB** = **13,056.00 MiB** at the
786,432 pool (ledger-exact) — exactly HALF the 27B's 34,816 B/token, which is
the whole reason three native windows fit where the 27B afforded only two
shortened ones. **f16 K would bill 26,112 MiB and cannot boot on a 24,123 MiB
card.** The 24 DeltaNet layers bill a fixed recurrent state per SLOT —
`RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)` ≈ 150 MiB each — so
pool arithmetic alone under-prices the shape.
**The shape (2026-10-17):** `--ctx-size 401408 --parallel 2` = TWO
conversations of 200,704 (Frank's cap; the harness `contextWindow` quotes
the per-slot number), `--flash-attn on`, **q8_0 K + q8_0 V**, batch 4096 /
**ubatch 2048** (the measured CUDA optimum carried from the MiMo/Swift A/Bs on
this card), `--log-verbosity 4`, `--mmproj` for vision, **speculative decoding
OFF** (`SPEC_TYPE=none`). Boot: **22,664 MiB used / 1,461 free**; the
projection (21,439) excludes the projector's 873.10 MiB worst-case, which
lands AFTER the fit check — read settled `nvidia-smi`, never the projection.
Measured: 76.6 t/s single-stream decode (the 9B ran 94.4), 1,258 t/s prefill
at 35k, 691.7 t/s at 166k, 109.4 t/s aggregate across two concurrent streams.
The **first request after boot is a warmup artifact** (2.6 t/s prefill / 37.7
t/s decode measured) — never report it.
**The shape (2026-10-18):** `--ctx-size 786432 --parallel 3` = THREE
conversations of 262,144, every one at the model's TRAINED window (Frank: "if
you can possibly fit a third context, do it"), `--flash-attn on`, **q8_0 K +
q8_0 V**, batch 4096 / **ubatch 2048** (the measured CUDA optimum carried from
the MiMo/Swift A/Bs on this card; it also sets the compute-buffer size, which
is what decides whether MTP fits), `--log-verbosity 4`, `--metrics`,
`--jinja`, `--mmproj` for vision, **speculative decoding OFF**
(`SPEC_TYPE=none`). Boot: **17,876 MiB used / 6,249 free**; compute 1,409 MiB
CUDA + 1,201 MiB CUDA_Host, projector 595 + 248 MiB. Measured on the same
boot: **167.2 t/s** greedy decode, 158.9 t/s single-stream chat, **280.2 t/s
aggregate across three concurrent streams**, **2,877 t/s prefill at 106,418**
tokens — the 27B record (76.6 decode / 109.4 aggregate / 691.7 prefill at 166k)
beaten on every axis. The **first request after boot is a warmup artifact**
(37.7 t/s class) — never report it.
**Why MTP is off**: the PQ2_0 export carries no MTP head (upstream ships a
separate `…-CRACK-MTP-GGUF`, +451 MB, for that), and MTP is a property of the
file, not of the flags.
**Why MTP is off even though the head is in the file.** The drafter is GOOD —
acceptance 0.798 / 0.617 per position, mean accepted length 2.41 — and the card
recommends depth 2. It still loses AT THIS SHAPE because the draft context
bills its OWN **f16** KV over the WHOLE pool (4,096 B/position × 786,432 =
**3,072.00 MiB**, 4,830 MiB with its compute buffers) and ubatch 2048 refuses to
give it up. Booted arms, same card, same traffic (2026-10-18):
**Vision**: the publisher's own Q8_0 mmproj via `--mmproj` (Qwen3-VL tower: 27
blocks, 768 px, patch 16, merge 2, `qwen3vl_merger`, deepstack). Probe traps
carry over: ≥256 px probe images or the model answers "video", and only the
OpenAI `image_url` content part reaches the model. **New trap:** thinking
tokens count against `max_tokens`, so a vision probe at 64 tokens returns
EMPTY `content` with `finish_reason: "length"` — budget ≥512 or the endpoint
looks blind while it is only mid-thought.
| slots | spec | ubatch | used / free | decode |
|---|---|---|---|---|
| 3 × 262,144 | off | 2048 | 17,876 / 6,249 | **167.2** |
| 3 × 262,144 | on | 2048 | 22,706 / 1,419 | 109.2 |
| 3 × 262,144 | on | 1024 | 21,400 / 2,725 | **213.5** |
| 2 × 262,144 | on | 2048 | 17,178 / 6,947 | 213.2 |
| 1 × 262,144 | on | 2048 | 11,652 / 12,473 | 214.2 |
**Base alignment**: Bonsai-2 is the censored BASE lineage (the 9B it replaced
was uncensored). The CRACK re-export on disk
(`/home/dank/models/Bonsai2-CRACK/`, header census proves byte-identical
geometry) is a one-line `MODEL_FILE` swap if a refusal ever surfaces.
**MTP is worth +28 % decode whenever the card has room**; the squeeze, not the
drafter, is what halves it at three windows. The arm that WOULD run it at three
windows is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` (213.5 t/s single-stream,
250.8 three-stream aggregate) and pays prefill −17 % and 2.7 GiB of headroom on
a card whose projector worst-case lands AFTER the engine's own fit check. This
endpoint is the fleet's compaction hub — prefill-bound, 2–3 concurrent
subagents — so the shipped arm takes the +17 % prefill (2,877 vs 2,395 t/s) and
the +12 % three-stream aggregate (280.2 vs 250.8) over single-stream speed, and
keeps 6.2 GiB free. `SPEC_TYPE=none` strips the WHOLE spec set (`--spec-type`
and every `--spec-draft-*`) — flags move as a set or the arm measures a crash
instead of an absence.
**Sampler**: the recipe lives in the checkpoint (`general.sampling.*` =
temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0,
confirmed in the engine's own `/props`) and llama.cpp reads it itself.
**Presence penalty is the one recipe value with no GGUF metadata key**, so
`PRESENCE_PENALTY=1.0` must reach the argv; `""` is the documented opt-out, and
the serve script therefore uses `${PRESENCE_PENALTY=1.0}` (NOT `:=`, which
would silently re-add the flag on an explicit empty value).
**Vision**: the publisher's own **Q8_0** mmproj via `--mmproj` — Q8_0 is
MANDATORY, not a preference: the card says the BF16 tower "can crash
llama-server when several image requests run at once", and `--parallel 3` is
exactly that shape. Probe traps carry over: ≥256 px probe images or the model
answers "video", only the OpenAI `image_url` content part reaches the model,
and thinking tokens count against `max_tokens` — budget ≥512 or the probe
returns EMPTY `content` with `finish_reason: "length"` and the endpoint looks
blind while it is only mid-thought.
**Base alignment**: Bonsai-2 is the censored BASE lineage. Unlike the 27B, the
CRACK re-export on disk (`/home/dank/models/Bonsai2-CRACK/`) is byte-identical
geometry to the 27B it matched, NOT to this 9B file — if a refusal ever
surfaces there is no one-line `MODEL_FILE` swap; a 9B dealigned export would
have to be fetched first.
**Compaction hub**: `models.yml` routes `compactionModel` for
`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here — 200,704 is the
only window in the fleet that fits a worst-case compaction input (~170k
tokens). With `--parallel 2` a compaction request no longer serializes behind
a live generation (the Gemma4 era's `--parallel 1` law). Retrieval through a
`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here —
262,144 (since the 2026-10-18 reslice) is the widest window in the fleet and
the only one that comfortably fits a worst-case compaction input (~170k
tokens). With `--parallel 3` a compaction request no longer serializes behind a
live generation (the Gemma4 era's `--parallel 1` law). Retrieval through a
25,960-token prompt was verified exact after the swap.
**Contract law learned at verbosity 4:** a raw `/completion` request prints
@@ -533,11 +577,13 @@ model closes a bare prompt with `` — cosmetic
(`probe_request.cpp:789`); the `/v1/chat/completions` path parses
`reasoning_content` correctly, which is the only path consumers use. Do not
lower `VERBOSITY` to silence it: the KV/buffer ledger is what VRAM forensics
read.
read. `--metrics` is required or the Prometheus scrape reads 501 = DOWN.
Runbook:
`serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md`.
Rollback: revert the swap commit — the Qwen3.5-9B Q6_K payload
`serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md`.
Rollback: revert the swap commit — the 27B payload
(`/home/dank/models/Bonsai2-PQ2_0/`) with the 10754 fork tree
(`~/GitPackages/llama-prism/`), the Qwen3.5-9B Q6_K payload
(`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`) with the master CUDA build
(`~/GitPackages/llama-master/build-cuda/`), the Gemma4 payload
(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload
@@ -960,6 +1006,21 @@ linked runbooks.
trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
36:29), which is the practical size of the effect. See
[[frank-medium-llamacpp-dflash2]].
- **A draft context bills its own f16 KV over the WHOLE POOL, and on a
single-card endpoint that squeeze can cost more than the drafter wins.** The
draft context is built at the target's `n_ctx`, so with `--parallel N` the
drafter pays the POOL, not the slot: on frank-zealot's 786,432-position
qwen35 pool, `--spec-type draft-mtp` added **3,072.00 MiB of f16 draft KV**
(4,096 B/position — the draft cache is never quantized) plus its own compute
buffers, 4,830 MiB all told. Booted arms, same card, same traffic
(2026-10-18): 3 slots + ubatch 2048 → **167.2 t/s decode with spec OFF,
109.2 with spec ON**; dropping `UBATCH_SIZE` to 1024 hands the buffers back
and spec ON gives **213.5** (+28 % over the spec-OFF arm), and at 1–2 slots
the SAME card runs MTP at ubatch 2048 and still gains ~28 %. So the MTP
question is never "does the file carry the head" — it is "what do draft KV
+ buffers cost against whatever the batch knobs keep for compute buffers",
and that answer is per-shape. Measure the arm, never the flag. See
[[frank-zealot-llama-prism-bonsai9-mtp]].
- **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau —
and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its
compute buffers land about TWICE as heavy on the card that holds the output
@@ -1185,15 +1246,16 @@ Active:
| Service | Runbook |
|---------|---------|
| frank-dragoon (flash-server 0.16.2, NATIVE v2 abliterated .hgn + MTP always-on, 2 slots over ONE 401,408 pool at ctx 200704, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md` |
| frank-zealot (llama.cpp PrismML fork, Bonsai-2-27B PQ2_0, 2×200,704 q8_0 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md` |
| frank-zealot (llama.cpp PrismML fork build 10777, Bonsai-2-**9B** PQ2_0 **with the MTP head**, 3×262,144 q8_0 KV, MTP off by measurement, vision ON via Q8_0 mmproj, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md` |
| frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18, was `amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
Retired (rollback targets and history):
| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
| llama.cpp PrismML fork build 10754, Ternary-Bonsai-2-**27B** PQ2_0, 2×200,704 q8_0 K+V, MTP off (no head in the file), vision ON (`frank-zealot`) | Dankclaw | 2026-10-17 → 10-18 (rollback target — payload + 10754 fork tree on disk; owns the 34,816 B/token 16-full-layer KV law, the 25,088 MiB f16 OOM wall and the two-boot VRAM pricing) | `frank-zealot-llama-prism-bonsai2-pq20.md` |
| radiance Swift1.5-Qwen3.8-Flash-Next GPTQ w4dn8, 3×200,000 fp8-KV slab 25600 (`frank-archon`) | Dankserver | 2026-10-08 → 10-15 (rollback target — `.rad` purged 2026-10-17, byte-exact re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`; owns the slab-slope ladder, KV dtype law, KLD campaign, and every shape law the iq4r swap carried verbatim) | `frank-archon-radiance-swift15-w4dn8.md` |
| llama.cpp CUDA (master) Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-17 (rollback target — payload + master CUDA build on disk; owns the 32 KiB/token 8-full-layer KV law and the 2→3-slot reslice law) | `frank-zealot-llama-qwen35-9b-q6k-2x200704.md` |
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-zealot-llama-gemma4-q4km-200704.md` |