feat(frank-zealot): Bonsai-2-9B PQ2_0 with the MTP head at 3 x 262,144

Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.

MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.

Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.

contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
This commit is contained in:
Frank
2026-10-18 13:15:18 -07:00
parent 2b127163c8
commit 466fd477d8
14 changed files with 1001 additions and 314 deletions
+68 -27
View File
@@ -11,15 +11,19 @@ TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
froggeric v22.5 template. Rollback: the in-house Swift1.5 w4dn8 `.rad` (on
disk); tags frank-archon-freetoken-pr587-20261005 and
frank-archon-vllm-shapeG-20261003),
froggeric v22.5 template. Rollback: revert the swap commit and re-download the
in-house Swift1.5 w4dn8 `.rad` byte-exact (purged from disk 2026-10-17; Hub
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`); tags
frank-archon-freetoken-pr587-20261005 and frank-archon-vllm-shapeG-20261003
(AGENTS.md Runbook Index; `find /home/dank/models -name '*.rad'` on
Dankserver 2026-10-18 lists only the iq4r file),
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
digest serving the NATIVE v2 conversion of the abliterated
Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine,
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
401,408-position KV pool at ctx 200704, vision ON)** and
200,704-position KV pool at ctx 200704 — the 2 × ctx = 401,408 raise is
committed 2026-10-18 but NOT yet deployed, vision ON)** and
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48
Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled,
@@ -54,7 +58,7 @@ gfx1151 iGPU 114 GiB GTT]
E[flash-server container gfx1151, digest-pinned image
frank-dragoon, port 7770]
F[flash-server v2 native abliterated .hgn + ngram table + vision tower
ONE 401408-position KV pool at ctx 200704, 2 slots, MTP always-on, image input]
ONE 200704-position KV pool at ctx 200704 (401408 committed, awaiting deploy), 2 slots, MTP always-on, image input]
G[llama.cpp PrismML fork CUDA build
frank-zealot, port 7771]
H[Ternary-Bonsai-2-27B PQ2_0 GGUF + Q8_0 mmproj
@@ -91,7 +95,11 @@ frank-zealot, port 7771]
**one pool behind two slots**, sized so TWO maximum-length requests coexist —
raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total
in-flight context, ~10.8 GiB reserved against the card's 262,144/262,144 =
7.2 GiB. The RAM comes out of the n-gram table's page cache, not out of a
7.2 GiB. **Deploy state 2026-10-18: the running container and the
host-deployed env still carry the 200,704-position pool** (boot ledger
`reserving the KV pool, 200704 positions (about 5.5 GiB)`; host
`frank-dragoon.env` `HALOGEN_KV_POOL_POSITIONS=200704`) — the doubling lands
on the next redeploy (commit `2b12716`).
longer window: the per-request ceiling stays 200,704, because 262,144 is the
native wall and the engine refuses more without `HALOGEN_ROPE_YARN`.
MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
@@ -120,17 +128,41 @@ frank-zealot, port 7771]
(`slow`/`advisor` → frank-archon) carried forward. Superseded by the
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
engine moved with the model
- [[frank-zealot-llama-prism-bonsai2-pq20]] — **ACTIVE since 2026-10-17**
- [[frank-zealot-llama-prism-bonsai9-mtp]] — **ACTIVE since 2026-10-18**
(root systemd unit, port 7771): the **PrismML fork** build of llama.cpp
(vendor 10754, commit `2459f68b5` — the only loader for the payload's custom
tensor type 142 + `prism.hadamard.*` transform) serving
prism-ml/Ternary-Bonsai-2-27B-gguf **PQ2_0** (2.13 bpw) on the RTX 3090 —
TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), **q8_0 K
+ q8_0 V** (34 KiB/token over the 16 full layers = 13,328 MiB; the f16 pair
needs 25,088 MiB and dies), settled 22,664 used / 1,461 free, vision via the
publisher's Q8_0 mmproj (+873 MiB, charged AFTER the engine's fit check), MTP
OFF (no MTP tensors). 76.6 t/s single-stream decode, 691.7 t/s prefill at
166k, 109.4 t/s aggregate at two streams. Supersedes the Qwen3.5-9B era below.
(vendor **10777**, commit `e3de71522` — the only loader for the payload's
custom PQ2_0 tensor type + `prism.hadamard.*` transform, and since this swap
also the minimum build that pairs MTP with a quantized V cache) serving
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head IN the file** on
the RTX 3090 — **THREE 262,144-token conversations**
(`--ctx-size 786432 --parallel 3`, every slot at the window the model was
trained at), **q8_0 K + q8_0 V** (17 KiB/token over the 8 full layers =
13,056 MiB ledger-exact; the f16 pair needs 26,112 MiB and dies at
`cudaMalloc`), settled **17,876 used / 6,249 free**, vision via the
publisher's **Q8_0** mmproj (Q8_0 is the publisher's requirement for
concurrent image requests, and this runs three slots), **MTP OFF by
measurement** — the draft context bills its own f16 KV over the WHOLE pool
(3,072 MiB) and at `UBATCH_SIZE=2048` that squeeze halves decode
(167.2 → 109.2 t/s); `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the
documented 213.5 t/s arm, declined for +17 % prefill and 6.2 GiB of headroom.
Measured: 167.2 t/s single-stream decode, 2,877 t/s prefill at 106,418,
280.2 t/s aggregate across three streams. `--metrics` rides (`METRICS=1`):
llama.cpp answers 501 on `/metrics` without it, which is why Prometheus
`up{job="frank-zealot"}` read 0 for its whole retained history until
2026-10-18. `contextWindow: 262144` is the **widest in the fleet**, so this
is the compaction hub. The `--presence-penalty` opt-out law (`${VAR=}`, not
`${VAR:=}`) and the `TimeoutStopSec=180` stop-drain law were both earned here.
- [[frank-zealot-llama-prism-bonsai2-pq20]] — **RETIRED 2026-10-17 → 10-18**
(payload + fork build 10754 tree on disk as rollback): the first PrismML-fork
era — prism-ml/Ternary-Bonsai-2-**27B**-gguf **PQ2_0**, TWO 200,704-token
conversations (`--ctx-size 401408 --parallel 2`), q8_0 K + q8_0 V
(34,816 B/token over the 16 full layers = 13,328 MiB; f16 needs 25,088 MiB and
dies), settled 22,664 used / 1,461 free, MTP **impossible** (that export
carries no head), 76.6 t/s decode, 691.7 t/s prefill at 166k, 109.4 t/s
aggregate at two streams. Owns the 34 KiB/token KV law, the f16 OOM wall, the
two-boot VRAM pricing, and the record of the 2026-10-18 bench window
(production unit stopped 03:38, hand-booted 9B candidate answering the port)
that closed with this swap.
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **RETIRED 2026-10-07 →
10-17** (payload + master CUDA build on disk as rollback): llama.cpp master
(CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive
@@ -197,10 +229,13 @@ frank-zealot, port 7771]
dankserver 2026-10-15 (Frank's one-time rm, volume for this payload;
rebuild via the fetch + build scripts named in the fp8-native runbook)
- [[frank-archon-radiance-swift15-w4dn8]] — **RETIRED 2026-10-08 → 10-15
(superseded by the publisher iq4r swap; payload on disk as the rollback)**:
(superseded by the publisher iq4r swap; the `.rad` was PURGED from disk
2026-10-17 — rollback = revert + byte-exact re-download of
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)**:
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
the FP8-native payload remains the rollback. **2026-10-13: fp8 KV, and the
the FP8-native payload was the second rollback until it too was deleted
2026-10-15. **2026-10-13: fp8 KV, and the
pair re-cut twice** — `--kv-cache-dtype fp8` first at
`--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
@@ -219,7 +254,9 @@ frank-zealot, port 7771]
reset the knob to its default (125,988 MiB) and the endpoint refused to boot
6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)**
w4dn8 swap replaced it; its `.rad` was DELETED from disk 2026-10-15 —
rebuild = `scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`,
`scripts/frank-archon.env:59-62`)**
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
@@ -248,8 +285,10 @@ frank-zealot, port 7771]
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
not boot on ROCm (pynccl JIT) — second card idle. Superseded by the
radiance swap for TP=2; the image + payload stay on disk until the
rollback window closes
radiance swap for TP=2. Disk state 2026-10-18: the image is still in
dankserver's rootless storage (`localhost/frank-primary-freetoken:pr587-77a7201`,
29.7 GB — the pre-rename name); the orcarouter FP8 payload is NO LONGER on
disk (no `*orcarouter*` path under /home/dank, `find` 2026-10-18)
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
(rollback target; the vLLM trio + env lore live in the tag)**: the
tcclaviger RDNA4
@@ -278,12 +317,14 @@ frank-zealot, port 7771]
35 tok/s target is NOT met, bounded by the fork's expert-gather read path
(~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
back; payload and R9V state dirs intact on disk, git tag
[[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
back; the R9V state dirs are on disk (`/home/dank/r9v-state`,
`~/.local/state/r9v`) but the payload GGUFs are not — `/home/dank/models`
holds only the iq4r `.rad` as of 2026-10-18; git tag
frank-archon-r9v-q8-20260929 is the last pre-cutover commit):
[[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
(same-day swap to the Q8_0 payload; rollback target, state + payload
intact on disk): the R9V radiance-vLLM fork serving
(same-day swap to the Q8_0 payload; state + payloads no longer on disk
2026-10-18, only the R9V state dirs remain): the R9V radiance-vLLM fork serving
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
@@ -505,8 +546,8 @@ frank-zealot, port 7771]
## CIRU Reference (Dankclaw)
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
and build history (on Dankclaw; the slot now serves the kingjones
ROCmFP4 model, see [[kingjones-strix-mtp]])
and build history (on Dankclaw; the gfx1151 slot now serves frank-dragoon's
flash-server v2-native payload — see [[frank-dragoon-flash-server-v2-native]])
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
identity, and experiment summary (retired 2026-08-22)