feat(frank-zealot): Bonsai-2-9B PQ2_0 with the MTP head at 3 x 262,144
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.
MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.
Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.
contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
This commit is contained in:
@@ -11,15 +11,19 @@ TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
|
||||
(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
|
||||
30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
|
||||
table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
|
||||
froggeric v22.5 template. Rollback: the in-house Swift1.5 w4dn8 `.rad` (on
|
||||
disk); tags frank-archon-freetoken-pr587-20261005 and
|
||||
frank-archon-vllm-shapeG-20261003),
|
||||
froggeric v22.5 template. Rollback: revert the swap commit and re-download the
|
||||
in-house Swift1.5 w4dn8 `.rad` byte-exact (purged from disk 2026-10-17; Hub
|
||||
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`); tags
|
||||
frank-archon-freetoken-pr587-20261005 and frank-archon-vllm-shapeG-20261003
|
||||
(AGENTS.md Runbook Index; `find /home/dank/models -name '*.rad'` on
|
||||
Dankserver 2026-10-18 lists only the iq4r file),
|
||||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||||
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||||
digest serving the NATIVE v2 conversion of the abliterated
|
||||
Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine,
|
||||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
|
||||
401,408-position KV pool at ctx 200704, vision ON)** and
|
||||
200,704-position KV pool at ctx 200704 — the 2 × ctx = 401,408 raise is
|
||||
committed 2026-10-18 but NOT yet deployed, vision ON)** and
|
||||
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
|
||||
Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48
|
||||
Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled,
|
||||
@@ -54,7 +58,7 @@ gfx1151 iGPU 114 GiB GTT]
|
||||
E[flash-server container gfx1151, digest-pinned image
|
||||
frank-dragoon, port 7770]
|
||||
F[flash-server v2 native abliterated .hgn + ngram table + vision tower
|
||||
ONE 401408-position KV pool at ctx 200704, 2 slots, MTP always-on, image input]
|
||||
ONE 200704-position KV pool at ctx 200704 (401408 committed, awaiting deploy), 2 slots, MTP always-on, image input]
|
||||
G[llama.cpp PrismML fork CUDA build
|
||||
frank-zealot, port 7771]
|
||||
H[Ternary-Bonsai-2-27B PQ2_0 GGUF + Q8_0 mmproj
|
||||
@@ -91,7 +95,11 @@ frank-zealot, port 7771]
|
||||
**one pool behind two slots**, sized so TWO maximum-length requests coexist —
|
||||
raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total
|
||||
in-flight context, ~10.8 GiB reserved against the card's 262,144/262,144 =
|
||||
7.2 GiB. The RAM comes out of the n-gram table's page cache, not out of a
|
||||
7.2 GiB. **Deploy state 2026-10-18: the running container and the
|
||||
host-deployed env still carry the 200,704-position pool** (boot ledger
|
||||
`reserving the KV pool, 200704 positions (about 5.5 GiB)`; host
|
||||
`frank-dragoon.env` `HALOGEN_KV_POOL_POSITIONS=200704`) — the doubling lands
|
||||
on the next redeploy (commit `2b12716`).
|
||||
longer window: the per-request ceiling stays 200,704, because 262,144 is the
|
||||
native wall and the engine refuses more without `HALOGEN_ROPE_YARN`.
|
||||
MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
|
||||
@@ -120,17 +128,41 @@ frank-zealot, port 7771]
|
||||
(`slow`/`advisor` → frank-archon) carried forward. Superseded by the
|
||||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||||
engine moved with the model
|
||||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — **ACTIVE since 2026-10-17**
|
||||
- [[frank-zealot-llama-prism-bonsai9-mtp]] — **ACTIVE since 2026-10-18**
|
||||
(root systemd unit, port 7771): the **PrismML fork** build of llama.cpp
|
||||
(vendor 10754, commit `2459f68b5` — the only loader for the payload's custom
|
||||
tensor type 142 + `prism.hadamard.*` transform) serving
|
||||
prism-ml/Ternary-Bonsai-2-27B-gguf **PQ2_0** (2.13 bpw) on the RTX 3090 —
|
||||
TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), **q8_0 K
|
||||
+ q8_0 V** (34 KiB/token over the 16 full layers = 13,328 MiB; the f16 pair
|
||||
needs 25,088 MiB and dies), settled 22,664 used / 1,461 free, vision via the
|
||||
publisher's Q8_0 mmproj (+873 MiB, charged AFTER the engine's fit check), MTP
|
||||
OFF (no MTP tensors). 76.6 t/s single-stream decode, 691.7 t/s prefill at
|
||||
166k, 109.4 t/s aggregate at two streams. Supersedes the Qwen3.5-9B era below.
|
||||
(vendor **10777**, commit `e3de71522` — the only loader for the payload's
|
||||
custom PQ2_0 tensor type + `prism.hadamard.*` transform, and since this swap
|
||||
also the minimum build that pairs MTP with a quantized V cache) serving
|
||||
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head IN the file** on
|
||||
the RTX 3090 — **THREE 262,144-token conversations**
|
||||
(`--ctx-size 786432 --parallel 3`, every slot at the window the model was
|
||||
trained at), **q8_0 K + q8_0 V** (17 KiB/token over the 8 full layers =
|
||||
13,056 MiB ledger-exact; the f16 pair needs 26,112 MiB and dies at
|
||||
`cudaMalloc`), settled **17,876 used / 6,249 free**, vision via the
|
||||
publisher's **Q8_0** mmproj (Q8_0 is the publisher's requirement for
|
||||
concurrent image requests, and this runs three slots), **MTP OFF by
|
||||
measurement** — the draft context bills its own f16 KV over the WHOLE pool
|
||||
(3,072 MiB) and at `UBATCH_SIZE=2048` that squeeze halves decode
|
||||
(167.2 → 109.2 t/s); `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the
|
||||
documented 213.5 t/s arm, declined for +17 % prefill and 6.2 GiB of headroom.
|
||||
Measured: 167.2 t/s single-stream decode, 2,877 t/s prefill at 106,418,
|
||||
280.2 t/s aggregate across three streams. `--metrics` rides (`METRICS=1`):
|
||||
llama.cpp answers 501 on `/metrics` without it, which is why Prometheus
|
||||
`up{job="frank-zealot"}` read 0 for its whole retained history until
|
||||
2026-10-18. `contextWindow: 262144` is the **widest in the fleet**, so this
|
||||
is the compaction hub. The `--presence-penalty` opt-out law (`${VAR=}`, not
|
||||
`${VAR:=}`) and the `TimeoutStopSec=180` stop-drain law were both earned here.
|
||||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — **RETIRED 2026-10-17 → 10-18**
|
||||
(payload + fork build 10754 tree on disk as rollback): the first PrismML-fork
|
||||
era — prism-ml/Ternary-Bonsai-2-**27B**-gguf **PQ2_0**, TWO 200,704-token
|
||||
conversations (`--ctx-size 401408 --parallel 2`), q8_0 K + q8_0 V
|
||||
(34,816 B/token over the 16 full layers = 13,328 MiB; f16 needs 25,088 MiB and
|
||||
dies), settled 22,664 used / 1,461 free, MTP **impossible** (that export
|
||||
carries no head), 76.6 t/s decode, 691.7 t/s prefill at 166k, 109.4 t/s
|
||||
aggregate at two streams. Owns the 34 KiB/token KV law, the f16 OOM wall, the
|
||||
two-boot VRAM pricing, and the record of the 2026-10-18 bench window
|
||||
(production unit stopped 03:38, hand-booted 9B candidate answering the port)
|
||||
that closed with this swap.
|
||||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **RETIRED 2026-10-07 →
|
||||
10-17** (payload + master CUDA build on disk as rollback): llama.cpp master
|
||||
(CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive
|
||||
@@ -197,10 +229,13 @@ frank-zealot, port 7771]
|
||||
dankserver 2026-10-15 (Frank's one-time rm, volume for this payload;
|
||||
rebuild via the fetch + build scripts named in the fp8-native runbook)
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **RETIRED 2026-10-08 → 10-15
|
||||
(superseded by the publisher iq4r swap; payload on disk as the rollback)**:
|
||||
(superseded by the publisher iq4r swap; the `.rad` was PURGED from disk
|
||||
2026-10-17 — rollback = revert + byte-exact re-download of
|
||||
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)**:
|
||||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
|
||||
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
|
||||
the FP8-native payload remains the rollback. **2026-10-13: fp8 KV, and the
|
||||
the FP8-native payload was the second rollback until it too was deleted
|
||||
2026-10-15. **2026-10-13: fp8 KV, and the
|
||||
pair re-cut twice** — `--kv-cache-dtype fp8` first at
|
||||
`--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
|
||||
with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
|
||||
@@ -219,7 +254,9 @@ frank-zealot, port 7771]
|
||||
reset the knob to its default (125,988 MiB) and the endpoint refused to boot
|
||||
6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
|
||||
w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)**
|
||||
w4dn8 swap replaced it; its `.rad` was DELETED from disk 2026-10-15 —
|
||||
rebuild = `scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`,
|
||||
`scripts/frank-archon.env:59-62`)**
|
||||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||||
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
|
||||
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
@@ -248,8 +285,10 @@ frank-zealot, port 7771]
|
||||
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
|
||||
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Superseded by the
|
||||
radiance swap for TP=2; the image + payload stay on disk until the
|
||||
rollback window closes
|
||||
radiance swap for TP=2. Disk state 2026-10-18: the image is still in
|
||||
dankserver's rootless storage (`localhost/frank-primary-freetoken:pr587-77a7201`,
|
||||
29.7 GB — the pre-rename name); the orcarouter FP8 payload is NO LONGER on
|
||||
disk (no `*orcarouter*` path under /home/dank, `find` 2026-10-18)
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
|
||||
(rollback target; the vLLM trio + env lore live in the tag)**: the
|
||||
tcclaviger RDNA4
|
||||
@@ -278,12 +317,14 @@ frank-zealot, port 7771]
|
||||
35 tok/s target is NOT met, bounded by the fork's expert-gather read path
|
||||
(~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
|
||||
Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
|
||||
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||||
back; payload and R9V state dirs intact on disk, git tag
|
||||
[[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||||
back; the R9V state dirs are on disk (`/home/dank/r9v-state`,
|
||||
`~/.local/state/r9v`) but the payload GGUFs are not — `/home/dank/models`
|
||||
holds only the iq4r `.rad` as of 2026-10-18; git tag
|
||||
frank-archon-r9v-q8-20260929 is the last pre-cutover commit):
|
||||
[[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
|
||||
(same-day swap to the Q8_0 payload; rollback target, state + payload
|
||||
intact on disk): the R9V radiance-vLLM fork serving
|
||||
(same-day swap to the Q8_0 payload; state + payloads no longer on disk
|
||||
2026-10-18, only the R9V state dirs remain): the R9V radiance-vLLM fork serving
|
||||
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
|
||||
blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
|
||||
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
|
||||
@@ -505,8 +546,8 @@ frank-zealot, port 7771]
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
||||
and build history (on Dankclaw; the slot now serves the kingjones
|
||||
ROCmFP4 model, see [[kingjones-strix-mtp]])
|
||||
and build history (on Dankclaw; the gfx1151 slot now serves frank-dragoon's
|
||||
flash-server v2-native payload — see [[frank-dragoon-flash-server-v2-native]])
|
||||
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
||||
identity, and experiment summary (retired 2026-08-22)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user