feat(frank-zealot): Bonsai-2-9B PQ2_0 with the MTP head at 3 x 262,144
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.
MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.
Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.
contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
This commit is contained in:
1 parent
2b127163c8
commit
466fd477d8
14 files changed
+1001
-314
No files matched your search
@@ -2108,3 +2108,52 @@ every restart since the cutover.
|
||||
satisfied, endpoint still absent), AGENTS.md key lesson, both index pages.
|
||||
bats 371/371. frank-dragoon and frank-zealot were live the whole window; only
|
||||
the reboot interrupted them (~40 s to listen, health 200).
|
||||
|
||||
## 2026-10-18 — frank-zealot: Bonsai-2-9B PQ2_0 with the MTP head (3 × 262,144)
|
||||
|
||||
- Swapped `prism-ml/Ternary-Bonsai-2-27B-gguf` → **`Ternary-Bonsai-2-9B-gguf`**
|
||||
at PQ2_0 (`Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2,675,410,720 B, Hub-LFS
|
||||
sha256 `c79fcdfaf6b7…` verified at fetch) plus the publisher's **Q8_0** mmproj
|
||||
(624,229,664 B, `e0cb6f69…`), into `/home/dank/models/Bonsai2-9B-MTP/`; the
|
||||
runtime moved to vendor fork build **10777** (`e3de71522`) because the MTP
|
||||
arm needs the build that pairs MTP with a quantized V cache.
|
||||
- The win is the **window, not the quant**: `--ctx-size 786432 --parallel 3`
|
||||
puts every conversation back at the trained 262,144 (the 27B era sold each
|
||||
slot 200,704), and the fleet's widest context makes :7771 the **compaction
|
||||
hub** — `models.yml` `contextWindow` 200704 → 262144. Measured vs the 27B:
|
||||
decode 167.2 vs 76.6 t/s single-stream, aggregate 280.2 vs 109.4 t/s across
|
||||
the (now three) concurrent streams, prefill 2,877 t/s at 106,418 depth.
|
||||
- **MTP armed, then OFF by measurement.** The head is in the file, the fork
|
||||
loads it, and the draft context bills its own **f16** KV over the WHOLE pool
|
||||
— 3,072 MiB at 786,432 positions, ledger-confirmed. At `UBATCH_SIZE=2048`
|
||||
that squeeze **halved decode (167.2 → 109.2 t/s)**. The documented MTP shape
|
||||
is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s; declined, because
|
||||
+17 % decode is not worth spending 6.2 GiB of the 6,249 MiB free. The arm is
|
||||
recorded in the env header, not left to rediscovery.
|
||||
- Two bugs the trial caught, both fixed in the repo: `${SPEC_PRESENCE_PENALTY:=0.0}`
|
||||
was a *default-assignment*, so the env value always won and the documented
|
||||
"MTP runs without `--presence-penalty`" arm was unreachable — now `${VAR=0.0}`,
|
||||
which lets `SPEC_TYPE=draft-mtp` drop the flag. And `up{job="frank-zealot"}`
|
||||
had read 0 for the **entire retained Prometheus history**: llama.cpp answers
|
||||
**501** on `/metrics` unless `--metrics` is passed, so the scrape failed beside
|
||||
a healthy `/health` and `LlmEndpointDown` fired for days. `METRICS=1` is now
|
||||
the default and the seam pins it.
|
||||
- Boot ledger: KV `13,056.00 MiB` (8 full-attention layers × 4 KV heads × 256
|
||||
dim, q8_0 K + q8_0 V over 786,432 positions — ledger-exact), settled
|
||||
**17,876 used / 6,249 free**; the f16 pair would need 26,112 MiB and cannot
|
||||
boot this card. `TimeoutStopSec=180` carries over (SIGTERM ignored past the
|
||||
old default during the 03:38 stop).
|
||||
- Header census that corrected my own first pass: `block_count` is **33**
|
||||
(32 trunk + the `blk.32` MTP block, which carries a full attention + FFN in
|
||||
q8_0 — that is why the draft context bills real KV), **202 of 442** tensors
|
||||
carry the fork's PQ2_0 type (norms f32, MTP projections q8_0), `hadamard`
|
||||
`block_size` **4096** with `gdn_grouped=True`. The "2.13 bpw / type 142 / 32
|
||||
layers" figures inherited from the 27B page were wrong for this file and are
|
||||
corrected in AGENTS.md, the env header and the runbooks.
|
||||
- Docs: new runbook `LLM/frank-zealot-llama-prism-bonsai9-mtp.md`; the 27B page
|
||||
re-banned **RETIRED 2026-10-17 → 10-18** with its bench-window section closed;
|
||||
both index pages, the homepage card, `Dankclaw/OpenWebUI.md` and the live
|
||||
picker row (DB `description` + context length 262,144) re-keyed. Rollback:
|
||||
revert the swap commit — the 27B payload and the build-10754 fork tree are
|
||||
both still on disk.
|
||||
|
||||
Reference in new issue
Block a user