feat(frank-zealot): Bonsai-2-9B PQ2_0 with the MTP head at 3 x 262,144
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.
MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.
Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.
contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
This commit is contained in:
Frank
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork), Bonsai-2-27B PQ2_0, 2 x 200704, q8_0 KV (ACTIVE)"
|
||||
description: "The ACTIVE frank-zealot runbook (since 2026-10-17) — the PrismML fork build of llama.cpp serving prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090: two conversations of 200,704 tokens with q8_0 K + q8_0 V (f16 KV does not fit this card at this window), vision through the publisher's Q8_0 Qwen3-VL projector, MTP impossible (no tensors in the file), and the measured four-arm VRAM census that priced the shape."
|
||||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10754), Bonsai-2-27B PQ2_0, 2 x 200704, q8_0 KV (RETIRED, rollback target)"
|
||||
description: "The frank-zealot runbook of 2026-10-17 → 2026-10-18 (RETIRED — rollback target) — the PrismML fork build 10754 of llama.cpp serving prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090: two conversations of 200,704 tokens with q8_0 K + q8_0 V (f16 KV does not fit this card at this window), vision through the publisher's Q8_0 Qwen3-VL projector, MTP impossible (no tensors in the file), and the measured four-arm VRAM census that priced the shape. Superseded by the 9B MTP export at 3 × 262,144."
|
||||
tags:
|
||||
- llm
|
||||
- frank-zealot
|
||||
@@ -14,12 +14,41 @@ timestamp: 2026-10-17
|
||||
|
||||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-27B PQ2_0, 2 × 200,704, q8_0 KV
|
||||
|
||||
Live since 2026-10-17, replacing the
|
||||
[[frank-zealot-llama-qwen35-9b-q6k-2x200704]] era (Qwen3.5-9B Q6_K, exact f16
|
||||
KV, same 2 × 200,704 shape). The trade: the ternary 27B costs ~3.4× the weights
|
||||
of the 9B and buys a 3× larger, stronger base at the SAME window contract —
|
||||
paid for by quantizing KV (f16 KV is arithmetically impossible for this
|
||||
payload on this card, measured below).
|
||||
**RETIRED 2026-10-17 → 2026-10-18** (AGENTS.md Runbook Index), superseded by
|
||||
[[frank-zealot-llama-prism-bonsai9-mtp]] — prism-ml/Ternary-Bonsai-2-**9B**
|
||||
PQ2_0 **with the MTP head**, THREE conversations at the native 262,144 window
|
||||
on fork build 10777. This era stays the **rollback target**: the payload
|
||||
(`/home/dank/models/Bonsai2-PQ2_0/`) and the 10754 fork tree
|
||||
(`~/GitPackages/llama-prism/`) are both still on disk. It owns the laws below —
|
||||
the 34,816 B/token KV figure over 16 full-attention layers, the 25,088 MiB f16
|
||||
OOM wall, and the two-boot VRAM pricing that every later shape on this card
|
||||
reused.
|
||||
|
||||
## 2026-10-18 bench window — production unit stopped, candidate arm on 7771
|
||||
|
||||
As of 2026-10-18 frank-zealot sits in a deliberate, dated **bench window**
|
||||
(fleet Benchmarking Policy: stop the production unit, bench candidate boots
|
||||
against the service itself). The journal shows the stop at 03:38:31 and the
|
||||
known llama.cpp shutdown shape — SIGTERM ignored past `TimeoutStopSec`,
|
||||
`State 'stop-sigterm' timed out. Killing.` at 03:40:01 — after which the unit
|
||||
reports `failed` (`is-enabled` → enabled); self-healing is merely parked
|
||||
while the window runs. What answers 7771 is a **hand-booted candidate arm**
|
||||
(`llama-prism-b10777/llama-server` on the 9B MTP file
|
||||
`Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, alias
|
||||
`frank-zealot`), not this runbook's payload: candidate-arm flags are not a
|
||||
state — the pid and shape moved between two censuses ten minutes apart (one
|
||||
argv read `--ctx-size 786432 --parallel 3 --spec-type draft-mtp`; a later
|
||||
`/props` read `n_ctx 262144`, `speculative.types=none`); quote neither, and
|
||||
expect `/v1/models` to report the candidate until the window closes. The
|
||||
window's remedy is the redeploy: `./scripts/deploy.sh dankclaw --service
|
||||
frank-zealot`. The production trio below is unchanged — the last PRODUCTION
|
||||
boot under the unit (03:14–03:38 the same morning) printed the contract
|
||||
exactly: `n_ctx = 401408`, `n_ctx_seq = 200704`, `CUDA0 KV buffer size =
|
||||
13328.00 MiB`, 2 slots (journal).
|
||||
|
||||
**Window CLOSED 2026-10-18:** that candidate became production. The redeploy
|
||||
shipped the 9B MTP export at `3 × 262,144` on fork build 10777 — see
|
||||
[[frank-zealot-llama-prism-bonsai9-mtp]]; this payload is the rollback target.
|
||||
|
||||
## Identity
|
||||
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 3 × 262144, q8_0 KV (ACTIVE)"
|
||||
description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: THREE conversations of 262,144 tokens (every slot at the model's trained window) over q8_0 K + q8_0 V, vision through the publisher's Q8_0 Qwen3-VL projector, and the five-arm boot census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head."
|
||||
tags:
|
||||
- llm
|
||||
- frank-zealot
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
- rtx3090
|
||||
- mtp
|
||||
- spec-decode
|
||||
timestamp: 2026-10-18
|
||||
---
|
||||
|
||||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 3 × 262,144, q8_0 KV
|
||||
|
||||
Live since 2026-10-18, replacing the
|
||||
[[frank-zealot-llama-prism-bonsai2-pq20]] era (Ternary-Bonsai-2-**27B** PQ2_0 at
|
||||
2 × 200,704). The trade in one sentence: the 9B bills **half** the KV per token
|
||||
of the 27B, and that is exactly what funds **three** conversations at the
|
||||
model's **native 262,144** window where the 27B afforded only two shortened
|
||||
ones (Frank's ask: *"if you can possibly fit a third context, do it"*). Every
|
||||
axis improved — decode, prefill, three-stream aggregate, and free VRAM.
|
||||
|
||||
## Payload identity
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork** build, `--version` → `0.2.0-dev (build 10777, commit e3de71522)` |
|
||||
| Checkpoint | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` — 2,675,410,720 B, Hub-LFS sha256 `c79fcdfaf6b7bf47089b50dadcfd11f9ce7d3e96adf3bbefde7912e5ef69e22b` |
|
||||
| Vision | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, Hub-LFS sha256 `e0cb6f6988cea323ef798cd77d019bbb819af75582f034eaab0e7232314953db` |
|
||||
| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (`--verify-only`; `-f`, byte-count and LFS sha256 per file) |
|
||||
| Alias | `frank-zealot` (`ALIAS`); the SERVICE/unit/scripts keep the StarCraft name, only the wire id is the model id |
|
||||
| Port | 7771, root systemd unit `frank-zealot.service` (relinked), `TimeoutStopSec=180` |
|
||||
|
||||
### The runtime lock has two tumblers now
|
||||
|
||||
1. **The tensor type.** `general.file_type 141` with **202 tensors of the fork's
|
||||
PQ2_0 type** (of 442 total; the rest are f32 norms/scalars and 8 q8_0 in the
|
||||
MTP block) plus `prism.hadamard.*` metadata
|
||||
(`normalized-sylvester-walsh-hadamard`, `block_size 4096`,
|
||||
`axis input-last-dimension`, `sign_mode explicit` over 201 weight names,
|
||||
inverse on `token_embd.weight`, `gdn_v_grouped true`). Stock llama.cpp
|
||||
refuses the custom type at load; a loader that ignored the Hadamard metadata
|
||||
would dequant **wrong**, silently. Same lock class as the ROCmFPX era.
|
||||
2. **The build floor.** The fork's card gates **MTP beside a quantized V cache**
|
||||
on `prism-b10776-d00e252` or newer, so the prebuilt moved **10754 → 10777**
|
||||
for this swap. The 10754 tree stays on disk (`~/GitPackages/llama-prism/`) as
|
||||
the 27B rollback runtime. `tests/frank_zealot_unit.bats` pins both paths.
|
||||
|
||||
## Header census (read from the GGUF, not the card)
|
||||
|
||||
`general.architecture qwen35`, `general.name Ternary-Bonsai-2-9B`,
|
||||
`general.size_label 9B`, `general.quantization_version 2`,
|
||||
`general.file_type 141`:
|
||||
|
||||
| Key | Value | Note |
|
||||
|---|---|---|
|
||||
| `qwen35.block_count` | **33** | 32 trunk blocks + `blk.32` = the MTP block |
|
||||
| `qwen35.full_attention_interval` | 4 | ⇒ **8 full-attention + 24 Gated DeltaNet** trunk layers |
|
||||
| `qwen35.context_length` | 262,144 | native; the deployed slot equals it |
|
||||
| `qwen35.embedding_length` / `feed_forward_length` | 4,096 / 12,288 | |
|
||||
| `attention.head_count` / `head_count_kv` | 16 / **4** | |
|
||||
| `attention.key_length` / `value_length` | 256 / 256 | |
|
||||
| `ssm.conv_kernel` / `state_size` / `group_count` | 4 / 128 / 16 | the 27B: 4 / 128 / 16 |
|
||||
| `ssm.time_step_rank` / `inner_size` | **32 / 4,096** | the 27B: 48 / 6,144 |
|
||||
| `rope.freq_base` / `dimension_count` / `dimension_sections` | 1e7 / 64 / `[11,11,10,0]` | |
|
||||
| `qwen35.nextn_predict_layers` | **1** | the MTP head is IN the file |
|
||||
| `general.sampling.*` | temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / repeat 1.0 | the card recipe; llama.cpp reads it itself |
|
||||
|
||||
**KV accounting.** Only the 8 full-attention layers scale with context:
|
||||
8 × 4 KV heads × 256 dim = **8,192 elements/token per cache**, so
|
||||
**q8_0 K + q8_0 V bills 17,408 B/token** — the 786,432-position pool is
|
||||
**13,056.00 MiB**, ledger-exact. That is exactly HALF the 27B's 34,816 B/token.
|
||||
**f16 KV would bill 26,112 MiB against a 24,123 MiB card** and dies at
|
||||
`cudaMalloc`. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
|
||||
instead — `RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)` ≈ 150 MiB
|
||||
each — so pool arithmetic alone under-prices the shape.
|
||||
|
||||
## The shape (`scripts/frank-zealot.env`)
|
||||
|
||||
`--ctx-size 786432 --parallel 3` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`,
|
||||
`--cache-type-k q8_0 --cache-type-v q8_0`, `--batch-size 4096`,
|
||||
`--ubatch-size 2048`, `--n-gpu-layers 99`, `--jinja`, `--reasoning auto`,
|
||||
`--reasoning-format deepseek`, `--metrics`, `--mmproj …`, `--log-verbosity 4`,
|
||||
`--presence-penalty 1.0`,
|
||||
**`SPEC_TYPE=none`** (which strips `--spec-type` AND every `--spec-draft-*`).
|
||||
|
||||
Per-slot window is the **contract** the harness quotes: `models.yml` carries
|
||||
`contextWindow: 262144`. llama.cpp refuses a prompt longer than the SLOT with a
|
||||
400 naming the limit — it never truncates — and since this swap the slot window
|
||||
equals the native window, so no YaRN and no cap is in play.
|
||||
|
||||
## The MTP verdict: the head is in the file and it is still OFF
|
||||
|
||||
The drafter is **good**: acceptance 0.798 / 0.617 per position, mean accepted
|
||||
length 2.41, and the card recommends depth 2. It loses **at this shape**
|
||||
because the draft context bills its own **f16** KV over the WHOLE pool —
|
||||
`blk.32` carries real attention (4 KV heads × 256 × 2 caches × 2 B =
|
||||
**4,096 B/position**) × 786,432 positions = **3,072.00 MiB**, 4,830 MiB with its
|
||||
compute buffers — and `--ubatch-size 2048` refuses to give that up.
|
||||
|
||||
Booted arms, same card, same traffic, 2026-10-18 (`/completion` `timings`,
|
||||
greedy 512 tokens; never the first request after a boot):
|
||||
|
||||
| slots | spec | ubatch | used / free | decode | 3-stream | prefill @106k |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **3 × 262,144** | **off** | **2048** | **17,876 / 6,249** | **167.2** | **280.2** | **2,877** |
|
||||
| 3 × 262,144 | on | 2048 | 22,706 / 1,419 | 109.2 | — | — |
|
||||
| 3 × 262,144 | on | 1024 | 21,400 / 2,725 | 213.5 | 250.8 | 2,395 |
|
||||
| 2 × 262,144 | on | 2048 | 17,178 / 6,947 | 213.2 | — | — |
|
||||
| 1 × 262,144 | on | 2048 | 11,652 / 12,473 | 214.2 | — | — |
|
||||
|
||||
**MTP is worth +28 % single-stream decode whenever the card has room** (rows
|
||||
4–5 prove the drafter itself is a win; row 2 proves the squeeze is what halves
|
||||
it). The arm that runs MTP at three windows is `SPEC_TYPE=draft-mtp` +
|
||||
`UBATCH_SIZE=1024` (row 3) and pays **prefill −17 %** and **2,725 MiB of
|
||||
headroom** on a card whose mmproj worst-case lands AFTER the engine's own fit
|
||||
check. This endpoint is the fleet's **compaction hub** — prefill-bound ~170k
|
||||
inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 %
|
||||
prefill and +12 % three-stream aggregate over single-stream speed, and keeps
|
||||
6.2 GiB free. Frank confirmed this arm on 2026-10-18.
|
||||
|
||||
## Boot ledger (deployed arm, verbosity 4)
|
||||
|
||||
```
|
||||
CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions
|
||||
CUDA0 RS buffer size = 452.25 MiB # 3 recurrent-state cells, 32 layers, 3 seqs
|
||||
compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host
|
||||
mmproj 595 MiB + 248 MiB # charged AFTER the fit check
|
||||
nvidia-smi settled: 17,876 MiB used / 6,249 MiB free of 24,123
|
||||
```
|
||||
|
||||
No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class
|
||||
numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`,
|
||||
never the projection**.
|
||||
|
||||
## Measured (deployed arm)
|
||||
|
||||
- **167.2 t/s** greedy decode (512 tokens, `/completion` `timings`); 158.9 t/s
|
||||
single-stream chat with the card's sampler.
|
||||
- **280.2 t/s aggregate** across three concurrent streams (≈93.8 each).
|
||||
- **2,877 t/s prefill** at 106,418 prompt tokens.
|
||||
- The 27B record it replaces: 76.6 decode, 109.4 aggregate across two streams,
|
||||
691.7 prefill at 166k. Beaten on every axis.
|
||||
- The **first request after a boot is a warmup artifact** (37.7 t/s class) —
|
||||
never report it.
|
||||
|
||||
## Contract traps
|
||||
|
||||
- **`--metrics` is mandatory.** Without it llama.cpp answers **501** on
|
||||
`/metrics`, so the Prometheus scrape reads DOWN while `/health` answers 200 —
|
||||
that is how this endpoint sat red across four payload generations
|
||||
(`up{job="frank-zealot"}` oldest retained sample, 2026-10-11, is already 0).
|
||||
- **Q8_0 mmproj, not BF16.** The publisher's card: the BF16 tower "can crash
|
||||
llama-server when several image requests run at once; Q8_0 is not affected" —
|
||||
and `--parallel 3` is precisely that shape.
|
||||
- **Only the `image_url` content part reaches the model.** The legacy top-level
|
||||
`images` field bumps `usage.prompt_tokens` and the model still invents an
|
||||
answer. Probe with a solid color and expect the exact word.
|
||||
- **≥256 px probe images** or the model answers "video".
|
||||
- **Thinking tokens count against `max_tokens`.** A vision probe at 64 tokens
|
||||
returns EMPTY `content` + `finish_reason: "length"` — the endpoint looks blind
|
||||
while it is only mid-thought. Budget ≥512.
|
||||
- **`presence_penalty` has no GGUF metadata key**, so the one recipe value that
|
||||
must reach the argv is this one. The serve script uses
|
||||
`${PRESENCE_PENALTY=1.0}` — **not `:=`** — because `""` is the documented
|
||||
opt-out and `:=` would silently re-add the flag (caught by the seam, fixed
|
||||
2026-10-18).
|
||||
- **Spec flags move as a SET.** A bare `--model-draft`/`--spec-type` arm without
|
||||
the whole set has segfaulted at load on this fork family in the past;
|
||||
`SPEC_TYPE=none` strips all of them.
|
||||
- **Base alignment is the censored base lineage, and the CRACK swap is NOT
|
||||
one line here.** `/home/dank/models/Bonsai2-CRACK/` matches the **27B**
|
||||
geometry, not this 9B file; a dealigned 9B would have to be fetched.
|
||||
- **A raw `/completion` request** logs `failed to parse reasoning content
|
||||
(… parse_reasoning=true)` when the model closes a bare prompt with `` —
|
||||
cosmetic (`probe_request.cpp:789`); `/v1/chat/completions` parses
|
||||
`reasoning_content` correctly. Do not lower `VERBOSITY`: the KV/buffer ledger
|
||||
is what VRAM forensics read.
|
||||
- **A failed stop leaves a `failed` unit, and an orphan keeps answering.** On
|
||||
2026-10-18 the trial stop hit `State 'stop-sigterm' timed out. Killing.` →
|
||||
`Active: failed (Result: timeout)`, while the hand-booted arm kept serving
|
||||
7771 — a live port with no unit behind it. `TimeoutStopSec=180` is now on the
|
||||
unit (llama.cpp needs the drain time to release slots and the card); the
|
||||
hand-booted arm must be killed by PID, never `pkill -f llama-server`.
|
||||
|
||||
## Compaction hub
|
||||
|
||||
`models.yml` routes `compactionModel` for `frank-archon`, `frank-dragoon` and
|
||||
`frank-zealot` here — 262,144 is the widest window in the fleet and the only one
|
||||
that comfortably fits a worst-case compaction input (~170k tokens; the clamp is
|
||||
`min(maxTokens 16384, contextWindow − est − 4096)`, compaction fires at 85 % of
|
||||
the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With `--parallel 3`
|
||||
a compaction request no longer serializes behind a live generation (the Gemma4
|
||||
era's `--parallel 1` law).
|
||||
|
||||
## Fallback rungs
|
||||
|
||||
1. `UBATCH_SIZE=1024` — frees **1,306 MiB** measured on this shape (prefill
|
||||
−17 %).
|
||||
2. `CACHE_TYPE_K/V=q4_0` — frees ~6.5 GiB at three windows (the 27B era measured
|
||||
the q4_0 rung at 16,392 used / 7,733 free at 2 × 200,704).
|
||||
3. Cut to 2 slots — but that breaks the third concurrent subagent, so it is a
|
||||
last resort, not a knob.
|
||||
|
||||
## Rollback
|
||||
|
||||
Revert the swap commit. On disk: the **27B payload**
|
||||
(`/home/dank/models/Bonsai2-PQ2_0/`) + the **10754** fork tree, the Qwen3.5-9B
|
||||
Q6_K payload (`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`) + master CUDA build,
|
||||
the Gemma4, MiMo, Swift and Ornith payloads. Then
|
||||
`./scripts/deploy.sh dankclaw --service frank-zealot`.
|
||||
|
||||
## Related
|
||||
|
||||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — the retired 27B era (rollback target)
|
||||
- [[Llama Cpp Serving Sizing]] — the two-boot VRAM pricing method
|
||||
- [[frank-dragoon-flash-server-v2-native]] — the other 2026-10-18 cutover
|
||||
- [[deployment]] — deploy flow and incidents
|
||||
@@ -11,15 +11,19 @@ TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
|
||||
(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
|
||||
30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
|
||||
table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
|
||||
froggeric v22.5 template. Rollback: the in-house Swift1.5 w4dn8 `.rad` (on
|
||||
disk); tags frank-archon-freetoken-pr587-20261005 and
|
||||
frank-archon-vllm-shapeG-20261003),
|
||||
froggeric v22.5 template. Rollback: revert the swap commit and re-download the
|
||||
in-house Swift1.5 w4dn8 `.rad` byte-exact (purged from disk 2026-10-17; Hub
|
||||
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`); tags
|
||||
frank-archon-freetoken-pr587-20261005 and frank-archon-vllm-shapeG-20261003
|
||||
(AGENTS.md Runbook Index; `find /home/dank/models -name '*.rad'` on
|
||||
Dankserver 2026-10-18 lists only the iq4r file),
|
||||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||||
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||||
digest serving the NATIVE v2 conversion of the abliterated
|
||||
Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine,
|
||||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
|
||||
401,408-position KV pool at ctx 200704, vision ON)** and
|
||||
200,704-position KV pool at ctx 200704 — the 2 × ctx = 401,408 raise is
|
||||
committed 2026-10-18 but NOT yet deployed, vision ON)** and
|
||||
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
|
||||
Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48
|
||||
Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled,
|
||||
@@ -54,7 +58,7 @@ gfx1151 iGPU 114 GiB GTT]
|
||||
E[flash-server container gfx1151, digest-pinned image
|
||||
frank-dragoon, port 7770]
|
||||
F[flash-server v2 native abliterated .hgn + ngram table + vision tower
|
||||
ONE 401408-position KV pool at ctx 200704, 2 slots, MTP always-on, image input]
|
||||
ONE 200704-position KV pool at ctx 200704 (401408 committed, awaiting deploy), 2 slots, MTP always-on, image input]
|
||||
G[llama.cpp PrismML fork CUDA build
|
||||
frank-zealot, port 7771]
|
||||
H[Ternary-Bonsai-2-27B PQ2_0 GGUF + Q8_0 mmproj
|
||||
@@ -91,7 +95,11 @@ frank-zealot, port 7771]
|
||||
**one pool behind two slots**, sized so TWO maximum-length requests coexist —
|
||||
raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total
|
||||
in-flight context, ~10.8 GiB reserved against the card's 262,144/262,144 =
|
||||
7.2 GiB. The RAM comes out of the n-gram table's page cache, not out of a
|
||||
7.2 GiB. **Deploy state 2026-10-18: the running container and the
|
||||
host-deployed env still carry the 200,704-position pool** (boot ledger
|
||||
`reserving the KV pool, 200704 positions (about 5.5 GiB)`; host
|
||||
`frank-dragoon.env` `HALOGEN_KV_POOL_POSITIONS=200704`) — the doubling lands
|
||||
on the next redeploy (commit `2b12716`).
|
||||
longer window: the per-request ceiling stays 200,704, because 262,144 is the
|
||||
native wall and the engine refuses more without `HALOGEN_ROPE_YARN`.
|
||||
MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
|
||||
@@ -120,17 +128,41 @@ frank-zealot, port 7771]
|
||||
(`slow`/`advisor` → frank-archon) carried forward. Superseded by the
|
||||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||||
engine moved with the model
|
||||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — **ACTIVE since 2026-10-17**
|
||||
- [[frank-zealot-llama-prism-bonsai9-mtp]] — **ACTIVE since 2026-10-18**
|
||||
(root systemd unit, port 7771): the **PrismML fork** build of llama.cpp
|
||||
(vendor 10754, commit `2459f68b5` — the only loader for the payload's custom
|
||||
tensor type 142 + `prism.hadamard.*` transform) serving
|
||||
prism-ml/Ternary-Bonsai-2-27B-gguf **PQ2_0** (2.13 bpw) on the RTX 3090 —
|
||||
TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), **q8_0 K
|
||||
+ q8_0 V** (34 KiB/token over the 16 full layers = 13,328 MiB; the f16 pair
|
||||
needs 25,088 MiB and dies), settled 22,664 used / 1,461 free, vision via the
|
||||
publisher's Q8_0 mmproj (+873 MiB, charged AFTER the engine's fit check), MTP
|
||||
OFF (no MTP tensors). 76.6 t/s single-stream decode, 691.7 t/s prefill at
|
||||
166k, 109.4 t/s aggregate at two streams. Supersedes the Qwen3.5-9B era below.
|
||||
(vendor **10777**, commit `e3de71522` — the only loader for the payload's
|
||||
custom PQ2_0 tensor type + `prism.hadamard.*` transform, and since this swap
|
||||
also the minimum build that pairs MTP with a quantized V cache) serving
|
||||
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head IN the file** on
|
||||
the RTX 3090 — **THREE 262,144-token conversations**
|
||||
(`--ctx-size 786432 --parallel 3`, every slot at the window the model was
|
||||
trained at), **q8_0 K + q8_0 V** (17 KiB/token over the 8 full layers =
|
||||
13,056 MiB ledger-exact; the f16 pair needs 26,112 MiB and dies at
|
||||
`cudaMalloc`), settled **17,876 used / 6,249 free**, vision via the
|
||||
publisher's **Q8_0** mmproj (Q8_0 is the publisher's requirement for
|
||||
concurrent image requests, and this runs three slots), **MTP OFF by
|
||||
measurement** — the draft context bills its own f16 KV over the WHOLE pool
|
||||
(3,072 MiB) and at `UBATCH_SIZE=2048` that squeeze halves decode
|
||||
(167.2 → 109.2 t/s); `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the
|
||||
documented 213.5 t/s arm, declined for +17 % prefill and 6.2 GiB of headroom.
|
||||
Measured: 167.2 t/s single-stream decode, 2,877 t/s prefill at 106,418,
|
||||
280.2 t/s aggregate across three streams. `--metrics` rides (`METRICS=1`):
|
||||
llama.cpp answers 501 on `/metrics` without it, which is why Prometheus
|
||||
`up{job="frank-zealot"}` read 0 for its whole retained history until
|
||||
2026-10-18. `contextWindow: 262144` is the **widest in the fleet**, so this
|
||||
is the compaction hub. The `--presence-penalty` opt-out law (`${VAR=}`, not
|
||||
`${VAR:=}`) and the `TimeoutStopSec=180` stop-drain law were both earned here.
|
||||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — **RETIRED 2026-10-17 → 10-18**
|
||||
(payload + fork build 10754 tree on disk as rollback): the first PrismML-fork
|
||||
era — prism-ml/Ternary-Bonsai-2-**27B**-gguf **PQ2_0**, TWO 200,704-token
|
||||
conversations (`--ctx-size 401408 --parallel 2`), q8_0 K + q8_0 V
|
||||
(34,816 B/token over the 16 full layers = 13,328 MiB; f16 needs 25,088 MiB and
|
||||
dies), settled 22,664 used / 1,461 free, MTP **impossible** (that export
|
||||
carries no head), 76.6 t/s decode, 691.7 t/s prefill at 166k, 109.4 t/s
|
||||
aggregate at two streams. Owns the 34 KiB/token KV law, the f16 OOM wall, the
|
||||
two-boot VRAM pricing, and the record of the 2026-10-18 bench window
|
||||
(production unit stopped 03:38, hand-booted 9B candidate answering the port)
|
||||
that closed with this swap.
|
||||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **RETIRED 2026-10-07 →
|
||||
10-17** (payload + master CUDA build on disk as rollback): llama.cpp master
|
||||
(CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive
|
||||
@@ -197,10 +229,13 @@ frank-zealot, port 7771]
|
||||
dankserver 2026-10-15 (Frank's one-time rm, volume for this payload;
|
||||
rebuild via the fetch + build scripts named in the fp8-native runbook)
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **RETIRED 2026-10-08 → 10-15
|
||||
(superseded by the publisher iq4r swap; payload on disk as the rollback)**:
|
||||
(superseded by the publisher iq4r swap; the `.rad` was PURGED from disk
|
||||
2026-10-17 — rollback = revert + byte-exact re-download of
|
||||
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)**:
|
||||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
|
||||
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
|
||||
the FP8-native payload remains the rollback. **2026-10-13: fp8 KV, and the
|
||||
the FP8-native payload was the second rollback until it too was deleted
|
||||
2026-10-15. **2026-10-13: fp8 KV, and the
|
||||
pair re-cut twice** — `--kv-cache-dtype fp8` first at
|
||||
`--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
|
||||
with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
|
||||
@@ -219,7 +254,9 @@ frank-zealot, port 7771]
|
||||
reset the knob to its default (125,988 MiB) and the endpoint refused to boot
|
||||
6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
|
||||
w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)**
|
||||
w4dn8 swap replaced it; its `.rad` was DELETED from disk 2026-10-15 —
|
||||
rebuild = `scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`,
|
||||
`scripts/frank-archon.env:59-62`)**
|
||||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||||
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
|
||||
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
@@ -248,8 +285,10 @@ frank-zealot, port 7771]
|
||||
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
|
||||
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Superseded by the
|
||||
radiance swap for TP=2; the image + payload stay on disk until the
|
||||
rollback window closes
|
||||
radiance swap for TP=2. Disk state 2026-10-18: the image is still in
|
||||
dankserver's rootless storage (`localhost/frank-primary-freetoken:pr587-77a7201`,
|
||||
29.7 GB — the pre-rename name); the orcarouter FP8 payload is NO LONGER on
|
||||
disk (no `*orcarouter*` path under /home/dank, `find` 2026-10-18)
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
|
||||
(rollback target; the vLLM trio + env lore live in the tag)**: the
|
||||
tcclaviger RDNA4
|
||||
@@ -278,12 +317,14 @@ frank-zealot, port 7771]
|
||||
35 tok/s target is NOT met, bounded by the fork's expert-gather read path
|
||||
(~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
|
||||
Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
|
||||
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||||
back; payload and R9V state dirs intact on disk, git tag
|
||||
[[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||||
back; the R9V state dirs are on disk (`/home/dank/r9v-state`,
|
||||
`~/.local/state/r9v`) but the payload GGUFs are not — `/home/dank/models`
|
||||
holds only the iq4r `.rad` as of 2026-10-18; git tag
|
||||
frank-archon-r9v-q8-20260929 is the last pre-cutover commit):
|
||||
[[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
|
||||
(same-day swap to the Q8_0 payload; rollback target, state + payload
|
||||
intact on disk): the R9V radiance-vLLM fork serving
|
||||
(same-day swap to the Q8_0 payload; state + payloads no longer on disk
|
||||
2026-10-18, only the R9V state dirs remain): the R9V radiance-vLLM fork serving
|
||||
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
|
||||
blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
|
||||
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
|
||||
@@ -505,8 +546,8 @@ frank-zealot, port 7771]
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
||||
and build history (on Dankclaw; the slot now serves the kingjones
|
||||
ROCmFP4 model, see [[kingjones-strix-mtp]])
|
||||
and build history (on Dankclaw; the gfx1151 slot now serves frank-dragoon's
|
||||
flash-server v2-native payload — see [[frank-dragoon-flash-server-v2-native]])
|
||||
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
||||
identity, and experiment summary (retired 2026-08-22)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user