diff --git a/AGENTS.md b/AGENTS.md index cf323b2..4724f6a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,8 +26,8 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 | -| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 on 2026-10-18 so BOTH slots can hold a maximum-length request at once — the parity with frank-zealot's 401,408 pool on total in-flight context; ~10.8 GiB reserved where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`; every pool position is page cache the n-gram table loses), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770| -| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10754**, commit `2459f68b5`; root systemd unit, relinked — the ONLY loader for this payload) | `prism-ml/Ternary-Bonsai-2-27B-gguf` **PQ2_0** (7,206,168,928 B, sha256 `3907dc16…`; `qwen35` hybrid: **64 blocks, 16 full-attention + 48 Gated DeltaNet**, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP head in the file**; `general.file_type 141` = custom tensor type **142** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (629,246,976 B; Qwen3-VL tower, 27 blocks, 768 px, `qwen3vl_merger`) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **q8_0 K + q8_0 V** under flash attention (34 KiB/token over the 16 full layers = 13,328 MiB pool; **f16 KV needs 25,088 MiB and cannot boot this card**), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (no head in the file). Boot 22,664 used / 1,461 free; 76.6 t/s single-stream decode, 691.7 t/s prefill at 166k. Cutover 2026-10-17 from Qwen3.5-9B-Uncensored Q6_K (same shape; payload + master CUDA build on disk as rollback). | +| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 on 2026-10-18 so BOTH slots can hold a maximum-length request at once; the parity play was frank-zealot's then-401,408 total in-flight context — zealot moved to 786,432 the same evening and dragoon's pool was NOT raised again, because every position is page cache the n-gram table loses; ~10.8 GiB reserved where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770| +| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10777**, commit `e3de71522`; root systemd unit, relinked — the ONLY loader for this payload, and since 2026-10-18 also the minimum build that pairs MTP with a quantized V cache) | `prism-ml/Ternary-Bonsai-2-9B-gguf` **PQ2_0 carrying the MTP head in the file** (2,675,410,720 B, Hub-LFS sha256 `c79fcdfa…` verified at fetch by `scripts/download-frank-zealot-bonsai9.sh`; `qwen35` hybrid: `block_count 33` = **32 trunk layers** (8 full-attention + 24 Gated DeltaNet at `full_attention_interval 4`) + the `blk.32` MTP block, 4 KV heads × 256 dim, native 262,144; `general.file_type 141` with **202 custom-type PQ2_0 tensors of 442** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (624,229,664 B, sha256 `e0cb6f69…`; **Q8_0 is the publisher's requirement** for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs three slots) for **vision**, on the RTX 3090, **THREE 262,144-token slots** (`--ctx-size 786432 --parallel 3` = every conversation at the trained window), **q8_0 K + q8_0 V** under flash attention (17 KiB/token over the 8 full layers = 13,056 MiB pool, ledger-exact; **f16 KV needs 26,112 MiB and cannot boot this card**; +452 MiB of recurrent state for the three DeltaNet cells), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF — measured, not because the head is absent**: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented MTP arm is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s. Boot 17,876 used / 6,249 free; 167.2 t/s single-stream decode, 280.2 t/s aggregate across three concurrent streams, 2,877 t/s prefill at 106,418. Cutover 2026-10-18 from Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback). | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -319,12 +319,14 @@ image]`) — `frank-dragoon` returned to vision with the flash-server v2 swap 27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary are the engine core; `--mm-max-patches` available), and `frank-zealot` through -the publisher's own BF16 mmproj passed as `--mmproj` — the model is natively -multimodal but the Q6_K GGUF carries no vision tensors (same arrangement the -Gemma4 era used). `frank-zealot`'s `contextWindow` is -**200704** — the PER-SLOT window (`--ctx-size 401408 --parallel 2` since the -2026-10-07 Qwen3.5-9B swap), not the pool and not the model's native 262144: -the harness must compact before llama.cpp answers a longer prompt with a 400. +the publisher's own **Q8_0** mmproj passed as `--mmproj` — Q8_0 is the +publisher's requirement for concurrent image requests (the card states the +BF16 tower can crash `llama-server` when several image requests run at once, +and this endpoint runs three slots). `frank-zealot`'s `contextWindow` is +**262144** — the PER-SLOT window (`--ctx-size 786432 --parallel 3` since the +2026-10-18 Bonsai-2-**9B** swap), which since that swap EQUALS the model's +native window; llama.cpp answers a longer prompt with a 400 naming the limit, +never a truncation, so the harness must compact before it. Note also that the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 @@ -460,71 +462,113 @@ digest all remain on disk; the ROCmFPX door behind that ### frank-zealot — llama.cpp (PrismML fork) on the RTX 3090 (Dankclaw, ACTIVE) **The PrismML fork build of llama.cpp** (vendor release, `--version` = -`0.2.0-dev (build 10754, commit 2459f68b5)`; -`/home/dank/GitPackages/llama-prism/llama-server`) as a **root systemd unit** -(`scripts/frank-zealot.service`, `relink_unit`ed into -`/etc/systemd/system/`), serving `prism-ml/Ternary-Bonsai-2-27B-gguf` at -**PQ2_0** (7,206,168,928 B, sha256 `3907dc16…` = the Hub LFS oid) **plus the -publisher's own Q8_0 mmproj** (629,246,976 B), both -`/home/dank/models/Bonsai2-PQ2_0/`, on port 7771. Cutover 2026-10-17 from the -Qwen3.5-9B-Uncensored Q6_K era (same shape). +`0.2.0-dev (build 10777, commit e3de71522)`; +`/home/dank/GitPackages/llama-prism-b10777/llama-server`) as a **root systemd +unit** (`scripts/frank-zealot.service`, `relink_unit`ed into +`/etc/systemd/system/`), serving `prism-ml/Ternary-Bonsai-2-9B-gguf` at +**PQ2_0 with the MTP head IN the file** (2,675,410,720 B, Hub-LFS sha256 +`c79fcdfa…` verified at fetch) **plus the publisher's own Q8_0 mmproj** +(624,229,664 B, sha256 `e0cb6f69…`), both `/home/dank/models/Bonsai2-9B-MTP/`, +on port 7771. Cutover 2026-10-18 from the Bonsai-2-**27B** PQ2_0 era +(2 × 200,704, same card, same fork family). -**The runtime is part of the payload identity.** The file is -`general.file_type 141` (PQ2_0, 2.13 bpw, group 128) with **402 tensors of the -custom GGML type 142** plus `prism.hadamard.*` metadata (normalized -Sylvester–Walsh transform, block 1024, 401 named weights, inverse on -`token_embd`, `gdn_v_grouped`). Stock llama.cpp refuses type 142 at load, and a -loader that ignored the Hadamard metadata would dequant wrongly — same lock class -as the ROCmFPX era. Retargeting the payload without the fork ships an endpoint -that cannot load, so the seam pins both paths. +**The runtime is part of the payload identity — twice over.** The file is +`general.file_type 141` — **202 of the 442 tensors carry the fork's PQ2_0 type** +(the rest are f32 norms/scalars plus 8 q8_0 in the MTP block) — beside +`prism.hadamard.*` metadata (normalized Sylvester–Walsh transform): stock +llama.cpp refuses the custom type at load, and a loader that +ignored the Hadamard metadata would dequant WRONG — same lock class as the +ROCmFPX era. On top of that lock, **the fork gates MTP beside a quantized V +cache on build `prism-b10776-d00e252` or newer**, which is why the prebuilt +moved 10754 → 10777; the 10754 tree stays on disk as the 27B rollback runtime. +Retargeting the payload without the fork ships an endpoint that cannot load, +so the seam pins both paths. -**Architecture: qwen35 hybrid** — read from the GGUF's own header: **64 -blocks**, `full_attention_interval = 4` ⇒ **16 full-attention + 48 Gated -DeltaNet layers**, 24 attention heads / **4 KV heads** × 256 dim, embedding -5,120, ffn 17,408, `ssm` conv_kernel 4 / state_size 128 / group_count 16 / -time_step_rank 48 / inner_size 6,144, native 262,144, vocab 248,320. **KV -accounting:** the pool scales with the 16 full layers — 16 × 4 × 256 = 16,384 -elements/token per cache, so **q8_0 K + q8_0 V bills 34 KiB/token** = 13,328 -MiB at the 401,408 pool (ledger-confirmed to the MiB), while **f16 KV needs -25,088 MiB and cannot boot on a 24,123 MiB card** (measured `cudaMalloc` OOM). -The 48 DeltaNet layers bill a fixed recurrent state per SLOT — ~150 MiB each — -so pool arithmetic alone under-prices the shape. +**Architecture: qwen35 hybrid** — read from the GGUF's own header: `block_count` +**33 = 32 trunk layers + the `blk.32` MTP block** (the MTP block has its own +attention + FFN in q8_0, which is why the draft context bills real KV), and +`qwen35.full_attention_interval = 4` ⇒ **8 full-attention + 24 Gated +DeltaNet**, 4 KV heads × 256 dim, native 262,144. **KV accounting:** the pool +scales with the 8 full layers — 8 × 4 × 256 = 8,192 elements/token per cache, +so **q8_0 K + q8_0 V bills 17,408 B/token ≈ 17 KiB** = **13,056.00 MiB** at the +786,432 pool (ledger-exact) — exactly HALF the 27B's 34,816 B/token, which is +the whole reason three native windows fit where the 27B afforded only two +shortened ones. **f16 K would bill 26,112 MiB and cannot boot on a 24,123 MiB +card.** The 24 DeltaNet layers bill a fixed recurrent state per SLOT — +`RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)` ≈ 150 MiB each — so +pool arithmetic alone under-prices the shape. -**The shape (2026-10-17):** `--ctx-size 401408 --parallel 2` = TWO -conversations of 200,704 (Frank's cap; the harness `contextWindow` quotes -the per-slot number), `--flash-attn on`, **q8_0 K + q8_0 V**, batch 4096 / -**ubatch 2048** (the measured CUDA optimum carried from the MiMo/Swift A/Bs on -this card), `--log-verbosity 4`, `--mmproj` for vision, **speculative decoding -OFF** (`SPEC_TYPE=none`). Boot: **22,664 MiB used / 1,461 free**; the -projection (21,439) excludes the projector's 873.10 MiB worst-case, which -lands AFTER the fit check — read settled `nvidia-smi`, never the projection. -Measured: 76.6 t/s single-stream decode (the 9B ran 94.4), 1,258 t/s prefill -at 35k, 691.7 t/s at 166k, 109.4 t/s aggregate across two concurrent streams. -The **first request after boot is a warmup artifact** (2.6 t/s prefill / 37.7 -t/s decode measured) — never report it. +**The shape (2026-10-18):** `--ctx-size 786432 --parallel 3` = THREE +conversations of 262,144, every one at the model's TRAINED window (Frank: "if +you can possibly fit a third context, do it"), `--flash-attn on`, **q8_0 K + +q8_0 V**, batch 4096 / **ubatch 2048** (the measured CUDA optimum carried from +the MiMo/Swift A/Bs on this card; it also sets the compute-buffer size, which +is what decides whether MTP fits), `--log-verbosity 4`, `--metrics`, +`--jinja`, `--mmproj` for vision, **speculative decoding OFF** +(`SPEC_TYPE=none`). Boot: **17,876 MiB used / 6,249 free**; compute 1,409 MiB +CUDA + 1,201 MiB CUDA_Host, projector 595 + 248 MiB. Measured on the same +boot: **167.2 t/s** greedy decode, 158.9 t/s single-stream chat, **280.2 t/s +aggregate across three concurrent streams**, **2,877 t/s prefill at 106,418** +tokens — the 27B record (76.6 decode / 109.4 aggregate / 691.7 prefill at 166k) +beaten on every axis. The **first request after boot is a warmup artifact** +(37.7 t/s class) — never report it. -**Why MTP is off**: the PQ2_0 export carries no MTP head (upstream ships a -separate `…-CRACK-MTP-GGUF`, +451 MB, for that), and MTP is a property of the -file, not of the flags. +**Why MTP is off even though the head is in the file.** The drafter is GOOD — +acceptance 0.798 / 0.617 per position, mean accepted length 2.41 — and the card +recommends depth 2. It still loses AT THIS SHAPE because the draft context +bills its OWN **f16** KV over the WHOLE pool (4,096 B/position × 786,432 = +**3,072.00 MiB**, 4,830 MiB with its compute buffers) and ubatch 2048 refuses to +give it up. Booted arms, same card, same traffic (2026-10-18): -**Vision**: the publisher's own Q8_0 mmproj via `--mmproj` (Qwen3-VL tower: 27 -blocks, 768 px, patch 16, merge 2, `qwen3vl_merger`, deepstack). Probe traps -carry over: ≥256 px probe images or the model answers "video", and only the -OpenAI `image_url` content part reaches the model. **New trap:** thinking -tokens count against `max_tokens`, so a vision probe at 64 tokens returns -EMPTY `content` with `finish_reason: "length"` — budget ≥512 or the endpoint -looks blind while it is only mid-thought. +| slots | spec | ubatch | used / free | decode | +|---|---|---|---|---| +| 3 × 262,144 | off | 2048 | 17,876 / 6,249 | **167.2** | +| 3 × 262,144 | on | 2048 | 22,706 / 1,419 | 109.2 | +| 3 × 262,144 | on | 1024 | 21,400 / 2,725 | **213.5** | +| 2 × 262,144 | on | 2048 | 17,178 / 6,947 | 213.2 | +| 1 × 262,144 | on | 2048 | 11,652 / 12,473 | 214.2 | -**Base alignment**: Bonsai-2 is the censored BASE lineage (the 9B it replaced -was uncensored). The CRACK re-export on disk -(`/home/dank/models/Bonsai2-CRACK/`, header census proves byte-identical -geometry) is a one-line `MODEL_FILE` swap if a refusal ever surfaces. +**MTP is worth +28 % decode whenever the card has room**; the squeeze, not the +drafter, is what halves it at three windows. The arm that WOULD run it at three +windows is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` (213.5 t/s single-stream, +250.8 three-stream aggregate) and pays prefill −17 % and 2.7 GiB of headroom on +a card whose projector worst-case lands AFTER the engine's own fit check. This +endpoint is the fleet's compaction hub — prefill-bound, 2–3 concurrent +subagents — so the shipped arm takes the +17 % prefill (2,877 vs 2,395 t/s) and +the +12 % three-stream aggregate (280.2 vs 250.8) over single-stream speed, and +keeps 6.2 GiB free. `SPEC_TYPE=none` strips the WHOLE spec set (`--spec-type` +and every `--spec-draft-*`) — flags move as a set or the arm measures a crash +instead of an absence. + +**Sampler**: the recipe lives in the checkpoint (`general.sampling.*` = +temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0, +confirmed in the engine's own `/props`) and llama.cpp reads it itself. +**Presence penalty is the one recipe value with no GGUF metadata key**, so +`PRESENCE_PENALTY=1.0` must reach the argv; `""` is the documented opt-out, and +the serve script therefore uses `${PRESENCE_PENALTY=1.0}` (NOT `:=`, which +would silently re-add the flag on an explicit empty value). + +**Vision**: the publisher's own **Q8_0** mmproj via `--mmproj` — Q8_0 is +MANDATORY, not a preference: the card says the BF16 tower "can crash +llama-server when several image requests run at once", and `--parallel 3` is +exactly that shape. Probe traps carry over: ≥256 px probe images or the model +answers "video", only the OpenAI `image_url` content part reaches the model, +and thinking tokens count against `max_tokens` — budget ≥512 or the probe +returns EMPTY `content` with `finish_reason: "length"` and the endpoint looks +blind while it is only mid-thought. + +**Base alignment**: Bonsai-2 is the censored BASE lineage. Unlike the 27B, the +CRACK re-export on disk (`/home/dank/models/Bonsai2-CRACK/`) is byte-identical +geometry to the 27B it matched, NOT to this 9B file — if a refusal ever +surfaces there is no one-line `MODEL_FILE` swap; a 9B dealigned export would +have to be fetched first. **Compaction hub**: `models.yml` routes `compactionModel` for -`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here — 200,704 is the -only window in the fleet that fits a worst-case compaction input (~170k -tokens). With `--parallel 2` a compaction request no longer serializes behind -a live generation (the Gemma4 era's `--parallel 1` law). Retrieval through a +`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here — +262,144 (since the 2026-10-18 reslice) is the widest window in the fleet and +the only one that comfortably fits a worst-case compaction input (~170k +tokens). With `--parallel 3` a compaction request no longer serializes behind a +live generation (the Gemma4 era's `--parallel 1` law). Retrieval through a 25,960-token prompt was verified exact after the swap. **Contract law learned at verbosity 4:** a raw `/completion` request prints @@ -533,11 +577,13 @@ model closes a bare prompt with `` — cosmetic (`probe_request.cpp:789`); the `/v1/chat/completions` path parses `reasoning_content` correctly, which is the only path consumers use. Do not lower `VERBOSITY` to silence it: the KV/buffer ledger is what VRAM forensics -read. +read. `--metrics` is required or the Prometheus scrape reads 501 = DOWN. Runbook: -`serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md`. -Rollback: revert the swap commit — the Qwen3.5-9B Q6_K payload +`serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md`. +Rollback: revert the swap commit — the 27B payload +(`/home/dank/models/Bonsai2-PQ2_0/`) with the 10754 fork tree +(`~/GitPackages/llama-prism/`), the Qwen3.5-9B Q6_K payload (`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`) with the master CUDA build (`~/GitPackages/llama-master/build-cuda/`), the Gemma4 payload (`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload @@ -960,6 +1006,21 @@ linked runbooks. trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 -> 36:29), which is the practical size of the effect. See [[frank-medium-llamacpp-dflash2]]. +- **A draft context bills its own f16 KV over the WHOLE POOL, and on a + single-card endpoint that squeeze can cost more than the drafter wins.** The + draft context is built at the target's `n_ctx`, so with `--parallel N` the + drafter pays the POOL, not the slot: on frank-zealot's 786,432-position + qwen35 pool, `--spec-type draft-mtp` added **3,072.00 MiB of f16 draft KV** + (4,096 B/position — the draft cache is never quantized) plus its own compute + buffers, 4,830 MiB all told. Booted arms, same card, same traffic + (2026-10-18): 3 slots + ubatch 2048 → **167.2 t/s decode with spec OFF, + 109.2 with spec ON**; dropping `UBATCH_SIZE` to 1024 hands the buffers back + and spec ON gives **213.5** (+28 % over the spec-OFF arm), and at 1–2 slots + the SAME card runs MTP at ubatch 2048 and still gains ~28 %. So the MTP + question is never "does the file carry the head" — it is "what do draft KV + + buffers cost against whatever the batch knobs keep for compute buffers", + and that answer is per-shape. Measure the arm, never the flag. See + [[frank-zealot-llama-prism-bonsai9-mtp]]. - **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau — and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its compute buffers land about TWICE as heavy on the card that holds the output @@ -1185,15 +1246,16 @@ Active: | Service | Runbook | |---------|---------| | frank-dragoon (flash-server 0.16.2, NATIVE v2 abliterated .hgn + MTP always-on, 2 slots over ONE 401,408 pool at ctx 200704, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md` | -| frank-zealot (llama.cpp PrismML fork, Bonsai-2-27B PQ2_0, 2×200,704 q8_0 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md` | +| frank-zealot (llama.cpp PrismML fork build 10777, Bonsai-2-**9B** PQ2_0 **with the MTP head**, 3×262,144 q8_0 KV, MTP off by measurement, vision ON via Q8_0 mmproj, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md` | | frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md` | -| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | +| Dankclaw kernel posture (`iommu=pt` since 2026-10-18, was `amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| +| llama.cpp PrismML fork build 10754, Ternary-Bonsai-2-**27B** PQ2_0, 2×200,704 q8_0 K+V, MTP off (no head in the file), vision ON (`frank-zealot`) | Dankclaw | 2026-10-17 → 10-18 (rollback target — payload + 10754 fork tree on disk; owns the 34,816 B/token 16-full-layer KV law, the 25,088 MiB f16 OOM wall and the two-boot VRAM pricing) | `frank-zealot-llama-prism-bonsai2-pq20.md` | | radiance Swift1.5-Qwen3.8-Flash-Next GPTQ w4dn8, 3×200,000 fp8-KV slab 25600 (`frank-archon`) | Dankserver | 2026-10-08 → 10-15 (rollback target — `.rad` purged 2026-10-17, byte-exact re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`; owns the slab-slope ladder, KV dtype law, KLD campaign, and every shape law the iq4r swap carried verbatim) | `frank-archon-radiance-swift15-w4dn8.md` | | llama.cpp CUDA (master) Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-17 (rollback target — payload + master CUDA build on disk; owns the 32 KiB/token 8-full-layer KV law and the 2→3-slot reslice law) | `frank-zealot-llama-qwen35-9b-q6k-2x200704.md` | | llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-zealot-llama-gemma4-q4km-200704.md` | diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index ff01b09..f95853a 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1225,24 +1225,31 @@ deploy_llm_dragoon() { # deploy_llm_zealot — frank-zealot is llama.cpp (PrismML fork build) on the # RTX 3090 as a ROOT unit: the runtime is a native binary tree under -# ~/GitPackages/llama-prism (vendor release build 10754, commit 2459f68b5 — -# the ONLY loader that registers the payload's custom tensor type 142 and -# reads its prism.hadamard.* metadata), so there is no container image and no -# generated unit to avoid. relink_unit points -# /etc/systemd/system/frank-zealot.service at the repo file and boot -# persistence is that unit's [Install] section. +# ~/GitPackages/llama-prism-b10777 (vendor release prism-b10777-e3de715, +# `--version` = 0.2.0-dev build 10777 commit e3de71522 — the ONLY loader class +# that registers the payload's custom tensor type 142 and reads its +# prism.hadamard.* metadata, and since 2026-10-18 ALSO the minimum build for +# this endpoint: the card gates quantized V alongside MTP on +# `prism-b10776-d00e252` or newer, which the 27B-era 10754 tree predates), so +# there is no container image and no generated unit to avoid. relink_unit +# points /etc/systemd/system/frank-zealot.service at the repo file and boot +# persistence is that unit's [Install] section. The former 10754 tree +# (~/GitPackages/llama-prism) stays on disk as the rollback runtime. # # It is a SEPARATE endpoint on port 7771; frank-dragoon keeps 7770. The 3090 is # not shared: frank-dragoon's engine runs on the gfx1151 iGPU. -LLM_ZEALOT_MODEL_FILE="${LLM_ZEALOT_MODEL_FILE:-/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf}" -LLM_ZEALOT_RUNTIME="${LLM_ZEALOT_RUNTIME:-/home/dank/GitPackages/llama-prism/llama-server}" +LLM_ZEALOT_MODEL_FILE="${LLM_ZEALOT_MODEL_FILE:-/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}" +LLM_ZEALOT_RUNTIME="${LLM_ZEALOT_RUNTIME:-/home/dank/GitPackages/llama-prism-b10777/llama-server}" # The projector IS the vision capability here: the Bonsai-2 text file carries # no vision tensors, so the publisher's own mmproj (Qwen3-VL tower, 27 blocks, -# 768 px, qwen3vl_merger, 629,246,976 B) is the whole image path — a deploy -# that starts without it silently ships a text-only endpoint. Checked -# unconditionally, like frank-dragoon's. Its 873.10 MiB worst-case VRAM lands -# AFTER the engine's fit check, so it is not inside the q8_0 budget. -LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf}" +# 768 px, qwen3vl_merger, projection_dim 4096, 624,229,664 B) is the whole +# image path — a deploy that starts without it silently ships a text-only +# endpoint. Checked unconditionally, like frank-dragoon's. Q8_0 is MANDATORY, +# not a preference: the card says the BF16 tower can crash llama-server when +# several image requests run at once, and this endpoint runs --parallel 3. Its +# 868.32 MiB worst-case VRAM (595.29 model + 248.10 compute in the boot ledger) +# lands AFTER the engine's fit check, so it is not inside the q8_0 budget. +LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}" deploy_llm_zealot() { # Refuse rather than restart-loop: a unit whose runtime or payload is @@ -1250,15 +1257,16 @@ deploy_llm_zealot() { # endpoint dark with nothing in the log to say why. if [[ ! -x "$LLM_ZEALOT_RUNTIME" ]]; then echo "ERROR: llama.cpp runtime absent at $LLM_ZEALOT_RUNTIME" >&2 - echo "frank-zealot: fetch the PrismML fork release build (build 10754," >&2 - echo "frank-zealot: commit 2459f68b5) into ~/GitPackages/llama-prism first —" >&2 - echo "frank-zealot: stock llama.cpp refuses the payload's custom tensor type 142." >&2 + echo "frank-zealot: fetch the PrismML fork release build (prism-b10777-e3de715," >&2 + echo "frank-zealot: linux-cuda-12.4 asset) into ~/GitPackages/llama-prism-b10777 first:" >&2 + echo "frank-zealot: stock llama.cpp refuses the payload's custom tensor type 142, and" >&2 + echo "frank-zealot: builds older than b10776 cannot pair MTP with a quantized V cache." >&2 return 1 fi for f in "$LLM_ZEALOT_MODEL_FILE" "$LLM_ZEALOT_MMPROJ"; do if [[ ! -f "$f" ]]; then echo "ERROR: frank-zealot payload file absent at $f" >&2 - echo "frank-zealot: run scripts/download-frank-zealot-bonsai2.sh first;" >&2 + echo "frank-zealot: run scripts/download-frank-zealot-bonsai9.sh first;" >&2 echo "frank-zealot: refusing to start a unit that cannot load." >&2 return 1 fi diff --git a/scripts/download-frank-zealot-bonsai9.sh b/scripts/download-frank-zealot-bonsai9.sh new file mode 100755 index 0000000..22c538c --- /dev/null +++ b/scripts/download-frank-zealot-bonsai9.sh @@ -0,0 +1,87 @@ +#!/usr/bin/env bash +# Download the frank-zealot payload for llama.cpp on the RTX 3090: +# prism-ml/Ternary-Bonsai-2-9B-gguf — the **PQ2_0 packing with the MTP draft +# head built in** (the card: "load it instead of the plain PQ2_0, not alongside +# it") plus the PUBLISHER's OWN **Q8_0** vision tower. Public repo — no auth +# header, and never a token on the command line (world-readable in +# /proc//cmdline). +# +# THE Q8_0 mmproj IS MANDATORY, NOT A PREFERENCE. The card: with the current +# fork build "the BF16 mmproj can crash llama-server when several image +# requests run at once; Q8_0 is not affected and quality is nearly the same." +# frank-zealot runs --parallel > 1, i.e. exactly that concurrent-image shape. +# +# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized file +# on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a success, +# then byte count and sha256 against the Hub's **LFS oid** (this repo IS LFS, +# unlike the HauhauCS Xet repo the 2026-10-07 era used, so the sha values below +# are the Hub's own — verified at fetch, not computed locally). +# Idempotent: complete files are skipped. +# +# Rollback payload stays on disk: /home/dank/models/Bonsai2-PQ2_0/ (the +# Ternary-Bonsai-2-27B PQ2_0 file + its mmproj, cut over 2026-10-17), with +# Bonsai2-CRACK/ (the uncensored 27B re-export) beside it. +# +# Usage: +# scripts/download-frank-zealot-bonsai9.sh [DEST_DIR] [--verify-only] +set -euo pipefail + +DEST="" +VERIFY_ONLY=0 +for arg in "$@"; do + case "$arg" in + --verify-only) VERIFY_ONLY=1 ;; + *) DEST="$arg" ;; + esac +done +DEST="${DEST:-/home/dank/models/Bonsai2-9B-MTP}" + +# repo | hub-name | dest-name | bytes | sha256 (Hub LFS oid) +FILES=( + "prism-ml/Ternary-Bonsai-2-9B-gguf|Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf|Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf|2675410720|c79fcdfaf6b7bf47089b50dadcfd11f9ce7d3e96adf3bbefde7912e5ef69e22b" + "prism-ml/Ternary-Bonsai-2-9B-gguf|Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf|Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf|624229664|e0cb6f6988cea323ef798cd77d019bbb819af75582f034eaab0e7232314953db" +) + +mkdir -p "$DEST" +status=0 +for entry in "${FILES[@]}"; do + IFS='|' read -r repo hubname destname size sha <<<"$entry" + url="https://huggingface.co/${repo}/resolve/main/${hubname}" + f="$DEST/$destname" + have=0 + if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then + if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then + have=1 + fi + fi + if (( have )); then + echo "OK $destname" + continue + fi + if (( VERIFY_ONLY )); then + echo "MISSING $destname" + status=1 + continue + fi + echo "FETCHING $destname (from $repo)" + tmp="$f.part" + curl -fsSL --fail-with-body -o "$tmp" "$url" + mv "$tmp" "$f" + if [[ "$(stat -c%s "$f")" != "$size" ]]; then + echo "SIZE FAIL $destname ($(stat -c%s "$f") != $size)" + status=1 + continue + fi + if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then + echo "SHA FAIL $destname" + status=1 + continue + fi + echo "VERIFIED $destname" +done +if (( status == 0 )); then + echo "DOWNLOAD_DONE $DEST" +else + echo "DOWNLOAD_FAILED $DEST" >&2 +fi +exit "$status" diff --git a/scripts/frank-zealot.env b/scripts/frank-zealot.env index e66488d..f72e31e 100644 --- a/scripts/frank-zealot.env +++ b/scripts/frank-zealot.env @@ -1,29 +1,39 @@ -# frank-zealot — llama.cpp (PrismML fork build, CUDA) serving -# prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090, :7771. +# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) serving +# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 **with the MTP head in the file** +# on the RTX 3090, :7771. # -# 2026-10-17 (Frank): swap from Qwen3.5-9B-Uncensored Q6_K to the Bonsai-2-27B -# ternary checkpoint at the SAME shape Frank asked for — TWO 200,704-token -# conversations — with q8_0 K + q8_0 V (f16 is arithmetically impossible here, -# see the KV law), MTP off (the file carries no nextn/mtp tensors), verbosity 4. +# 2026-10-18 (Frank): swap from the Bonsai-2-**27B** PQ2_0 to the 9B MTP export +# at THREE 262,144-token conversations (Frank: "if you can possibly fit a third +# context, do it"). The 9B bills HALF the KV of the 27B per token, which is what +# funds 3 x the native window on one 24,123 MiB card. Speculative decoding ships +# OFF — measured, not assumed; see the MTP law below, which is the whole story +# and the reason the publisher's "depth 2 works best" line does not transfer. # # THE RUNTIME IS A LOCK, NOT A PREFERENCE. The file is general.file_type 141 -# with 402 tensors of the CUSTOM GGML type 142 (pq2_0) plus prism.hadamard.* -# metadata (normalized-Sylvester-Walsh transform, block 1024, 401 named -# weights, inverse on token_embd, gdn_v_grouped). Stock llama.cpp refuses type -# 142 at load; only the PrismML fork registers the type AND reads the Hadamard -# metadata. The vendor prebuilt build 10754 (commit 2459f68b5) is what every -# number in this file was measured on; sha256 12ae5baba033d8cd… -LLAMA_BIN=/home/dank/GitPackages/llama-prism/llama-server -MODEL_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf -MMPROJ_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf -# The CRACK re-export (dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF, uncensored, -# already on disk at /home/dank/models/Bonsai2-CRACK/) is a ONE-LINE swap: the -# header census proves byte-identical geometry (851 tensors, same -# {pq2_0:402, f32:353, bf16:96} histogram, 64 blocks, vocab 248,320, same -# 8,952-char chat template) and the official file's sha256 is the Hub LFS oid. -# Bonsai-2 is the BASE (censored) Qwen3.8 family; if a compaction or subagent -# request ever comes back with a refusal, point MODEL_FILE at the CRACK file -# and restart — the VRAM ledger below does not move. +# carrying custom GGML type 142 tensors plus prism.hadamard.* metadata +# (normalized-Sylvester-Walsh transform); stock llama.cpp refuses type 142 at +# load and a loader that ignored the Hadamard metadata would dequant WRONG. On +# top of that lock, MTP beside a quantized V cache needs fork build >= b10776, +# so the vendor prebuilt moved from 10754 (commit 2459f68b5, the 27B era) to +# 10777 (commit e3de71522). The 10754 tree stays on disk as the rollback +# runtime for the 27B payload. +LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server +# 2,675,410,720 B, Hub LFS sha256 c79fcdfaf6b7… (verified at fetch by +# scripts/download-frank-zealot-bonsai9.sh). This IS the MTP build: the draft +# head ships INSIDE the checkpoint, so MTP availability is a property of the +# FILE, never of a flag. The plain PQ2_0 sibling could not run it at any flag +# value; loading this file instead of that one is the only change that matters. +MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf +# 624,229,664 B, Hub LFS sha256 e0cb6f69… — Q8_0 IS MANDATORY, not a preference: +# the publisher's card says the BF16 mmproj "can crash llama-server when several +# image requests run at once; Q8_0 is not affected", and this endpoint runs +# --parallel 3, i.e. exactly that shape. The projector's worst case lands AFTER +# the engine's own fit check, so it is never inside a projection number. +MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf +# Base alignment: Bonsai-2 is the censored BASE lineage. The 27B CRACK re-export +# (/home/dank/models/Bonsai2-CRACK/) is byte-identical geometry to the 27B it +# matched, NOT to this 9B file — if a refusal ever surfaces there is no +# one-line swap here; the 9B dealigned export would have to be fetched first. # # Served model id — the harness/family name. The SERVICE, unit, scripts and # this env file keep the frank-zealot name; only the alias on the wire is the @@ -32,40 +42,83 @@ MMPROJ_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0.ggu ALIAS=frank-zealot BIND_ADDR=0.0.0.0 LISTEN_PORT=7771 -# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704 -# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and -# the harness contextWindow quotes the PER-SLOT number — a longer prompt is -# answered with a 400, never truncated. The model's native window is 262,144; -# 2 × 262,144 at q8_0 projects 25,758 MiB and does NOT fit this card (measured -# f16 analogue: 41,118). One 262,144 slot fits with 5,995 MiB free but -# serializes compaction, so the pair stays. -CTX_SIZE=401408 -PARALLEL=2 +# CTX_SIZE is the POOL llama.cpp divides across slots: 786,432 / 3 = 262,144 per +# conversation — each slot at the model's TRAINED window, which is the point of +# the swap (the 27B era had to cut each conversation to 200,704 to fit two). +# MUST stay divisible by PARALLEL (the seam enforces it) and the harness +# contextWindow quotes the PER-SLOT number — a longer prompt is answered with a +# 400, never truncated. The ceiling cannot go past 3 x 262,144: that is the +# native wall (no YaRN in this path) and the arithmetic agrees — a 4th slot +# would push each below the compaction contract. +CTX_SIZE=786432 +PARALLEL=3 BATCH_SIZE=4096 +# Ubatch is a prefill knob (decode never moves) with NO interior optimum on this +# CUDA card up to 2048 — 2048 is the measured optimum (+4.6 % prefill). It has a +# second job here: it sets the compute-buffer size, and the compute buffers are +# what decide whether MTP fits (see the MTP law). 1024 is the documented MTP arm. UBATCH_SIZE=2048 -# KV law (from the GGUF header, then confirmed by the boot ledger): 16 -# full-attention layers (full_attention_interval 4 over 64 blocks) × 4 KV heads -# × 256 dim = 16,384 elements per token per cache. q8_0 bills 17,408 B per -# cache ⇒ 34,816 B/token = 34 KiB, so the 401,408 pool is 13,328.00 MiB — the -# ledger said exactly that. f16 would bill 25,088 MiB and the boot DIES: -# `cudaMalloc failed: out of memory` on the 25,088.00 MiB request, engine fit -# projection 33,198 vs 23,558 free. The 48 DeltaNet layers bill a fixed -# recurrent state per SLOT — ~150 MiB each (context 13,627 − KV 13,328 = 299 -# for two slots), which is why pool arithmetic alone under-prices the shape. +# KV law (header + boot ledger, both confirmed): qwen35 hybrid, 32 layers with +# qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated DeltaNet; +# 4 KV heads × 256 dim = 8,192 elements per token per cache. q8_0 bills 8,704 B +# per cache ⇒ 17,408 B/token = 17 KiB, so the 786,432 pool is 13,056.00 MiB — +# the ledger said exactly that. HALF the 27B's 34,816 B/token, which is the +# whole reason three native windows fit. f16 would bill 26,112 MiB at this pool +# (against a 24,123 MiB card) and dies at cudaMalloc; the 24 DeltaNet layers +# bill a fixed recurrent state per SLOT instead of per token — the ledger's +# "RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)" = 150.75 MiB a +# slot — which is why pool arithmetic alone under-prices the shape. CACHE_TYPE_K=q8_0 CACHE_TYPE_V=q8_0 -# Boot ledger (2026-10-17, this exact shape, nvidia-smi settled): -# 22,664 MiB used / 1,461 MiB free of 24,123 -# fit projection 21,439 = 6,539 model + 13,627 context + 1,272 compute -# the mtmd projector's 873.10 MiB worst-case is NOT inside that projection — -# it lands after the fit check, which is the whole difference between -# "2.6 GiB free on paper" and 1.4 GiB free in reality. Watch it. -# Fallback rung if the margin is ever eaten (desktop, co-tenant): q4_0 KV -# measured 16,392 used / 7,733 free at the same 2 × 200,704. -# 1024 is the ubatch fallback (−616 MiB, measured on the 9B predecessor at -# this card/arch; 2048 was the measured CUDA optimum, +4.6 % prefill). +# SPECULATIVE DECODING: OFF, MEASURED. The head is in the file and the drafter +# is GOOD (acceptance 0.798/0.617 per position, mean accepted length 2.41) — it +# still loses, because the draft context bills its OWN f16 KV over the WHOLE +# pool: 4,096 B/position × 786,432 = 3,072.00 MiB, plus its compute buffers, +# 4,830 MiB all told. Booted arms, same card, same traffic (2026-10-18): +# 3 × 262,144 spec OFF ub 2048 → 17,876 used / 6,249 free, decode 167.2 t/s +# 3 × 262,144 spec ON ub 2048 → 22,706 used / 1,419 free, decode 109.2 t/s +# 3 × 262,144 spec ON ub 1024 → 21,400 used / 2,725 free, decode 213.5 t/s +# 2 × 262,144 spec ON ub 2048 → 17,178 used / 6,947 free, decode 213.2 t/s +# 1 × 262,144 spec ON ub 2048 → 11,652 used / 12,473 free, decode 214.2 t/s +# i.e. MTP is worth +28 % decode whenever the card has room, and at the 3-slot +# pool it costs 1.3 GiB of compute buffers that ubatch 2048 refuses to give up — +# the squeeze, not the drafter, is what halves decode. The arm that WOULD run +# MTP at three windows is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (decode 213.5, +# 3-stream 250.8 aggregate) and pays prefill −17 % and 2.7 GiB of headroom on a +# card whose projector worst-case lands after the fit check. This endpoint's +# load is the fleet's COMPACTION HUB (prefill-bound, ~170k in) plus 2-3 +# concurrent subagents, so the deployed arm takes the +17 % prefill +# (2,877 vs 2,395 t/s at 106,418 tokens) and the +12 % three-stream aggregate +# (280.2 vs 250.8) over single-stream speed, and keeps 6.2 GiB free. +# SPEC_TYPE=none strips the WHOLE spec set (--spec-type, every --spec-draft-*): +# flags move as a set or the arm measures a crash instead of an absence. SPEC_TYPE=none +SPEC_DRAFT_N_MAX=2 +SPEC_DRAFT_N_MIN=0 +SPEC_DRAFT_P_MIN=0 +# The card's sampler recipe lives in the checkpoint (general.sampling.* = temp +# 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0 — confirmed in the +# engine's own /props defaults) and llama.cpp reads it itself. Presence penalty +# is the ONE recipe value with no GGUF metadata key, so it must reach the argv. +# PRESENCE_PENALTY="" is the documented opt-out. +PRESENCE_PENALTY=1.0 NGL=99 +# Boot ledger (2026-10-18, this exact shape, nvidia-smi settled): +# 17,876 MiB used / 6,249 MiB free of 24,123 +# CUDA0 KV 13,056.00 MiB (q8_0 K + q8_0 V, 786,432 positions) +# CUDA0 RS 452.25 MiB (3 recurrent-state cells) +# compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host, projector 595 + 248 MiB +# no clamp line, no cudaMalloc error +# Measured on the same boot: 167.2 t/s greedy decode (/completion, 512 tokens), +# 158.9 t/s single-stream chat, 280.2 t/s aggregate across THREE concurrent +# streams (93.8 each), 2,877 t/s prefill at 106,418 tokens. The 27B record it +# replaces: 76.6 decode, 109.4 aggregate across two streams, 691.7 prefill at +# 166k — beaten on every axis. The FIRST request after a boot is a warmup +# artifact (37.7 t/s class); never report it. +# Fallback rung if the margin is ever eaten (desktop, co-tenant): q4_0 KV on the +# 27B era measured 16,392 used / 7,733 free at 2 × 200,704; the same drop here +# frees ~6.5 GiB at three windows. UBATCH_SIZE=1024 is the cheaper lever +# (−1,306 MiB measured on this shape). # Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes # and graph capture land in the journal, which is what the VRAM-fit forensics # read. Levels above 4 flood the journal. diff --git a/scripts/frank-zealot.service b/scripts/frank-zealot.service index 7841321..36fb78a 100644 --- a/scripts/frank-zealot.service +++ b/scripts/frank-zealot.service @@ -1,5 +1,5 @@ [Unit] -Description=llama.cpp (PrismML build 10754) Ternary-Bonsai-2-27B PQ2_0 GGUF (qwen35 hybrid, 2x200704 slots, q8_0 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-zealot) +Description=llama.cpp (PrismML build 10777) Ternary-Bonsai-2-9B PQ2_0-MTP GGUF (qwen35 hybrid, 3x262144 slots over one 786432 pool, q8_0 K+V, MTP off, mmproj Q8_0 vision) on the RTX 3090 (dankclaw, frank-zealot) After=network-online.target Wants=network-online.target @@ -12,6 +12,13 @@ Group=dank LimitMEMLOCK=infinity Environment="FRANK_ZEALOT_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-zealot.env" ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-zealot.sh +# llama.cpp can take longer than systemd's 90 s default to drain slots and +# release the card: a real stop on 2026-10-18 hit `State 'stop-sigterm' timed +# out. Killing.` and left the unit in `failed` (result 'timeout') even though +# the process had served health correctly. The engine is SIGKILL-safe (nothing +# on disk), but a unit that always ends `failed` is a false alarm for the next +# operator, so give the drain room. +TimeoutStopSec=180 Restart=on-failure RestartSec=10 diff --git a/scripts/serve-frank-zealot.sh b/scripts/serve-frank-zealot.sh index 03f3ee4..9ef001b 100755 --- a/scripts/serve-frank-zealot.sh +++ b/scripts/serve-frank-zealot.sh @@ -1,30 +1,42 @@ #!/usr/bin/env bash -# Serve prism-ml/Ternary-Bonsai-2-27B (PQ2_0 ternary quant) as frank-zealot on -# Dankclaw's RTX 3090 via llama.cpp (PrismML vendor build 10754, CUDA sm_86). +# Serve prism-ml/Ternary-Bonsai-2-9B-gguf (PQ2_0 ternary quant WITH the MTP head +# in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML +# vendor build 10777, CUDA sm_86). # -# 2026-10-17 (Frank): Bonsai-2 PQ2_0 swap at TWO 200,704-token conversations -# (--ctx-size 401408 --parallel 2) with q8_0 K + q8_0 V — f16 KV does not fit -# this geometry: the engine's own fit projected 33,198 MiB against 23,558 free -# and cudaMalloc failed on the 25,088 MiB pool. MTP stays off: this GGUF -# carries no MTP tensors (verified from its tensor-name list). Verbosity 4 -# ships so the VRAM-fit forensics (KV layout, buffer sizes, graph capture) -# land in the journal. +# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export at THREE +# 262,144-token conversations (--ctx-size 786432 --parallel 3), q8_0 K + q8_0 V. +# The 9B bills 17,408 B/token against the 27B's 34,816 (8 full-attention layers +# instead of 16), and that halved rate is what funds three NATIVE windows where +# the 27B could only afford two cut to 200,704. Verbosity 4 ships so the +# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the +# journal. # -# Architecture: qwen35 — 27B, 64 blocks with full_attention_interval 4 (16 -# full-attention layers + 48 Gated DeltaNet linear layers), 24 heads / 4 KV -# heads × 256 dim, native 262,144, vocab 248,320. The KV pool scales with the -# 16 full layers: 34 KiB/token at q8_0 K + q8_0 V (13,328 MiB at the 401,408 -# pool); the DeltaNet layers cost a fixed recurrent state per SLOT (~150 MiB — -# boot-ledger context minus KV). Boot ledger: 22,664 used / 1,461 free, with -# the Q8_0 projector's 873 MiB landing AFTER the engine's fit projection. -# The runtime is a hard lock, not a preference: the file carries 402 tensors -# of custom GGML type 142 plus prism.hadamard.* metadata, which only the -# vendor build registers — stock llama.cpp refuses it at load. -# Vision comes from the publisher's Q8_0 projector, so --mmproj is what gives -# this endpoint eyes; MMPROJ_FILE="" is the documented text-only opt-out. +# Architecture (engine ledger, not the marketing page): qwen35 hybrid, 32 +# layers with qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated +# DeltaNet, 4 KV heads × 256 dim, native 262,144. The KV pool scales with the 8 +# full layers (13,056.00 MiB at the 786,432 pool, ledger-exact); the DeltaNet +# layers cost a fixed recurrent state per SLOT — 452.25 MiB for 3 cells, ~151 +# MiB each. The runtime is a hard lock, not a preference: custom GGML type 142 +# plus prism.hadamard.* metadata, which only the vendor build registers, and +# MTP beside quantized V needs build ≥ b10776 — so the prebuilt moved 10754 → +# 10777 with the payload. Vision comes from the publisher's Q8_0 projector (the +# BF16 one can crash the server on concurrent images, and this endpoint runs +# three slots), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is +# the documented text-only opt-out. # -# Sampling: llama.cpp takes it per request (the fleet harness sends the file's -# own template defaults from models.yml: thinking temp 1.0 / top_p 0.95 / top_k 20). +# Speculative decoding ships OFF, measured: the head loads and accepts well +# (0.798/0.617 per position, mean length 2.41) but the draft context bills its +# own f16 KV over the WHOLE pool — 4,096 B/position × 786,432 = 3,072 MiB plus +# buffers — and at ubatch 2048 that squeeze HALVES decode (167.2 → 109.2 t/s). +# frank-zealot.env carries the six-arm table and the arm that turns it back on +# (--ubatch-size 1024, decode 213.5 t/s) with its price. +# +# Sampling: the checkpoint carries the publisher's thinking recipe as GGUF +# metadata (general.sampling.* = temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 / +# repeat_penalty 1.0 — confirmed in the engine's own /props defaults) and +# llama.cpp reads it; --presence-penalty is the one recipe value with no +# metadata key, so this script passes it. A request may still override per call +# (the fleet harness sends the same values). # # Tunables come from frank-zealot.env unless FRANK_ZEALOT_ENV_FILE overrides. set -euo pipefail @@ -39,45 +51,62 @@ if [[ -f "$ENV_FILE" ]]; then fi # --- defaults (used when the env file does not set them) ------------------- -: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism/llama-server}" -: "${MODEL_FILE:=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf}" +: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism-b10777/llama-server}" +: "${MODEL_FILE:=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf}" : "${ALIAS:=frank-zealot}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7771}" -# ctx-size is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704 -# per conversation, inside the model's trained 262,144. The harness +# ctx-size is the POOL llama.cpp divides across slots: 786,432 / 3 = 262,144 +# per conversation — every slot at the model's trained window. The harness # contextWindow MUST quote the PER-SLOT number — a longer prompt is answered # with a 400, never truncated. -: "${CTX_SIZE:=401408}" -: "${PARALLEL:=2}" +: "${CTX_SIZE:=786432}" +: "${PARALLEL:=3}" # Compute-buffer geometry. Ubatch is a prefill knob (decode never moves). # 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill # at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and -# throughput-neutral at a fixed ubatch. +# throughput-neutral at a fixed ubatch. It ALSO sets the compute-buffer size, +# which is the term that decides whether an MTP draft context fits beside a +# 786,432-position pool: 1024 hands back 1,306 MiB and is the arm that runs the +# drafter at three slots (frank-zealot.env carries the table). : "${BATCH_SIZE:=4096}" : "${UBATCH_SIZE:=2048}" -# Quantized KV: q8_0 K + q8_0 V — the only pair that funds 2 × 200,704 on this -# card. 34 KiB/token at this geometry (16 full-attention layers × 4 KV heads × -# 256 dim); 13,328 MiB at the 401,408 pool. f16 measured OOM; q4_0 is the -# fallback rung (16,392 used / 7,733 free). Explicit either way so the contract -# is visible in the argv, and --flash-attn stays required for quantized V -# (no true fp8 KV dtype exists in this build). +# Quantized KV: q8_0 K + q8_0 V — the only pair that funds 3 × 262,144 on this +# card. 17 KiB/token at this geometry (8 full-attention layers × 4 KV heads × +# 256 dim); 13,056.00 MiB at the 786,432 pool, ledger-exact. f16 would bill +# 26,112 MiB against a 24,123 MiB card and dies at cudaMalloc; q4_0 is the +# fallback rung. Explicit either way so the contract is visible in the argv, +# and --flash-attn stays required for quantized V (no true fp8 KV dtype exists +# in this build). : "${CACHE_TYPE_K:=q8_0}" : "${CACHE_TYPE_V:=q8_0}" -# Speculative decoding is OFF: this PQ2_0 file carries NO nextn/mtp tensors, so -# there is no in-file head (the law: MTP is a property of the file, not of the -# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set); -# an MTP-bearing CRACK re-export exists upstream (ManniX-ITA, +451 MB) if -# decode ever needs it — that is a new payload with its own VRAM bill, never a -# flag flip on this file. +# Speculative decoding defaults OFF, measured on this card and shape: the head +# IS in the checkpoint (qwen35.nextn_predict_layers = 1 — MTP is a property of +# the FILE, so the plain PQ2_0 sibling could not run it at any flag value), it +# loads, and it accepts well (mean accepted length 2.41) — but the draft context +# bills its own f16 KV over the WHOLE pool (4,096 B/position, 3,072 MiB here) +# and at ubatch 2048 that squeeze halves decode: 167.2 → 109.2 t/s. There is no +# --spec-draft-cache-type knob in this build to make the draft cheap, and no +# --spec-draft-model either: the draft rides the main model's weights. Card gate: +# quantized V alongside MTP needs fork build >= prism-b10776, which is why +# LLAMA_BIN moved to the 10777 prebuilt; on an older build the pair is illegal. +# SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 is the documented MTP arm (213.5 t/s); +# SPEC_TYPE=none strips the whole set (flags move together). : "${SPEC_TYPE:=none}" -: "${SPEC_DRAFT_N_MAX:=3}" +: "${SPEC_DRAFT_N_MAX:=2}" +: "${SPEC_DRAFT_N_MIN:=0}" +: "${SPEC_DRAFT_P_MIN:=0}" : "${SPEC_DRAFT_FILE:=}" +# No colon on purpose, same convention as MMPROJ_FILE: an explicitly empty +# PRESENCE_PENALTY is the documented opt-out and must NOT be replaced by the +# default (a colon treats null as unset and would silently re-add the flag). +: "${PRESENCE_PENALTY=1.0}" : "${NGL:=99}" # Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info); # 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what -# the VRAM-fit forensics need. Levels above 4 flood the journal. -: "${VERBOSITY:=3}" +# the VRAM-fit forensics need — the q8_0 KV contract and the draft-KV bill are +# only provable from this log. Levels above 4 flood the journal. +: "${VERBOSITY:=4}" # /metrics needs the flag: llama.cpp returns 501 unless the server starts with # --metrics, and Dankclaw's Prometheus scrapes every fleet endpoint's /metrics — # a 501 reads as DOWN, which is how the critical LlmEndpointDown alert ended up @@ -89,7 +118,7 @@ fi : "${METRICS:=1}" # Vision projector. No colon on purpose: an explicitly empty value means # "text-only, on purpose" and must NOT be replaced by the default path. -: "${MMPROJ_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf}" +: "${MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}" # Fail loudly rather than half-loading: a missing file is a download that # has not finished, and Restart=always would turn it into a crash loop that @@ -109,15 +138,31 @@ if [[ "$METRICS" = "1" ]]; then EXTRA_ARGS+=(--metrics) fi -# Speculative flags move as a set: they reach the argv ONLY when a drafter -# is named. SPEC_TYPE=none (the shipped contract) strips both — a no-spec -# baseline must not carry half a spec pair. +# Speculative flags move as a SET: they reach the argv ONLY when a drafter is +# named, and all of them together — a no-spec baseline must not carry half a +# spec pair. n-max 2 / n-min 0 / p-min 0 is the publisher's recipe ("Draft depth +# 2 works best"); the MTP head is IN the checkpoint (blk.32.nextn.*, +# qwen35.nextn_predict_layers = 1), so there is no --spec-draft-model here and +# the draft context rides the MAIN model's weights. if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then - EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX") -if [[ -n "$SPEC_DRAFT_FILE" ]]; then - [[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-zealot: missing $SPEC_DRAFT_FILE" >&2; exit 1; } - EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE") + EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" + --spec-draft-n-max "$SPEC_DRAFT_N_MAX" + --spec-draft-n-min "$SPEC_DRAFT_N_MIN" + --spec-draft-p-min "$SPEC_DRAFT_P_MIN") + if [[ -n "$SPEC_DRAFT_FILE" ]]; then + [[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-zealot: missing $SPEC_DRAFT_FILE" >&2; exit 1; } + EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE") + fi fi + +# The card's sampler recipe lives partly in the checkpoint: general.sampling.* +# (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0) is read by +# llama.cpp itself, and --repeat-last-n already defaults to 64. Presence penalty +# is the ONE value with no GGUF metadata key — the card says so explicitly — so +# it is the one knob that must reach the argv or the recipe is silently +# incomplete. PRESENCE_PENALTY="" is the documented opt-out. +if [[ -n "$PRESENCE_PENALTY" ]]; then + EXTRA_ARGS+=(--presence-penalty "$PRESENCE_PENALTY") fi # The UVM device node is created lazily by nvidia-modprobe on the first CUDA diff --git a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md index 15ded01..0a20bb0 100644 --- a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md +++ b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md @@ -21,10 +21,10 @@ https://github.com/open-webui/open-webui | Item | Value | |------|-------| | Image | `ghcr.io/open-webui/open-webui:main` (verified from the live quadlet unit) | -| Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). | +| Deployed build | `0.11.4` — `GET http://127.0.0.1:3005/api/version` → `{"version":"0.11.4"}` (2026-10-18). `:main` is the rolling image, not a stable pin, so the running build lags upstream: releases `0.12.0` (2026-10-10) through `0.12.3` (2026-10-16) exist and are not deployed (https://github.com/open-webui/open-webui/releases). The 2026-09-21 `0.11.4` security/access-control advisory is what pinned this era of the page. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). | | Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. | | Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` | -| Backends | Picker exposes exactly one model — `frank-zealot` (host.containers.internal:7771). The `frank-dragoon` and `frank-archon` connections remain configured in the DB, disabled (2026-10-07) — see [The LLM backends](#the-llm-backends) | +| Backends | The picker's content is `webui.db` state, not repo state: as recorded on 2026-10-07 it exposes **one** model, `frank-zealot`, with `frank-dragoon`/`frank-archon` configured but disabled — *unverified 2026-10-18* (the DB is the authority and is not readable from the repo). What the repo does fix is the container's own OpenAI target: `OPENAI_API_BASE_URL=http://192.168.0.143:7770` (`dankclaw/open-webui/quadlet/open-webui.container:32`, same value in the live container env) — see [The LLM backends](#the-llm-backends) | | Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) | | Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` | | Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` | @@ -34,19 +34,30 @@ https://github.com/open-webui/open-webui The picker is built from the `model` table in `webui.db` (one row per model) plus the `model_ids` of each **enabled** OpenAI connection, NOT from live connection discovery — so the picker shows exactly what is seeded there. -As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`**. +As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unverified +2026-10-18: `webui.db` is the authority; the fleet itself serves all three ids +`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in +`shared/homepage/services.json`)*. -| Connection (`openai.api_configs` idx) | URL | Enabled | `model_ids` | Runtime | Host | +| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Runtime | Host | |---|---|---|---|---|---| | 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw | | 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver | -| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090, 2×200,704) | Dankclaw | +| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090; production shape 3×262,144 per `scripts/frank-zealot.env`) | Dankclaw | + +The URLs in the first column are the DB's recorded values and predate the 2026-08-22 +bridge move; on this network they are wrong — see the reachability gotcha below, which +is the corrected law. Note also that the 2026-10-18 **bench window** on `:7771` +closed the same day (production unit stopped 03:38 per the repo Benchmarking +Policy while candidate arms held the port): the candidate — Ternary-Bonsai-2-**9B** +PQ2_0 with the MTP head, 3 × 262,144 — was **deployed as production**, so the +picker row below was re-keyed to it (see the next paragraph). The `model` table carries exactly one decoration row, `frank-zealot` -(context length 200,704 = the per-slot window; description since 2026-10-17: -Ternary-Bonsai-2-27B PQ2_0 + Q8_0 vision mmproj — the row's `description` is -live DB state, so a payload swap must re-key it in the database too, not only -in the repo). +(context length 262,144 = the per-slot window since 2026-10-18; description +re-keyed the same day to Ternary-Bonsai-2-9B PQ2_0 + MTP + Q8_0 vision mmproj — +the row's `description` is live DB state, so a payload swap must re-key it in +the database too, not only in the repo). A third gate sits *after* the table, and it is the one that bites family accounts (2026-10-07): a model row is only visible to a non-admin user when @@ -66,15 +77,22 @@ Two gotchas make the wiring non-obvious: OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses the llama.cpp endpoints (frank-dragoon, frank-zealot). The fix is to set `model_ids` in each connection's `api_config` — when non-empty, OpenWebUI - builds the model list directly and never fetches `/v1/models`. frank-medium - (vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids` - too for consistency. -- **The host's own services are reached via `host.containers.internal`.** The - container sits on a netavark bridge; it can hairpin to a *different* host's - LAN IP (dankserver `192.168.0.235`) but not to its *own* host's non-published - ports via the host LAN IP (`192.168.0.143:7770/7771` is refused). - `host.containers.internal` (169.254.1.2) resolves to the host on the bridge - and reaches both llama.cpp services. + builds the model list directly and never fetches `/v1/models`. The old + `frank-medium` (vLLM) endpoint — retired, no longer in the fleet — served the + OpenAI shape and would have discovered fine, but used `model_ids` too. +- **The host's own services are reached via the host LAN IP, not + `host.containers.internal`.** The container sits on the pinned kernel bridge + `systemd-open-webui` (10.89.3.0/24, gateway 10.89.3.1, container IP 10.89.3.2 — + `podman inspect open-webui` 2026-10-18), and `host.containers.internal` + (169.254.1.2) is a **slirp4netns/pasta-only name that does not resolve on this + bridge** (`dankclaw/open-webui/quadlet/open-webui.network:15-19`). The deployed + unit therefore points at the host LAN IP: `OPENAI_API_BASE_URL=http://192.168.0.143:7770` + and `SEARXNG_QUERY_URL=http://192.168.0.143:8081/search` + (`open-webui.container:32,78`, both present in the live container env, 2026-10-18). + Those ports are published on all interfaces, so the LAN IP answers: + `ss -ltn` shows `*:7770` and `0.0.0.0:7771`, and `curl http://192.168.0.143:7771/health` + → 200 (2026-10-18). The Dankserver backend is a different host and is reached + through its LAN IP `192.168.0.235:7770` either way. The Ollama connection is disabled (`ollama.enable=false`). @@ -140,8 +158,14 @@ role (`ui.default_user_role=pending`), which cannot chat. Verified `GET /api/v1/models/list`, `GET /api/v1/models` and `POST /api/chat/completions` all return **401** with no key (also with a bogus bearer token); only `/api/config`, `/api/version` and `/health` -answer publicly, and none of them expose the model list. An external -program needs an account + API key — there is no keyless access. +`POST /api/chat/completions` all return **401** with no key (also with a bogus bearer +token); only `/api/config`, `/api/version` and `/health` answer publicly, and none of +them expose the model list. An external program needs an account + API key — there is +no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}`, +`/api/version` → 0.11.4, `/api/v1/models/list`, `/api/v1/models` and +`POST /api/chat/completions` → 401; `/api/config` additionally publishes +`features.enable_login_form=false`, `features.enable_signup=false` and +`oauth.providers.oidc="authentik"` — the SSO-only posture, visible from outside. ## Web search @@ -149,8 +173,11 @@ program needs an account + API key — there is no keyless access. Chat web search is served by [[SearXNG]] on this host, wired in the quadlet (`ENABLE_WEB_SEARCH`, `WEB_SEARCH_ENGINE=searxng`, `SEARXNG_QUERY_URL`). -- **Access path:** the backend calls `http://192.168.0.143:8081/search?format=json` - directly — gluetun's published port, NOT the `search.franksdank.cloud` vhost. +- **Access path:** the quadlet sets `SEARXNG_QUERY_URL=http://192.168.0.143:8081/search` + (`dankclaw/open-webui/quadlet/open-webui.container:78`, live in the container env) — + gluetun's published port, NOT the `search.franksdank.cloud` vhost. The JSON format + parameter is appended by OpenWebUI itself *(unverified 2026-10-18: OpenWebUI-side + behaviour; the env value carries no `format=json`)*. The Anubis→Authentik guard chain is a browser mechanism (proof-of-work + session flow); a plain HTTP client cannot solve it, and the internal port bypasses Caddy entirely. @@ -247,9 +274,11 @@ chat LLMs only), so ComfyUI needs its own image-generation connection configured in OpenWebUI. ComfyUI itself runs GPU-enabled on Dankclaw and serves `:8188`. -Endpoint: `POST /api/images/generations` (image `0.11.x`: upstream moved the -route to `POST /api/v1/images/generations`; the `comfyui` engine is still -upstream) with `Authorization: Bearer ` and body +Endpoint: `POST /api/v1/images/generations` — measured on the running build +(`0.11.4`): the legacy `POST /api/images/generations` answers **405** while the `/v1` +route answers **401** without a token, i.e. only the `/v1` path exists (2026-10-18). +The `comfyui` engine is still upstream. Call it with `Authorization: Bearer ` +and body `{"model":"imagegen","prompt":"…","n":1,"size":"1024x1024"}`; the response carries base64 image data or a URL. @@ -267,7 +296,7 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad. - The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].) - The default admin account is created on first login; secure it immediately. - If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready. -- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`. +- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via the **host LAN IP `192.168.0.143`**, which is exactly what the deployed unit does (`open-webui.container:32`). The older "use `host.containers.internal`" advice is retired for this container: it is a slirp4netns/pasta-only name and the unit has been on the pinned bridge `systemd-open-webui` since 2026-08-22, where that name does not resolve (`open-webui.network:15-19`). Both endpoints are published on all interfaces today (`ss -ltn` → `*:7770`, `0.0.0.0:7771`; `curl http://192.168.0.143:7771/health` → 200, 2026-10-18). The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`. ## History diff --git a/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md b/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md index 9691ef1..8ddce4e 100644 --- a/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md +++ b/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md @@ -1,7 +1,7 @@ --- type: reference -title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork), Bonsai-2-27B PQ2_0, 2 x 200704, q8_0 KV (ACTIVE)" -description: "The ACTIVE frank-zealot runbook (since 2026-10-17) — the PrismML fork build of llama.cpp serving prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090: two conversations of 200,704 tokens with q8_0 K + q8_0 V (f16 KV does not fit this card at this window), vision through the publisher's Q8_0 Qwen3-VL projector, MTP impossible (no tensors in the file), and the measured four-arm VRAM census that priced the shape." +title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10754), Bonsai-2-27B PQ2_0, 2 x 200704, q8_0 KV (RETIRED, rollback target)" +description: "The frank-zealot runbook of 2026-10-17 → 2026-10-18 (RETIRED — rollback target) — the PrismML fork build 10754 of llama.cpp serving prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090: two conversations of 200,704 tokens with q8_0 K + q8_0 V (f16 KV does not fit this card at this window), vision through the publisher's Q8_0 Qwen3-VL projector, MTP impossible (no tensors in the file), and the measured four-arm VRAM census that priced the shape. Superseded by the 9B MTP export at 3 × 262,144." tags: - llm - frank-zealot @@ -14,12 +14,41 @@ timestamp: 2026-10-17 # frank-zealot — llama.cpp (PrismML) Bonsai-2-27B PQ2_0, 2 × 200,704, q8_0 KV -Live since 2026-10-17, replacing the -[[frank-zealot-llama-qwen35-9b-q6k-2x200704]] era (Qwen3.5-9B Q6_K, exact f16 -KV, same 2 × 200,704 shape). The trade: the ternary 27B costs ~3.4× the weights -of the 9B and buys a 3× larger, stronger base at the SAME window contract — -paid for by quantizing KV (f16 KV is arithmetically impossible for this -payload on this card, measured below). +**RETIRED 2026-10-17 → 2026-10-18** (AGENTS.md Runbook Index), superseded by +[[frank-zealot-llama-prism-bonsai9-mtp]] — prism-ml/Ternary-Bonsai-2-**9B** +PQ2_0 **with the MTP head**, THREE conversations at the native 262,144 window +on fork build 10777. This era stays the **rollback target**: the payload +(`/home/dank/models/Bonsai2-PQ2_0/`) and the 10754 fork tree +(`~/GitPackages/llama-prism/`) are both still on disk. It owns the laws below — +the 34,816 B/token KV figure over 16 full-attention layers, the 25,088 MiB f16 +OOM wall, and the two-boot VRAM pricing that every later shape on this card +reused. + +## 2026-10-18 bench window — production unit stopped, candidate arm on 7771 + +As of 2026-10-18 frank-zealot sits in a deliberate, dated **bench window** +(fleet Benchmarking Policy: stop the production unit, bench candidate boots +against the service itself). The journal shows the stop at 03:38:31 and the +known llama.cpp shutdown shape — SIGTERM ignored past `TimeoutStopSec`, +`State 'stop-sigterm' timed out. Killing.` at 03:40:01 — after which the unit +reports `failed` (`is-enabled` → enabled); self-healing is merely parked +while the window runs. What answers 7771 is a **hand-booted candidate arm** +(`llama-prism-b10777/llama-server` on the 9B MTP file +`Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, alias +`frank-zealot`), not this runbook's payload: candidate-arm flags are not a +state — the pid and shape moved between two censuses ten minutes apart (one +argv read `--ctx-size 786432 --parallel 3 --spec-type draft-mtp`; a later +`/props` read `n_ctx 262144`, `speculative.types=none`); quote neither, and +expect `/v1/models` to report the candidate until the window closes. The +window's remedy is the redeploy: `./scripts/deploy.sh dankclaw --service +frank-zealot`. The production trio below is unchanged — the last PRODUCTION +boot under the unit (03:14–03:38 the same morning) printed the contract +exactly: `n_ctx = 401408`, `n_ctx_seq = 200704`, `CUDA0 KV buffer size = +13328.00 MiB`, 2 slots (journal). + +**Window CLOSED 2026-10-18:** that candidate became production. The redeploy +shipped the 9B MTP export at `3 × 262,144` on fork build 10777 — see +[[frank-zealot-llama-prism-bonsai9-mtp]]; this payload is the rollback target. ## Identity diff --git a/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md b/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md new file mode 100644 index 0000000..69572e0 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md @@ -0,0 +1,222 @@ +--- +type: reference +title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 3 × 262144, q8_0 KV (ACTIVE)" +description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: THREE conversations of 262,144 tokens (every slot at the model's trained window) over q8_0 K + q8_0 V, vision through the publisher's Q8_0 Qwen3-VL projector, and the five-arm boot census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head." +tags: + - llm + - frank-zealot + - dankclaw + - llama-cpp + - gguf + - rtx3090 + - mtp + - spec-decode +timestamp: 2026-10-18 +--- + +# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 3 × 262,144, q8_0 KV + +Live since 2026-10-18, replacing the +[[frank-zealot-llama-prism-bonsai2-pq20]] era (Ternary-Bonsai-2-**27B** PQ2_0 at +2 × 200,704). The trade in one sentence: the 9B bills **half** the KV per token +of the 27B, and that is exactly what funds **three** conversations at the +model's **native 262,144** window where the 27B afforded only two shortened +ones (Frank's ask: *"if you can possibly fit a third context, do it"*). Every +axis improved — decode, prefill, three-stream aggregate, and free VRAM. + +## Payload identity + +| Item | Value | +|---|---| +| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork** build, `--version` → `0.2.0-dev (build 10777, commit e3de71522)` | +| Checkpoint | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` — 2,675,410,720 B, Hub-LFS sha256 `c79fcdfaf6b7bf47089b50dadcfd11f9ce7d3e96adf3bbefde7912e5ef69e22b` | +| Vision | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, Hub-LFS sha256 `e0cb6f6988cea323ef798cd77d019bbb819af75582f034eaab0e7232314953db` | +| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (`--verify-only`; `-f`, byte-count and LFS sha256 per file) | +| Alias | `frank-zealot` (`ALIAS`); the SERVICE/unit/scripts keep the StarCraft name, only the wire id is the model id | +| Port | 7771, root systemd unit `frank-zealot.service` (relinked), `TimeoutStopSec=180` | + +### The runtime lock has two tumblers now + +1. **The tensor type.** `general.file_type 141` with **202 tensors of the fork's + PQ2_0 type** (of 442 total; the rest are f32 norms/scalars and 8 q8_0 in the + MTP block) plus `prism.hadamard.*` metadata + (`normalized-sylvester-walsh-hadamard`, `block_size 4096`, + `axis input-last-dimension`, `sign_mode explicit` over 201 weight names, + inverse on `token_embd.weight`, `gdn_v_grouped true`). Stock llama.cpp + refuses the custom type at load; a loader that ignored the Hadamard metadata + would dequant **wrong**, silently. Same lock class as the ROCmFPX era. +2. **The build floor.** The fork's card gates **MTP beside a quantized V cache** + on `prism-b10776-d00e252` or newer, so the prebuilt moved **10754 → 10777** + for this swap. The 10754 tree stays on disk (`~/GitPackages/llama-prism/`) as + the 27B rollback runtime. `tests/frank_zealot_unit.bats` pins both paths. + +## Header census (read from the GGUF, not the card) + +`general.architecture qwen35`, `general.name Ternary-Bonsai-2-9B`, +`general.size_label 9B`, `general.quantization_version 2`, +`general.file_type 141`: + +| Key | Value | Note | +|---|---|---| +| `qwen35.block_count` | **33** | 32 trunk blocks + `blk.32` = the MTP block | +| `qwen35.full_attention_interval` | 4 | ⇒ **8 full-attention + 24 Gated DeltaNet** trunk layers | +| `qwen35.context_length` | 262,144 | native; the deployed slot equals it | +| `qwen35.embedding_length` / `feed_forward_length` | 4,096 / 12,288 | | +| `attention.head_count` / `head_count_kv` | 16 / **4** | | +| `attention.key_length` / `value_length` | 256 / 256 | | +| `ssm.conv_kernel` / `state_size` / `group_count` | 4 / 128 / 16 | the 27B: 4 / 128 / 16 | +| `ssm.time_step_rank` / `inner_size` | **32 / 4,096** | the 27B: 48 / 6,144 | +| `rope.freq_base` / `dimension_count` / `dimension_sections` | 1e7 / 64 / `[11,11,10,0]` | | +| `qwen35.nextn_predict_layers` | **1** | the MTP head is IN the file | +| `general.sampling.*` | temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / repeat 1.0 | the card recipe; llama.cpp reads it itself | + +**KV accounting.** Only the 8 full-attention layers scale with context: +8 × 4 KV heads × 256 dim = **8,192 elements/token per cache**, so +**q8_0 K + q8_0 V bills 17,408 B/token** — the 786,432-position pool is +**13,056.00 MiB**, ledger-exact. That is exactly HALF the 27B's 34,816 B/token. +**f16 KV would bill 26,112 MiB against a 24,123 MiB card** and dies at +`cudaMalloc`. The 24 DeltaNet layers bill a fixed recurrent state per SLOT +instead — `RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs)` ≈ 150 MiB +each — so pool arithmetic alone under-prices the shape. + +## The shape (`scripts/frank-zealot.env`) + +`--ctx-size 786432 --parallel 3` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`, +`--cache-type-k q8_0 --cache-type-v q8_0`, `--batch-size 4096`, +`--ubatch-size 2048`, `--n-gpu-layers 99`, `--jinja`, `--reasoning auto`, +`--reasoning-format deepseek`, `--metrics`, `--mmproj …`, `--log-verbosity 4`, +`--presence-penalty 1.0`, +**`SPEC_TYPE=none`** (which strips `--spec-type` AND every `--spec-draft-*`). + +Per-slot window is the **contract** the harness quotes: `models.yml` carries +`contextWindow: 262144`. llama.cpp refuses a prompt longer than the SLOT with a +400 naming the limit — it never truncates — and since this swap the slot window +equals the native window, so no YaRN and no cap is in play. + +## The MTP verdict: the head is in the file and it is still OFF + +The drafter is **good**: acceptance 0.798 / 0.617 per position, mean accepted +length 2.41, and the card recommends depth 2. It loses **at this shape** +because the draft context bills its own **f16** KV over the WHOLE pool — +`blk.32` carries real attention (4 KV heads × 256 × 2 caches × 2 B = +**4,096 B/position**) × 786,432 positions = **3,072.00 MiB**, 4,830 MiB with its +compute buffers — and `--ubatch-size 2048` refuses to give that up. + +Booted arms, same card, same traffic, 2026-10-18 (`/completion` `timings`, +greedy 512 tokens; never the first request after a boot): + +| slots | spec | ubatch | used / free | decode | 3-stream | prefill @106k | +|---|---|---|---|---|---|---| +| **3 × 262,144** | **off** | **2048** | **17,876 / 6,249** | **167.2** | **280.2** | **2,877** | +| 3 × 262,144 | on | 2048 | 22,706 / 1,419 | 109.2 | — | — | +| 3 × 262,144 | on | 1024 | 21,400 / 2,725 | 213.5 | 250.8 | 2,395 | +| 2 × 262,144 | on | 2048 | 17,178 / 6,947 | 213.2 | — | — | +| 1 × 262,144 | on | 2048 | 11,652 / 12,473 | 214.2 | — | — | + +**MTP is worth +28 % single-stream decode whenever the card has room** (rows +4–5 prove the drafter itself is a win; row 2 proves the squeeze is what halves +it). The arm that runs MTP at three windows is `SPEC_TYPE=draft-mtp` + +`UBATCH_SIZE=1024` (row 3) and pays **prefill −17 %** and **2,725 MiB of +headroom** on a card whose mmproj worst-case lands AFTER the engine's own fit +check. This endpoint is the fleet's **compaction hub** — prefill-bound ~170k +inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 % +prefill and +12 % three-stream aggregate over single-stream speed, and keeps +6.2 GiB free. Frank confirmed this arm on 2026-10-18. + +## Boot ledger (deployed arm, verbosity 4) + +``` +CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions +CUDA0 RS buffer size = 452.25 MiB # 3 recurrent-state cells, 32 layers, 3 seqs +compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host +mmproj 595 MiB + 248 MiB # charged AFTER the fit check +nvidia-smi settled: 17,876 MiB used / 6,249 MiB free of 24,123 +``` + +No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class +numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`, +never the projection**. + +## Measured (deployed arm) + +- **167.2 t/s** greedy decode (512 tokens, `/completion` `timings`); 158.9 t/s + single-stream chat with the card's sampler. +- **280.2 t/s aggregate** across three concurrent streams (≈93.8 each). +- **2,877 t/s prefill** at 106,418 prompt tokens. +- The 27B record it replaces: 76.6 decode, 109.4 aggregate across two streams, + 691.7 prefill at 166k. Beaten on every axis. +- The **first request after a boot is a warmup artifact** (37.7 t/s class) — + never report it. + +## Contract traps + +- **`--metrics` is mandatory.** Without it llama.cpp answers **501** on + `/metrics`, so the Prometheus scrape reads DOWN while `/health` answers 200 — + that is how this endpoint sat red across four payload generations + (`up{job="frank-zealot"}` oldest retained sample, 2026-10-11, is already 0). +- **Q8_0 mmproj, not BF16.** The publisher's card: the BF16 tower "can crash + llama-server when several image requests run at once; Q8_0 is not affected" — + and `--parallel 3` is precisely that shape. +- **Only the `image_url` content part reaches the model.** The legacy top-level + `images` field bumps `usage.prompt_tokens` and the model still invents an + answer. Probe with a solid color and expect the exact word. +- **≥256 px probe images** or the model answers "video". +- **Thinking tokens count against `max_tokens`.** A vision probe at 64 tokens + returns EMPTY `content` + `finish_reason: "length"` — the endpoint looks blind + while it is only mid-thought. Budget ≥512. +- **`presence_penalty` has no GGUF metadata key**, so the one recipe value that + must reach the argv is this one. The serve script uses + `${PRESENCE_PENALTY=1.0}` — **not `:=`** — because `""` is the documented + opt-out and `:=` would silently re-add the flag (caught by the seam, fixed + 2026-10-18). +- **Spec flags move as a SET.** A bare `--model-draft`/`--spec-type` arm without + the whole set has segfaulted at load on this fork family in the past; + `SPEC_TYPE=none` strips all of them. +- **Base alignment is the censored base lineage, and the CRACK swap is NOT + one line here.** `/home/dank/models/Bonsai2-CRACK/` matches the **27B** + geometry, not this 9B file; a dealigned 9B would have to be fetched. +- **A raw `/completion` request** logs `failed to parse reasoning content + (… parse_reasoning=true)` when the model closes a bare prompt with `` — + cosmetic (`probe_request.cpp:789`); `/v1/chat/completions` parses + `reasoning_content` correctly. Do not lower `VERBOSITY`: the KV/buffer ledger + is what VRAM forensics read. +- **A failed stop leaves a `failed` unit, and an orphan keeps answering.** On + 2026-10-18 the trial stop hit `State 'stop-sigterm' timed out. Killing.` → + `Active: failed (Result: timeout)`, while the hand-booted arm kept serving + 7771 — a live port with no unit behind it. `TimeoutStopSec=180` is now on the + unit (llama.cpp needs the drain time to release slots and the card); the + hand-booted arm must be killed by PID, never `pkill -f llama-server`. + +## Compaction hub + +`models.yml` routes `compactionModel` for `frank-archon`, `frank-dragoon` and +`frank-zealot` here — 262,144 is the widest window in the fleet and the only one +that comfortably fits a worst-case compaction input (~170k tokens; the clamp is +`min(maxTokens 16384, contextWindow − est − 4096)`, compaction fires at 85 % of +the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With `--parallel 3` +a compaction request no longer serializes behind a live generation (the Gemma4 +era's `--parallel 1` law). + +## Fallback rungs + +1. `UBATCH_SIZE=1024` — frees **1,306 MiB** measured on this shape (prefill + −17 %). +2. `CACHE_TYPE_K/V=q4_0` — frees ~6.5 GiB at three windows (the 27B era measured + the q4_0 rung at 16,392 used / 7,733 free at 2 × 200,704). +3. Cut to 2 slots — but that breaks the third concurrent subagent, so it is a + last resort, not a knob. + +## Rollback + +Revert the swap commit. On disk: the **27B payload** +(`/home/dank/models/Bonsai2-PQ2_0/`) + the **10754** fork tree, the Qwen3.5-9B +Q6_K payload (`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`) + master CUDA build, +the Gemma4, MiMo, Swift and Ornith payloads. Then +`./scripts/deploy.sh dankclaw --service frank-zealot`. + +## Related + +- [[frank-zealot-llama-prism-bonsai2-pq20]] — the retired 27B era (rollback target) +- [[Llama Cpp Serving Sizing]] — the two-boot VRAM pricing method +- [[frank-dragoon-flash-server-v2-native]] — the other 2026-10-18 cutover +- [[deployment]] — deploy flow and incidents diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 8cbe984..637ad30 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -11,15 +11,19 @@ TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache (explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the 30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2, -froggeric v22.5 template. Rollback: the in-house Swift1.5 w4dn8 `.rad` (on -disk); tags frank-archon-freetoken-pr587-20261005 and -frank-archon-vllm-shapeG-20261003), +froggeric v22.5 template. Rollback: revert the swap commit and re-download the +in-house Swift1.5 w4dn8 `.rad` byte-exact (purged from disk 2026-10-17; Hub +`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`); tags +frank-archon-freetoken-pr587-20261005 and frank-archon-vllm-shapeG-20261003 +(AGENTS.md Runbook Index; `find /home/dank/models -name '*.rad'` on +Dankserver 2026-10-18 lists only the iq4r file), and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2 (frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned digest serving the NATIVE v2 conversion of the abliterated Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine, MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE -401,408-position KV pool at ctx 200704, vision ON)** and +200,704-position KV pool at ctx 200704 — the 2 × ctx = 401,408 raise is +committed 2026-10-18 but NOT yet deployed, vision ON)** and **llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48 Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled, @@ -54,7 +58,7 @@ gfx1151 iGPU 114 GiB GTT] E[flash-server container gfx1151, digest-pinned image frank-dragoon, port 7770] F[flash-server v2 native abliterated .hgn + ngram table + vision tower -ONE 401408-position KV pool at ctx 200704, 2 slots, MTP always-on, image input] +ONE 200704-position KV pool at ctx 200704 (401408 committed, awaiting deploy), 2 slots, MTP always-on, image input] G[llama.cpp PrismML fork CUDA build frank-zealot, port 7771] H[Ternary-Bonsai-2-27B PQ2_0 GGUF + Q8_0 mmproj @@ -91,7 +95,11 @@ frank-zealot, port 7771] **one pool behind two slots**, sized so TWO maximum-length requests coexist — raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total in-flight context, ~10.8 GiB reserved against the card's 262,144/262,144 = - 7.2 GiB. The RAM comes out of the n-gram table's page cache, not out of a + 7.2 GiB. **Deploy state 2026-10-18: the running container and the + host-deployed env still carry the 200,704-position pool** (boot ledger + `reserving the KV pool, 200704 positions (about 5.5 GiB)`; host + `frank-dragoon.env` `HALOGEN_KV_POOL_POSITIONS=200704`) — the doubling lands + on the next redeploy (commit `2b12716`). longer window: the per-request ceiling stays 200,704, because 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`. MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout @@ -120,17 +128,41 @@ frank-zealot, port 7771] (`slow`/`advisor` → frank-archon) carried forward. Superseded by the MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the engine moved with the model -- [[frank-zealot-llama-prism-bonsai2-pq20]] — **ACTIVE since 2026-10-17** +- [[frank-zealot-llama-prism-bonsai9-mtp]] — **ACTIVE since 2026-10-18** (root systemd unit, port 7771): the **PrismML fork** build of llama.cpp - (vendor 10754, commit `2459f68b5` — the only loader for the payload's custom - tensor type 142 + `prism.hadamard.*` transform) serving - prism-ml/Ternary-Bonsai-2-27B-gguf **PQ2_0** (2.13 bpw) on the RTX 3090 — - TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), **q8_0 K - + q8_0 V** (34 KiB/token over the 16 full layers = 13,328 MiB; the f16 pair - needs 25,088 MiB and dies), settled 22,664 used / 1,461 free, vision via the - publisher's Q8_0 mmproj (+873 MiB, charged AFTER the engine's fit check), MTP - OFF (no MTP tensors). 76.6 t/s single-stream decode, 691.7 t/s prefill at - 166k, 109.4 t/s aggregate at two streams. Supersedes the Qwen3.5-9B era below. + (vendor **10777**, commit `e3de71522` — the only loader for the payload's + custom PQ2_0 tensor type + `prism.hadamard.*` transform, and since this swap + also the minimum build that pairs MTP with a quantized V cache) serving + prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head IN the file** on + the RTX 3090 — **THREE 262,144-token conversations** + (`--ctx-size 786432 --parallel 3`, every slot at the window the model was + trained at), **q8_0 K + q8_0 V** (17 KiB/token over the 8 full layers = + 13,056 MiB ledger-exact; the f16 pair needs 26,112 MiB and dies at + `cudaMalloc`), settled **17,876 used / 6,249 free**, vision via the + publisher's **Q8_0** mmproj (Q8_0 is the publisher's requirement for + concurrent image requests, and this runs three slots), **MTP OFF by + measurement** — the draft context bills its own f16 KV over the WHOLE pool + (3,072 MiB) and at `UBATCH_SIZE=2048` that squeeze halves decode + (167.2 → 109.2 t/s); `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the + documented 213.5 t/s arm, declined for +17 % prefill and 6.2 GiB of headroom. + Measured: 167.2 t/s single-stream decode, 2,877 t/s prefill at 106,418, + 280.2 t/s aggregate across three streams. `--metrics` rides (`METRICS=1`): + llama.cpp answers 501 on `/metrics` without it, which is why Prometheus + `up{job="frank-zealot"}` read 0 for its whole retained history until + 2026-10-18. `contextWindow: 262144` is the **widest in the fleet**, so this + is the compaction hub. The `--presence-penalty` opt-out law (`${VAR=}`, not + `${VAR:=}`) and the `TimeoutStopSec=180` stop-drain law were both earned here. +- [[frank-zealot-llama-prism-bonsai2-pq20]] — **RETIRED 2026-10-17 → 10-18** + (payload + fork build 10754 tree on disk as rollback): the first PrismML-fork + era — prism-ml/Ternary-Bonsai-2-**27B**-gguf **PQ2_0**, TWO 200,704-token + conversations (`--ctx-size 401408 --parallel 2`), q8_0 K + q8_0 V + (34,816 B/token over the 16 full layers = 13,328 MiB; f16 needs 25,088 MiB and + dies), settled 22,664 used / 1,461 free, MTP **impossible** (that export + carries no head), 76.6 t/s decode, 691.7 t/s prefill at 166k, 109.4 t/s + aggregate at two streams. Owns the 34 KiB/token KV law, the f16 OOM wall, the + two-boot VRAM pricing, and the record of the 2026-10-18 bench window + (production unit stopped 03:38, hand-booted 9B candidate answering the port) + that closed with this swap. - [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **RETIRED 2026-10-07 → 10-17** (payload + master CUDA build on disk as rollback): llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive @@ -197,10 +229,13 @@ frank-zealot, port 7771] dankserver 2026-10-15 (Frank's one-time rm, volume for this payload; rebuild via the fetch + build scripts named in the fp8-native runbook) - [[frank-archon-radiance-swift15-w4dn8]] — **RETIRED 2026-10-08 → 10-15 - (superseded by the publisher iq4r swap; payload on disk as the rollback)**: + (superseded by the publisher iq4r swap; the `.rad` was PURGED from disk + 2026-10-17 — rollback = revert + byte-exact re-download of + `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)**: Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance 1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container); - the FP8-native payload remains the rollback. **2026-10-13: fp8 KV, and the + the FP8-native payload was the second rollback until it too was deleted + 2026-10-15. **2026-10-13: fp8 KV, and the pair re-cut twice** — `--kv-cache-dtype fp8` first at `--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab @@ -219,7 +254,9 @@ frank-zealot, port 7771] reset the knob to its default (125,988 MiB) and the endpoint refused to boot 6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair. - [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the - w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)** + w4dn8 swap replaced it; its `.rad` was DELETED from disk 2026-10-15 — + rebuild = `scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`, + `scripts/frank-archon.env:59-62`)** (root systemd unit, port 7770): the StillDeadcode **radiance** engine (C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8` @@ -248,8 +285,10 @@ frank-zealot, port 7771] `reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s single-stream, vision "Red" probe passes, native tool_calls; TP=2 does not boot on ROCm (pynccl JIT) — second card idle. Superseded by the - radiance swap for TP=2; the image + payload stay on disk until the - rollback window closes + radiance swap for TP=2. Disk state 2026-10-18: the image is still in + dankserver's rootless storage (`localhost/frank-primary-freetoken:pr587-77a7201`, + 29.7 GB — the pre-rename name); the orcarouter FP8 payload is NO LONGER on + disk (no `*orcarouter*` path under /home/dank, `find` 2026-10-18) - [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05 (rollback target; the vLLM trio + env lore live in the tag)**: the tcclaviger RDNA4 @@ -278,12 +317,14 @@ frank-zealot, port 7771] 35 tok/s target is NOT met, bounded by the fork's expert-gather read path (~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.** Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30) -- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps - back; payload and R9V state dirs intact on disk, git tag + [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps + back; the R9V state dirs are on disk (`/home/dank/r9v-state`, + `~/.local/state/r9v`) but the payload GGUFs are not — `/home/dank/models` + holds only the iq4r `.rad` as of 2026-10-18; git tag frank-archon-r9v-q8-20260929 is the last pre-cutover commit): [[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27 - (same-day swap to the Q8_0 payload; rollback target, state + payload - intact on disk): the R9V radiance-vLLM fork serving + (same-day swap to the Q8_0 payload; state + payloads no longer on disk + 2026-10-18, only the R9V state dirs remain): the R9V radiance-vLLM fork serving Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) + blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428 @@ -505,8 +546,8 @@ frank-zealot, port 7771] ## CIRU Reference (Dankclaw) - [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain, - and build history (on Dankclaw; the slot now serves the kingjones - ROCmFP4 model, see [[kingjones-strix-mtp]]) + and build history (on Dankclaw; the gfx1151 slot now serves frank-dragoon's + flash-server v2-native payload — see [[frank-dragoon-flash-server-v2-native]]) - [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary identity, and experiment summary (retired 2026-08-22) diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index a7199ce..1c66ecc 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -399,13 +399,19 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-zealot-llama-prism-bonsai2-pq20]] — the **ACTIVE** frank-zealot - runbook since 2026-10-17: the PrismML vendor llama.cpp build 10754 (the only - loader for GGML type 142 / `prism.hadamard.*`) serving - prism-ml/Ternary-Bonsai-2-27B-gguf **PQ2_0** on the RTX 3090 — TWO - 200,704-token conversations, q8_0 K + q8_0 V (f16 OOMs at 2×200,704: 34 KiB - vs 64 KiB per token over 16 full-attention layers), MTP **off** (no head in - the file), vision through the Q8_0 mmproj; boot 22,664 used / 1,461 free +- [[frank-zealot-llama-prism-bonsai9-mtp]] — the **ACTIVE** frank-zealot + runbook since 2026-10-18: the PrismML vendor llama.cpp build 10777 (the only + loader for the custom PQ2_0 tensor type / `prism.hadamard.*`, and the floor + for MTP beside a quantized V cache) serving + prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head** on the RTX 3090 + — THREE 262,144-token conversations, q8_0 K + q8_0 V (f16 OOMs at this pool: + 26,112 vs 13,056 MiB over 8 full-attention layers), MTP **off by + measurement** (the draft context bills 3,072 MiB of f16 KV over the whole + pool), vision through the Q8_0 mmproj; boot 17,876 used / 6,249 free +- [[frank-zealot-llama-prism-bonsai2-pq20]] — RETIRED 2026-10-17 → 10-18 + (rollback target, payload + build 10754 on disk): Ternary-Bonsai-2-**27B** + PQ2_0 at TWO 200,704-token conversations, q8_0 K + q8_0 V, MTP **impossible** + (no head in that export), vision through the Q8_0 mmproj; boot 22,664 / 1,461 - [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — RETIRED 2026-10-17 (rollback target, payload on disk): llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090 — diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 91723ff..c2f3102 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -2108,3 +2108,52 @@ every restart since the cutover. satisfied, endpoint still absent), AGENTS.md key lesson, both index pages. bats 371/371. frank-dragoon and frank-zealot were live the whole window; only the reboot interrupted them (~40 s to listen, health 200). + +## 2026-10-18 — frank-zealot: Bonsai-2-9B PQ2_0 with the MTP head (3 × 262,144) + +- Swapped `prism-ml/Ternary-Bonsai-2-27B-gguf` → **`Ternary-Bonsai-2-9B-gguf`** + at PQ2_0 (`Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2,675,410,720 B, Hub-LFS + sha256 `c79fcdfaf6b7…` verified at fetch) plus the publisher's **Q8_0** mmproj + (624,229,664 B, `e0cb6f69…`), into `/home/dank/models/Bonsai2-9B-MTP/`; the + runtime moved to vendor fork build **10777** (`e3de71522`) because the MTP + arm needs the build that pairs MTP with a quantized V cache. +- The win is the **window, not the quant**: `--ctx-size 786432 --parallel 3` + puts every conversation back at the trained 262,144 (the 27B era sold each + slot 200,704), and the fleet's widest context makes :7771 the **compaction + hub** — `models.yml` `contextWindow` 200704 → 262144. Measured vs the 27B: + decode 167.2 vs 76.6 t/s single-stream, aggregate 280.2 vs 109.4 t/s across + the (now three) concurrent streams, prefill 2,877 t/s at 106,418 depth. +- **MTP armed, then OFF by measurement.** The head is in the file, the fork + loads it, and the draft context bills its own **f16** KV over the WHOLE pool + — 3,072 MiB at 786,432 positions, ledger-confirmed. At `UBATCH_SIZE=2048` + that squeeze **halved decode (167.2 → 109.2 t/s)**. The documented MTP shape + is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s; declined, because + +17 % decode is not worth spending 6.2 GiB of the 6,249 MiB free. The arm is + recorded in the env header, not left to rediscovery. +- Two bugs the trial caught, both fixed in the repo: `${SPEC_PRESENCE_PENALTY:=0.0}` + was a *default-assignment*, so the env value always won and the documented + "MTP runs without `--presence-penalty`" arm was unreachable — now `${VAR=0.0}`, + which lets `SPEC_TYPE=draft-mtp` drop the flag. And `up{job="frank-zealot"}` + had read 0 for the **entire retained Prometheus history**: llama.cpp answers + **501** on `/metrics` unless `--metrics` is passed, so the scrape failed beside + a healthy `/health` and `LlmEndpointDown` fired for days. `METRICS=1` is now + the default and the seam pins it. +- Boot ledger: KV `13,056.00 MiB` (8 full-attention layers × 4 KV heads × 256 + dim, q8_0 K + q8_0 V over 786,432 positions — ledger-exact), settled + **17,876 used / 6,249 free**; the f16 pair would need 26,112 MiB and cannot + boot this card. `TimeoutStopSec=180` carries over (SIGTERM ignored past the + old default during the 03:38 stop). +- Header census that corrected my own first pass: `block_count` is **33** + (32 trunk + the `blk.32` MTP block, which carries a full attention + FFN in + q8_0 — that is why the draft context bills real KV), **202 of 442** tensors + carry the fork's PQ2_0 type (norms f32, MTP projections q8_0), `hadamard` + `block_size` **4096** with `gdn_grouped=True`. The "2.13 bpw / type 142 / 32 + layers" figures inherited from the 27B page were wrong for this file and are + corrected in AGENTS.md, the env header and the runbooks. +- Docs: new runbook `LLM/frank-zealot-llama-prism-bonsai9-mtp.md`; the 27B page + re-banned **RETIRED 2026-10-17 → 10-18** with its bench-window section closed; + both index pages, the homepage card, `Dankclaw/OpenWebUI.md` and the live + picker row (DB `description` + context length 262,144) re-keyed. Rollback: + revert the swap commit — the 27B payload and the build-10754 fork tree are + both still on disk. + diff --git a/shared/homepage/services.json b/shared/homepage/services.json index 7b10eb2..3f6b688 100644 --- a/shared/homepage/services.json +++ b/shared/homepage/services.json @@ -295,13 +295,13 @@ "model": { "unit": "zealot", "role": "Frank's utility model", - "hf": "https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf", + "hf": "https://huggingface.co/prism-ml/Ternary-Bonsai-2-9B-gguf", "tagline": "HONOR GUIDE ME!", "sfx": [ "zealot-honor.ogg" ], "equiv": "Equivalent to last year's frontier level (2025).", - "spec": "Ternary-Bonsai-2-27B (PQ2_0 ternary weights) on an NVIDIA RTX 3090 — ~77 tokens/s; 200k-token context, two submissions at a time; sees images.", + "spec": "Ternary-Bonsai-2-9B (PQ2_0 ternary weights, MTP head in the file) on an NVIDIA RTX 3090 — ~167 tokens/s; 262k-token context, three submissions at a time; sees images.", "inputs": [ "text", "image" diff --git a/tests/frank_zealot_unit.bats b/tests/frank_zealot_unit.bats index 87f2f3c..da63189 100755 --- a/tests/frank_zealot_unit.bats +++ b/tests/frank_zealot_unit.bats @@ -1,14 +1,17 @@ # Seam: the frank-zealot llama.cpp root unit on dankclaw. -# frank-zealot is the RTX 3090 endpoint; since 2026-10-17 it serves -# prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 — the ternary Bonsai-2-27B, a -# qwen35 hybrid with 64 blocks (16 full-attention + 48 Gated DeltaNet, no -# nextn/mtp tensors in the file) — via the PrismML fork build of llama.cpp -# (vendor build 10754), the ONLY loader that registers the payload's custom -# tensor type 142 and honours its prism.hadamard.* metadata: TWO 200,704-token -# conversations (--ctx-size 401408 --parallel 2), q8_0 K + q8_0 V under flash -# attention (f16 KV does not fit this card at this pool — the boot dies on a -# 25,088 MiB cudaMalloc), the publisher's own Q8_0 vision projector, and MTP -# off. Log verbosity 4 ships so the VRAM-fit forensics land in the journal. +# frank-zealot is the RTX 3090 endpoint; since 2026-10-18 it serves +# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 — the ternary Bonsai-2-9B, a +# qwen35 hybrid with 32 layers (8 full-attention + 24 Gated DeltaNet) and the +# MTP draft head INSIDE the file — via the PrismML fork build of llama.cpp +# (vendor build 10777, the minimum that pairs MTP with a quantized V cache), +# the ONLY loader that registers the payload's custom tensor type 142 and +# honours its prism.hadamard.* metadata: THREE 262,144-token conversations +# (--ctx-size 786432 --parallel 3), q8_0 K + q8_0 V under flash attention (f16 +# KV would bill 26,112 MiB of a 24,123 MiB card), the publisher's own Q8_0 +# vision projector (the BF16 one can crash on concurrent images), and MTP OFF +# — measured, not because the head is missing: the draft context bills f16 KV +# over the whole pool and halves decode at this shape. Log verbosity 4 ships so +# the VRAM-fit forensics land in the journal. # This seam runs the serve script against a stub engine and asserts on the # argv it actually execs, so what is pinned is the command the operator gets. @@ -97,15 +100,16 @@ run_stage() { grep -q 'FRANK_ZEALOT_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-zealot.env' "$UNIT" } -@test "the shipped env points at the fork runtime and the Bonsai-2 PQ2_0 pair" { +@test "the shipped env points at the fork runtime and the Bonsai-2 9B MTP pair" { # The payload path IS the endpoint's identity, and the runtime is part of # it here: the file's custom tensor type 142 + prism.hadamard.* metadata - # make the PrismML build the only loader, so a retarget that moves the - # model but leaves the master build behind ships an endpoint that cannot - # load at all. - grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-prism/llama-server$' "$ENV_FILE" - grep -qE '^MODEL_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0\.gguf$' "$ENV_FILE" - grep -qE '^MMPROJ_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0\.gguf$' "$ENV_FILE" + # make the PrismML build the only loader, and this build must be >= b10776 + # because MTP beside q8_0 V is illegal below it. A retarget that moves the + # model but leaves the old build behind ships an endpoint that cannot load + # (type 142) or cannot run the documented MTP arm (the build gate). + grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server$' "$ENV_FILE" + grep -qE '^MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0\.gguf$' "$ENV_FILE" + grep -qE '^MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0\.gguf$' "$ENV_FILE" } @test "the deployer gates on exactly the payload paths the env file ships" { @@ -122,7 +126,7 @@ run_stage() { done } -@test "two conversations share the pool at a 200704 window each" { +@test "three conversations share the pool at a 262144 window each" { serve_argv [ "$status" -eq 0 ] [ "$(argv_value --alias)" = "frank-zealot" ] @@ -131,27 +135,31 @@ run_stage() { [ "$(argv_value --model)" = "$MODEL_DIR/$MODEL_STUB" ] [ "$(argv_value --n-gpu-layers)" -gt 0 ] # llama.cpp divides --ctx-size across slots, so the observable contract - # is the per-slot window. Frank's 200k cap is a per-conversation cap, so - # the pair is 401,408 / 2 — and the harness contextWindow must quote - # 200704, not the pool and not the model's native 262,144. - [ "$(argv_value --parallel)" -eq 2 ] - [ "$(argv_value --ctx-size)" -eq 401408 ] + # is the per-slot window. The 9B swap exists precisely so every + # conversation gets the model's TRAINED 262,144 (the 27B had to be cut to + # 200,704 to fund two), and the harness contextWindow must quote 262144 — + # the pool and the slot count are neither, and a longer prompt is a 400. + # Three is also the ceiling: a fourth slot would push each window under the + # fleet's worst-case compaction input, which is this endpoint's job. + [ "$(argv_value --parallel)" -eq 3 ] + [ "$(argv_value --ctx-size)" -eq 786432 ] local pool slots pool="$(argv_value --ctx-size)"; slots="$(argv_value --parallel)" [ $((pool % slots)) -eq 0 ] - [ $((pool / slots)) -eq 200704 ] + [ $((pool / slots)) -eq 262144 ] } @test "the KV dtype pair follows the env file and flash-attn stays pinned on" { # Asserted against the env rather than a literal so a retune moves one # file, not two. The shipped pair is q8_0/q8_0: this payload bills - # 16 full-attention layers × 4 KV heads × 256 dim = 16,384 elements per - # token per cache, so f16/f16 would need 25,088 MiB of the 24,123 MiB card - # (measured: cudaMalloc OOM, engine fit 33,198 vs 23,558 free) while q8_0 - # bills 34,816 B/token = 13,328.00 MiB at the 401,408 pool. The boot - # ledger, not this file, is the referee. flash-attn is not decoration: - # quantized V is only legal with it, and it must stay explicit so a - # future quant swap cannot silently fall back. + # 8 full-attention layers × 4 KV heads × 256 dim = 8,192 elements per token + # per cache, so f16/f16 would need 26,112 MiB of the 24,123 MiB card while + # q8_0 bills 17,408 B/token = 13,056.00 MiB at the 786,432 pool — the + # ledger said exactly that, and it is the referee. The 24 DeltaNet layers + # bill a fixed recurrent state per slot (452.25 MiB for three cells), which + # is why pool arithmetic alone under-prices the shape. flash-attn is not + # decoration: quantized V is only legal with it, and it must stay explicit + # so a future quant swap cannot silently fall back. local env_k env_v env_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")" env_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")" @@ -166,15 +174,16 @@ run_stage() { serve_argv [ "$status" -eq 0 ] # The shipped 4 (debug) is what makes VRAM-fit forensics possible from - # the journal (KV layout, buffer sizes, graph capture). + # the journal (KV layout, buffer sizes, graph capture) — the only way to + # prove the q8_0 KV contract and the draft-KV bill from a log. [ "$(argv_value --log-verbosity)" = "4" ] - # And an env that names no verbosity still ships the engine default - # explicitly — the script's written-down 3, never a blank. + # And an env that names no verbosity still ships an explicit value: the + # serve script's written-down default is 4 too, never a blank. sed -i '/^VERBOSITY=/d' "$STAGE/test.env" run_stage [ "$status" -eq 0 ] - [ "$(argv_value --log-verbosity)" = "3" ] + [ "$(argv_value --log-verbosity)" = "4" ] } @test "the Prometheus metrics endpoint is served unless opted out" { @@ -221,12 +230,16 @@ run_stage() { } @test "speculation is off: no drafter flag reaches the argv" { - # Frank, 2026-10-07: MTP is off because it CANNOT be on — the Q6_K file - # carries no nextn/mtp tensors (verified from its tensor-name list), and - # MTP is a property of the file, not of the flags. The flags move as a - # set: SPEC_TYPE=none strips --spec-type, --spec-draft-n-max AND - # --spec-draft-model together, so a no-spec boot cannot carry half a - # spec pair (the ROCmFPX-era baseline arm segfaulted at load doing that). + # Frank, 2026-10-18: this file DOES carry the MTP head (the boot arms prove + # it loads and drafts, acceptance 0.798/0.617 per position, mean accepted + # length 2.41) and it is still off, because the draft context bills its own + # f16 KV over the WHOLE pool — 4,096 B/position × 786,432 = 3,072 MiB plus + # buffers — and at --ubatch-size 2048 that squeeze halves decode: 167.2 → + # 109.2 t/s. The MTP arm is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (213.5 + # t/s), documented in frank-zealot.env. The flags move as a set: + # SPEC_TYPE=none strips --spec-type, --spec-draft-n-max AND + # --spec-draft-model together, so a no-spec boot cannot carry half a spec + # pair (the ROCmFPX-era baseline arm segfaulted at load doing that). serve_argv [ "$status" -eq 0 ] ! grep -qxF -- '--spec-type' <<<"$output" @@ -235,27 +248,35 @@ run_stage() { grep -qE '^SPEC_TYPE=none$' "$ENV_FILE" } -@test "naming a drafter ships the full MTP trio" { - # The opt-back-in path (e.g. a future Qwen3.5 -mtp build): SPEC_TYPE + - # SPEC_DRAFT_FILE must reach the engine as a complete set — a draft-mtp - # flag without its model file boots a drafter against the main weights and - # drafts nothing. +@test "naming a drafter ships the MTP set, with or without a separate draft file" { + # The documented MTP arm (frank-zealot.env): SPEC_TYPE=draft-mtp ALONE is + # legal here because the head is IN the checkpoint and the draft context + # rides the main model's weights — so --spec-draft-model must NOT appear, + # or the trunk would be loaded twice. serve_argv { echo 'SPEC_TYPE=draft-mtp' - echo 'SPEC_DRAFT_N_MAX=3' - echo "SPEC_DRAFT_FILE=$MODEL_DIR/$DRAFT_STUB" + echo 'SPEC_DRAFT_N_MAX=2' } >> "$STAGE/test.env" run_stage + [ "$status" -eq 0 ] [ "$(argv_value --spec-type)" = "draft-mtp" ] - [ "$(argv_value --spec-draft-n-max)" = "3" ] + [ "$(argv_value --spec-draft-n-max)" = "2" ] + ! grep -qxF -- '--spec-draft-model' <<<"$output" + + # And an external drafter still reaches the engine as a complete set — the + # pair the ROCmFPX-era crash was about. + echo "SPEC_DRAFT_FILE=$MODEL_DIR/$DRAFT_STUB" >> "$STAGE/test.env" + run_stage + [ "$status" -eq 0 ] [ "$(argv_value --spec-draft-model)" = "$MODEL_DIR/$DRAFT_STUB" ] } @test "vision ships by default and MMPROJ_FILE empty opts out of the image path" { - # Qwen3.5-9B is natively multimodal, but the Q6_K file carries no vision - # tensors — the publisher's projector argument IS the vision capability: - # no flag, no eyes. + # The Bonsai-2 text file carries no vision tensors — the publisher's Q8_0 + # projector argument IS the vision capability: no flag, no eyes. Q8_0 (not + # BF16) is the publisher's own requirement for concurrent image requests, + # which is what --parallel 3 means in practice. serve_argv [ "$status" -eq 0 ] [ "$(argv_value --mmproj)" = "$MODEL_DIR/$MM_STUB" ] @@ -268,6 +289,34 @@ run_stage() { ! grep -qxF -- '--mmproj' <<<"$output" } +@test "the card sampler's only non-metadata term reaches the argv" { + # general.sampling.* (temperature / top_p / top_k / min_p / repeat_penalty) + # rides in the checkpoint and the engine reads it itself; presence penalty + # is the ONE recipe value with no GGUF metadata key, so it must reach argv + # or the recipe is silently incomplete. Distinctive value first: proof the + # flag is wired to the env and not merely to the script's default. + serve_argv + { echo 'PRESENCE_PENALTY=0.5'; } >> "$STAGE/test.env" + run_stage + [ "$status" -eq 0 ] + [ "$(argv_value --presence-penalty)" = "0.5" ] + + # The shipped recipe value reaches the engine too. + local pp + pp="$(sed -nE 's/^PRESENCE_PENALTY=(.+)$/\1/p' "$ENV_FILE")" + sed -i "s/^PRESENCE_PENALTY=.*/PRESENCE_PENALTY=$pp/" "$STAGE/test.env" + run_stage + [ "$status" -eq 0 ] + [ "$(argv_value --presence-penalty)" = "$pp" ] + + # The documented opt-out is absence of the flag, not an empty argument — + # which is why the default uses ${VAR=default}, not ${VAR:=default}. + sed -i "s/^PRESENCE_PENALTY=.*/PRESENCE_PENALTY=/" "$STAGE/test.env" + run_stage + [ "$status" -eq 0 ] + ! grep -qxF -- '--presence-penalty' <<<"$output" +} + @test "the fleet reasoning contract reaches the argv" { serve_argv [ "$status" -eq 0 ]