Compare commits
20
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d5a43b7192 | ||
|
|
d457985b8d | ||
|
|
5550d32a2e | ||
|
|
71bc1735d3 | ||
|
|
68afc6dbaf | ||
|
|
100b62ba22 | ||
|
|
9a17b93e8a | ||
|
|
3e779823e0 | ||
|
|
30e3d36b1e | ||
|
|
5fbbea2c2a | ||
|
|
96b7687b34 | ||
|
|
7889a68497 | ||
|
|
b590695e2a | ||
|
|
b91873578b | ||
|
|
0d70f115d1 | ||
|
|
7e391d4325 | ||
|
|
5768680542 | ||
|
|
d2ce6693da | ||
|
|
2dc31ff09c | ||
|
|
0315a62d84 |
@@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
|
||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 and DEPLOYED 2026-10-18 so BOTH slots can hold a maximum-length request at once; the parity play was frank-zealot's then-401,408 total in-flight context — zealot moved to 786,432 the same evening and dragoon's pool was NOT raised again, because every position is page cache the n-gram table loses; **11.0 GiB reserved, measured on that deploy** (the preflight formula bills 11.3) where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
|
||||
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10777**, commit `e3de71522`; root systemd unit, relinked — the ONLY loader for this payload, and since 2026-10-18 also the minimum build that pairs MTP with a quantized V cache) | `prism-ml/Ternary-Bonsai-2-9B-gguf` **PQ2_0 carrying the MTP head in the file** (2,675,410,720 B, Hub-LFS sha256 `c79fcdfa…` verified at fetch by `scripts/download-frank-zealot-bonsai9.sh`; `qwen35` hybrid: `block_count 33` = **32 trunk layers** (8 full-attention + 24 Gated DeltaNet at `full_attention_interval 4`) + the `blk.32` MTP block, 4 KV heads × 256 dim, native 262,144; `general.file_type 141` with **202 custom-type PQ2_0 tensors of 442** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (624,229,664 B, sha256 `e0cb6f69…`; **Q8_0 is the publisher's requirement** for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs three slots) for **vision**, on the RTX 3090, **THREE 262,144-token slots** (`--ctx-size 786432 --parallel 3` = every conversation at the trained window), **q8_0 K + q8_0 V** under flash attention (17 KiB/token over the 8 full layers = 13,056 MiB pool, ledger-exact; **f16 KV needs 26,112 MiB and cannot boot this card**; +151 MiB of recurrent state for the three DeltaNet cells — the SPEC-ARMED arms print 452 MiB, because MTP depth 2 bills three seqs per slot), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF — measured, not because the head is absent**: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented MTP arm is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s. Production boot 17,922 used / 6,203 free; **168.7 t/s** single-stream greedy decode (three runs, 168.4–168.7) and **~300 t/s** aggregate across three concurrent streams (102.8 each); 2,877 t/s prefill at 106,418 was the trial arm. Cutover 2026-10-18 from Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback). | 7771 |
|
||||
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10777**, commit `e3de71522`; root systemd unit, relinked — the ONLY loader for this payload; the b10776 floor that gates MTP beside a quantized V cache is moot under exact KV, but the custom tensor type alone keeps the build pinned) | `prism-ml/Ternary-Bonsai-2-9B-gguf` **PQ2_0 carrying the MTP head in the file** (2,675,410,720 B, Hub-LFS sha256 `c79fcdfa…` verified at fetch by `scripts/download-frank-zealot-bonsai9.sh`; `qwen35` hybrid: `block_count 33` = **32 trunk layers** (8 full-attention + 24 Gated DeltaNet at `full_attention_interval 4`) + the `blk.32` MTP block, 4 KV heads × 256 dim, native 262,144; `general.file_type 141` with **202 custom-type PQ2_0 tensors of 442** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (624,229,664 B, sha256 `e0cb6f69…`; **Q8_0 is the publisher's requirement** for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs several slots) for **vision**, on the RTX 3090, **TWO 262,144-token slots** (`--ctx-size 524288 --parallel 2` = every conversation STILL at the trained window — the per-slot contract never moved) over **unquantized f16 K + f16 V** (32 KiB/token over the 8 full layers = **16,384 MiB** pool; the POOL halved because exact KV at the old 786,432 pool is 24,576 MiB against the 24,123 MiB the driver leaves addressable — the repo's older "26,112 MiB" figure had doubled the q8_0 number including its 1/32 scale overhead; +100.5 MiB of recurrent state for the two DeltaNet cells, ~50.25 MiB a cell), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF — measured, and now doubly so**: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB at the old pool, 2,048 MiB at this one) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented q8_0-era MTP arm is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s. Last MEASURED boot (the 3 × 262,144 q8_0 arm this replaces): 17,922 used / 6,203 free, **168.7 t/s** single-stream greedy decode (three runs, 168.4–168.7), **~300 t/s** aggregate across three concurrent streams (102.8 each), 2,877 t/s prefill at 106,418 — the f16 boot's ledger and rates land in the runbook once measured. Cutover 2026-10-18 from Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback). | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -252,8 +252,12 @@ archive.
|
||||
real users, and a bench beside a live service starves the GPU and corrupts
|
||||
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
|
||||
endpoints; a live llama.cpp server (frank-zealot) also reports honest
|
||||
per-request numbers in its `/completion` `timings` object (prefill/decode
|
||||
rates, `draft_n`, `draft_n_accepted`), which is how frank-zealot was measured.
|
||||
per-request numbers in a `timings` object (prefill/decode rates, `draft_n`,
|
||||
`draft_n_accepted`) — but on the PrismML fork build 10777 that object is only
|
||||
reaching you on **`/v1/chat/completions`**: a greedy `ignore_eos` run on the raw
|
||||
`/completion` path there answers **500 "The model produced output that does not
|
||||
match the expected Content-only format"**, so measure through the chat path
|
||||
(measured 2026-10-18 on the exact-KV reslice).
|
||||
halogen has NO `/completion` path at all — frank-dragoon's rates come from
|
||||
wall-clock around `/v1/chat/completions` plus the boot ledger. Rolling your own
|
||||
bench requires asking first.
|
||||
@@ -350,9 +354,9 @@ and `frank-zealot` through
|
||||
the publisher's own **Q8_0** mmproj passed as `--mmproj` — Q8_0 is the
|
||||
publisher's requirement for concurrent image requests (the card states the
|
||||
BF16 tower can crash `llama-server` when several image requests run at once,
|
||||
and this endpoint runs three slots). `frank-zealot`'s `contextWindow` is
|
||||
**262144** — the PER-SLOT window (`--ctx-size 786432 --parallel 3` since the
|
||||
2026-10-18 Bonsai-2-**9B** swap), which since that swap EQUALS the model's
|
||||
and this endpoint runs several slots). `frank-zealot`'s `contextWindow` is
|
||||
**262144** — the PER-SLOT window (`--ctx-size 524288 --parallel 2` since the
|
||||
2026-10-18 reslice to exact f16 KV), which EQUALS the model's
|
||||
native window; llama.cpp answers a longer prompt with a 400 naming the limit,
|
||||
never a truncation, so the harness must compact before it.
|
||||
Note also
|
||||
@@ -510,7 +514,8 @@ llama.cpp refuses the custom type at load, and a loader that
|
||||
ignored the Hadamard metadata would dequant WRONG — same lock class as the
|
||||
ROCmFPX era. On top of that lock, **the fork gates MTP beside a quantized V
|
||||
cache on build `prism-b10776-d00e252` or newer**, which is why the prebuilt
|
||||
moved 10754 → 10777; the 10754 tree stays on disk as the 27B rollback runtime.
|
||||
moved 10754 → 10777 (exact f16 KV makes that gate moot — the custom type alone
|
||||
pins the runtime); the 10754 tree stays on disk as the 27B rollback runtime.
|
||||
Retargeting the payload without the fork ships an endpoint that cannot load,
|
||||
so the seam pins both paths.
|
||||
|
||||
@@ -520,36 +525,47 @@ attention + FFN in q8_0, which is why the draft context bills real KV), and
|
||||
`qwen35.full_attention_interval = 4` ⇒ **8 full-attention + 24 Gated
|
||||
DeltaNet**, 4 KV heads × 256 dim, native 262,144. **KV accounting:** the pool
|
||||
scales with the 8 full layers — 8 × 4 × 256 = 8,192 elements/token per cache,
|
||||
so **q8_0 K + q8_0 V bills 17,408 B/token ≈ 17 KiB** = **13,056.00 MiB** at the
|
||||
786,432 pool (ledger-exact) — exactly HALF the 27B's 34,816 B/token, which is
|
||||
the whole reason three native windows fit where the 27B afforded only two
|
||||
shortened ones. **f16 K would bill 26,112 MiB and cannot boot on a 24,123 MiB
|
||||
card.** The 24 DeltaNet layers bill a fixed recurrent state per SLOT — the
|
||||
production ledger prints `RS buffer size = 150.75 MiB (3 cells, 32 layers,
|
||||
3 seqs)`, while the SPEC-ARMED arms print 452.25 MiB (MTP depth 2 bills three
|
||||
seqs per slot, so 9 cells) — so pool arithmetic alone under-prices the shape.
|
||||
so **the deployed exact pair f16 K + f16 V bills 32,768 B/token = 32 KiB** =
|
||||
**16,384 MiB** at the 524,288 pool. The q8_0 pair this endpoint shipped first
|
||||
billed **17,408 B/token** (1.0625 B/element: 32 codes plus one 2-byte scale per
|
||||
32-block) = **13,056.00 MiB** at the old 786,432 pool, ledger-exact — exactly
|
||||
HALF the 27B's 34,816 B/token. **Exact KV at that 786,432 pool is 24,576 MiB,
|
||||
more than the 24,123 MiB the driver leaves addressable, so the POOL is what
|
||||
pays for f16 — the window never moved.** (The "26,112 MiB" figure this repo
|
||||
carried doubled the q8_0 number *including* its scale overhead; the true ratio
|
||||
is 1.882×, not 2×.) The 24 DeltaNet layers bill a fixed recurrent state per
|
||||
SLOT — ~50.25 MiB a cell, so the 3-slot ledger printed `RS buffer size =
|
||||
150.75 MiB (3 cells, 32 layers, 3 seqs)` and two slots bill 100.50, while the
|
||||
SPEC-ARMED arms print 452.25 MiB (MTP depth 2 bills three seqs per slot, so 9
|
||||
cells) — so pool arithmetic alone under-prices the shape.
|
||||
|
||||
**The shape (2026-10-18):** `--ctx-size 786432 --parallel 3` = THREE
|
||||
conversations of 262,144, every one at the model's TRAINED window (Frank: "if
|
||||
you can possibly fit a third context, do it"), `--flash-attn on`, **q8_0 K +
|
||||
q8_0 V**, batch 4096 / **ubatch 2048** (the measured CUDA optimum carried from
|
||||
the MiMo/Swift A/Bs on this card; it also sets the compute-buffer size, which
|
||||
is what decides whether MTP fits), `--log-verbosity 4`, `--metrics`,
|
||||
`--jinja`, `--mmproj` for vision, **speculative decoding OFF**
|
||||
(`SPEC_TYPE=none`). Boot: **17,876 MiB used / 6,249 free**; compute 1,409 MiB
|
||||
CUDA + 1,201 MiB CUDA_Host, projector 595 + 248 MiB. Measured on the same
|
||||
boot: **167.2 t/s** greedy decode, 158.9 t/s single-stream chat, **280.2 t/s
|
||||
aggregate across three concurrent streams**, **2,877 t/s prefill at 106,418**
|
||||
tokens — the 27B record (76.6 decode / 109.4 aggregate / 691.7 prefill at 166k)
|
||||
beaten on every axis. The **first request after boot is a warmup artifact**
|
||||
(37.7 t/s class) — never report it.
|
||||
**The shape (2026-10-18, resliced the same evening):** `--ctx-size 524288
|
||||
--parallel 2` = TWO conversations of 262,144, every one still at the model's
|
||||
TRAINED window (the first arm was THREE — Frank: "if you can possibly fit a
|
||||
third context, do it" — and the exact-KV reslice handed one back),
|
||||
`--flash-attn on`, **unquantized f16 K + f16 V**, batch 4096 / **ubatch 2048**
|
||||
(the measured CUDA optimum carried from the MiMo/Swift A/Bs on this card; it
|
||||
also sets the compute-buffer size, which is what decides whether MTP fits),
|
||||
`--log-verbosity 4`, `--metrics`, `--jinja`, `--mmproj` for vision,
|
||||
**speculative decoding OFF** (`SPEC_TYPE=none`). The MEASURED record still
|
||||
belongs to the arm this replaced (3 × 262,144, q8_0 K + q8_0 V): boot **17,876
|
||||
MiB used / 6,249 free**; compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host, projector
|
||||
595 + 248 MiB; **167.2 t/s** greedy decode, 158.9 t/s single-stream chat,
|
||||
**280.2 t/s aggregate across three concurrent streams**, **2,877 t/s prefill at
|
||||
106,418** tokens — the 27B record (76.6 decode / 109.4 aggregate / 691.7 prefill
|
||||
at 166k) beaten on every axis. The exact-KV arm's ledger and rates go in the
|
||||
runbook once booted; its projection is ~21,200 MiB used / ~2,900 free, since
|
||||
only the KV pool (+3,328) and one recurrent-state cell (−50) move. The **first
|
||||
request after boot is a warmup artifact** (37.7 t/s class) — never report it.
|
||||
|
||||
**Why MTP is off even though the head is in the file.** The drafter is GOOD —
|
||||
acceptance 0.798 / 0.617 per position, mean accepted length 2.41 — and the card
|
||||
recommends depth 2. It still loses AT THIS SHAPE because the draft context
|
||||
bills its OWN **f16** KV over the WHOLE pool (4,096 B/position × 786,432 =
|
||||
**3,072.00 MiB**, 4,830 MiB with its compute buffers) and ubatch 2048 refuses to
|
||||
give it up. Booted arms, same card, same traffic (2026-10-18):
|
||||
recommends depth 2. It still loses because the draft context bills its OWN
|
||||
**f16** KV over the WHOLE
|
||||
pool (4,096 B/position — **3,072.00 MiB** at the 786,432 pool, 4,830 MiB with
|
||||
its compute buffers; **2,048.00 MiB** at the shipped 524,288 pool) and ubatch
|
||||
2048 refuses to give it up. Booted arms, same card, same traffic (2026-10-18,
|
||||
every row q8_0 K + q8_0 V at the 786,432 pool):
|
||||
|
||||
| slots | spec | ubatch | used / free | decode |
|
||||
|---|---|---|---|---|
|
||||
@@ -563,13 +579,17 @@ give it up. Booted arms, same card, same traffic (2026-10-18):
|
||||
drafter, is what halves it at three windows. The arm that WOULD run it at three
|
||||
windows is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` (213.5 t/s single-stream,
|
||||
250.8 three-stream aggregate) and pays prefill −17 % and 2.7 GiB of headroom on
|
||||
a card whose projector worst-case lands AFTER the engine's own fit check. This
|
||||
endpoint is the fleet's compaction hub — prefill-bound, 2–3 concurrent
|
||||
subagents — so the shipped arm takes the +17 % prefill (2,877 vs 2,395 t/s) and
|
||||
the +12 % three-stream aggregate (280.2 vs 250.8) over single-stream speed, and
|
||||
keeps 6.2 GiB free. `SPEC_TYPE=none` strips the WHOLE spec set (`--spec-type`
|
||||
and every `--spec-draft-*`) — flags move as a set or the arm measures a crash
|
||||
instead of an absence.
|
||||
a card whose projector worst-case lands AFTER the engine's own fit check. Under
|
||||
**exact KV the drafter is arithmetically out**: the trunk projects ~21,200 MiB
|
||||
of the 24,123 MiB addressable before the draft context's 2,048 MiB and its
|
||||
buffers are even asked for, so the doors back to speculation are ONE window per
|
||||
conversation, or KV back to q8_0.
|
||||
The compaction-hub load is what makes exact KV affordable: prefill is
|
||||
weight-bandwidth bound and does not re-read the cache per generated token, so
|
||||
the dtype is paid in decode and concurrency (two conversations now, not three),
|
||||
not in the work this endpoint exists to do. `SPEC_TYPE=none` strips the WHOLE
|
||||
spec set (`--spec-type` and every `--spec-draft-*`) — flags move as a set or the
|
||||
arm measures a crash instead of an absence.
|
||||
|
||||
**Sampler**: the recipe lives in the checkpoint (`general.sampling.*` =
|
||||
temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0,
|
||||
@@ -581,7 +601,7 @@ would silently re-add the flag on an explicit empty value).
|
||||
|
||||
**Vision**: the publisher's own **Q8_0** mmproj via `--mmproj` — Q8_0 is
|
||||
MANDATORY, not a preference: the card says the BF16 tower "can crash
|
||||
llama-server when several image requests run at once", and `--parallel 3` is
|
||||
llama-server when several image requests run at once", and `--parallel 2` is
|
||||
exactly that shape. Probe traps carry over: ≥256 px probe images or the model
|
||||
answers "video", only the OpenAI `image_url` content part reaches the model,
|
||||
and thinking tokens count against `max_tokens` — budget ≥512 or the probe
|
||||
@@ -596,10 +616,12 @@ have to be fetched first.
|
||||
|
||||
**Compaction hub**: `models.yml` routes `compactionModel` for
|
||||
`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here —
|
||||
262,144 (since the 2026-10-18 reslice) is the widest window in the fleet and
|
||||
the only one that comfortably fits a worst-case compaction input (~170k
|
||||
tokens). With `--parallel 3` a compaction request no longer serializes behind a
|
||||
live generation (the Gemma4 era's `--parallel 1` law). Retrieval through a
|
||||
262,144 (still the PER-SLOT window after the exact-KV reslice) is the widest
|
||||
window in the fleet and the only one that comfortably fits a worst-case
|
||||
compaction input (~170k tokens). With `--parallel 2` a compaction request still
|
||||
runs beside ONE live generation, and with two already busy it waits — the Gemma4
|
||||
era's `--parallel 1` law at half strength, which is the concurrency price of
|
||||
exact KV. Retrieval through a
|
||||
25,960-token prompt was verified exact after the swap.
|
||||
|
||||
**Contract law learned at verbosity 4:** a raw `/completion` request prints
|
||||
@@ -1040,18 +1062,34 @@ linked runbooks.
|
||||
- **A draft context bills its own f16 KV over the WHOLE POOL, and on a
|
||||
single-card endpoint that squeeze can cost more than the drafter wins.** The
|
||||
draft context is built at the target's `n_ctx`, so with `--parallel N` the
|
||||
drafter pays the POOL, not the slot: on frank-zealot's 786,432-position
|
||||
qwen35 pool, `--spec-type draft-mtp` added **3,072.00 MiB of f16 draft KV**
|
||||
(4,096 B/position — the draft cache is never quantized) plus its own compute
|
||||
buffers, 4,830 MiB all told. Booted arms, same card, same traffic
|
||||
(2026-10-18): 3 slots + ubatch 2048 → **167.2 t/s decode with spec OFF,
|
||||
109.2 with spec ON**; dropping `UBATCH_SIZE` to 1024 hands the buffers back
|
||||
and spec ON gives **213.5** (+28 % over the spec-OFF arm), and at 1–2 slots
|
||||
the SAME card runs MTP at ubatch 2048 and still gains ~28 %. So the MTP
|
||||
question is never "does the file carry the head" — it is "what do draft KV
|
||||
+ buffers cost against whatever the batch knobs keep for compute buffers",
|
||||
and that answer is per-shape. Measure the arm, never the flag. See
|
||||
drafter pays the POOL, not the slot: on frank-zealot's qwen35 pool
|
||||
`--spec-type draft-mtp` adds **4,096 B/position of draft KV at the default
|
||||
f16 dtype** (`--spec-draft-type-k/-v` can quantize the draft — untested here),
|
||||
**3,072.00 MiB** at the 786,432 pool it shipped with and **2,048.00 MiB** at
|
||||
the 524,288 pool it runs now, plus its own compute buffers (4,830 MiB all told
|
||||
at the old pool). Booted arms, same card, same traffic (2026-10-18, every row
|
||||
q8_0 KV): 3 slots + ubatch 2048 → **167.2 t/s decode with spec OFF, 109.2 with
|
||||
spec ON**; dropping `UBATCH_SIZE` to 1024 hands the buffers back and spec ON
|
||||
gives **213.5** (+28 % over the spec-OFF arm), and at 1–2 slots the SAME card
|
||||
runs MTP at ubatch 2048 and still gains ~28 %. So the MTP question is never
|
||||
"does the file carry the head" — it is "what do draft KV + buffers cost
|
||||
against whatever the batch knobs keep for compute buffers", and that answer is
|
||||
per-shape. Measure the arm, never the flag. Production corollary: beside
|
||||
exact KV an **f16-draft** arm has no room — the trunk projects ~21,200 MiB of
|
||||
the 24,123 addressable before the drafter is asked for anything; the untested
|
||||
door is `--spec-draft-type-k/-v q8_0` (1,088 MiB). See
|
||||
[[frank-zealot-llama-prism-bonsai9-mtp]].
|
||||
- **A quant's scale overhead belongs in its byte rate, and doubling a measured
|
||||
pool figure to "get the f16 number" invents 6.25 %.** This repo carried "f16
|
||||
KV would bill 26,112 MiB" for weeks by doubling the q8_0 ledger figure
|
||||
(13,056.00 MiB). q8_0 is 1.0625 B/element (32 codes plus one 2-byte scale per
|
||||
32-block), f16 is exactly 2.0 ⇒ the true ratio is **1.882×**, so exact KV over
|
||||
that pool is **24,576 MiB**. The conclusion (three f16 windows cannot boot a
|
||||
card that leaves 24,123 MiB addressable) survived; the number did not — and
|
||||
the honest rate is what made the resize calculable, because it shows 524,288
|
||||
positions is the largest exact-KV pool that fits with room for the projector.
|
||||
Derive byte rates from element count × block layout, then check them against
|
||||
the ledger.
|
||||
- **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau —
|
||||
and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its
|
||||
compute buffers land about TWICE as heavy on the card that holds the output
|
||||
|
||||
@@ -7,7 +7,10 @@ Consumers (the omp harness, OpenWebUI, open-notebook) speak the OpenAI
|
||||
directory under ``$VOICES_DIR`` and ``reference.wav`` inside it is what makes
|
||||
the name real. A name that has no reference is not a voice — the shim answers
|
||||
400 with the list that is, instead of silently narrating in a fallback voice
|
||||
(silent fallback is how a wrong-voice endpoint stays wrong for weeks).
|
||||
(silent fallback is how a wrong-voice endpoint stays wrong for weeks). A request
|
||||
that names NO voice at all takes ``$DEFAULT_VOICE``: that is a spoken choice in
|
||||
the env file rather than a surprise in the code, and the journal marks it
|
||||
``(default)`` so a name nobody asked for is never mistaken for one that was.
|
||||
|
||||
Two backends sit behind the one surface, chosen by ``TTS_BACKEND``:
|
||||
|
||||
@@ -70,7 +73,7 @@ class Config:
|
||||
|
||||
def __init__(self, voices_dir, backend, upstream, upstream_model, llama_bin,
|
||||
llama_model, llama_mmproj, llama_lang, llama_threads, port,
|
||||
max_chars, timeout):
|
||||
max_chars, timeout, default_voice=""):
|
||||
self.voices_dir = Path(voices_dir)
|
||||
self.backend = backend
|
||||
self.upstream = upstream
|
||||
@@ -83,6 +86,7 @@ class Config:
|
||||
self.port = port
|
||||
self.max_chars = max_chars
|
||||
self.timeout = timeout
|
||||
self.default_voice = default_voice
|
||||
|
||||
@classmethod
|
||||
def from_env(cls, environ=None):
|
||||
@@ -100,6 +104,7 @@ class Config:
|
||||
port=_int(env, "SHIM_PORT", 7772),
|
||||
max_chars=_int(env, "SHIM_MAX_CHARS", 15000),
|
||||
timeout=float(_int(env, "SHIM_TIMEOUT", 600)),
|
||||
default_voice=(env.get("DEFAULT_VOICE") or "").strip(),
|
||||
)
|
||||
|
||||
|
||||
@@ -271,8 +276,13 @@ class SpeechHandler(BaseHTTPRequestHandler):
|
||||
"endpoint serves wav and mp3" % fmt))
|
||||
return
|
||||
|
||||
# `voice_id` is the alias the omp harness forwards for `voice`.
|
||||
voice = body.get("voice") or body.get("voice_id")
|
||||
# `voice_id` is the alias the omp harness forwards for `voice`. A request
|
||||
# that names nothing takes the declared default — the harness has no
|
||||
# setting that feeds a voice, so voice-less calls are the normal shape
|
||||
# there, not a client bug.
|
||||
requested = body.get("voice") or body.get("voice_id")
|
||||
defaulted = not requested and bool(cfg.default_voice)
|
||||
voice = requested or cfg.default_voice
|
||||
reference = resolve_voice(cfg.voices_dir, voice)
|
||||
if reference is None:
|
||||
available = list_voices(cfg.voices_dir)
|
||||
@@ -297,8 +307,9 @@ class SpeechHandler(BaseHTTPRequestHandler):
|
||||
|
||||
content_type = MP3_CONTENT_TYPE if fmt == "mp3" else WAV_CONTENT_TYPE
|
||||
self._bytes(200, audio, content_type)
|
||||
self.log_message("speech voice=%s format=%s chars=%d -> %d bytes %s",
|
||||
voice, fmt, len(text), len(audio), content_type)
|
||||
self.log_message("speech voice=%s%s format=%s chars=%d -> %d bytes %s",
|
||||
voice, " (default)" if defaulted else "", fmt,
|
||||
len(text), len(audio), content_type)
|
||||
|
||||
# -- backends ----------------------------------------------------------
|
||||
def _speech_http(self, cfg, model, text, voice, reference):
|
||||
|
||||
@@ -56,11 +56,14 @@
|
||||
# the registration body. The façade mounts the same host directory read-only for
|
||||
# its own name list, so one directory stays the single source of a voice name.
|
||||
#
|
||||
# COLD-VOICE LAW, THE REASON EXECSTARTPOST EXISTS. The FIRST speech request for
|
||||
# a registered voice decodes the whole reference clip to snapshot codec state:
|
||||
# 6.2-8.0 s, first audio byte at 5,530 ms. Every later request restores the
|
||||
# snapshot in 0.5 ms. So the unit warms every voice at boot, serially (~6-8 s
|
||||
# each), and `TimeoutStartSec=900` is what keeps a growing voice library from
|
||||
# REGISTRY LAW, THE REASON EXECSTARTPOST EXISTS. The voice registry is memory
|
||||
# only: a restart forgets every voice, and re-registering does not warm one —
|
||||
# only synthesizing does — so a boot without this hook leaves the façade
|
||||
# answering 400 for every name. The per-voice cold penalty is real but small, and
|
||||
# not the trial's 6.2-8.0 s: measured on the built container 2026-10-18 against an
|
||||
# empty registry, the first synthesis of a freshly registered voice cost 1,883 ms
|
||||
# against 1,226-1,283 ms for the requests after it. The warm pass runs serially
|
||||
# (two short syntheses per voice, ~5 s each) and `TimeoutStartSec=900` keeps a growing library from
|
||||
# tripping systemd's default 90 s start deadline. The leading `-` on
|
||||
# ExecStartPost is load-bearing: with `Restart=always`, a warmer that exits
|
||||
# non-zero (a pack missing its reference.wav, the engine still loading past the
|
||||
@@ -104,7 +107,7 @@ HealthStartPeriod=120s
|
||||
[Service]
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
# Model load (tens of seconds) plus one cold decode per registered voice.
|
||||
# Model load (tens of seconds) plus two short syntheses per registered voice.
|
||||
TimeoutStartSec=900
|
||||
ExecStartPost=-/usr/bin/podman exec frank-tts-engine /app/warm-voices.sh
|
||||
|
||||
|
||||
@@ -37,8 +37,9 @@
|
||||
# has a second backend (`TTS_BACKEND=llama-tts-cli`) that needs no upstream, and
|
||||
# an engine that will not start must not take this endpoint's unit down with it.
|
||||
# When both boot, this comes up after the engine, whose ExecStartPost warm pass
|
||||
# is part of its start — so the first request of the day meets a warm voice
|
||||
# registry instead of paying the 5.5 s cold decode.
|
||||
# is part of its start — so the first request of the day meets a REGISTERED voice
|
||||
# (the registry is memory-only) rather than a 400, and skips the ~0.6 s penalty
|
||||
# the first synthesis of an unwarmed voice costs.
|
||||
#
|
||||
# VOICES. /home/dank/voices is host state, never repo state, and is mounted
|
||||
# READ-ONLY: this endpoint has no business writing voice packs. Because the
|
||||
|
||||
@@ -1459,7 +1459,7 @@ LLM_ZEALOT_RUNTIME="${LLM_ZEALOT_RUNTIME:-/home/dank/GitPackages/llama-prism-b10
|
||||
# image path — a deploy that starts without it silently ships a text-only
|
||||
# endpoint. Checked unconditionally, like frank-dragoon's. Q8_0 is MANDATORY,
|
||||
# not a preference: the card says the BF16 tower can crash llama-server when
|
||||
# several image requests run at once, and this endpoint runs --parallel 3. Its
|
||||
# several image requests run at once, and this endpoint runs --parallel 2. Its
|
||||
# 868.32 MiB worst-case VRAM (595.29 model + 248.10 compute in the boot ledger)
|
||||
# lands AFTER the engine's fit check, so it is not inside the q8_0 budget.
|
||||
LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}"
|
||||
|
||||
@@ -27,18 +27,17 @@
|
||||
# below stop describing the process you are running. Nothing in this fleet
|
||||
# pins this container; that is a decision, not an oversight.
|
||||
#
|
||||
# THE COLD-VOICE LAW (why the unit has an ExecStartPost at all). The first
|
||||
# speech request for a registered voice decodes the ENTIRE reference clip to
|
||||
# snapshot its codec state: 6.2-8.0 s total, first audio byte at 5,530 ms. Every
|
||||
# later request for that voice restores the snapshot in 0.5 ms. Registrations
|
||||
# live in memory only, so a restart loses them and re-registration alone does
|
||||
# NOT warm the voice — the warm pass has to synthesize something. So
|
||||
# dankclaw/voiceservice/tts-engine-warm.sh (in the image at
|
||||
# /app/warm-voices.sh) runs on every start and speaks one short sentence per
|
||||
# voice. Skip it and the first person to ask for audio that day waits 5.5 s in
|
||||
# silence. It runs INSIDE this container — `podman exec`, from the unit's
|
||||
# ExecStartPost — so it reads /voices here and the keys at the bottom of this
|
||||
# file are its knobs too.
|
||||
# THE REGISTRY LAW (why the unit has an ExecStartPost at all). Registrations live
|
||||
# in memory only, so a restart loses them, and re-registration alone does NOT warm
|
||||
# a voice — the warm pass has to synthesize something. Without it the façade
|
||||
# answers 400 for every name until somebody asks for audio. So
|
||||
# dankclaw/voiceservice/tts-engine-warm.sh (in the image at /app/warm-voices.sh)
|
||||
# runs on every start and speaks a short sentence twice per voice. The penalty it
|
||||
# removes is about 0.6 s, NOT the trial's 6.2-8.0 s: measured on the built
|
||||
# container 2026-10-18 against an empty registry, the first synthesis cost
|
||||
# 1,883 ms and the requests after it 1,226-1,283 ms. The hook runs INSIDE this
|
||||
# container — `podman exec`, from the unit's ExecStartPost — so it reads /voices
|
||||
# here and the keys at the bottom of this file are its knobs too.
|
||||
#
|
||||
# STREAMING IS NOT BUFFERED. The same seed gives audio that differs in 92.2 %
|
||||
# of samples (-41.6 dB RMS) between `response_format=pcm` (the streaming path,
|
||||
@@ -125,8 +124,8 @@ HEALTH_WAIT=300
|
||||
# The sentence each voice speaks to snapshot its codec state, and the text the
|
||||
# second (warm-timing) pass repeats. Fixed on purpose: the cold=/warm= numbers in
|
||||
# the journal are comparable across boots only if the input is the same. Kept
|
||||
# short because warming a library is serial — one cold decode per voice, 6-8 s
|
||||
# each, inside the unit's start deadline.
|
||||
# short because warming a library is serial — two syntheses per voice, ~5 s each,
|
||||
# inside the unit's start deadline.
|
||||
# QUOTED, and the quoting means two different things to the two readers of this
|
||||
# file, which is the trap here:
|
||||
# * bash (`set -a; source`, how the repo's seams and any human read it) needs
|
||||
|
||||
+13
-2
@@ -35,8 +35,9 @@
|
||||
# shim resolves the voice and re-encodes to mp3 when asked. The
|
||||
# shipped engine (frank-tts-engine) speaks a voice it has
|
||||
# REGISTERED on itself, from the same /home/dank/voices packs,
|
||||
# and warms them at boot — see [[Frank TTS Engine]] for why an
|
||||
# unwarmed voice costs the first caller 5.5 s.
|
||||
# and warms them at boot — see [[Frank TTS Engine]] for what that
|
||||
# boot warm is for (a memory-only registry: without it every name
|
||||
# is a 400) and what it saves (~0.6 s).
|
||||
# llama-tts-cli run LLAMA_TTS_BIN once per request with
|
||||
# --tts-speaker-file <reference.wav>, nice 19 so a narration
|
||||
# never starves the text endpoints on this box. Needs
|
||||
@@ -56,6 +57,16 @@ TTS_BACKEND=omni-http
|
||||
TTS_UPSTREAM=http://frank-tts-engine:8880
|
||||
TTS_UPSTREAM_MODEL=voxcpm2
|
||||
|
||||
# THE DEFAULT VOICE — what a request that names NO voice speaks. The omp harness
|
||||
# has no setting that feeds a voice to its speech device (`voice_id` is a
|
||||
# per-call argument; `speech.voice` belongs to the TUI's Kokoro speak-aloud and
|
||||
# never reaches an HTTP request), so a harness voiceover arrives voice-less and
|
||||
# would otherwise be a 400 for every caller that does not name one. An UNKNOWN
|
||||
# name still 400s with the list that exists — this covers the absent case only —
|
||||
# and the journal marks such a line `(default)` so an unrequested voice never
|
||||
# reads as a requested one. Any pack directory in /home/dank/voices is legal.
|
||||
DEFAULT_VOICE=narrator1
|
||||
|
||||
# CLI-backend knobs, inert while TTS_BACKEND=omni-http. Empty means "not
|
||||
# configured": the endpoint answers a 500 that says so instead of running
|
||||
# `nice env -m ...` with an empty binary name.
|
||||
|
||||
+76
-57
@@ -2,12 +2,14 @@
|
||||
# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 **with the MTP head in the file**
|
||||
# on the RTX 3090, :7771.
|
||||
#
|
||||
# 2026-10-18 (Frank): swap from the Bonsai-2-**27B** PQ2_0 to the 9B MTP export
|
||||
# at THREE 262,144-token conversations (Frank: "if you can possibly fit a third
|
||||
# context, do it"). The 9B bills HALF the KV of the 27B per token, which is what
|
||||
# funds 3 x the native window on one 24,123 MiB card. Speculative decoding ships
|
||||
# OFF — measured, not assumed; see the MTP law below, which is the whole story
|
||||
# and the reason the publisher's "depth 2 works best" line does not transfer.
|
||||
# 2026-10-18 (Frank): swap from the Bonsai-2-**27B** PQ2_0 to the 9B MTP export,
|
||||
# and the same evening the shape moved to TWO 262,144-token conversations over an
|
||||
# UNQUANTIZED (f16) KV cache — "shift frank-zealot to 2 parallels of unquantized
|
||||
# kv". Exact K and V is the retrieval-fidelity ask; the price is one concurrent
|
||||
# conversation, because f16 owns the card. Speculative decoding ships OFF —
|
||||
# measured, not assumed; see the MTP law below, which is the whole story and the
|
||||
# reason the publisher's "depth 2 works best" line does not transfer (and why
|
||||
# exact KV makes the drafter illegal outright).
|
||||
#
|
||||
# THE RUNTIME IS A LOCK, NOT A PREFERENCE. The file is general.file_type 141
|
||||
# carrying custom GGML type 142 tensors plus prism.hadamard.* metadata
|
||||
@@ -27,7 +29,7 @@ MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.g
|
||||
# 624,229,664 B, Hub LFS sha256 e0cb6f69… — Q8_0 IS MANDATORY, not a preference:
|
||||
# the publisher's card says the BF16 mmproj "can crash llama-server when several
|
||||
# image requests run at once; Q8_0 is not affected", and this endpoint runs
|
||||
# --parallel 3, i.e. exactly that shape. The projector's worst case lands AFTER
|
||||
# --parallel 2, i.e. exactly that shape. The projector's worst case lands AFTER
|
||||
# the engine's own fit check, so it is never inside a projection number.
|
||||
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
|
||||
# Base alignment: Bonsai-2 is the censored BASE lineage. The 27B CRACK re-export
|
||||
@@ -42,16 +44,18 @@ MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.ggu
|
||||
ALIAS=frank-zealot
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
# CTX_SIZE is the POOL llama.cpp divides across slots: 786,432 / 3 = 262,144 per
|
||||
# conversation — each slot at the model's TRAINED window, which is the point of
|
||||
# the swap (the 27B era had to cut each conversation to 200,704 to fit two).
|
||||
# MUST stay divisible by PARALLEL (the seam enforces it) and the harness
|
||||
# contextWindow quotes the PER-SLOT number — a longer prompt is answered with a
|
||||
# 400, never truncated. The ceiling cannot go past 3 x 262,144: that is the
|
||||
# native wall (no YaRN in this path) and the arithmetic agrees — a 4th slot
|
||||
# would push each below the compaction contract.
|
||||
CTX_SIZE=786432
|
||||
PARALLEL=3
|
||||
# CTX_SIZE is the POOL llama.cpp divides across slots: 524,288 / 2 = 262,144 per
|
||||
# conversation — each slot STILL at the model's TRAINED window; the per-slot
|
||||
# contract did not move. MUST stay divisible by PARALLEL (the seam enforces it)
|
||||
# and the harness contextWindow quotes the PER-SLOT number — a longer prompt is
|
||||
# answered with a 400, never truncated. THE POOL CANNOT STAY AT 786,432 UNDER
|
||||
# EXACT KV: f16 bills 32,768 B/token, so that pool is 24,576 MiB — more than the
|
||||
# 24,123 MiB the driver leaves addressable, before a single weight loads.
|
||||
# Halving the POOL to 524,288 (16,384 MiB) is what funds exact KV. The window
|
||||
# cannot go past 262,144 a slot either: that is the native wall (no YaRN in this
|
||||
# path), so a 3rd slot would be a KV-dtype cut, not more context.
|
||||
CTX_SIZE=524288
|
||||
PARALLEL=2
|
||||
BATCH_SIZE=4096
|
||||
# Ubatch is a prefill knob (decode never moves) with NO interior optimum on this
|
||||
# CUDA card up to 2048 — 2048 is the measured optimum (+4.6 % prefill). It has a
|
||||
@@ -60,38 +64,47 @@ BATCH_SIZE=4096
|
||||
UBATCH_SIZE=2048
|
||||
# KV law (header + boot ledger, both confirmed): qwen35 hybrid, 32 layers with
|
||||
# qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated DeltaNet;
|
||||
# 4 KV heads × 256 dim = 8,192 elements per token per cache. q8_0 bills 8,704 B
|
||||
# per cache ⇒ 17,408 B/token = 17 KiB, so the 786,432 pool is 13,056.00 MiB —
|
||||
# the ledger said exactly that. HALF the 27B's 34,816 B/token, which is the
|
||||
# whole reason three native windows fit. f16 would bill 26,112 MiB at this pool
|
||||
# (against a 24,123 MiB card) and dies at cudaMalloc; the 24 DeltaNet layers
|
||||
# bill a fixed recurrent state per SLOT instead of per token — the PRODUCTION
|
||||
# ledger prints "RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)"
|
||||
# (~50 MiB a cell), while the SPEC-ARMED arms print 452.25 MiB, because MTP
|
||||
# depth 2 bills three seqs per slot (9 cells) — which is why pool arithmetic
|
||||
# alone under-prices the shape.
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=q8_0
|
||||
# 4 KV heads × 256 dim (n_embd_k_gqa = n_embd_v_gqa = 1,024 per layer) = 8,192
|
||||
# elements per token per cache. EXACT KV: f16 = 2 B/element ⇒ 16,384 B per cache
|
||||
# ⇒ 32,768 B/token = 32 KiB, so the 524,288 pool is 16,384.00 MiB. q8_0 bills
|
||||
# 1.0625 B/element (32 codes + one 2-byte scale per 32-block) ⇒ 17,408 B/token,
|
||||
# which is the 13,056.00 MiB the 3-slot ledger printed at the 786,432 pool.
|
||||
# THE RATIO IS 1.882×, NOT 2×: the old note here said "f16 would bill 26,112
|
||||
# MiB" because it doubled the q8_0 figure with its scale overhead included;
|
||||
# exact arithmetic says 24,576 MiB at that pool. The wall stands, only its
|
||||
# height moves. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
|
||||
# instead of per token — the 3-slot ledger printed "RS buffer size = 150.75 MiB
|
||||
# (3 cells, 32 layers, 3 seqs)", ~50.25 MiB a cell so two slots bill 100.50 —
|
||||
# while the SPEC-ARMED arms print 452.25 MiB, because MTP depth 2 bills three
|
||||
# seqs per slot (9 cells). Pool arithmetic alone under-prices the shape.
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
# SPECULATIVE DECODING: OFF, MEASURED. The head is in the file and the drafter
|
||||
# is GOOD (acceptance 0.798/0.617 per position, mean accepted length 2.41) — it
|
||||
# still loses, because the draft context bills its OWN f16 KV over the WHOLE
|
||||
# pool: 4,096 B/position × 786,432 = 3,072.00 MiB, plus its compute buffers,
|
||||
# 4,830 MiB all told. Booted arms, same card, same traffic (2026-10-18):
|
||||
# pool at its default dtype. That dtype IS a knob (--spec-draft-type-k /
|
||||
# --spec-draft-type-v, both default f16 — verified against this build's --help),
|
||||
# but SPEC_TYPE=none never sends them. Under the
|
||||
# 3-slot q8_0 shape that bill was 4,096 B/position × 786,432 = 3,072.00 MiB plus
|
||||
# ~1,758 MiB of buffers. Booted arms, same card, same traffic (2026-10-18, the
|
||||
# WHOLE table is q8_0 K + q8_0 V — it predates exact KV):
|
||||
# 3 × 262,144 spec OFF ub 2048 → 17,876 used / 6,249 free, decode 167.2 t/s
|
||||
# 3 × 262,144 spec ON ub 2048 → 22,706 used / 1,419 free, decode 109.2 t/s
|
||||
# 3 × 262,144 spec ON ub 1024 → 21,400 used / 2,725 free, decode 213.5 t/s
|
||||
# 2 × 262,144 spec ON ub 2048 → 17,178 used / 6,947 free, decode 213.2 t/s
|
||||
# 1 × 262,144 spec ON ub 2048 → 11,652 used / 12,473 free, decode 214.2 t/s
|
||||
# i.e. MTP is worth +28 % decode whenever the card has room, and at the 3-slot
|
||||
# pool it costs 1.3 GiB of compute buffers that ubatch 2048 refuses to give up —
|
||||
# the squeeze, not the drafter, is what halves decode. The arm that WOULD run
|
||||
# MTP at three windows is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (decode 213.5,
|
||||
# 3-stream 250.8 aggregate) and pays prefill −17 % and 2.7 GiB of headroom on a
|
||||
# card whose projector worst-case lands after the fit check. This endpoint's
|
||||
# load is the fleet's COMPACTION HUB (prefill-bound, ~170k in) plus 2-3
|
||||
# concurrent subagents, so the deployed arm takes the +17 % prefill
|
||||
# (2,877 vs 2,395 t/s at 106,418 tokens) and the +12 % three-stream aggregate
|
||||
# (280.2 vs 250.8) over single-stream speed, and keeps 6.2 GiB free.
|
||||
# MTP is worth +28 % decode whenever the card has room. At the SHIPPED exact-KV
|
||||
# shape an f16-draft arm has none: the draft KV alone is 4,096 × 524,288 =
|
||||
# 2,048.00 MiB plus its buffers, on a trunk that already projects ~21,200 MiB of
|
||||
# the 24,123 the driver leaves addressable. DOORS BACK TO THE DRAFTER: one
|
||||
# window per conversation; KV back to q8_0 with UBATCH_SIZE=1024 (213.5 t/s);
|
||||
# or the UNMEASURED arm --spec-draft-type-k q8_0 --spec-draft-type-v q8_0, which
|
||||
# halves the draft bill to 1,088 MiB and is legal on this build (>= b10776).
|
||||
# Nobody has booted that beside exact KV — measure before promising it.
|
||||
# This endpoint's load is still the fleet's COMPACTION HUB (prefill-bound,
|
||||
# ~170k in): prefill is weight-bandwidth bound and does not read the KV cache
|
||||
# per generated token, which is why exact KV is affordable here at all; what it
|
||||
# buys back is concurrency — TWO conversations, not three.
|
||||
# SPEC_TYPE=none strips the WHOLE spec set (--spec-type, every --spec-draft-*):
|
||||
# flags move as a set or the arm measures a crash instead of an absence.
|
||||
SPEC_TYPE=none
|
||||
@@ -105,27 +118,33 @@ SPEC_DRAFT_P_MIN=0
|
||||
# PRESENCE_PENALTY="" is the documented opt-out.
|
||||
PRESENCE_PENALTY=1.0
|
||||
NGL=99
|
||||
# Boot ledger (2026-10-18 PRODUCTION boot under the unit, verbosity 4):
|
||||
# 17,922 MiB used / 6,203 MiB free of 24,123 (nvidia-smi settled)
|
||||
# Boot ledger REFERENCE — the previous production boot (2026-10-18 13:19,
|
||||
# 3 × 262,144, q8_0 K + q8_0 V, verbosity 4), i.e. the shape this file replaced:
|
||||
# 17,956 MiB used / 6,169 MiB free of 24,123 addressable (nvidia-smi settled;
|
||||
# its total column reads 24,576 — the driver reserves the difference, and
|
||||
# older notes quoting "a 24,123 MiB card" mean this addressable figure)
|
||||
# CUDA0 model 2,036.71 MiB + CPU_Mapped 257.66 MiB
|
||||
# CUDA0 KV 13,056.00 MiB (q8_0 K + q8_0 V, 786,432 positions)
|
||||
# CUDA0 RS 150.75 MiB (3 cells, 32 layers, 3 seqs)
|
||||
# compute 1,409.02 MiB CUDA + 1,088.87 MiB CUDA_Host (+2.84 MiB output buf)
|
||||
# mmproj worst-case estimate 868.32 MiB (248.10 CUDA + 24.93 CPU compute
|
||||
# meta) — charged AFTER the engine's own fit check, as always
|
||||
# mmproj 595 MiB weights + 248.10 MiB CUDA compute meta; estimated worst case
|
||||
# 868.32 MiB — charged AFTER the engine's own fit check, as always
|
||||
# no clamp line, no cudaMalloc error
|
||||
# Measured on this boot (/completion timings, greedy, ignore_eos, 512 tokens):
|
||||
# 168.7 t/s decode (three runs 168.4-168.7) and ~300 t/s aggregate across
|
||||
# THREE concurrent streams (102.8 each). The trial arm's record (167.2 decode,
|
||||
# 280.2 aggregate, 2,877 t/s prefill at 106,418) stands; the 27B it replaces
|
||||
# gave 76.6 / 109.4 / 691.7 at 166k. The FIRST request after a boot is a
|
||||
# warmup artifact (37.7 t/s class) and a natural generation that stops early
|
||||
# makes a wall-clock aggregate read 4x low - measure with ignore_eos and read
|
||||
# the engine's own timings.
|
||||
# Fallback rung if the margin is ever eaten (desktop, co-tenant): q4_0 KV on the
|
||||
# 27B era measured 16,392 used / 7,733 free at 2 × 200,704; the same drop here
|
||||
# frees ~6.5 GiB at three windows. UBATCH_SIZE=1024 is the cheaper lever
|
||||
# (−1,306 MiB measured on this shape).
|
||||
# Measured on THAT boot (/completion timings, greedy, ignore_eos, 512 tokens):
|
||||
# 168.7 t/s decode (three runs 168.4-168.7), ~300 t/s aggregate across THREE
|
||||
# concurrent streams (102.8 each), 2,877 t/s prefill at 106,418. The 27B it
|
||||
# replaced gave 76.6 / 109.4 / 691.7 at 166k. These are the numbers the exact-KV
|
||||
# arm is judged against: decode reads 1.882× the KV bytes per token, so a decode
|
||||
# drop is EXPECTED and must be measured, not assumed, while prefill should hold.
|
||||
# Every non-KV term of that ledger survives this reslice except RS (−50.25 MiB,
|
||||
# one fewer cell) and one slot's worth of scheduler state, so the f16 arm
|
||||
# PROJECTS 17,956 − 13,056 + 16,384 − 50 = 21,234 MiB used / 2,889 free, and
|
||||
# ~2.0 GiB free through the projector's worst case. REPLACE THIS PROJECTION WITH
|
||||
# THE MEASURED PAIR after the boot: read the new ledger AND settled nvidia-smi.
|
||||
# Fallback rungs if that margin gets eaten (desktop, co-tenant), cheapest first:
|
||||
# CACHE_TYPE_V=q8_0 (exact K, quantized V) frees 8,192 MiB at this pool;
|
||||
# CTX_SIZE=401408 (two 200,704 conversations) puts exact KV at 12,544.00 MiB and
|
||||
# buys ~6.7 GiB back; UBATCH_SIZE=1024 hands 1,306 MiB of compute buffers back.
|
||||
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
|
||||
# and graph capture land in the journal, which is what the VRAM-fit forensics
|
||||
# read. Levels above 4 flood the journal.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=llama.cpp (PrismML build 10777) Ternary-Bonsai-2-9B PQ2_0-MTP GGUF (qwen35 hybrid, 3x262144 slots over one 786432 pool, q8_0 K+V, MTP off, mmproj Q8_0 vision) on the RTX 3090 (dankclaw, frank-zealot)
|
||||
Description=llama.cpp (PrismML build 10777) Ternary-Bonsai-2-9B PQ2_0-MTP GGUF (qwen35 hybrid, 2x262144 slots over one 524288 pool, unquantized f16 K+V, MTP off, mmproj Q8_0 vision) on the RTX 3090 (dankclaw, frank-zealot)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
|
||||
@@ -3,33 +3,35 @@
|
||||
# in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
|
||||
# vendor build 10777, CUDA sm_86).
|
||||
#
|
||||
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export at THREE
|
||||
# 262,144-token conversations (--ctx-size 786432 --parallel 3), q8_0 K + q8_0 V.
|
||||
# The 9B bills 17,408 B/token against the 27B's 34,816 (8 full-attention layers
|
||||
# instead of 16), and that halved rate is what funds three NATIVE windows where
|
||||
# the 27B could only afford two cut to 200,704. Verbosity 4 ships so the
|
||||
# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the
|
||||
# journal.
|
||||
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
|
||||
# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
|
||||
# an UNQUANTIZED f16 KV cache. The 9B bills 8,192 elements per token per cache
|
||||
# (8 full-attention layers instead of the 27B's 16), so exact K and V costs
|
||||
# 32,768 B/token — 16,384.00 MiB at the 524,288 pool, which is what a 24,123 MiB
|
||||
# addressable card can carry beside the trunk. Verbosity 4 ships so the VRAM-fit
|
||||
# forensics (KV layout, buffer sizes, graph capture) land in the journal.
|
||||
#
|
||||
# Architecture (engine ledger, not the marketing page): qwen35 hybrid, 32
|
||||
# layers with qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated
|
||||
# DeltaNet, 4 KV heads × 256 dim, native 262,144. The KV pool scales with the 8
|
||||
# full layers (13,056.00 MiB at the 786,432 pool, ledger-exact); the DeltaNet
|
||||
# layers cost a fixed recurrent state per SLOT — 452.25 MiB for 3 cells, ~151
|
||||
# MiB each. The runtime is a hard lock, not a preference: custom GGML type 142
|
||||
# plus prism.hadamard.* metadata, which only the vendor build registers, and
|
||||
# MTP beside quantized V needs build ≥ b10776 — so the prebuilt moved 10754 →
|
||||
# 10777 with the payload. Vision comes from the publisher's Q8_0 projector (the
|
||||
# BF16 one can crash the server on concurrent images, and this endpoint runs
|
||||
# three slots), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is
|
||||
# the documented text-only opt-out.
|
||||
# full layers (16,384.00 MiB of f16 at the 524,288 pool); the DeltaNet layers
|
||||
# cost a fixed recurrent state per SLOT — ~50.25 MiB a cell, so 100.50 MiB for
|
||||
# two. The runtime is a hard lock, not a preference: custom GGML type 142 plus
|
||||
# prism.hadamard.* metadata, which only the vendor build registers (the b10776
|
||||
# gate that moved the prebuilt 10754 → 10777 was MTP beside quantized V, which
|
||||
# exact KV now makes moot — the type-142 lock alone keeps the build pinned).
|
||||
# Vision comes from the publisher's Q8_0 projector (the BF16 one can crash the
|
||||
# server on concurrent images, and this endpoint runs two slots), so --mmproj is
|
||||
# what gives this endpoint eyes; MMPROJ_FILE="" is the documented text-only
|
||||
# opt-out.
|
||||
#
|
||||
# Speculative decoding ships OFF, measured: the head loads and accepts well
|
||||
# (0.798/0.617 per position, mean length 2.41) but the draft context bills its
|
||||
# own f16 KV over the WHOLE pool — 4,096 B/position × 786,432 = 3,072 MiB plus
|
||||
# buffers — and at ubatch 2048 that squeeze HALVES decode (167.2 → 109.2 t/s).
|
||||
# frank-zealot.env carries the six-arm table and the arm that turns it back on
|
||||
# (--ubatch-size 1024, decode 213.5 t/s) with its price.
|
||||
# own f16 KV over the WHOLE pool — 4,096 B/position × 524,288 = 2,048 MiB plus
|
||||
# buffers — and the trunk already projects 21,234 MiB with exact KV. Exact KV
|
||||
# and MTP cannot share this card; frank-zealot.env carries the q8_0-era arm table
|
||||
# and the two doors back to the drafter (one window, or KV back to q8_0 at
|
||||
# --ubatch-size 1024 for 213.5 t/s).
|
||||
#
|
||||
# Sampling: the checkpoint carries the publisher's thinking recipe as GGUF
|
||||
# metadata (general.sampling.* = temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
|
||||
@@ -56,40 +58,46 @@ fi
|
||||
: "${ALIAS:=frank-zealot}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7771}"
|
||||
# ctx-size is the POOL llama.cpp divides across slots: 786,432 / 3 = 262,144
|
||||
# per conversation — every slot at the model's trained window. The harness
|
||||
# ctx-size is the POOL llama.cpp divides across slots: 524,288 / 2 = 262,144
|
||||
# per conversation — every slot still at the model's trained window. The harness
|
||||
# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered
|
||||
# with a 400, never truncated.
|
||||
: "${CTX_SIZE:=786432}"
|
||||
: "${PARALLEL:=3}"
|
||||
# with a 400, never truncated. Under exact KV the pool cannot stay at 786,432:
|
||||
# 32,768 B/token × 786,432 = 24,576 MiB, more than the card's whole addressable
|
||||
# 24,123 MiB before a single weight loads.
|
||||
: "${CTX_SIZE:=524288}"
|
||||
: "${PARALLEL:=2}"
|
||||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
|
||||
# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill
|
||||
# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and
|
||||
# throughput-neutral at a fixed ubatch. It ALSO sets the compute-buffer size,
|
||||
# which is the term that decides whether an MTP draft context fits beside a
|
||||
# 786,432-position pool: 1024 hands back 1,306 MiB and is the arm that runs the
|
||||
# drafter at three slots (frank-zealot.env carries the table).
|
||||
# which is the term that decides whether an MTP draft context fits beside a KV
|
||||
# pool: 1024 hands back 1,306 MiB and was the arm that ran the drafter at three
|
||||
# q8_0 windows (frank-zealot.env carries the table).
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=2048}"
|
||||
# Quantized KV: q8_0 K + q8_0 V — the only pair that funds 3 × 262,144 on this
|
||||
# card. 17 KiB/token at this geometry (8 full-attention layers × 4 KV heads ×
|
||||
# 256 dim); 13,056.00 MiB at the 786,432 pool, ledger-exact. f16 would bill
|
||||
# 26,112 MiB against a 24,123 MiB card and dies at cudaMalloc; q4_0 is the
|
||||
# fallback rung. Explicit either way so the contract is visible in the argv,
|
||||
# and --flash-attn stays required for quantized V (no true fp8 KV dtype exists
|
||||
# in this build).
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=q8_0}"
|
||||
# Exact KV: f16 K + f16 V — no quantization error anywhere in the attention
|
||||
# state. 32 KiB/token at this geometry (8 full-attention layers × 4 KV heads ×
|
||||
# 256 dim × 2 B × 2 caches); 16,384.00 MiB at the 524,288 pool. That is the whole
|
||||
# reason the slot count is 2: the same dtype at the old 786,432 pool is 24,576
|
||||
# MiB and dies at cudaMalloc. q8_0 K + q8_0 V (17,408 B/token, 13,056.00 MiB at
|
||||
# 786,432) is the documented fallback rung and funds three windows; CACHE_TYPE_V
|
||||
# alone is the halfway door. Explicit either way so the contract is visible in
|
||||
# the argv, and --flash-attn stays pinned because quantized V — the fallback —
|
||||
# is illegal without it (no true fp8 KV dtype exists in this build).
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
# Speculative decoding defaults OFF, measured on this card and shape: the head
|
||||
# IS in the checkpoint (qwen35.nextn_predict_layers = 1 — MTP is a property of
|
||||
# the FILE, so the plain PQ2_0 sibling could not run it at any flag value), it
|
||||
# loads, and it accepts well (mean accepted length 2.41) — but the draft context
|
||||
# bills its own f16 KV over the WHOLE pool (4,096 B/position, 3,072 MiB here)
|
||||
# and at ubatch 2048 that squeeze halves decode: 167.2 → 109.2 t/s. There is no
|
||||
# --spec-draft-cache-type knob in this build to make the draft cheap, and no
|
||||
# --spec-draft-model either: the draft rides the main model's weights. Card gate:
|
||||
# quantized V alongside MTP needs fork build >= prism-b10776, which is why
|
||||
# LLAMA_BIN moved to the 10777 prebuilt; on an older build the pair is illegal.
|
||||
# bills its own f16 KV over the WHOLE pool (4,096 B/position, 2,048 MiB at the
|
||||
# 524,288 pool) and at ubatch 2048 that squeeze halves decode: 167.2 → 109.2 t/s
|
||||
# on the q8_0 3-slot shape. The draft cache dtype IS a knob (--spec-draft-type-k
|
||||
# / --spec-draft-type-v, both default f16 — verified in this build's --help), so
|
||||
# an armed drafter can halve its own bill; there is no --spec-draft-model because
|
||||
# the draft rides the main model's weights. Card gate: quantized V alongside MTP
|
||||
# needs fork build >= prism-b10776, which is why LLAMA_BIN moved to the 10777
|
||||
# prebuilt; exact KV removes that gate but NOT the type-142 loader lock.
|
||||
# SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 is the documented MTP arm (213.5 t/s);
|
||||
# SPEC_TYPE=none strips the whole set (flags move together).
|
||||
: "${SPEC_TYPE:=none}"
|
||||
@@ -104,7 +112,7 @@ fi
|
||||
: "${NGL:=99}"
|
||||
# Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info);
|
||||
# 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what
|
||||
# the VRAM-fit forensics need — the q8_0 KV contract and the draft-KV bill are
|
||||
# the VRAM-fit forensics need — the exact-KV size and the draft-KV bill are
|
||||
# only provable from this log. Levels above 4 flood the journal.
|
||||
: "${VERBOSITY:=4}"
|
||||
# /metrics needs the flag: llama.cpp returns 501 unless the server starts with
|
||||
@@ -169,11 +177,11 @@ fi
|
||||
# open after boot; make sure it exists before the server needs it.
|
||||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||||
|
||||
# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
|
||||
# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
|
||||
# the q8_0 KV contract in frank-zealot.env is a real path — without this
|
||||
# flag or that build option the engine silently falls back and the
|
||||
# contract becomes a lie.
|
||||
# Flash attention stays pinned on. Exact f16 KV runs fine with or without it,
|
||||
# but the documented fallback rung (quantized V) is ILLEGAL without it, and the
|
||||
# build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17) so that rung is a
|
||||
# real path — dropping the flag here would leave the retreat route silently
|
||||
# falling back to something the env file does not describe.
|
||||
ARGS=(
|
||||
--model "$MODEL_FILE"
|
||||
--alias "$ALIAS"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
type: reference
|
||||
description: "frank-tts-engine is the arm A TTS engine on Dankclaw: qwentts.cpp's resident tts-server with the 1.7 B base talker, on a private podman bridge behind frank-tts — repo-owned, not yet deployed."
|
||||
description: "frank-tts-engine is the arm A TTS engine on Dankclaw: qwentts.cpp's resident tts-server with the 1.7 B base talker, on a private podman bridge behind frank-tts — deployed 2026-10-18, and it publishes no port on purpose."
|
||||
tags:
|
||||
- dankclaw
|
||||
- tts
|
||||
@@ -14,7 +14,7 @@ timestamp: 2026-10-18
|
||||
|
||||
`frank-tts-engine` is the **arm A text-to-speech engine** on Dankclaw: [qwentts.cpp](https://github.com/ServeurpersoCom/qwentts.cpp)'s resident `tts-server` HTTP process, holding the Qwen3-TTS **1.7 B base talker (Q4_K_M)** and the **12 Hz tokenizer (Q8_0)** in RAM once and synthesizing from there. It listens on container port **8880** and is **deliberately not a fleet endpoint** — the endpoint consumers address is `frank-tts` on **7772**, which resolves voice names, re-encodes to mp3 and owns the timeout budget. This page is the engine runbook; the endpoint's contract lives with its own page.
|
||||
|
||||
**Nothing here is deployed.** This page documents repo-owned configuration — the Quadlet unit and network, the image build, the weight fetcher, the boot warmer, the deployer path — and the measured facts those files were written against. No container, image, weight file or network exists on Dankclaw yet, and the engine publishes no port at all, so there is nothing for the homepage checker or a browser to reach. Deploying it is a deliberate next step, in the order at the bottom of this page.
|
||||
**Deployed 2026-10-18.** The chain ran in the order at the bottom of this page — fetch (1.4 GiB, byte + LFS oid verified, 18 s), image build against the pinned commit, unit + network install, container-health gate, warm pass — and the live state still reads the way this page describes it: unit `active` with no drop-ins, container healthy 31 s after the build, **4.0 GiB** resident on a host that kept 17 GiB available, `3 voice(s) registered and warm` in the journal, and no published port anywhere. Two things the first real boot taught: the warm hook's registration body must reach curl through a **file**, not an argument (Troubleshooting), and `STEP 3/13 RUN useradd` is the first honest proof of the image's non-root user.
|
||||
|
||||
**Host:** Dankclaw (`192.168.0.143`), rootless Podman, user `dank` — **Port:** 8880, **never published** — **Network:** the private `systemd-frank-tts-engine` bridge (10.89.5.0/24) and no other — **Deployer:** `frank-tts-engine` — **Consumers:** `frank-tts` (7772) and, through it, every voice request in the fleet including [[Open Notebook]]'s podcast renders.
|
||||
|
||||
@@ -96,25 +96,27 @@ The **façade** on 7772 is the pair's public surface and carries the probe. Note
|
||||
|---|---|
|
||||
| `dankclaw/voiceservice/Containerfile.engine` | two-stage build of the pinned commit, proven cmake line, digest-pinned base, ships both scripts |
|
||||
| `dankclaw/voiceservice/tts-engine-start.sh` | image entrypoint (`/app/start.sh`): env → flags, refuses to boot without weights |
|
||||
| `dankclaw/voiceservice/tts-engine-warm.sh` | boot warmer (`/app/warm-voices.sh`): register from `reference.wav`, speak one sentence, report cold/warm |
|
||||
| `dankclaw/voiceservice/tts-engine-warm.sh` | boot warmer (`/app/warm-voices.sh`): register from `reference.wav` (body via a temp file), speak a fixed sentence twice, report cold/warm |
|
||||
| `dankclaw/voiceservice/quadlet/frank-tts-engine.container` | the unit: bridge only, no publish, env file, two read-only mounts, health probe, warm hook, `Restart=always` |
|
||||
| `dankclaw/voiceservice/quadlet/frank-tts-engine.network` | the private 10.89.5.0/24 bridge, with the subnet registry |
|
||||
| `scripts/frank-tts-engine.env` | paths and numbers only, with the thread law and the cold-voice law in its header |
|
||||
| `scripts/frank-tts-engine.env` | paths and numbers only, with the thread law and the registry law in its header |
|
||||
| `scripts/download-frank-tts-engine-qwentts.sh` | weight fetcher: no resume, byte count + LFS oid check, `--verify-only` |
|
||||
| `scripts/deploy-remote.sh` → `deploy_frank_tts_engine` | weights gate → build → quadlet → restart → container health → warm |
|
||||
| `tests/frank_tts_engine.bats` | the seam: generator output, entrypoint argv, warmer against a stub engine, planner and routing maps |
|
||||
|
||||
## Deploy order (host steps, in order)
|
||||
## Deploy record (host steps, in the order they ran)
|
||||
|
||||
Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblocks the podcast pipeline.
|
||||
Run 2026-10-18 as `./scripts/deploy.sh dankclaw --service frank-tts-engine`, then
|
||||
`--service frank-tts`. Steps 1–6 are this service and the façade; only step 7 is
|
||||
still open.
|
||||
|
||||
1. **Fetch the weights** — `scripts/download-frank-tts-engine-qwentts.sh` (1.4 GiB, byte + sha256 verified, idempotent; `--verify-only` is what the deployer calls).
|
||||
2. **Deploy the engine** — `./scripts/deploy.sh dankclaw --service frank-tts-engine` (`--service` is the only way to name one service; `deploy-remote.sh` is the remote half and takes `<target> <git-revision> [service]...`, so never call it by hand with a bare service name). Builds the image (needs github.com once), installs the unit and the network file, restarts, gates on the container's health status, then warms.
|
||||
2. **Deploy the engine** — `./scripts/deploy.sh dankclaw --service frank-tts-engine` (`--service` is the only way to name one service; `deploy-remote.sh` is the remote half and takes `<target> <git-revision> [service]...`, so never call it by hand with a bare service name). It builds the image (needs github.com once), installs the unit and the network file, gates on the container's health status, then warms.
|
||||
3. **Confirm the endpoint is alive** — `podman exec frank-tts-engine curl -fsS http://127.0.0.1:8880/health` → `{"status":"ok"}`, and `systemctl --user status frank-tts-engine` showing the unit `active` with the warm pass behind it.
|
||||
4. **Confirm the voices are known** — the warmer's journal lines must name every pack; once the façade is up, `curl http://127.0.0.1:7772/v1/audio/voices` is the authoritative list (the façade has exactly three routes: `/health`, `/v1/audio/voices`, `/v1/audio/speech`).
|
||||
5. **Prove the cold-voice law is paid at boot, not by a listener** — two identical synthesis requests for one voice, back to back, from inside the engine's namespace; the **second** must answer well under a second (`warm=` in the journal is already that number; the manual pair is the confirmation on the real artifact).
|
||||
6. Then the façade on **7772** (its own page): nothing may be published there until the engine answers, because open-notebook's render fails the moment the façade is up and the engine is not.
|
||||
7. Nothing on 7772 exists yet — that is why open-notebook's podcast render cannot run. See [[Open Notebook]].
|
||||
4. **Confirm the voices are known** — the warmer's journal lines name every pack, and once the façade is up `curl http://127.0.0.1:7772/v1/audio/voices` is the authoritative list (the façade has exactly three routes: `/health`, `/v1/audio/voices`, `/v1/audio/speech`).
|
||||
5. **Measure the ladder instead of trusting the trial** — two identical requests for one voice back to back. The design expected 6.2-8.0 s then 0.5 ms; the deployed container gave **1,883 ms then 1,226-1,283 ms**, so the hook buys ~0.6 s and stays only for the registry reason (law 1). The retired numbers must not be quoted.
|
||||
6. **Then the façade on 7772** (its own page), after the engine answered — that order is the whole point of step 6: publishing the façade in front of a dead engine would make every consumer fail at the first voice line.
|
||||
7. **Open:** [[Open Notebook]] is still not installed, so nothing renders a podcast yet; the voice leg it waits on now exists.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
type: reference
|
||||
description: "frank-tts is the fleet's cloned-voice speech endpoint on Dankclaw :7772 — the one voice surface consumers address, in front of frank-tts-engine; repo-owned, not yet deployed."
|
||||
description: "frank-tts is the fleet's cloned-voice speech endpoint on Dankclaw :7772 — the one voice surface consumers address, in front of frank-tts-engine; deployed 2026-10-18 and wired as the harness speech role."
|
||||
tags:
|
||||
- dankclaw
|
||||
- tts
|
||||
@@ -16,10 +16,15 @@ audio in that cloned voice. Everything else in the voice stack sits behind it
|
||||
the engine ([[Frank TTS Engine]]) publishes no port on purpose, and consumers
|
||||
never learn a path.
|
||||
|
||||
**Nothing here is deployed.** No host was touched while this was written: no
|
||||
image built, no container started, nothing listening on 7772. That is also why
|
||||
[[Open Notebook]] cannot render a podcast episode yet — its pipeline calls this
|
||||
endpoint.
|
||||
**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts`,
|
||||
both units `active`, no drop-ins, and the homepage card's `wip` gate came off in
|
||||
the same change. Probes on the deployed endpoint: `/health` answers
|
||||
`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}`,
|
||||
and two different voices rendered the same sentence into **different** bytes
|
||||
(24 kHz mono PCM16, 4.24 s, healthy RMS) — so name resolution is real, not a
|
||||
pass-through of one default. The operator harness routes its `speech` role here;
|
||||
see [[#The harness speech role]]. What still waits is [[Open Notebook]]: its
|
||||
render needs the app installed, not this endpoint.
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
@@ -30,6 +35,7 @@ endpoint.
|
||||
| Networks | `dankclaw_default` first (it owns the default route and the consumers), then `frank-tts-engine.network` (10.89.5.0/24, the bridge to the engine) |
|
||||
| Voice packs | `/home/dank/voices` mounted read-only at `/voices` — host state, never repo state |
|
||||
| Upstream | `http://frank-tts-engine:8880` (`scripts/frank-tts.env`, `TTS_UPSTREAM`) |
|
||||
| Default voice | `DEFAULT_VOICE` (`scripts/frank-tts.env`) answers a request that names none; the journal marks that line `(default)`, and an **unknown** name is still a 400 |
|
||||
| Rollback | `TTS_BACKEND=llama-tts-cli` runs a CLI binary per request and needs no upstream at all |
|
||||
|
||||
## Naming a voice
|
||||
@@ -37,7 +43,9 @@ endpoint.
|
||||
A voice **is** a directory name. `voice=narrator1` resolves to
|
||||
`/voices/narrator1/reference.wav`, and that file — not a manifest, not a config
|
||||
entry — is what makes the name real. omp's TTS device may send `voice_id`; the
|
||||
shim accepts either.
|
||||
shim accepts either. A call that sends **neither** — the harness's normal shape,
|
||||
because no harness setting carries a voice for this transport — speaks
|
||||
`$DEFAULT_VOICE`; empty default makes it a 400 again.
|
||||
|
||||
The usable shape is `^[a-z0-9][a-z0-9._-]*$`: lowercase, starting with a letter
|
||||
or digit, then letters, digits, `.`, `_`, `-`, at most 64 characters. No slash,
|
||||
@@ -109,7 +117,7 @@ shows up as a failed synthesis, never as a down probe.
|
||||
| `dankclaw/voiceservice/Containerfile` | python 3.12-alpine pinned by digest + ffmpeg, uid 10000 |
|
||||
| `dankclaw/voiceservice/quadlet/frank-tts.container` | the unit: two networks, `Restart=always`, `HealthCmd`, `[Install]` |
|
||||
| `dankclaw/voiceservice/quadlet/frank-tts-engine.network` | the shared bridge, named after the engine unit and attached by both |
|
||||
| `scripts/frank-tts.env` | backend, upstream, port, limits — paths and numbers, no credentials |
|
||||
| `scripts/frank-tts.env` | backend, upstream, port, limits, and the default voice — paths and numbers, no credentials |
|
||||
| `tests/frank_tts_unit.bats` | the shim against stub engines over real sockets |
|
||||
| `tests/frank_tts_engine.bats` | the pair's wiring: generator output, planner, homepage, deployer |
|
||||
|
||||
@@ -138,8 +146,51 @@ A `/health` probe job would therefore be structurally never-green, and because
|
||||
`frank-tts` target would page on Signal every few minutes for an endpoint that
|
||||
is fine — the frank-archon lesson pointed the other way. The engine cannot be
|
||||
scraped at all: it publishes nothing. Status lives in `systemctl --user status`,
|
||||
`podman healthcheck run`, the two unit journals, and the homepage card, which is
|
||||
gated `wip: down` until the deploy and probes `127.0.0.1:7772`.
|
||||
`podman healthcheck run`, the two unit journals, and the homepage card, which
|
||||
probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the
|
||||
2026-10-18 deploy.
|
||||
|
||||
## The harness speech role
|
||||
|
||||
The operator harness reaches this endpoint through
|
||||
`~/.omp/agent/models.yml` — provider `frank-tts (dankclaw)`,
|
||||
`baseUrl: http://192.168.0.143:7772/v1`, `auth: none`, and the runner api
|
||||
`openai-speech` on the **model** row (naming a runner api at *provider* level
|
||||
fails the whole file's validation and disables every custom provider, which is
|
||||
how the fleet's chat endpoints disappear) — plus `config.yml`
|
||||
`modelRoles.speech: frank-tts (dankclaw)/frank-tts`. Four laws, read out of the
|
||||
harness bundle and measured the day of the flip:
|
||||
|
||||
- **The only voice the harness can send is the `voice_id` argument of the call.**
|
||||
The transport builds `{model, input, response_format, voice?}` and adds `voice`
|
||||
**only** when the caller passed `voice_id`; no setting feeds it.
|
||||
`tts.localVoice` is the *local Kokoro backend's* voice (it is what answered
|
||||
`voice=kokoro/bm_george, backend=local` in the control test), and
|
||||
`speech.voice` is the TUI's **speak-aloud** voice — its enum is Kokoro's list,
|
||||
and it never reaches an HTTP request. So a harness call arrives voice-less by
|
||||
default, which is exactly why the endpoint grew `DEFAULT_VOICE`: the declared
|
||||
pack answers a voice-less request and the journal marks that line `(default)`,
|
||||
while an **unknown** name still 400s with the list that exists.
|
||||
- **`speech.voice` must stay a Kokoro name.** Setting it to a pack name does not
|
||||
select a pack; it hands the vocalizer a value outside its enum. (The first
|
||||
draft of this section said the opposite and was reverted the same day, after a
|
||||
fresh session's call came back `unknown voice None`.)
|
||||
- **There is no silent fallback.** An explicitly configured `modelRoles.speech`
|
||||
yields exactly one candidate, so a dead 7772 fails the speech device with
|
||||
*"No available speech model matches the speech role"* instead of quietly
|
||||
degrading to Kokoro. That is the price of `auth: none` (which makes a dead
|
||||
endpoint read as available) and it is the wanted shape: a narration job should
|
||||
fail loudly rather than ship a voice Frank never asked for.
|
||||
- **A session older than the models.yml row cannot see it.** `models.yml` loads
|
||||
at process start while `config.yml` is read live, so a long-lived session
|
||||
carrying the flipped role fails exactly like a dead endpoint while a fresh
|
||||
process succeeds in the same minute. Measured 2026-10-18: the open session
|
||||
failed twice (provider/id form, then the bare id), `local/kokoro` succeeded in
|
||||
that same session, and a fresh `omp -p --tools tts` returned `Saved 180524
|
||||
bytes (model=frank-tts (dankclaw)/frank-tts, voice=narrator1, codec=wav)`
|
||||
while the façade logged the matching `speech voice=narrator1 format=wav
|
||||
chars=51 -> 180524 bytes` from the podman bridge. Rule the stale registry out
|
||||
before blaming the service.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@@ -148,5 +199,6 @@ gated `wip: down` until the deploy and probes `127.0.0.1:7772`.
|
||||
| `400 unknown voice 'x'; available voices: …` | the name is not a pack: case, spelling, missing `reference.wav`, or an empty/root-owned `/home/dank/voices` |
|
||||
| `(none: no voice pack with a reference.wav in /voices)` | the mount, not the voices: `podman inspect frank-tts --format '{{json .Mounts}}'`, then the directory owner |
|
||||
| Endpoint answers, synthesis fails | the engine: `podman healthcheck run frank-tts-engine`, then `systemctl --user status frank-tts-engine.service` |
|
||||
| First request of the day takes ~5.5 s | the warm pass did not run — cold-voice law in [[Frank TTS Engine]] |
|
||||
| First request of the day is ~0.6 s slower than the rest — or every name returns 400 | the warm pass did not run; it carries `-`, so it cannot fail the unit. `journalctl --user -u frank-tts-engine` for the warmer's lines, then `podman exec frank-tts-engine /app/warm-voices.sh` — the registry is memory-only (law 1 of [[Frank TTS Engine]]) |
|
||||
| The unit never appears after a deploy | the generator dropped it: `frank-tts-engine.network` is missing from `~/.config/containers/systemd/` |
|
||||
| A consumer rejects `narrator1` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] |
|
||||
|
||||
@@ -59,7 +59,7 @@ podman exec open-notebook curl -sS http://frank-tts:7772/health
|
||||
podman exec open-notebook curl -sS http://host.containers.internal:7771/v1/models
|
||||
```
|
||||
|
||||
**The blocker to state plainly:** frank-tts is built, named and tested but **not deployed** — nothing answers on 7772 (`ss -ltn`, Dankclaw, 2026-10-18; [[Local TTS and Voice Cloning]] records the same census). open-notebook deployed today writes scripts and fails at the first voice line. frank-tts ships first, or podcast generation waits.
|
||||
**The blocker moved; it did not vanish.** frank-tts and frank-tts-engine are **deployed** (2026-10-18) and answer on 7772, so the voice leg works end to end — `podman exec open-notebook curl -sS http://frank-tts:7772/health` is the check to run the day the app exists. What still blocks a render is open-notebook itself: not installed on this host yet. When it is, the two provider rows above are the only wiring, and the seeded speaker profiles carrying OpenAI voice names are the first trap.
|
||||
|
||||
**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). llama.cpp's OpenAI-compatible endpoint is a known-good target for that; frank-dragoon's halogen runtime has **unverified** structured-output behaviour, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout.
|
||||
|
||||
|
||||
@@ -43,14 +43,16 @@ As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unve
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
|
||||
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver |
|
||||
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090; production shape 3×262,144 per `scripts/frank-zealot.env`) | Dankclaw |
|
||||
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090; production shape 2×262,144 exact f16 KV per `scripts/frank-zealot.env`) | Dankclaw |
|
||||
|
||||
The URLs in the first column are the DB's recorded values and predate the 2026-08-22
|
||||
bridge move; on this network they are wrong — see the reachability gotcha below, which
|
||||
is the corrected law. Note also that the 2026-10-18 **bench window** on `:7771`
|
||||
closed the same day (production unit stopped 03:38 per the repo Benchmarking
|
||||
Policy while candidate arms held the port): the candidate — Ternary-Bonsai-2-**9B**
|
||||
PQ2_0 with the MTP head, 3 × 262,144 — was **deployed as production**, so the
|
||||
PQ2_0 with the MTP head, deployed that afternoon at 3 × 262,144 q8_0 KV and
|
||||
resliced the same evening to **2 × 262,144 over exact f16 KV** — was **deployed
|
||||
as production**, so the
|
||||
picker row below was re-keyed to it (see the next paragraph).
|
||||
|
||||
The `model` table carries exactly two decoration rows, `frank-zealot` and
|
||||
@@ -292,6 +294,42 @@ carries base64 image data or a URL.
|
||||
Failure modes: connection refused ⇒ OpenWebUI or ComfyUI is down; timeouts
|
||||
on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
|
||||
|
||||
## Text-to-speech (frank-tts)
|
||||
|
||||
TTS is the fleet's one OpenAI-compatible speech endpoint —
|
||||
`http://host.containers.internal:7772/v1` (`audio.tts.openai.api_base_url`
|
||||
in the config DB, 2026-10-18), model `frank-tts`, voice `narrator1`. That
|
||||
port is the rootless quadlet `frank-tts` (this repo:
|
||||
`dankclaw/voiceservice/quadlet/frank-tts.container`,
|
||||
`scripts/frank-tts.env`), and its shim resolves `voice=<name>` against
|
||||
`/home/dank/voices/<name>/reference.wav` — a name without a reference is a
|
||||
400 listing the voices that exist, never a silent fallback.
|
||||
|
||||
**The 29-voice `af_* … bm_fable` list is NOT frank-tts.** That enum belongs
|
||||
to **Kokoro.js running in the browser**: the deployed front end (0.11.4)
|
||||
ships an in-page Kokoro implementation (chunk carrying a frozen voice table
|
||||
`{af_heart:{name:"Heart",…}}` and `_validate_voice`, model
|
||||
`onnx-community/Kokoro-82M-v1.0-ONNX`, plus a personal *Kokoro.js dtype*
|
||||
setting). Its engine select takes `""` (Default) or `browser-kokoro`; when
|
||||
`browser-kokoro` is chosen the page synthesizes locally and rejects any
|
||||
voice outside its fixed pack — so `narrator1` fails with
|
||||
`Voice: "narrator1" not found. Should be one of: af_heart, …` **without a
|
||||
single request reaching the server**. Personal audio settings are per
|
||||
browser: no `user.settings` row and no container env carries an audio
|
||||
override, yet the toast still appears ⇒ the choice lives in that browser's
|
||||
own storage. Fix in the UI: Settings → Audio → TTS Engine → **Default**,
|
||||
which hands synthesis to the admin config (frank-tts); picking `af_heart`
|
||||
instead keeps Kokoro, and Kokoro cannot clone — fixed pack only.
|
||||
|
||||
Verified 2026-10-18 from inside the container with the exact OpenWebUI body
|
||||
(`{"model":"frank-tts","input":…,"voice":"narrator1",
|
||||
"response_format":"mp3","speed":1.0}`) → **200, `audio/mpeg`, 7,629 B**,
|
||||
while `podman logs open-webui` recorded **zero** `audio/speech` requests for
|
||||
the failing click — that contrast is the instrument: a server-path failure
|
||||
leaves a log line and a frank-tts error, a `browser-kokoro` failure leaves
|
||||
neither.
|
||||
|
||||
|
||||
## Retired instances
|
||||
|
||||
- The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
|
||||
@@ -349,6 +387,7 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Frank TTS]] — the speech endpoint behind *Text-to-speech (frank-tts)*
|
||||
- [[Hermes]] — the agent OpenWebUI can talk to
|
||||
- [[SearXNG]] — the search engine on this host
|
||||
- [LLM knowledge base](../LLM/index.md) — the vLLM and llama.cpp endpoints
|
||||
|
||||
@@ -26,7 +26,7 @@ timestamp: 2026-10-03
|
||||
> the PrismML 27B era of [[frank-zealot-llama-prism-bonsai2-pq20]] (build 10754,
|
||||
> 2 × 200,704 q8_0 KV, MTP off) — retired 2026-10-17 → 10-18 as a rollback
|
||||
> target; the active row today is build **10777** on
|
||||
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` at 3 × 262,144 q8_0 KV, MTP off by
|
||||
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` at 2 × 262,144 exact f16 KV, MTP off by
|
||||
> measurement (AGENTS.md Runbook Index). The Q5_K_M
|
||||
> payload is still on disk (`stat -c %s` 2026-10-18 → 6876124704).
|
||||
|
||||
|
||||
@@ -20,13 +20,16 @@ timestamp: 2026-10-01
|
||||
> [[frank-zealot-llama-mimo-q5km]], the same runtime and shape one quant rung
|
||||
> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
|
||||
> target (on disk 2026-10-18, `stat -c %s` → 7793710624). :7771 today runs
|
||||
> [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
|
||||
> `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704, q8_0 K + q8_0 V, MTP off).
|
||||
> [[frank-zealot-llama-prism-bonsai9-mtp]] (llama.cpp PrismML fork build
|
||||
> **10777**, `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144,
|
||||
> exact f16 K + f16 V, MTP off by measurement).
|
||||
>
|
||||
> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF
|
||||
> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp
|
||||
> exposes per-tensor. The per-tensor lever stayed; this pairing did not — the
|
||||
> live endpoint bills q8_0 K + q8_0 V (`scripts/frank-zealot.env:55-56`).
|
||||
> live endpoint bills **exact f16 K + f16 V** — 32,768 B/token = 16,384 MiB
|
||||
> over the 524,288-position pool of two 262,144 windows
|
||||
> (`scripts/frank-zealot.env:57-58,80-81`).
|
||||
|
||||
## Identity
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 3 × 262144, q8_0 KV (ACTIVE)"
|
||||
description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: THREE conversations of 262,144 tokens (every slot at the model's trained window) over q8_0 K + q8_0 V, vision through the publisher's Q8_0 Qwen3-VL projector, and the five-arm boot census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head."
|
||||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 2 × 262144, exact f16 KV (ACTIVE)"
|
||||
description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: TWO conversations of 262,144 tokens (every slot at the model's trained window) over an UNQUANTIZED f16 K + V cache, vision through the publisher's Q8_0 Qwen3-VL projector, and the boot-arm census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head."
|
||||
tags:
|
||||
- llm
|
||||
- frank-zealot
|
||||
@@ -14,15 +14,22 @@ tags:
|
||||
timestamp: 2026-10-18
|
||||
---
|
||||
|
||||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 3 × 262,144, q8_0 KV
|
||||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 2 × 262,144, exact f16 KV
|
||||
|
||||
Live since 2026-10-18, replacing the
|
||||
[[frank-zealot-llama-prism-bonsai2-pq20]] era (Ternary-Bonsai-2-**27B** PQ2_0 at
|
||||
2 × 200,704). The trade in one sentence: the 9B bills **half** the KV per token
|
||||
of the 27B, and that is exactly what funds **three** conversations at the
|
||||
model's **native 262,144** window where the 27B afforded only two shortened
|
||||
ones (Frank's ask: *"if you can possibly fit a third context, do it"*). Every
|
||||
axis improved — decode, prefill, three-stream aggregate, and free VRAM.
|
||||
2 × 200,704). The original trade: the 9B bills **half** the KV per token of the
|
||||
27B, and that is what funded **three** conversations at the model's **native
|
||||
262,144** window where the 27B afforded only two shortened ones (Frank's ask:
|
||||
*"if you can possibly fit a third context, do it"*).
|
||||
|
||||
The shape moved again the same evening (Frank: *"shift frank-zealot to 2
|
||||
parallels of unquantized kv"*). The KV cache went **exact** — f16 K + f16 V, no
|
||||
quantization error anywhere in the attention state — and the **slot count paid
|
||||
for it: 3 → 2**. The per-slot contract never moved; every conversation is still
|
||||
262,144, still the trained window. What the dtype bought is fidelity, and what
|
||||
it costs is one concurrent conversation plus the bytes attention reads per
|
||||
generated token.
|
||||
|
||||
## Payload identity
|
||||
|
||||
@@ -46,9 +53,12 @@ axis improved — decode, prefill, three-stream aggregate, and free VRAM.
|
||||
refuses the custom type at load; a loader that ignored the Hadamard metadata
|
||||
would dequant **wrong**, silently. Same lock class as the ROCmFPX era.
|
||||
2. **The build floor.** The fork's card gates **MTP beside a quantized V cache**
|
||||
on `prism-b10776-d00e252` or newer, so the prebuilt moved **10754 → 10777**
|
||||
for this swap. The 10754 tree stays on disk (`~/GitPackages/llama-prism/`) as
|
||||
the 27B rollback runtime. `tests/frank_zealot_unit.bats` pins both paths.
|
||||
on `prism-b10776-d00e252` or newer, which is why the prebuilt moved
|
||||
**10754 → 10777** for the swap. Exact f16 KV makes that gate moot — there is
|
||||
no quantized V to pair with the drafter — but tumbler 1 pins the runtime on
|
||||
its own: stock llama.cpp cannot load this file at any flag value. The 10754
|
||||
tree stays on disk (`~/GitPackages/llama-prism/`) as the 27B rollback
|
||||
runtime. `tests/frank_zealot_unit.bats` pins both paths.
|
||||
|
||||
## Header census (read from the GGUF, not the card)
|
||||
|
||||
@@ -71,41 +81,66 @@ axis improved — decode, prefill, three-stream aggregate, and free VRAM.
|
||||
| `general.sampling.*` | temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / repeat 1.0 | the card recipe; llama.cpp reads it itself |
|
||||
|
||||
**KV accounting.** Only the 8 full-attention layers scale with context:
|
||||
8 × 4 KV heads × 256 dim = **8,192 elements/token per cache**, so
|
||||
**q8_0 K + q8_0 V bills 17,408 B/token** — the 786,432-position pool is
|
||||
**13,056.00 MiB**, ledger-exact. That is exactly HALF the 27B's 34,816 B/token.
|
||||
**f16 KV would bill 26,112 MiB against a 24,123 MiB card** and dies at
|
||||
`cudaMalloc`. The 24 DeltaNet layers bill a fixed recurrent state per SLOT
|
||||
instead: the PRODUCTION ledger prints `RS buffer size = 150.75 MiB (3 cells,
|
||||
32 layers, 3 seqs)` ≈ 50 MiB a cell, while the SPEC-ARMED arms printed
|
||||
452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples
|
||||
even the recurrent state. Pool arithmetic alone under-prices the shape.
|
||||
8 × 4 KV heads × 256 dim = **8,192 elements/token per cache** (the engine prints
|
||||
`n_embd_k_gqa = n_embd_v_gqa = 1024`). **The deployed pair is exact: f16 K +
|
||||
f16 V bills 32,768 B/token** (2 B/element × 8,192 elements × 2 caches), so the
|
||||
524,288-position pool is **16,384 MiB**. The q8_0 pair this endpoint shipped
|
||||
first billed **17,408 B/token** — 1.0625 B/element, 32 codes plus one 2-byte
|
||||
scale per 32-block — the **13,056.00 MiB** its ledger printed at the 786,432
|
||||
pool, exactly HALF the 27B's 34,816 B/token.
|
||||
|
||||
**Corrected arithmetic, and it matters.** The note that lived here and in
|
||||
`frank-zealot.env` — "f16 would bill 26,112 MiB" — doubled the q8_0 figure
|
||||
*including* its scale overhead, overstating exact KV by 6.25 %. Exact f16 at the
|
||||
786,432 pool is **24,576 MiB** against the **24,123 MiB** the driver leaves
|
||||
addressable (`nvidia-smi`'s total column reads 24,576; used + free reads 24,123
|
||||
— the difference is driver reservation). The wall stands in both readings; the
|
||||
honest rate is what makes the fit calculable instead of superstitious. **Halving
|
||||
the POOL is what funds exact KV** — the window is already at the native ceiling
|
||||
and there is no YaRN in this path.
|
||||
|
||||
The 24 DeltaNet layers bill a fixed recurrent state per SLOT instead: the 3-slot
|
||||
ledger printed `RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)` ≈
|
||||
**50.25 MiB a cell**, so two slots bill 100.50. The SPEC-ARMED arms printed
|
||||
452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples even
|
||||
the recurrent state. Pool arithmetic alone under-prices the shape.
|
||||
|
||||
## The shape (`scripts/frank-zealot.env`)
|
||||
|
||||
`--ctx-size 786432 --parallel 3` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`,
|
||||
`--cache-type-k q8_0 --cache-type-v q8_0`, `--batch-size 4096`,
|
||||
`--ctx-size 524288 --parallel 2` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`,
|
||||
`--cache-type-k f16 --cache-type-v f16`, `--batch-size 4096`,
|
||||
`--ubatch-size 2048`, `--n-gpu-layers 99`, `--jinja`, `--reasoning auto`,
|
||||
`--reasoning-format deepseek`, `--metrics`, `--mmproj …`, `--log-verbosity 4`,
|
||||
`--presence-penalty 1.0`,
|
||||
**`SPEC_TYPE=none`** (which strips `--spec-type` AND every `--spec-draft-*`).
|
||||
|
||||
Per-slot window is the **contract** the harness quotes: `models.yml` carries
|
||||
`contextWindow: 262144`. llama.cpp refuses a prompt longer than the SLOT with a
|
||||
400 naming the limit — it never truncates — and since this swap the slot window
|
||||
equals the native window, so no YaRN and no cap is in play.
|
||||
`contextWindow: 262144`, and it did **not** move with the slot count because
|
||||
llama.cpp divides `--ctx-size` across slots. It refuses a prompt longer than the
|
||||
SLOT with a 400 naming the limit — never a truncation — and since this swap the
|
||||
slot window equals the native window, so no YaRN and no cap is in play.
|
||||
`tests/frank_zealot_unit.bats` now gates the arithmetic: the shipped dtype ×
|
||||
pool plus the measured non-KV ledger terms must fit 24,123 MiB with ≥1 GiB of
|
||||
slack, and the gate is proven to REFUSE the impossible arm (exact KV at the old
|
||||
786,432 pool = 24,576 MiB of cache alone).
|
||||
|
||||
## The MTP verdict: the head is in the file and it is still OFF
|
||||
|
||||
The drafter is **good**: acceptance 0.798 / 0.617 per position, mean accepted
|
||||
length 2.41, and the card recommends depth 2. It loses **at this shape**
|
||||
because the draft context bills its own **f16** KV over the WHOLE pool —
|
||||
`blk.32` carries real attention (4 KV heads × 256 × 2 caches × 2 B =
|
||||
**4,096 B/position**) × 786,432 positions = **3,072.00 MiB**, 4,830 MiB with its
|
||||
compute buffers — and `--ubatch-size 2048` refuses to give that up.
|
||||
length 2.41, and the card recommends depth 2. It is OFF because the draft
|
||||
context bills its own **f16** KV over the WHOLE pool — `blk.32` carries real
|
||||
attention (4 KV heads × 256 × 2 caches × 2 B = **4,096 B/position** at the
|
||||
default dtype). The draft dtype is itself a knob — `--spec-draft-type-k` /
|
||||
`--spec-draft-type-v`, both default f16, verified against this build's `--help`
|
||||
(so the "no flag exists" claim that lived here is FALSE) — and quantizing the
|
||||
draft to q8_0 would bill 2,176 B/position, **1,088 MiB** at the shipped pool.
|
||||
**No measured arm runs a drafter beside exact KV**; every row below is f16 draft.
|
||||
|
||||
Booted arms, same card, same traffic, 2026-10-18 (`/completion` `timings`,
|
||||
greedy 512 tokens; never the first request after a boot):
|
||||
The census below is the **q8_0 era**: every row is q8_0 K + q8_0 V at the
|
||||
786,432 pool, where the draft bill was 3,072.00 MiB and 4,830 MiB with its
|
||||
compute buffers. Booted arms, same card, same traffic, 2026-10-18
|
||||
(`/completion` `timings`, greedy 512 tokens; never the first request after a
|
||||
boot):
|
||||
|
||||
| slots | spec | ubatch | used / free | decode | 3-stream | prefill @106k |
|
||||
|---|---|---|---|---|---|---|
|
||||
@@ -117,15 +152,26 @@ greedy 512 tokens; never the first request after a boot):
|
||||
|
||||
**MTP is worth +28 % single-stream decode whenever the card has room** (rows
|
||||
4–5 prove the drafter itself is a win; row 2 proves the squeeze is what halves
|
||||
it). The arm that runs MTP at three windows is `SPEC_TYPE=draft-mtp` +
|
||||
`UBATCH_SIZE=1024` (row 3) and pays **prefill −17 %** and **2,725 MiB of
|
||||
it). The arm that ran it at three windows was `SPEC_TYPE=draft-mtp` +
|
||||
`UBATCH_SIZE=1024` (row 3), paying **prefill −17 %** and **2,725 MiB of
|
||||
headroom** on a card whose mmproj worst-case lands AFTER the engine's own fit
|
||||
check. This endpoint is the fleet's **compaction hub** — prefill-bound ~170k
|
||||
inputs plus 2–3 concurrent subagents — so the shipped arm takes the +17 %
|
||||
prefill and +12 % three-stream aggregate over single-stream speed, and keeps
|
||||
6.2 GiB free. Frank confirmed this arm on 2026-10-18.
|
||||
check.
|
||||
|
||||
## Boot ledger (production boot 2026-10-18 13:19, under the unit, verbosity 4)
|
||||
**Under exact KV an f16-draft arm is priced out of this card.** The draft
|
||||
context is built at the TARGET's pool, so at 524,288 positions it bills
|
||||
4,096 × 524,288 = **2,048.00 MiB** plus its own buffers, on a trunk that already
|
||||
projects **~21,200 MiB** of the 24,123 MiB addressable. The doors back to
|
||||
speculation are **one window per conversation** (row 5's class of shape), **KV
|
||||
back to q8_0** with `UBATCH_SIZE=1024`, or the **unmeasured**
|
||||
`--spec-draft-type-k q8_0 --spec-draft-type-v q8_0` arm — 1,088 MiB of draft,
|
||||
legal on b10777, never booted beside exact KV. Frank chose exact KV over the
|
||||
drafter on 2026-10-18, and the endpoint's actual job is what made that
|
||||
affordable: the fleet's **compaction hub** is prefill-bound on ~170k inputs,
|
||||
prefill is weight-bandwidth bound and does not re-read the cache per generated
|
||||
token — so the dtype is paid in decode and concurrency, not in the work this
|
||||
service exists to do.
|
||||
|
||||
## Boot ledger — the q8_0 3-slot production boot (2026-10-18 13:19, under the unit, verbosity 4)
|
||||
|
||||
```
|
||||
CUDA0 model buffer size = 2036.71 MiB # + 257.66 MiB CPU_Mapped (mmap)
|
||||
@@ -141,7 +187,30 @@ No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class
|
||||
numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`,
|
||||
never the projection**.
|
||||
|
||||
## Measured (production boot, `/completion` `timings`, greedy, `ignore_eos`)
|
||||
**The exact-KV boot, measured** (2026-10-18 evening, PR #21, under the unit,
|
||||
verbosity 4 — the live shape):
|
||||
|
||||
```
|
||||
CUDA0 model buffer size = 2036.71 MiB # + 257.66 MiB CPU_Mapped (mmap)
|
||||
CUDA0 KV buffer size = 16384.00 MiB # f16 K + f16 V, 524,288 positions
|
||||
CUDA0 RS buffer size = 100.50 MiB # 2 cells, 32 layers, 2 seqs
|
||||
CUDA0 compute buffer = 1408.06 MiB # + 1088.07 MiB CUDA_Host
|
||||
mmproj worst-case (est.) = 868.32 MiB # 248.10 CUDA + 24.93 CPU, AFTER fit
|
||||
nvidia-smi settled: 21,176 MiB used / 2,949 MiB free of 24,123
|
||||
n_slots = 2, n_ctx_slot = 262144, kv_unified = false
|
||||
```
|
||||
|
||||
The pre-flight projection was right to the MiB, because the two terms it moved
|
||||
are exact arithmetic: **16,384.00 MiB of KV is precisely 32,768 B/token ×
|
||||
524,288 positions** — rate derivation and ledger agree with no slop, which is
|
||||
what buries the old 26,112 MiB figure for good — and the recurrent state is one
|
||||
50.25 MiB cell lighter (100.50 exactly). Settled card state landed at **21,176
|
||||
used / 2,949 free**, within 24 MiB of the 21,200 / 2,920 projection,
|
||||
`DropInPaths` empty, no clamp line, no `cudaMalloc` error. It is a desktop
|
||||
margin, though: the mmproj's 868 MiB worst case is still charged AFTER the
|
||||
engine's own fit check, so ~2 GiB is what an image storm has to live inside.
|
||||
|
||||
## Measured — the q8_0 3-slot boot (`/completion` `timings`, greedy, `ignore_eos`)
|
||||
|
||||
- **168.7 t/s** decode — three runs of 512 tokens: 168.4 / 168.7 / 168.7
|
||||
(the trial arm printed 167.2; same shape, same noise band).
|
||||
@@ -156,15 +225,52 @@ never the projection**.
|
||||
stops on its own: a 24-token countdown measured 72 t/s across three streams
|
||||
while the same shape with `ignore_eos` gave 300.
|
||||
|
||||
## Measured — the exact-KV boot (floors, not a clean A/B)
|
||||
|
||||
Instrument: `/v1/chat/completions` `timings` (this fork refuses the raw
|
||||
`/completion` path for greedy runs — see *Contract traps*), greedy,
|
||||
`ignore_eos`, 512 tokens, first request discarded.
|
||||
|
||||
|Arm|q8_0 3-slot record|f16 2-slot, live co-tenancy|
|
||||
|---|---|---|
|
||||
|single-stream decode|168.7|**134.1 – 141.5** over five back-to-back runs; 145.8 and 163.2 in a lighter minute|
|
||||
|concurrent aggregate|≈300 across 3 (102.8 each)|**256.6 / 262.4 / 265.1** across 2 (≈128–133 each)|
|
||||
|prefill at ~106k|2,877 at 106,418|**3,159.5 at 106,282**|
|
||||
|decode at 106k depth|never measured|**71.5**|
|
||||
|
||||
**These are floors, not the shape's rates.** Someone else was driving the
|
||||
endpoint through the whole window — 56 new prompts on the other slot in three
|
||||
minutes, `nvidia-smi` at 96 % GPU-util — which is the exact condition the
|
||||
Benchmarking Policy forbids, and a contended deep arm even came back truncated
|
||||
(a 512 KB body accepted as `prompt_n = 4`). What the floors already support:
|
||||
**per-conversation concurrency improved** (two streams hold ~130 t/s each where
|
||||
three held 102.8), endpoint aggregate sits ~12 % under the 3-slot record,
|
||||
prefill beat the record (+9.8 %) while contended, and single-stream decode is
|
||||
between parity and −20 % against it. The 1.882× KV byte bill is real; its true
|
||||
size on this card is still unmeasured, and quoting 134 t/s or 163 t/s as "the"
|
||||
number would be a guess in either direction. Re-run the same arms with the box
|
||||
quiet before the figure is carried anywhere else.
|
||||
|
||||
## Contract traps
|
||||
|
||||
- **`--metrics` is mandatory.** Without it llama.cpp answers **501** on
|
||||
`/metrics`, so the Prometheus scrape reads DOWN while `/health` answers 200 —
|
||||
that is how this endpoint sat red across four payload generations
|
||||
|
||||
(`up{job="frank-zealot"}` oldest retained sample, 2026-10-11, is already 0).
|
||||
- **Bare `/completion` is a dead instrument on this fork build.** A greedy
|
||||
`ignore_eos` run answers **500 — "The model produced output that does not
|
||||
match the expected Content-only format"**, because the fork guards the raw
|
||||
path the moment reasoning markup leaks (older builds only logged `failed to
|
||||
parse reasoning content` and answered anyway). The honest per-request
|
||||
instrument on THIS build is `/v1/chat/completions`, whose non-stream response
|
||||
carries the same `timings` object (`prompt_n`, `prompt_per_second`,
|
||||
`predicted_n`, `predicted_per_second`). `AGENTS.md` still names the
|
||||
`/completion` `timings` object as this endpoint's instrument — on build 10777
|
||||
read it off the chat path.
|
||||
- **Q8_0 mmproj, not BF16.** The publisher's card: the BF16 tower "can crash
|
||||
llama-server when several image requests run at once; Q8_0 is not affected" —
|
||||
and `--parallel 3` is precisely that shape.
|
||||
and `--parallel 2` is still precisely that shape.
|
||||
- **Only the `image_url` content part reaches the model.** The legacy top-level
|
||||
`images` field bumps `usage.prompt_tokens` and the model still invents an
|
||||
answer. Probe with a solid color and expect the exact word.
|
||||
@@ -216,9 +322,11 @@ never the projection**.
|
||||
`frank-zealot` here — 262,144 is the widest window in the fleet and the only one
|
||||
that comfortably fits a worst-case compaction input (~170k tokens; the clamp is
|
||||
`min(maxTokens 16384, contextWindow − est − 4096)`, compaction fires at 85 % of
|
||||
the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). With `--parallel 3`
|
||||
a compaction request no longer serializes behind a live generation (the Gemma4
|
||||
era's `--parallel 1` law).
|
||||
the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). What the reslice to
|
||||
**two** slots does cost is here: a compaction request still runs beside ONE live
|
||||
generation, but with two conversations already busy a third waits — the Gemma4
|
||||
era's `--parallel 1` serialization law at half strength. The window itself is
|
||||
untouched, and the window is the term the compaction contract depends on.
|
||||
|
||||
## Deploy ledger (2026-10-18)
|
||||
|
||||
@@ -263,12 +371,24 @@ era's `--parallel 1` law).
|
||||
|
||||
## Fallback rungs
|
||||
|
||||
1. `UBATCH_SIZE=1024` — frees **1,306 MiB** measured on this shape (prefill
|
||||
−17 %).
|
||||
2. `CACHE_TYPE_K/V=q4_0` — frees ~6.5 GiB at three windows (the 27B era measured
|
||||
the q4_0 rung at 16,392 used / 7,733 free at 2 × 200,704).
|
||||
3. Cut to 2 slots — but that breaks the third concurrent subagent, so it is a
|
||||
last resort, not a knob.
|
||||
Cheapest first; every one is a single line of `scripts/frank-zealot.env`, and
|
||||
the KV figures are for the 524,288-position pool unless stated otherwise:
|
||||
|
||||
1. `UBATCH_SIZE=1024` — hands back **1,306 MiB** of compute buffers measured on
|
||||
this card (prefill −17 %).
|
||||
2. `CACHE_TYPE_V=q8_0` — exact K, quantized V: the pool goes 16,384 →
|
||||
**12,544 MiB**, freeing 3,840, and it is the halfway door back.
|
||||
`--flash-attn on` is what keeps quantized V legal.
|
||||
3. `CTX_SIZE=401408` — two 200,704 conversations: exact KV at
|
||||
**12,544.00 MiB**, same 3,840 freed, but `models.yml` `contextWindow` MUST
|
||||
follow the per-slot number or long turns start 400-ing.
|
||||
4. Revert to `q8_0` K + `q8_0` V at `CTX_SIZE=786432 PARALLEL=3` — the shape
|
||||
this one replaced: 13,056.00 MiB of cache and the THIRD conversation back.
|
||||
5. `CACHE_TYPE_K/V=q4_0` — the far rung: 10,240 B/token ⇒ **5,120 MiB** here
|
||||
(the 27B era measured q4_0 at 16,392 used / 7,733 free at 2 × 200,704).
|
||||
|
||||
Any retune has to keep the seam's fit gate green — it is the only instrument
|
||||
that can see a boot that would die at `cudaMalloc`.
|
||||
|
||||
## Rollback
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ timestamp: 2026-10-07
|
||||
[[frank-zealot-llama-prism-bonsai2-pq20]] (27B PQ2_0, 2 × 200,704 q8_0 KV, build
|
||||
10754), itself retired 2026-10-17 → 10-18 and now the rollback target: :7771
|
||||
today is llama.cpp PrismML build **10777** on
|
||||
`Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 3 × 262,144 q8_0 KV, MTP off by
|
||||
`Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144 exact f16 KV, MTP off by
|
||||
measurement (AGENTS.md Runbook Index active row; `scripts/frank-zealot.env`).
|
||||
Transient 2026-10-18: :7771 is inside a bench window — the unit is `failed`
|
||||
since 03:14 and a hand-booted process answers the port; restore with
|
||||
|
||||
@@ -22,7 +22,7 @@ timestamp: 2026-10-06
|
||||
> frank-small the same night). Superseded by
|
||||
> [[frank-zealot-llama-qwen35-9b-q6k-2x200704]]; the production shape on :7771
|
||||
> today is llama.cpp PrismML build **10777** on
|
||||
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 3 × 262,144 q8_0 KV, MTP off by
|
||||
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144 exact f16 KV, MTP off by
|
||||
> measurement (AGENTS.md Runbook Index active row); the 27B PQ2_0 / 2 × 200,704
|
||||
> shape of [[frank-zealot-llama-prism-bonsai2-pq20]] is that era's rollback
|
||||
> target. Transient 2026-10-18 bench window on :7771 — the unit is `failed`
|
||||
|
||||
@@ -20,8 +20,9 @@ timestamp: 2026-10-01
|
||||
> **RETIRED 2026-10-01, hours after cutover** (AGENTS.md Runbook Index). Replaced by
|
||||
> [[frank-zealot-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
|
||||
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express. :7771
|
||||
> today runs [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork
|
||||
> build 10754, `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off).
|
||||
> today runs [[frank-zealot-llama-prism-bonsai9-mtp]] (llama.cpp PrismML fork
|
||||
> build **10777**, `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144,
|
||||
> exact f16 K + f16 V, MTP off by measurement).
|
||||
> The venv (`/home/dank/GitPackages/sglang-cu128/`) is still on disk; the INT8
|
||||
> payload was staged to
|
||||
> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 and
|
||||
@@ -59,8 +60,8 @@ timestamp: 2026-10-01
|
||||
the two-way concurrency contract. **That compaction arithmetic is
|
||||
specific to a 262,144-token slot**: the shape that followed on 2026-10-02
|
||||
ran three slots of 180,224, which 183,501 tokens already exceeds — see *What
|
||||
three slots cost* in [[frank-zealot-llama-mimo-q5km]]; :7771 runs two 200,704
|
||||
slots today.
|
||||
three slots cost* in [[frank-zealot-llama-mimo-q5km]]; :7771 today runs two
|
||||
262,144-token conversations over one 524,288-position pool with exact f16 KV.
|
||||
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
|
||||
dim x 1 B), pool 9.31 GiB.
|
||||
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
|
||||
|
||||
@@ -26,9 +26,10 @@ MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
|
||||
200704, vision ON)** and
|
||||
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
|
||||
Ternary-Bonsai-2-9B at **PQ2_0** — 9B qwen35 hybrid (8 full-attention + 24
|
||||
Gated DeltaNet layers + the blk.32 MTP block), THREE 262,144-token
|
||||
conversations, MTP present in the file but OFF by measurement,
|
||||
q8_0 K + q8_0 V (f16 KV does not fit the RTX 3090 at this window), vision via
|
||||
Gated DeltaNet layers + the blk.32 MTP block), TWO slots over ONE
|
||||
524,288-position pool (2 × the 262,144 window), MTP present in the file but
|
||||
OFF by measurement, **exact f16 K + f16 V** — 32,768 B/token = **16,384 MiB**
|
||||
of KV (three 262,144 windows at exact KV do not fit the RTX 3090), vision via
|
||||
the publisher's own Q8_0 mmproj, on the RTX 3090 — and the custom
|
||||
tensor type 141 + `prism.hadamard.*` metadata mean no other loader can open the
|
||||
file)**.
|
||||
@@ -63,7 +64,7 @@ ONE 401408-position KV pool at ctx 200704 = 2 slots x ctx, MTP always-on, image
|
||||
G[llama.cpp PrismML fork CUDA build
|
||||
frank-zealot, port 7771]
|
||||
H[Ternary-Bonsai-2-9B PQ2_0 GGUF + Q8_0 mmproj
|
||||
3 x 262144 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision]
|
||||
2 slots over ONE 524288-position pool at ctx 262144, exact f16 K + f16 V, MTP off, mmproj vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -132,19 +133,26 @@ frank-zealot, port 7771]
|
||||
custom PQ2_0 tensor type + `prism.hadamard.*` transform, and since this swap
|
||||
also the minimum build that pairs MTP with a quantized V cache) serving
|
||||
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head IN the file** on
|
||||
the RTX 3090 — **THREE 262,144-token conversations**
|
||||
(`--ctx-size 786432 --parallel 3`, every slot at the window the model was
|
||||
trained at), **q8_0 K + q8_0 V** (17 KiB/token over the 8 full layers =
|
||||
13,056 MiB ledger-exact; the f16 pair needs 26,112 MiB and dies at
|
||||
`cudaMalloc`), settled **17,876 used / 6,249 free**, vision via the
|
||||
publisher's **Q8_0** mmproj (Q8_0 is the publisher's requirement for
|
||||
concurrent image requests, and this runs three slots), **MTP OFF by
|
||||
measurement** — the draft context bills its own f16 KV over the WHOLE pool
|
||||
(3,072 MiB) and at `UBATCH_SIZE=2048` that squeeze halves decode
|
||||
(167.2 → 109.2 t/s); `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the
|
||||
documented 213.5 t/s arm, declined for +17 % prefill and 6.2 GiB of headroom.
|
||||
Measured: 167.2 t/s single-stream decode, 2,877 t/s prefill at 106,418,
|
||||
280.2 t/s aggregate across three streams. `--metrics` rides (`METRICS=1`):
|
||||
the RTX 3090 — **TWO 262,144-token conversations**
|
||||
(`--ctx-size 524288 --parallel 2`, every slot at the window the model was
|
||||
trained at), **unquantized f16 K + f16 V** (32,768 B/token over the 8 full
|
||||
layers = **16,384 MiB** exact; the POOL halved because that rate over the
|
||||
OLD 786,432 pool bills 24,576 MiB against the 24,123 the driver leaves
|
||||
addressable, and the old "26,112 MiB" note was wrong — it doubled the
|
||||
q8_0 figure (17,408 B/token = 13,056 MiB) *including* its 1/32 scale
|
||||
overhead, so f16/q8_0 is **1.882×**, not 2×), the q8_0-era boot settled
|
||||
**17,876 used / 6,249 free**, vision via the publisher's **Q8_0** mmproj
|
||||
(Q8_0 is the publisher's requirement for concurrent image requests, and this
|
||||
runs two slots), **MTP OFF by measurement** — the draft context bills its own
|
||||
f16 KV over the WHOLE pool (4,096 B/position × 524,288 = 2,048 MiB) and at
|
||||
`UBATCH_SIZE=2048` that squeeze halves decode (167.2 → 109.2 t/s);
|
||||
`SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the documented 213.5 t/s arm,
|
||||
declined for +17 % prefill and 6.2 GiB of headroom — and exact KV leaves an
|
||||
f16-draft arm no room at all (a quantized draft, `--spec-draft-type-* q8_0`,
|
||||
is the unmeasured door).
|
||||
Measured at the q8_0-era shape: 167.2 t/s single-stream decode, 2,877 t/s
|
||||
prefill at 106,418, 280.2 t/s aggregate across three streams. `--metrics`
|
||||
rides (`METRICS=1`):
|
||||
llama.cpp answers 501 on `/metrics` without it, which is why Prometheus
|
||||
`up{job="frank-zealot"}` read 0 for its whole retained history until
|
||||
2026-10-18. `contextWindow: 262144` is the **widest in the fleet**, so this
|
||||
|
||||
@@ -328,9 +328,11 @@ a live stream mid-sentence.
|
||||
**Chosen: arm A, 1.7 B talker, resident `tts-server` in a `frank-tts-engine`
|
||||
container, with `TTS_BACKEND=llama-tts-cli` kept in the repo as the rollback.**
|
||||
Three consequences are configuration, not opinion: the unit **boot-warms every
|
||||
voice** (5.5 s cold first byte otherwise), it **does not pin a cpuset** casually
|
||||
(threads are `hardware_concurrency()/2`, no flag), and every pack registers with
|
||||
the **WAV** because the engine rejects a lone embedding.
|
||||
voice** — the registry is memory-only, so an unwarm'd name is a 400 and the first
|
||||
synthesis otherwise costs about 0.6 s more than the ones after it, as corrected
|
||||
above —, it **does not pin a cpuset** casually (threads are
|
||||
`hardware_concurrency()/2`, no flag), and every pack registers with the **WAV**
|
||||
because the engine rejects a lone embedding.
|
||||
|
||||
Two follow-ups stay open on purpose, and neither blocks the deploy:
|
||||
|
||||
|
||||
@@ -80,8 +80,8 @@ macOS mini) as quadlets inside its podman machine — not on Dankserver.
|
||||
- [[Seerr]] — unified *arr request manager (guard-chained; fronts FrankBook-Pro's stack)
|
||||
- [[Marreta]] — content-cleaning proxy (guard-chained; shares SearXNG's Gluetun egress)
|
||||
- [[Open Notebook]] — podcast generator (scripts by frank-zealot, voices by frank-tts); repo-owned config, not deployed
|
||||
- [[Frank TTS Engine]] — the arm A TTS engine (`qwentts.cpp` resident `tts-server`, 1.7 B talker) behind frank-tts on a private bridge; repo-owned config, not deployed
|
||||
- [[Frank TTS]] — the 7772 cloned-voice endpoint consumers address: voice naming, pack layout, why the upstream is a container name; repo-owned config, not deployed
|
||||
- [[Frank TTS Engine]] — the arm A TTS engine (`qwentts.cpp` resident `tts-server`, 1.7 B talker) behind frank-tts on a private bridge; deployed on Dankclaw, CPU-only, publishes no port by design
|
||||
- [[Frank TTS]] — the 7772 cloned-voice endpoint consumers address: voice naming, pack layout, why the upstream is a container name; deployed, and it is the harness `speech` role
|
||||
- [[Voice Packs]] — how a frank-tts voice is built: reference.wav + the pinned engine's speaker artifacts, cut past the 10-second source-material gate
|
||||
|
||||
## Infrastructure
|
||||
@@ -409,10 +409,15 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
loader for the custom PQ2_0 tensor type / `prism.hadamard.*`, and the floor
|
||||
for MTP beside a quantized V cache) serving
|
||||
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head** on the RTX 3090
|
||||
— THREE 262,144-token conversations, q8_0 K + q8_0 V (f16 OOMs at this pool:
|
||||
26,112 vs 13,056 MiB over 8 full-attention layers), MTP **off by
|
||||
measurement** (the draft context bills 3,072 MiB of f16 KV over the whole
|
||||
pool), vision through the Q8_0 mmproj; boot 17,876 used / 6,249 free
|
||||
— TWO 262,144-token conversations (`--ctx-size 524288 --parallel 2`), exact
|
||||
f16 K + f16 V = **16,384 MiB** over 8 full-attention layers (32,768 B/token;
|
||||
the per-slot window is UNCHANGED at 262,144 — what halved is the POOL,
|
||||
because exact f16 over the OLD 786,432 pool bills **24,576 MiB** against the
|
||||
**24,123** the driver leaves addressable; the old "26,112 vs 13,056 MiB" note
|
||||
doubled the q8_0 figure with its 1/32 scale overhead included, so f16/q8_0 is
|
||||
1.882×, not 2×), MTP **off by measurement** (the draft context bills 2,048
|
||||
MiB of f16 KV over the whole pool, and exact KV leaves no room for one),
|
||||
vision through the Q8_0 mmproj; boot 17,876 used / 6,249 free (the q8_0 era)
|
||||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — RETIRED 2026-10-17 → 10-18
|
||||
(rollback target, payload + build 10754 on disk): Ternary-Bonsai-2-**27B**
|
||||
PQ2_0 at TWO 200,704-token conversations, q8_0 K + q8_0 V, MTP **impossible**
|
||||
|
||||
@@ -1,5 +1,32 @@
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-10-18
|
||||
- **frank-zealot resliced to two windows of exact KV (Frank's call, PR #21).**
|
||||
`--ctx-size 786432 --parallel 3` at q8_0 K+V → **`524288 / 2` at f16 K+V**, so
|
||||
both remaining conversations keep the native 262,144 window and the
|
||||
compaction hub reads exact K and V. Halving the pool is what funds it: the
|
||||
f16 pair is 32,768 B/token over the 8 full-attention layers = 16,384.00 MiB,
|
||||
while the old 786,432-position pool would have needed ~24,576 MiB of KV alone
|
||||
on a card that leaves 24,576 addressable. Measured boot: KV **16,384.00**,
|
||||
RS **100.50** (2 cells), compute 1,408.06 + 1,088.07, settled `nvidia-smi`
|
||||
**21,176 used / 2,949 free**, `n_ctx_slot = 262144`, `DropInPaths` empty. The
|
||||
doc's old "f16 needs 26,112 MiB and cannot boot" figure was the 27B's
|
||||
34,816 B/token applied to the 9B — corrected in the runbook.
|
||||
- **Rates are floors, and that is recorded as such.** The endpoint had another
|
||||
consumer for the whole measurement window (56 prompts on the other slot in
|
||||
three minutes, 96 % GPU-util), so single-stream decode came back 134.1–141.5
|
||||
(145.8/163.2 in a lighter minute) against the 3-slot record of 168.7,
|
||||
two-stream aggregate 256.6–265.1 against ≈300 across three, and prefill
|
||||
**3,159.5 at 106,282** beat the record's 2,877 while contended. Per-stream
|
||||
concurrency improved (≈130 each vs 102.8); a clean re-measure needs the box
|
||||
quiet.
|
||||
- **Instrument correction (fleet law).** On the PrismML fork build 10777 the raw
|
||||
`/completion` path answers **500 "The model produced output that does not
|
||||
match the expected Content-only format"** for a greedy `ignore_eos` run, so
|
||||
the honest per-request `timings` for this endpoint must be read off
|
||||
`/v1/chat/completions`. `AGENTS.md` said `/completion`; it now says the chat
|
||||
path.
|
||||
|
||||
## 2026-10-13
|
||||
- **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17
|
||||
power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`
|
||||
@@ -2682,3 +2709,142 @@ every restart since the cutover.
|
||||
probe for the voice pair (neither container exposes `/metrics`; the blackbox
|
||||
question is still open), and the podcast render — open-notebook is on `main` but
|
||||
not installed.
|
||||
|
||||
- Sweep afterwards: the retired ladder still lived in six places the first pass
|
||||
missed — both quadlet files, `scripts/frank-tts-engine.env`,
|
||||
`scripts/frank-tts.env`, the [[Frank TTS]] troubleshooting row, the research
|
||||
brief's consequences paragraph, and one test comment. All re-keyed to the
|
||||
measured claim: about **0.6 s**, and the 400-for-every-name reason the pages
|
||||
should have led with. The trial entry above is left as written — a dated worklog
|
||||
entry is a record — and this line is the correction that points at it.
|
||||
|
||||
## 2026-10-18 — 7772 is live and the harness speaks through it
|
||||
|
||||
- Merged **#15** (`0315a62`, the warmer's argument-cap fix) and **#16**
|
||||
(`d2ce669`, the comment sweep), then deployed both voice services in order:
|
||||
`--service frank-tts-engine` (image rebuilt from the fixed warmer, unit
|
||||
`active`, zero drop-ins, warm pass in the unit's own start — `narrator1
|
||||
cold=1542ms warm=1709ms`, `narrator2 1329/1405`, `narrator3 1451/1394`, `3
|
||||
voice(s) registered and warm`) and `--service frank-tts`. The Playwright gate
|
||||
ran at the end of each deploy: 33/33 twice.
|
||||
- Deployed-state proof from a LAN client: `/health` → `omni-http` with all three
|
||||
voices; the same sentence in `narrator1` and `narrator2` came back with
|
||||
**different sha256** and different RMS (4.24 s, 24 kHz mono PCM16), so the name
|
||||
actually selects a voice. Engine-side ledger for the harness request:
|
||||
`TTFA 67.5 ms`, `Total 2158.8 ms`, `audio 3.76 s, RTF 0.574`.
|
||||
- **The harness `speech` role moved off Kokoro**: `config.yml`
|
||||
`modelRoles.speech: frank-tts (dankclaw)/frank-tts`. No voice key moves with it.
|
||||
The bundle is unambiguous: the speech device forwards the call's `voice_id` and
|
||||
nothing else, `tts.localVoice` belongs to the local backend (it is what answered
|
||||
in the control test), and `speech.voice` is the TUI's speak-aloud enum.
|
||||
- Two harness laws learned by breaking it first (both now on [[Frank TTS]]): an
|
||||
explicit speech role has **no fallback candidate**, so a dead endpoint errors
|
||||
instead of degrading; and `models.yml` loads at process start while
|
||||
`config.yml` is read live, so a session older than the `frank-tts` row fails
|
||||
exactly like a dead endpoint. Fresh-process proof in the same minute:
|
||||
`Saved 180524 bytes (model=frank-tts (dankclaw)/frank-tts, voice=narrator1)`
|
||||
with the matching façade log line `speech voice=narrator1 format=wav chars=51
|
||||
-> 180524 bytes`. This window's own session is that stale case and knows it.
|
||||
- Status sweep, because six pages still said the endpoint did not exist: both
|
||||
vault descriptions, the two `index.md` bullets, the Open Notebook blocker (the
|
||||
voice leg works; open-notebook itself is what is missing), the engine page's
|
||||
"Deploy order — nothing below has been run" is now a **Deploy record**, and the
|
||||
homepage card lost `wip: down` so the admin panel probes 7772 for real.
|
||||
- Corrected the same evening, by breaking it in a restarted session: a harness
|
||||
call that names no voice reaches 7772 with **no `voice` key at all** — `400
|
||||
unknown voice None; available voices: narrator1, narrator2, narrator3` —
|
||||
because no setting feeds one. The earlier `Saved 180524 bytes
|
||||
(voice=narrator1)` succeeded only because that run was *told* to pass
|
||||
`voice_id`. The endpoint therefore grew `DEFAULT_VOICE` (its own PR): a
|
||||
voice-less request speaks the declared pack and the journal line carries
|
||||
`(default)`, while an unknown name still 400s with the list. Reverting the role
|
||||
to Kokoro is one key, not two, and `speech.voice: narrator1` — briefly recorded
|
||||
above as wiring — went back to `am_fenrir`: that key is the vocalizer's Kokoro
|
||||
enum, so a pack name there breaks speak-aloud and does nothing for synthesis.
|
||||
|
||||
## 2026-10-18 — the default voice shipped: a harness voiceover needs no voice
|
||||
|
||||
- Merged **#17** (`b590695`) and **#18** (`7889a68`), then deployed both affected
|
||||
services in one dispatch — `--service frank-tts --service homepage`, 159 s,
|
||||
Playwright gate 33/33. The shim image rebuilt with the new handler; the
|
||||
homepage checker took the card on its next timer tick: `frank-tts → up`,
|
||||
probed for real rather than gated.
|
||||
- The call that had failed is the one that proves the fix. Same device, same
|
||||
restarted session, **no** `voice_id`: `Saved 430124 bytes (model=frank-tts
|
||||
(dankclaw)/frank-tts, voice=default, codec=wav)`. The façade journal says what
|
||||
actually spoke — `speech voice=narrator1 (default) format=wav chars=148 ->
|
||||
430124 bytes` — with the engine's ledger behind it (`audio 8.96 s, RTF 0.551`).
|
||||
The harness prints `voice=default` because it never had a name to print; the
|
||||
voice that answered is the operator's line in `scripts/frank-tts.env`, and the
|
||||
journal keeps them distinguishable forever after.
|
||||
- The page gained a **Default voice** row in its contract table and
|
||||
[[#Naming a voice]] now states that a call naming nothing is the harness's
|
||||
normal shape, not a client bug — the misconception that produced the 400 in
|
||||
the first place.
|
||||
|
||||
## 2026-10-18 — OpenWebUI "voice narrator1 not found": the rejector is in the browser
|
||||
|
||||
- Frank's toast — `Error: Voice "narrator1" not found. Should be one of:
|
||||
af_heart, …, bm_fable` — looked like frank-tts refusing the name. It is
|
||||
not: that 29-name enum is the voice pack of **Kokoro.js running in the
|
||||
page**, and OpenWebUI 0.11.4 ships that implementation in its own front
|
||||
end (frozen table `{af_heart:{name:"Heart",…}}`, `_validate_voice`, model
|
||||
`onnx-community/Kokoro-82M-v1.0-ONNX`, a personal *Kokoro.js dtype*
|
||||
setting, engine values `""`/`browser-kokoro`).
|
||||
- Instruments that separated the two paths, in order: the admin config DB is
|
||||
correct (`audio.tts.engine=openai`,
|
||||
`audio.tts.openai.api_base_url=http://host.containers.internal:7772/v1`,
|
||||
`model=frank-tts`, `voice=narrator1`, `split_on=punctuation`); from inside
|
||||
the container the exact OpenWebUI body
|
||||
(`{"model":"frank-tts","input":…,"voice":"narrator1","response_format":"mp3","speed":1.0}`)
|
||||
returns **200 `audio/mpeg` 7,629 B**; and `podman logs open-webui` holds
|
||||
**zero** `audio/speech` requests for the failing click. A server-path
|
||||
failure leaves a log line and a frank-tts error — a `browser-kokoro`
|
||||
failure leaves neither. No container env carries an `AUDIO_*`/`TTS`
|
||||
variable and no `user.settings` row (all four users) carries an audio
|
||||
override, so the engine choice is per-browser storage, not fleet state.
|
||||
- Fix is a UI click, not a deploy: Settings → Audio → TTS Engine →
|
||||
**Default** hands synthesis back to frank-tts and the cloned voices;
|
||||
choosing `af_heart` keeps in-browser Kokoro, which cannot clone (fixed
|
||||
pack). Recorded on [[OpenWebUI]] under *Text-to-speech (frank-tts)*;
|
||||
suite green 384/384 (wall 65.9 s).
|
||||
- Also in the window: **PR #22 merged** (`0c7fe8b`, `dank/command-center`) so
|
||||
`agent/extensions/dankvoice.ts` landed on the auto-load path — proven in a
|
||||
plain `omp` session with no flags: `/say merged build check` → 1.20 s WAV
|
||||
to the player, temp dir cleaned; the audible pass through the default
|
||||
`pw-play` detection confirmed by Frank. Sessions open before the merge need
|
||||
a restart to pick the command up.
|
||||
|
||||
## 2026-10-18 — frank-zealot: 2 × 262,144 over unquantized f16 KV
|
||||
|
||||
- Frank's instruction: **"shift frank-zealot to 2 parallels of unquantized
|
||||
kv"**. The env delta is four lines: `CTX_SIZE=524288`, `PARALLEL=2`,
|
||||
`CACHE_TYPE_K=f16`, `CACHE_TYPE_V=f16`. Everything else holds — batch
|
||||
**4096** / ubatch **2048**, `--flash-attn on`, `--jinja`, `--metrics`,
|
||||
`--presence-penalty 1.0`, `SPEC_TYPE=none`, the Q8_0 mmproj, port 7771, and
|
||||
the per-slot window **262,144** — `models.yml` `contextWindow` untouched.
|
||||
- **Why the pool had to halve.** Exact KV bills **32,768 B/token** (8
|
||||
full-attention layers × 4 KV heads × 256 dim × 2 B × 2 caches), so the old
|
||||
786,432 pool asks **24,576 MiB** against the **24,123 MiB** the driver leaves
|
||||
addressable — impossible, not tight. 524,288 positions ⇒ **16,384 MiB** and
|
||||
exactly two windows of 262,144.
|
||||
- Arithmetic correction to the repo's long-standing figure: "the f16 pair would
|
||||
need **26,112 MiB**" doubled the q8_0 number *including* its 1/32 scale
|
||||
overhead — q8_0 is 17,408 B/token (1.0625 B/element), so f16/q8_0 is
|
||||
**1.882×**, not 2×, and exact f16 over 786,432 is **24,576 MiB**. The
|
||||
conclusion stands (three f16 windows cannot boot this card); the number was
|
||||
wrong.
|
||||
- The cost: **concurrency 3 → 2 conversations**, and MTP is now **doubly** out.
|
||||
The 13:16 measurement still says off, and exact KV adds a second reason — the
|
||||
draft context bills its own f16 KV over the WHOLE pool (**4,096 B/position**
|
||||
× 524,288 = **2,048 MiB**) on a trunk projecting ~21,200 used.
|
||||
- **Projection, explicitly not a measurement:** ~**21,200 MiB** used /
|
||||
~**2,920 free**, derived from the 13:19 ledger's 17,922 used because only two
|
||||
terms move — the KV pool (**+3,328**) and one recurrent-state cell
|
||||
(**−50.25**, 3 cells → 2). The measured boot ledger and the decode/prefill
|
||||
re-measure land in the follow-up commit, after the first f16 boot.
|
||||
- New gate in `tests/frank_zealot_unit.bats`: dtype × pool plus the measured
|
||||
non-KV terms must fit **24,123 MiB** with **≥ 1 GiB** of slack — and the same
|
||||
gate proves it refuses the impossible arm (**f16 at 786,432**), not merely
|
||||
passes the shipped one. Runbook re-keyed to the new shape:
|
||||
`LLM/frank-zealot-llama-prism-bonsai9-mtp.md`.
|
||||
|
||||
@@ -541,12 +541,12 @@
|
||||
"name": "frank-tts",
|
||||
"category": "admin",
|
||||
"section": "AI",
|
||||
"description": "The fleet's cloned-voice speech endpoint: POST /v1/audio/speech with voice=<name>, audio back in that voice. Public on 7772; the engine behind it (frank-tts-engine) is unpublished on purpose. Hidden until deployed.",
|
||||
"description": "The fleet's cloned-voice speech endpoint: POST /v1/audio/speech with voice=<name>, audio back in that voice. Published on 7772; the engine behind it (frank-tts-engine) is unpublished on purpose. It is also the harness speech role.",
|
||||
"url": "",
|
||||
"docs": "",
|
||||
"probe": "127.0.0.1:7772",
|
||||
"method": "http",
|
||||
"wip": "down",
|
||||
"wip": false,
|
||||
"showLink": false,
|
||||
"art": "/img/apps/mono-ft.svg"
|
||||
}
|
||||
|
||||
@@ -510,8 +510,8 @@ PY
|
||||
# two GGUFs (1.2 GiB + 291 MB) are still being read and the HTTP thread has
|
||||
# not bound yet. The warmer is launched against an address with nothing on
|
||||
# it, the listener appears a beat later, and the requirement is that the
|
||||
# warmer notices and does the work — an early exit here is the day's first
|
||||
# caller paying 5.5 s of silence for a voice nobody warmed.
|
||||
# warmer notices and does the work — an early exit leaves every voice
|
||||
# unregistered, and the façade then answers 400 for all of them.
|
||||
port="$(free_port)"
|
||||
( ENGINE_URL="http://127.0.0.1:$port" VOICES_DIR="$VOICES" HEALTH_WAIT=60 \
|
||||
bash "$WARM" > "$TEST_TMP/race.log" 2>&1; echo "$?" > "$TEST_TMP/race.rc" ) &
|
||||
@@ -943,10 +943,14 @@ DRV
|
||||
|| { echo "editing app.py must rebuild the façade image"; false; }
|
||||
}
|
||||
|
||||
@test "the voice stack's homepage state: one card for the pair, and it is hidden while undeployed" {
|
||||
@test "the voice stack's homepage state: one probed card for the pair, and no link" {
|
||||
# The engine is not on the homepage and must not be: it publishes no port, so
|
||||
# there is no address the checker could dial, and a card that can never be
|
||||
# green is worse than no card. The façade carries the pair's signal.
|
||||
# green is worse than no card. The façade carries the pair's signal — and
|
||||
# since the 2026-10-18 deploy that signal is a real probe (`wip: false`), not
|
||||
# a placeholder. It stays `showLink: false` and admin-only: an HTTP API has no
|
||||
# page to open, and the checker counts ANY HTTP response as up, so the card is
|
||||
# honest without a link and without a `/health` path in the probe.
|
||||
python3 - "$REPO_ROOT" <<'PY'
|
||||
import json, os, sys
|
||||
root = sys.argv[1]
|
||||
@@ -955,8 +959,12 @@ assert "frank-tts-engine" not in cards, \
|
||||
"the engine publishes no port: a card for it could never be probed green"
|
||||
fts = cards["frank-tts"]
|
||||
assert fts["probe"] == "127.0.0.1:7772", fts["probe"]
|
||||
assert fts["wip"] == "down" and fts["showLink"] is False, \
|
||||
"nothing listens on 7772 until the deploy, so a visible card reads as down forever"
|
||||
assert fts["wip"] is False, \
|
||||
"the endpoint is deployed; a wip card would hide its real state: " + repr(fts.get("wip"))
|
||||
assert fts["showLink"] is False and fts["url"] == "", \
|
||||
"an API endpoint has no UI to link to — it is a status card, not a tile"
|
||||
assert fts["category"] == "admin", \
|
||||
"consumers dial this; it is not an app the user homepage should advertise"
|
||||
assert os.path.exists(f"{root}/shared/homepage{fts['art']}"), f"missing art {fts['art']}"
|
||||
PY
|
||||
}
|
||||
|
||||
@@ -183,6 +183,37 @@ assert 'voice_id' not in body, body
|
||||
PY
|
||||
}
|
||||
|
||||
@test "a voice-less request speaks the declared default and marks the log" {
|
||||
# The omp harness has no setting that feeds a voice to its speech device, so
|
||||
# a voice-less call is its NORMAL shape rather than a client bug. With the
|
||||
# operator's default armed the call must produce that pack's audio, and the
|
||||
# journal line must say (default) — otherwise an unrequested voice reads,
|
||||
# weeks later, exactly like one somebody asked for.
|
||||
arm_omni
|
||||
export DEFAULT_VOICE=narrator1
|
||||
start_app
|
||||
speech '{"model":"tts-1","input":"hello"}'
|
||||
[ "$output" = "200" ] || { cat "$TEST_TMP/app.log"; false; }
|
||||
cmp "$TEST_TMP/out.bin" "$TEST_TMP/upstream.wav"
|
||||
python3 - "$TEST_TMP/capture.json" <<'PY'
|
||||
import json, sys
|
||||
body = json.load(open(sys.argv[1]))
|
||||
assert body['voice'] == 'narrator1', 'the declared default never reached the engine'
|
||||
PY
|
||||
grep -q 'speech voice=narrator1 (default) format=wav' "$TEST_TMP/app.log" \
|
||||
|| { cat "$TEST_TMP/app.log"; false; }
|
||||
}
|
||||
|
||||
@test "with no default declared a voice-less request stays a 400" {
|
||||
# Strictness is still the code default: an endpoint that invents a voice
|
||||
# nobody configured is exactly how a wrong-voice deployment stays wrong.
|
||||
arm_omni
|
||||
start_app
|
||||
speech '{"model":"tts-1","input":"hello"}'
|
||||
[ "$output" = "400" ] || { cat "$TEST_TMP/app.log"; false; }
|
||||
grep -q 'narrator1' "$TEST_TMP/out.bin"
|
||||
}
|
||||
|
||||
@test "an unknown voice is a 400 naming the voices that exist" {
|
||||
arm_omni
|
||||
start_app
|
||||
|
||||
@@ -3,15 +3,16 @@
|
||||
# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 — the ternary Bonsai-2-9B, a
|
||||
# qwen35 hybrid with 32 layers (8 full-attention + 24 Gated DeltaNet) and the
|
||||
# MTP draft head INSIDE the file — via the PrismML fork build of llama.cpp
|
||||
# (vendor build 10777, the minimum that pairs MTP with a quantized V cache),
|
||||
# the ONLY loader that registers the payload's custom tensor type 142 and
|
||||
# honours its prism.hadamard.* metadata: THREE 262,144-token conversations
|
||||
# (--ctx-size 786432 --parallel 3), q8_0 K + q8_0 V under flash attention (f16
|
||||
# KV would bill 26,112 MiB of a 24,123 MiB card), the publisher's own Q8_0
|
||||
# vision projector (the BF16 one can crash on concurrent images), and MTP OFF
|
||||
# — measured, not because the head is missing: the draft context bills f16 KV
|
||||
# over the whole pool and halves decode at this shape. Log verbosity 4 ships so
|
||||
# the VRAM-fit forensics land in the journal.
|
||||
# (vendor build 10777), the ONLY loader that registers the payload's custom
|
||||
# tensor type 142 and honours its prism.hadamard.* metadata: TWO 262,144-token
|
||||
# conversations (--ctx-size 524288 --parallel 2) over an UNQUANTIZED f16 KV
|
||||
# cache. Exact KV bills 32,768 B/token here, so the pool is what pays for it —
|
||||
# the same dtype at the old 786,432 pool is 24,576 MiB against the 24,123 MiB
|
||||
# the driver leaves addressable, and the slot count is the term that moved. The
|
||||
# publisher's own Q8_0 vision projector rides along (the BF16 one can crash on
|
||||
# concurrent images), and MTP stays OFF: the head is present, but exact KV
|
||||
# leaves no room for a draft context that bills f16 KV over the whole pool. Log
|
||||
# verbosity 4 ships so the VRAM-fit forensics land in the journal.
|
||||
# This seam runs the serve script against a stub engine and asserts on the
|
||||
# argv it actually execs, so what is pinned is the command the operator gets.
|
||||
|
||||
@@ -76,6 +77,22 @@ argv_value() {
|
||||
grep -A1 -xF -- "$1" <<<"$output" | tail -1
|
||||
}
|
||||
|
||||
# kv_bytes_per_token DTYPE — bytes ONE cache costs per token at this payload's
|
||||
# geometry: 8 full-attention layers × 4 KV heads × 256 dim = 8,192 elements
|
||||
# (measured from the GGUF header: n_embd_k_gqa = n_embd_v_gqa = 1024). f16 is
|
||||
# 2 B/element; q8_0 is 1.0625 (32 codes plus one 2-byte scale per 32-block);
|
||||
# q4_0 is 0.625 (codes + scale + min per 32-block). Unknown dtype is a FAILURE,
|
||||
# never a silent zero — the fit gate below is only honest if it cannot be
|
||||
# satisfied by a rung it does not understand.
|
||||
kv_bytes_per_token() {
|
||||
case "$1" in
|
||||
f16|bf16) echo 16384 ;;
|
||||
q8_0) echo 8704 ;;
|
||||
q4_0) echo 5120 ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# run_stage — exec the CURRENT $STAGE/test.env as-is. serve_argv REWRITES
|
||||
# that file from the shipped env, so any test that mutates it (drop a line,
|
||||
# append a drafter) must exec through this, not serve_argv — which silently
|
||||
@@ -126,7 +143,7 @@ run_stage() {
|
||||
done
|
||||
}
|
||||
|
||||
@test "three conversations share the pool at a 262144 window each" {
|
||||
@test "two conversations share the pool at a 262144 window each" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --alias)" = "frank-zealot" ]
|
||||
@@ -134,15 +151,13 @@ run_stage() {
|
||||
[ "$(argv_value --host)" = "0.0.0.0" ]
|
||||
[ "$(argv_value --model)" = "$MODEL_DIR/$MODEL_STUB" ]
|
||||
[ "$(argv_value --n-gpu-layers)" -gt 0 ]
|
||||
# llama.cpp divides --ctx-size across slots, so the observable contract
|
||||
# is the per-slot window. The 9B swap exists precisely so every
|
||||
# conversation gets the model's TRAINED 262,144 (the 27B had to be cut to
|
||||
# 200,704 to fund two), and the harness contextWindow must quote 262144 —
|
||||
# the pool and the slot count are neither, and a longer prompt is a 400.
|
||||
# Three is also the ceiling: a fourth slot would push each window under the
|
||||
# fleet's worst-case compaction input, which is this endpoint's job.
|
||||
[ "$(argv_value --parallel)" -eq 3 ]
|
||||
[ "$(argv_value --ctx-size)" -eq 786432 ]
|
||||
# llama.cpp divides --ctx-size across slots, so the observable contract is
|
||||
# the per-slot window. Exact KV costs the SLOT COUNT, not the window: every
|
||||
# conversation still gets the model's TRAINED 262,144, and the harness
|
||||
# contextWindow must quote that number — the pool and the slot count are
|
||||
# neither, and a longer prompt is a 400, never a truncation.
|
||||
[ "$(argv_value --parallel)" -eq 2 ]
|
||||
[ "$(argv_value --ctx-size)" -eq 524288 ]
|
||||
local pool slots
|
||||
pool="$(argv_value --ctx-size)"; slots="$(argv_value --parallel)"
|
||||
[ $((pool % slots)) -eq 0 ]
|
||||
@@ -151,15 +166,16 @@ run_stage() {
|
||||
|
||||
@test "the KV dtype pair follows the env file and flash-attn stays pinned on" {
|
||||
# Asserted against the env rather than a literal so a retune moves one
|
||||
# file, not two. The shipped pair is q8_0/q8_0: this payload bills
|
||||
# 8 full-attention layers × 4 KV heads × 256 dim = 8,192 elements per token
|
||||
# per cache, so f16/f16 would need 26,112 MiB of the 24,123 MiB card while
|
||||
# q8_0 bills 17,408 B/token = 13,056.00 MiB at the 786,432 pool — the
|
||||
# ledger said exactly that, and it is the referee. The 24 DeltaNet layers
|
||||
# bill a fixed recurrent state per slot (452.25 MiB for three cells), which
|
||||
# is why pool arithmetic alone under-prices the shape. flash-attn is not
|
||||
# decoration: quantized V is only legal with it, and it must stay explicit
|
||||
# so a future quant swap cannot silently fall back.
|
||||
# file, not two. The shipped pair is f16/f16: exact attention state, and the
|
||||
# slot count is what makes it affordable — this payload bills 8,192 elements
|
||||
# per token per cache (8 full-attention layers × 4 KV heads × 256 dim), so
|
||||
# exact KV is 32,768 B/token = 16,384.00 MiB at the 524,288 pool, where the
|
||||
# q8_0 pair this file used to ship billed 17,408 B/token = 13,056.00 MiB at
|
||||
# the old 786,432 pool (ledger-exact both times). The 24 DeltaNet layers
|
||||
# bill a fixed recurrent state per slot (~50.25 MiB a cell), which is why
|
||||
# pool arithmetic alone under-prices the shape. flash-attn is not
|
||||
# decoration: the quantized-V fallback is only legal with it, so it stays
|
||||
# explicit and a dtype retune cannot silently fall back.
|
||||
local env_k env_v
|
||||
env_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
env_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
@@ -170,6 +186,38 @@ run_stage() {
|
||||
[ "$(argv_value --flash-attn)" = "on" ]
|
||||
}
|
||||
|
||||
@test "the shipped KV dtype and pool actually fit the card this boots on" {
|
||||
# No health gate can see this mistake: llama.cpp sizes the KV pool as
|
||||
# rate × pool-positions and cudaMalloc fails at boot ON THE LIVE ENDPOINT.
|
||||
# The fit is arithmetic off the payload geometry plus the non-KV terms
|
||||
# measured off the boot ledger (weights 2,036.71 + compute buffers 1,409.02
|
||||
# + projector 595 + 248.10 + CUDA context ≈ 460 = 4,749 MiB, dtype- and
|
||||
# slot-independent at this ubatch), plus one ~51 MiB recurrent-state cell
|
||||
# per slot. 24,123 MiB is what the driver leaves addressable (nvidia-smi
|
||||
# reports it as used + free; its total column reads 24,576).
|
||||
local env_k env_v pool slots rate_k rate_v kv total
|
||||
env_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
env_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
pool="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
slots="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
rate_k="$(kv_bytes_per_token "$env_k")"
|
||||
rate_v="$(kv_bytes_per_token "$env_v")"
|
||||
kv="$(( pool * (rate_k + rate_v) / 1048576 ))"
|
||||
total="$(( kv + 4749 + 51 * slots ))"
|
||||
# The shipped arm has to leave real room: the projector's worst case is
|
||||
# charged AFTER the engine's own fit check, so a projection that lands on
|
||||
# the ceiling is a crash waiting for the first image request.
|
||||
[ "$total" -le $(( 24123 - 1024 )) ]
|
||||
|
||||
# And the gate must be able to REFUSE: the same exact KV at the pool this
|
||||
# shape left behind is 24,576 MiB of cache alone — a boot that dies before
|
||||
# /health ever answers. Proving the gate bites is what makes passing one
|
||||
# meaningful.
|
||||
local over
|
||||
over="$(( 786432 * (rate_k + rate_v) / 1048576 + 4749 + 51 * 3 ))"
|
||||
[ "$over" -gt 24123 ]
|
||||
}
|
||||
|
||||
@test "log verbosity follows the env file" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
@@ -233,10 +281,11 @@ run_stage() {
|
||||
# Frank, 2026-10-18: this file DOES carry the MTP head (the boot arms prove
|
||||
# it loads and drafts, acceptance 0.798/0.617 per position, mean accepted
|
||||
# length 2.41) and it is still off, because the draft context bills its own
|
||||
# f16 KV over the WHOLE pool — 4,096 B/position × 786,432 = 3,072 MiB plus
|
||||
# buffers — and at --ubatch-size 2048 that squeeze halves decode: 167.2 →
|
||||
# 109.2 t/s. The MTP arm is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (213.5
|
||||
# t/s), documented in frank-zealot.env. The flags move as a set:
|
||||
# f16 KV over the WHOLE pool — 4,096 B/position × 524,288 = 2,048 MiB plus
|
||||
# buffers — on a trunk that already projects 21,234 MiB with exact KV. The
|
||||
# MTP arms in frank-zealot.env are all q8_0-era measurements; the doors back
|
||||
# to the drafter are one window per conversation, or KV back to q8_0 with
|
||||
# UBATCH_SIZE=1024 (213.5 t/s). The flags move as a set:
|
||||
# SPEC_TYPE=none strips --spec-type, --spec-draft-n-max AND
|
||||
# --spec-draft-model together, so a no-spec boot cannot carry half a spec
|
||||
# pair (the ROCmFPX-era baseline arm segfaulted at load doing that).
|
||||
@@ -276,7 +325,7 @@ run_stage() {
|
||||
# The Bonsai-2 text file carries no vision tensors — the publisher's Q8_0
|
||||
# projector argument IS the vision capability: no flag, no eyes. Q8_0 (not
|
||||
# BF16) is the publisher's own requirement for concurrent image requests,
|
||||
# which is what --parallel 3 means in practice.
|
||||
# which is what --parallel 2 means in practice.
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --mmproj)" = "$MODEL_DIR/$MM_STUB" ]
|
||||
|
||||
Reference in New Issue
Block a user