docs(frank-zealot): Prism Bonsai-2 PQ2_0 runbook; indexes and log re-keyed
- new runbook: runtime lock (build 10754 / type 142 door), geometry census, four-arm VRAM table (f16 wall, q8_0 deployed, q4_0 rung, 2x262,144), measured rate ladder, behavior contract, ops notes - LLM/index.md + vault root index: new ACTIVE entry, 9B era retired in place - log.md: the trial+cutover wave record, incl. the env-var-name law
This commit is contained in:
@@ -0,0 +1,174 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork), Bonsai-2-27B PQ2_0, 2 x 200704, q8_0 KV (ACTIVE)"
|
||||
description: "The ACTIVE frank-zealot runbook (since 2026-10-17) — the PrismML fork build of llama.cpp serving prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090: two conversations of 200,704 tokens with q8_0 K + q8_0 V (f16 KV does not fit this card at this window), vision through the publisher's Q8_0 Qwen3-VL projector, MTP impossible (no tensors in the file), and the measured four-arm VRAM census that priced the shape."
|
||||
tags:
|
||||
- llm
|
||||
- frank-zealot
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
- rtx3090
|
||||
timestamp: 2026-10-17
|
||||
---
|
||||
|
||||
# frank-zealot — llama.cpp (PrismML) Bonsai-2-27B PQ2_0, 2 × 200,704, q8_0 KV
|
||||
|
||||
Live since 2026-10-17, replacing the
|
||||
[[frank-zealot-llama-qwen35-9b-q6k-2x200704]] era (Qwen3.5-9B Q6_K, exact f16
|
||||
KV, same 2 × 200,704 shape). The trade: the ternary 27B costs ~3.4× the weights
|
||||
of the 9B and buys a 3× larger, stronger base at the SAME window contract —
|
||||
paid for by quantizing KV (f16 KV is arithmetically impossible for this
|
||||
payload on this card, measured below).
|
||||
|
||||
## Identity
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB (24,123 MiB visible to the engine) |
|
||||
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS`), `/v1` OpenAI surface |
|
||||
| Runtime | **PrismML fork build** `/home/dank/GitPackages/llama-prism/llama-server` — vendor release, `--version` = `0.2.0-dev (build 10754, commit 2459f68b5)`, sha256 `12ae5baba033d8cd…`. NOT a preference: see *Why the fork* |
|
||||
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
|
||||
| Weights | `/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf` — 7,206,168,928 B, sha256 `3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1` |
|
||||
| Projector | `Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf` — 629,246,976 B (full sha256 pinned in the fetch script), Qwen3-VL tower; **873.10 MiB worst-case VRAM, loaded AFTER the engine's fit check** |
|
||||
| Fetch | [`scripts/download-frank-zealot-bonsai2.sh`](../../scripts/download-frank-zealot-bonsai2.sh) — `-f`, byte-count + sha256 verified, idempotent, `--verify-only`, no token on the command line |
|
||||
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine) |
|
||||
|
||||
## Why the fork — the format is the lock
|
||||
|
||||
Read from the file's own metadata, not the model card:
|
||||
|
||||
|Key|Value|
|
||||
|---|---|
|
||||
|`general.architecture`|`qwen35` (hybrid: Gated DeltaNet + softmax attention)|
|
||||
|`qwen35.block_count`|64, `full_attention_interval` 4 ⇒ **16 full-attention + 48 DeltaNet layers**|
|
||||
|Attention|24 heads / **4 KV heads** × key 256 / value 256; `rope.dimension_sections` [11,11,10,0], `freq_base` 1e7, partial rope `dimension_count` 64|
|
||||
|Widths|embedding 5,120, feed-forward 17,408|
|
||||
|`qwen35.ssm.*`|conv_kernel 4, state_size 128, group_count 16, **time_step_rank 48**, **inner_size 6,144**|
|
||||
|Window / vocab|`context_length` 262,144; vocab 248,320 (gpt2 BPE, pre `qwen35`), eos 248046|
|
||||
|`general.file_type`|**141 = PQ2_0 — 2.13 bpw (group 128)**|
|
||||
|Tensor histogram|851 tensors: **`pq2_0` ×402** (custom GGML type **142**), f32 ×353, bf16 ×96|
|
||||
|`prism.hadamard.*`|transform `normalized-sylvester-walsh-hadamard`, block_size 1024, axis `input-last-dimension`, sign_mode explicit, **401 named weights**, sign widths [5120, 6144, 17408], 28,672 sign values, inverse applied on `token_embd.weight`, `gdn_v_grouped` true|
|
||||
|Sampler metadata|`general.sampling.temp` 1.0 / `top_p` 0.95 / `top_k` 20|
|
||||
|Chat template|8,952 chars; reads `enable_thinking`, **`reasoning_effort` ∈ {`xhigh` (default), `medium`, `low` — anything else is `raise_exception`**, `preserve_thinking`, `tools`/`tool_call`, image + video content parts|
|
||||
|
||||
Stock llama.cpp cannot load this file: type 142 is not in its enum (`gguf-py`'s
|
||||
stock `GGMLQuantizationType` raises `ValueError: 142 is not a valid…` on the
|
||||
same read), and even a loader that parsed it would ignore the
|
||||
`prism.hadamard.*` dequant transform the weights depend on. Same class of lock
|
||||
as the ROCmFPX custom-type era. The vendor build 10754 already carries the
|
||||
September upstream fixes (AVX-512 load segfault, and a `GGML_CUDA_FA_ALL_QUANTS`
|
||||
-capable CUDA backend — proven by q8_0 V serving under flash attention at the
|
||||
correct 34,816 B/token, below).
|
||||
|
||||
## VRAM census (2026-10-17, four arms, production stopped)
|
||||
|
||||
The engine's own `common_memory_breakdown_print` ledger + settled `nvidia-smi`,
|
||||
all arms `--n-gpu-layers 99 --flash-attn on --batch-size 4096 --ubatch-size 2048`,
|
||||
mmproj attached:
|
||||
|
||||
|Shape|KV|projected `model + context + compute`|settled used / free|verdict|
|
||||
|---|---|---|---|---|
|
||||
|**2 × 200,704 (pool 401,408)**|**q8_0 / q8_0**|21,439 = 6,539 + 13,627 + 1,272|**22,664 / 1,461**|**DEPLOYED**|
|
||||
|2 × 200,704|f16 / f16|33,198 = 6,539 + 25,387 + 1,272|—|`cudaMalloc failed: out of memory` on the **25,088.00 MiB** KV request|
|
||||
|1 × 262,144|q8_0 / q8_0|16,905 = 6,539 + 8,853 + 1,512|18,130 / 5,995|fits, but one slot serializes compaction|
|
||||
|2 × 262,144|q8_0 / q8_0|≈25,758 (arithmetic, same terms)|—|does not fit (f16 analogue measured 41,118)|
|
||||
|2 × 200,704|q4_0 / q4_0|15,167 = 6,539 + 7,355 + 1,272|16,392 / 7,733|the roomy fallback rung|
|
||||
|
||||
**KV rate law (confirmed to the MiB):** 16 full layers × 4 KV heads × 256 dim =
|
||||
16,384 elements/token/cache; q8_0 bills 17,408 B per cache ⇒ **34,816 B/token**,
|
||||
so the 401,408 pool = **13,328.00 MiB** — the ledger printed exactly that
|
||||
(`CUDA0 KV buffer size = 13328.00 MiB`). f16 doubles it to 25,088 MiB, which is
|
||||
the wall. The 48 DeltaNet layers bill a fixed recurrent state per **slot**:
|
||||
context 13,627 − KV 13,328 = **299 MiB for two slots (~150 MiB each)** — pool
|
||||
arithmetic alone under-prices the shape by that much.
|
||||
|
||||
**The projector is outside the fit check.** `[mtmd] estimated worst-case memory
|
||||
usage of mmproj is 873.10 MiB` is printed before fitting but not billed in it:
|
||||
settled used (22,664) − projection (21,439) ≈ 1.2 GiB is that projector plus the
|
||||
CUDA context. This is the difference between "2.6 GiB free on paper" and the
|
||||
real **1.4 GiB** — the same late-allocation failure class that killed the
|
||||
DSpark drafter boot in the llama.cpp eras. Any resize must re-read the settled
|
||||
`nvidia-smi`, never the projection.
|
||||
|
||||
## Measured rates (greedy, `Server-Timing`-free `/completion` `timings`)
|
||||
|
||||
|Probe|Result|
|
||||
|---|---|
|
||||
|Short, warm, single stream (128 tok)|**76.6 t/s** decode (the 9B Q6_K predecessor ran 94.4 — the ternary 27B is ~19 % slower, weight bandwidth + Hadamard dequant)|
|
||||
|First request after boot|2.6 t/s prefill / 37.7 t/s decode — **warmup artifact; never report the first request** (measured twice on this payload)|
|
||||
|Prefill at ~35k depth|**1,258.4 t/s** (27.8 s)|
|
||||
|Prefill at 166,027-token depth|**691.7 t/s** (240.0 s), zero errors, slot intact|
|
||||
|Two concurrent streams, 256 tok each|**54.7 + 54.7 = 109.4 t/s aggregate** — `--parallel 2` really batches|
|
||||
|Long-context retrieval (25,960-token prompt, needle at position 180 of 300 passages)|exact: *"Frank keeps the fleet in git, only through it."*|
|
||||
|
||||
## Behavior contract (probes, not assumptions)
|
||||
|
||||
- **Reasoning** lands in `reasoning_content` (`--reasoning-format deepseek`) ✓.
|
||||
- **Tool calls**: a `get_weather` request returns `finish_reason: tool_calls`
|
||||
with `arguments: {"city":"Seattle"}` — valid JSON, no loop, no HTTP 500. The
|
||||
fork's `KNOWN_ISSUES.md` warning about malformed/looping tool calls did not
|
||||
reproduce on build 10754 at this shape.
|
||||
- **Vision**: a 512 px solid-red PNG answers **"Red"** — but ONLY with a
|
||||
thinking budget ≥ ~512 tokens. At `max_tokens: 64` the request returns
|
||||
`finish_reason: "length"` with EMPTY `content` and half a thought in
|
||||
`reasoning_content`: thinking tokens count against the budget, so a
|
||||
short-budget vision probe reads as "the model is blind". Images do reach the
|
||||
model (a 512 px image bills ~280 prompt tokens: 322 total vs 42 for the text
|
||||
alone). Keep the ≥256 px probe rule from the earlier eras.
|
||||
- **Log noise law**: `/completion` (text context) at `VERBOSITY=4` prints
|
||||
`failed to parse reasoning content (response_format json_schema is currently
|
||||
only supported with parse_reasoning=true), content will be stored in
|
||||
'content' field` (`probe_request.cpp:789`) whenever the model closes a raw
|
||||
prompt with ``. It is **cosmetic** — the full chain of thought
|
||||
is in `content`. The OpenAI `/v1/chat/completions` path parses it into
|
||||
`reasoning_content` correctly, which is the path the harness and every
|
||||
consumer uses. Do not diagnose this as a fault, and do not drop verbosity
|
||||
below 4 — the KV/buffer ledger in the journal is what VRAM forensics read.
|
||||
- **Long raw-prompt completions can come back empty**: at ~26k+ tokens a bare
|
||||
`/completion` request generated 0–1 tokens with no error (the model emits an
|
||||
end-of-turn marker immediately, and a text context has no role frame to keep
|
||||
it talking). Chat traffic is unaffected; nothing in the harness posts raw
|
||||
`/completion` at depth.
|
||||
|
||||
## Ops notes
|
||||
|
||||
- **Trial-window law (cost one boot window on 2026-10-17):** a scratch env file
|
||||
must use the serve script's OWN variable names — `CACHE_TYPE_K`/`CACHE_TYPE_V`,
|
||||
`BIND_ADDR`, `NGL`. An invented `KV_CACHE_TYPE` is silently swallowed by the
|
||||
script's `: "${CACHE_TYPE_K:=f16}"` default and the arm boots **f16**, which
|
||||
for this payload is an OOM that looks like a model problem.
|
||||
- **Base alignment:** Bonsai-2 is the BASE (censored) Qwen3.8-27B lineage, so a
|
||||
refusal is possible on a compaction or subagent request. The CRACK re-export
|
||||
(`dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF`, already on disk at
|
||||
`/home/dank/models/Bonsai2-CRACK/Bonsai-2-27B-PQ2_0-CRACK.gguf`, sha256
|
||||
`5b24ea3e…`) is a **one-line `MODEL_FILE` swap**: the header census proves
|
||||
byte-identical geometry — 851 tensors, the same `{142:402, f32:353, bf16:96}`
|
||||
histogram, 64 blocks, vocab 248,320, the same 8,952-char template. The VRAM
|
||||
ledger does not move. If a refusal ever surfaces, that is the fix; there is
|
||||
also an MTP-bearing CRACK re-export upstream
|
||||
(`ManniX-ITA/…-CRACK-MTP-GGUF`, 7,657,489,920 B, +451 MB head) if decode ever
|
||||
needs the drafter — untested here.
|
||||
- **Compaction hub unchanged:** all three endpoints still route
|
||||
`compactionModel` here; 200,704 per slot is the fleet's only window that fits
|
||||
a worst-case compaction input, and `--parallel 2` keeps compaction off the
|
||||
critical path of a live generation.
|
||||
- **Harness side** (`~/.omp/agent/models.yml`): `contextWindow: 200704` and
|
||||
`maxTokens: 16384` unchanged; `extraBody` now quotes the payload's own
|
||||
1.0 / 0.95 / 20 with `preserve_thinking: true`. The `reasoningEffortMap` fold
|
||||
became **load-bearing** (the template raises on any level outside
|
||||
`xhigh|medium|low`) — an unmapped level is a hard 500 on the role's first
|
||||
request, naming Jinja rather than the model.
|
||||
- **Rollback:** revert the swap commit. The Qwen3.5-9B Q6_K payload, its mmproj
|
||||
and the master CUDA build all remain on disk
|
||||
(`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`,
|
||||
`~/GitPackages/llama-master/build-cuda/`), and the retired era keeps its own
|
||||
page: [[frank-zealot-llama-qwen35-9b-q6k-2x200704]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — the replaced era (rollback target)
|
||||
- [[frank-dragoon-flash-server-v2-swift15]] — the abliterated sibling on the iGPU
|
||||
- [[frank-archon-radiance-qwen38-iq4r]] — the censored-base flagship this payload shares base alignment with
|
||||
- [[deployment]] — deploy flow and incidents
|
||||
@@ -20,11 +20,13 @@ digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
|
||||
Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine,
|
||||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192
|
||||
ctx, vision ON)** and
|
||||
**llama.cpp CUDA (frank-zealot, port 7771: HauhauCS Gemma4-26B-A4B QAT
|
||||
Uncensored Q4_K_M GGUF — 26B MoE hybrid (25 sliding-window + 5 full-attention
|
||||
layers), ONE 200,704-token conversation, MTP disabled,
|
||||
q8_0 K + q8_0 V (FP8 class), vision via the publisher's own BF16
|
||||
mmproj, on the RTX 3090)**.
|
||||
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
|
||||
Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48
|
||||
Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled,
|
||||
q8_0 K + q8_0 V (f16 KV does not fit the RTX 3090 at this window), vision via
|
||||
the publisher's own Q8_0 Qwen3-VL mmproj, on the RTX 3090 — and the custom
|
||||
tensor type 142 + `prism.hadamard.*` metadata mean no other loader can open the
|
||||
file)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -106,13 +108,25 @@ frank-zealot, port 7771]
|
||||
(`slow`/`advisor` → frank-archon) carried forward. Superseded by the
|
||||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||||
engine moved with the model
|
||||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07**
|
||||
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090
|
||||
— TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), exact
|
||||
f16 K + f16 V, vision via the publisher's BF16 mmproj. MTP is OFF because
|
||||
the file has no MTP tensors at all. Priced from the MiMo Q6_K boots (same
|
||||
qwen35 geometry, same card, same shape). Supersedes the Gemma4 era (below).
|
||||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — **ACTIVE since 2026-10-17**
|
||||
(root systemd unit, port 7771): the **PrismML fork** build of llama.cpp
|
||||
(vendor 10754, commit `2459f68b5` — the only loader for the payload's custom
|
||||
tensor type 142 + `prism.hadamard.*` transform) serving
|
||||
prism-ml/Ternary-Bonsai-2-27B-gguf **PQ2_0** (2.13 bpw) on the RTX 3090 —
|
||||
TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), **q8_0 K
|
||||
+ q8_0 V** (34 KiB/token over the 16 full layers = 13,328 MiB; the f16 pair
|
||||
needs 25,088 MiB and dies), settled 22,664 used / 1,461 free, vision via the
|
||||
publisher's Q8_0 mmproj (+873 MiB, charged AFTER the engine's fit check), MTP
|
||||
OFF (no MTP tensors). 76.6 t/s single-stream decode, 691.7 t/s prefill at
|
||||
166k, 109.4 t/s aggregate at two streams. Supersedes the Qwen3.5-9B era below.
|
||||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **RETIRED 2026-10-07 →
|
||||
10-17** (payload + master CUDA build on disk as rollback): llama.cpp master
|
||||
(CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive
|
||||
**Q6_K** on the RTX 3090 — TWO 200,704-token conversations
|
||||
(`--ctx-size 401408 --parallel 2`), exact f16 K + f16 V, vision via the
|
||||
publisher's BF16 mmproj. MTP is OFF because the file has no MTP tensors at
|
||||
all. Priced from the MiMo Q6_K boots (same qwen35 geometry, same card, same
|
||||
shape). Supersedes the Gemma4 era (below).
|
||||
- [[frank-zealot-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
|
||||
day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
|
||||
master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
|
||||
|
||||
@@ -388,8 +388,15 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — the **ACTIVE** frank-zealot
|
||||
runbook since 2026-10-07 (late): llama.cpp master (CUDA sm_86) serving
|
||||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — the **ACTIVE** frank-zealot
|
||||
runbook since 2026-10-17: the PrismML vendor llama.cpp build 10754 (the only
|
||||
loader for GGML type 142 / `prism.hadamard.*`) serving
|
||||
prism-ml/Ternary-Bonsai-2-27B-gguf **PQ2_0** on the RTX 3090 — TWO
|
||||
200,704-token conversations, q8_0 K + q8_0 V (f16 OOMs at 2×200,704: 34 KiB
|
||||
vs 64 KiB per token over 16 full-attention layers), MTP **off** (no head in
|
||||
the file), vision through the Q8_0 mmproj; boot 22,664 used / 1,461 free
|
||||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — RETIRED 2026-10-17 (rollback
|
||||
target, payload on disk): llama.cpp master (CUDA sm_86) serving
|
||||
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090 —
|
||||
TWO 200,704-token conversations, exact f16 K + f16 V, MTP **impossible**
|
||||
(the file carries no `nextn`/`mtp` tensors), vision through the publisher's
|
||||
|
||||
@@ -1799,3 +1799,56 @@ every restart since the cutover.
|
||||
mints their accounts as role `pending` on first login.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md` — human-user template
|
||||
census extended with pk 22–23.
|
||||
|
||||
## 2026-10-17 — frank-zealot: Bonsai-2-27B PQ2_0 trial + cutover (2 × 200,704, q8_0 KV)
|
||||
|
||||
- Ask: evaluate `prism-ml/Ternary-Bonsai-2-27B-gguf` (PQ2_0) as frank-zealot's
|
||||
payload, and specifically measure **2 × 200,704 with q8_0 K + q8_0 V** on the
|
||||
RTX 3090. Fetch script `scripts/download-frank-zealot-bonsai2.sh` (exact
|
||||
bytes + sha256, `--verify-only`) landed first; payload verified at
|
||||
`/home/dank/models/Bonsai2-PQ2_0/` (7,206,168,928 B + 629,246,976 B mmproj).
|
||||
- Runtime door: the file is `file_type 141` with 402 tensors of custom GGML
|
||||
type **142** plus `prism.hadamard.*` metadata — stock llama.cpp refuses it,
|
||||
so the PrismML vendor build 10754 (`2459f68b5`) is forced, not chosen. A
|
||||
source build with `GGML_CUDA_FA_ALL_QUANTS=ON` was started and then
|
||||
cancelled as unnecessary: the vendor build already serves q8_0 V under flash
|
||||
attention at the exact 34,816 B/token (ledger-confirmed).
|
||||
- Trial-window law cost one boot: the scratch env invented `KV_CACHE_TYPE`
|
||||
where the serve script reads `CACHE_TYPE_K`/`CACHE_TYPE_V`, so the script's
|
||||
`:=f16` default silently won and every arm booted f16. That failed window is
|
||||
now the measured f16 wall: `cudaMalloc` OOM on a 25,088.00 MiB KV request,
|
||||
engine fit 33,198 vs 23,558 free.
|
||||
- Four-arm census (production stopped, trap-guarded restore, `bg_12`):
|
||||
**2 × 200,704 q8_0 = 22,664 used / 1,461 free (DEPLOYED)**; 1 × 262,144
|
||||
q8_0 = 18,130 / 5,995 (fits, but serializes compaction); 2 × 262,144 q8_0
|
||||
≈ 25,758 projected (no fit); 2 × 200,704 q4_0 = 16,392 / 7,733 (fallback
|
||||
rung). Recurrent DeltaNet state bills ~150 MiB per SLOT (context 13,627 −
|
||||
KV 13,328 for two slots), which pool arithmetic alone misses.
|
||||
- Behavior probes green: exact-instruction chat, `reasoning_content` present,
|
||||
`tool_calls` with valid JSON arguments (no fork `KNOWN_ISSUES` loop/500),
|
||||
512 px solid-red PNG answered **"Red"**, needle retrieval exact through a
|
||||
25,960-token prompt, 166,027-token prefill clean in 240.0 s. Rates: 76.6 t/s
|
||||
single-stream (9B ran 94.4), 1,258 t/s at 35k, 691.7 t/s at 166k,
|
||||
**109.4 t/s aggregate** at two concurrent streams.
|
||||
- New fleet laws recorded (AGENTS.md + runbook): the mmproj's 873.10 MiB
|
||||
worst-case is NOT inside the engine's fit projection; the first request after
|
||||
a boot is a warmup artifact (2.6 / 37.7 t/s measured); a 64-token thinking
|
||||
budget makes a thinking model look blind; the Bonsai-2 template reads
|
||||
`reasoning_effort` (xhigh/medium/low, raises otherwise) so the harness fold
|
||||
became load-bearing on this endpoint, and it reads `preserve_thinking`.
|
||||
- Cutover: `scripts/frank-zealot.env` (payload + `CACHE_TYPE_*`=q8_0 + ledger
|
||||
lore), `scripts/deploy-remote.sh` (runtime/model/mmproj gates re-keyed, fork
|
||||
hint), `tests/frank_zealot_unit.bats` (payload pair pinned; KV assertions now
|
||||
env-derived instead of an f16 literal), harness `models.yml` (extraBody 1.0
|
||||
/0.95/20 + `preserve_thinking`, effort-map comment corrected —
|
||||
`contextWindow` 200704 and `maxTokens` 16384 unchanged), new runbook
|
||||
[[frank-zealot-llama-prism-bonsai2-pq20]], indexes re-keyed, 9B era retired
|
||||
in place. Full bats green (371).
|
||||
- Base alignment: **Frank chose the OFFICIAL censored payload** — publisher
|
||||
provenance over refusal-freedom. The 9B it replaces was uncensored, so the
|
||||
homepage card and its warning contract were re-keyed with it: zealot's card
|
||||
now carries the base-alignment notice, the shared legal warning belongs to
|
||||
dragoon alone, and the `homepage_config.bats` equality
|
||||
(`zealot.warn == dragoon.warn`) retired with the payload it described.
|
||||
The CRACK re-export (byte-identical geometry per header census) stays on
|
||||
disk as the one-line `MODEL_FILE` fallback if refusals bite.
|
||||
|
||||
Reference in New Issue
Block a user