Frank
64185d63b9
- env: Bonsai2 payload pair, CACHE_TYPE_K/V=q8_0 (f16 OOMs at 2x200,704: 34 KiB vs 64 KiB/token over the 16 full-attention layers), measured boot ledger 22,664 used / 1,461 free incl. the 873 MiB mmproj that lands AFTER the engine fit check, q4_0 fallback rung, CRACK one-line MODEL_FILE swap - deployer: runtime/model/mmproj gates re-keyed to the PrismML vendor build (the only loader for GGML tensor type 142 + prism.hadamard.* metadata) - seam: payload trio pinned; KV argv assertions now derived from CACHE_TYPE_* in the env file — the wrong-var-name f16 fallback that produced the first failed trial window can no longer pass silently
73 lines
4.2 KiB
Bash
73 lines
4.2 KiB
Bash
# frank-zealot — llama.cpp (PrismML fork build, CUDA) serving
|
||
# prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090, :7771.
|
||
#
|
||
# 2026-10-17 (Frank): swap from Qwen3.5-9B-Uncensored Q6_K to the Bonsai-2-27B
|
||
# ternary checkpoint at the SAME shape Frank asked for — TWO 200,704-token
|
||
# conversations — with q8_0 K + q8_0 V (f16 is arithmetically impossible here,
|
||
# see the KV law), MTP off (the file carries no nextn/mtp tensors), verbosity 4.
|
||
#
|
||
# THE RUNTIME IS A LOCK, NOT A PREFERENCE. The file is general.file_type 141
|
||
# with 402 tensors of the CUSTOM GGML type 142 (pq2_0) plus prism.hadamard.*
|
||
# metadata (normalized-Sylvester-Walsh transform, block 1024, 401 named
|
||
# weights, inverse on token_embd, gdn_v_grouped). Stock llama.cpp refuses type
|
||
# 142 at load; only the PrismML fork registers the type AND reads the Hadamard
|
||
# metadata. The vendor prebuilt build 10754 (commit 2459f68b5) is what every
|
||
# number in this file was measured on; sha256 12ae5baba033d8cd…
|
||
LLAMA_BIN=/home/dank/GitPackages/llama-prism/llama-server
|
||
MODEL_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf
|
||
MMPROJ_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf
|
||
# The CRACK re-export (dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF, uncensored,
|
||
# already on disk at /home/dank/models/Bonsai2-CRACK/) is a ONE-LINE swap: the
|
||
# header census proves byte-identical geometry (851 tensors, same
|
||
# {pq2_0:402, f32:353, bf16:96} histogram, 64 blocks, vocab 248,320, same
|
||
# 8,952-char chat template) and the official file's sha256 is the Hub LFS oid.
|
||
# Bonsai-2 is the BASE (censored) Qwen3.8 family; if a compaction or subagent
|
||
# request ever comes back with a refusal, point MODEL_FILE at the CRACK file
|
||
# and restart — the VRAM ledger below does not move.
|
||
#
|
||
# Served model id — the harness/family name. The SERVICE, unit, scripts and
|
||
# this env file keep the frank-zealot name; only the alias on the wire is the
|
||
# model id. Consumers that key on the served id: OpenWebUI (model row,
|
||
# access_grant, api_configs model_ids) and the harness models.yml.
|
||
ALIAS=frank-zealot
|
||
BIND_ADDR=0.0.0.0
|
||
LISTEN_PORT=7771
|
||
# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
|
||
# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
|
||
# the harness contextWindow quotes the PER-SLOT number — a longer prompt is
|
||
# answered with a 400, never truncated. The model's native window is 262,144;
|
||
# 2 × 262,144 at q8_0 projects 25,758 MiB and does NOT fit this card (measured
|
||
# f16 analogue: 41,118). One 262,144 slot fits with 5,995 MiB free but
|
||
# serializes compaction, so the pair stays.
|
||
CTX_SIZE=401408
|
||
PARALLEL=2
|
||
BATCH_SIZE=4096
|
||
UBATCH_SIZE=2048
|
||
# KV law (from the GGUF header, then confirmed by the boot ledger): 16
|
||
# full-attention layers (full_attention_interval 4 over 64 blocks) × 4 KV heads
|
||
# × 256 dim = 16,384 elements per token per cache. q8_0 bills 17,408 B per
|
||
# cache ⇒ 34,816 B/token = 34 KiB, so the 401,408 pool is 13,328.00 MiB — the
|
||
# ledger said exactly that. f16 would bill 25,088 MiB and the boot DIES:
|
||
# `cudaMalloc failed: out of memory` on the 25,088.00 MiB request, engine fit
|
||
# projection 33,198 vs 23,558 free. The 48 DeltaNet layers bill a fixed
|
||
# recurrent state per SLOT — ~150 MiB each (context 13,627 − KV 13,328 = 299
|
||
# for two slots), which is why pool arithmetic alone under-prices the shape.
|
||
CACHE_TYPE_K=q8_0
|
||
CACHE_TYPE_V=q8_0
|
||
# Boot ledger (2026-10-17, this exact shape, nvidia-smi settled):
|
||
# 22,664 MiB used / 1,461 MiB free of 24,123
|
||
# fit projection 21,439 = 6,539 model + 13,627 context + 1,272 compute
|
||
# the mtmd projector's 873.10 MiB worst-case is NOT inside that projection —
|
||
# it lands after the fit check, which is the whole difference between
|
||
# "2.6 GiB free on paper" and 1.4 GiB free in reality. Watch it.
|
||
# Fallback rung if the margin is ever eaten (desktop, co-tenant): q4_0 KV
|
||
# measured 16,392 used / 7,733 free at the same 2 × 200,704.
|
||
# 1024 is the ubatch fallback (−616 MiB, measured on the 9B predecessor at
|
||
# this card/arch; 2048 was the measured CUDA optimum, +4.6 % prefill).
|
||
SPEC_TYPE=none
|
||
NGL=99
|
||
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
|
||
# and graph capture land in the journal, which is what the VRAM-fit forensics
|
||
# read. Levels above 4 flood the journal.
|
||
VERBOSITY=4
|