Frank
acbb539b97
Rebuilt the fork from origin/prism at 024474448 with the same configure the 10777 cutover used (Release, GGML_CUDA=ON, sm_86, gcc-13 host compiler, Ninja), installed to /home/dank/GitPackages/llama-prism-0244744. The payload gate holds: libllama carries 42 prism.hadamard strings (b10777 had 29) and libggml-base still registers PQ2_0, so type 142 and the hadamard metadata still load. LLAMA_BIN inherits a bin/lib install layout now, so the install needs an RPATH ($ORIGIN;$ORIGIN/../lib) — cmake's install step strips the build rpath and the binary otherwise dies on libllama-server-impl.so. The unit test that pinned the 10777 path now asserts the invariant its own comment described (LLAMA_BIN names a llama-prism-* build) instead of a directory one rebuild happens to install; the suite runs on a box with no /home/dank, so existence is not checkable there. Rollback: repoint LLAMA_BIN at llama-prism-b10777/llama-server (still on disk).
86 lines
5.1 KiB
Bash
86 lines
5.1 KiB
Bash
# frank-zealot — llama.cpp (PrismML fork build 12118 = 0.7.0-dev, commit
|
||
# 024474448, CUDA) on the RTX 3090, :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
|
||
# the refusal-direction adapter applied at load.
|
||
#
|
||
# ONE OWNER: shape, payload identity, eras, laws and every measured number
|
||
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
|
||
# knobs and only the reason a reader editing THIS line cannot recover
|
||
# elsewhere. Where the two disagree, this file wins on values.
|
||
LLAMA_BIN=/home/dank/GitPackages/llama-prism-0244744/bin/llama-server
|
||
# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
|
||
# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
|
||
# metadata dequantizes wrong, silently.
|
||
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
|
||
# Q8_0 IS MANDATORY: the publisher states the BF16 tower can crash llama-server
|
||
# under concurrent image requests, which is exactly --parallel 3.
|
||
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
|
||
# Refusal-direction adapter (a rank-1 LoRA, 3.2 MB) applied at load; it edits
|
||
# the compute graph, it is not a second model. Scale 1 is the card's "held-out
|
||
# refusals 0 %" point. LORA_FILE="" is the documented opt-out — the plain pack
|
||
# again, no rebuild — and LORA_SCALE=0 is the same result through the flag.
|
||
# PROVENANCE ASYMMETRY: the adapter's own prism.abliterate.base_gguf names
|
||
# Ternary-Bonsai-2-9B-PTQ1_0.gguf as the pack its direction was fitted against,
|
||
# which is NOT this packing and is not on this host. The tensor names match, so
|
||
# it applies; the strength was calibrated on a sibling quantization, so the
|
||
# effect is proven by the endpoint's behaviour, never by the metadata.
|
||
LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf
|
||
LORA_SCALE=1
|
||
# Served id — the harness/family name, not a fact about the payload.
|
||
# Id-keyed consumers: OpenWebUI (model row, access_grant, api_configs), models.yml.
|
||
ALIAS=frank-zealot
|
||
BIND_ADDR=0.0.0.0
|
||
LISTEN_PORT=7771
|
||
# CTX_SIZE is the POOL llama.cpp divides across slots: 602,112 / 3 = 200,704 a
|
||
# conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
|
||
# models.yml contextWindow quotes the PER-SLOT number — a longer prompt is a
|
||
# 400, never a truncation. The pool is bought, not granted: K+V at 25,088
|
||
# B/token is 14,406.00 MiB here and the MTP draft context bills its own f16 KV
|
||
# over the WHOLE pool (4,096 B/position ⇒ 2,352.00 MiB), so the old 786,432
|
||
# pool cannot carry a drafter at any KV dtype — 18,816 + 3,072 of cache alone
|
||
# against the 24,123 MiB the driver leaves addressable.
|
||
CTX_SIZE=602112
|
||
PARALLEL=3
|
||
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch (the law
|
||
# on [[frank-zealot]]); 1024×6 keeps it a clean multiple of the ubatch.
|
||
BATCH_SIZE=6144
|
||
# Ubatch is a prefill knob (decode never moves with it) and it sets the
|
||
# compute-buffer size, which decides whether a drafter fits. 1024 is the rung
|
||
# every measured MTP arm here ran on (213.5 t/s against 109.2 at 2048). Its
|
||
# VRAM credit is SHAPE-specific: the 786,432-pool arm census showed ~1,306 MiB
|
||
# back, this boot measured 2,089.88 MiB of target compute against 2,496.13 at
|
||
# ubatch 2048 — 406 MiB back, with the split moving (CUDA0 +257, CUDA_Host
|
||
# −663). The drafter is funded by CACHE_TYPE_V below, not by this knob.
|
||
UBATCH_SIZE=1024
|
||
# K exact, V in the 8-bit class: 16,384 + 8,704 = 25,088 B/token (8,192
|
||
# elements per token per cache × 2 B for f16, × 1.0625 for q8_0 — there is no
|
||
# true fp8 KV dtype in this build). That 7,680 B/token saved against exact V is
|
||
# what funds BOTH the third conversation and the drafter. Quantized V is
|
||
# illegal without --flash-attn, which the serve script pins. Rate law and the
|
||
# arithmetic: [[frank-zealot]] → Laws.
|
||
CACHE_TYPE_K=f16
|
||
CACHE_TYPE_V=q8_0
|
||
# Speculation ON (Frank, 2026-10-19): the head is in the file
|
||
# (qwen35.nextn_predict_layers = 1, blk.32.nextn.*) and rides the MAIN model's
|
||
# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
|
||
# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
|
||
# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
|
||
# beside MTP needs fork build >= b10776; LLAMA_BIN is 12118 (b10777 is kept on
|
||
# disk as the rollback path).
|
||
# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
|
||
# every --spec-draft-*); half a set has segfaulted at load.
|
||
SPEC_TYPE=draft-mtp
|
||
SPEC_DRAFT_N_MAX=2
|
||
SPEC_DRAFT_N_MIN=0
|
||
SPEC_DRAFT_P_MIN=0
|
||
# The sampler recipe lives in the checkpoint (general.sampling.*) and
|
||
# llama.cpp reads it itself. Presence penalty is the one recipe value with no
|
||
# GGUF metadata key, so it must reach argv. "" is the documented opt-out.
|
||
PRESENCE_PENALTY=1.0
|
||
NGL=99
|
||
# 4 = debug: KV layout, buffer sizes and graph capture land in the journal, and
|
||
# that is what VRAM forensics read. Above 4 floods the journal.
|
||
VERBOSITY=4
|
||
# Without --metrics llama.cpp answers 501 on /metrics, which Prometheus reads
|
||
# as DOWN while /health stays 200. 0 opts out and re-breaks the scrape.
|
||
METRICS=1
|