Frank
3eaba57925
Frank ordered the adapter; this adds it to the shape as one knob and one flag. - `LORA_FILE` / `LORA_SCALE` in the env, `--lora-scaled <file>:<scale>` as ONE argv token so a scale can never reach the engine without its adapter. The rollback follows the mmproj convention exactly: `LORA_FILE=` (explicitly empty) drops the flag entirely, no rebuild, and `LORA_SCALE=0` is the card's own "inert but present" value. - New fetcher `scripts/download-frank-zealot-uncensored-lora.sh`: `-f`, byte count, sha256, `--verify-only`, idempotent. The file is NOT LFS (a plain git object), so the pinned sha is the content hash, not an LFS oid. - The deployer gates on the adapter beside the pack and the projector, and its error names the ADAPTER's fetcher — sending an operator to the bonsai9 script for a 3 MB file is how a payload ships without it. - Seam: path+scale pinned as one token, the scale knob including 0, absence on the empty opt-out, a refusal to boot on a missing adapter, and the env ↔ deployer anti-drift pair extended to LORA_FILE/LLM_ZEALOT_LORA. 454/454. Provenance, stated where it will be read: the adapter's own metadata names `Ternary-Bonsai-2-9B-PTQ1_0.gguf` (3561df7a…) as the pack the direction was fitted against. This endpoint serves the PQ2_0+MTP+Q8_0 packing (c79fcdfa…), which is not that file and is not on this host. The loader matches tensor names, not hashes, so it applies — and build 10777 is the build the card verified on — but the strength is calibrated on a sibling quantization. The proof is therefore the refusal probe against the live endpoint, captured before and after, and not the card's table.
85 lines
5.0 KiB
Bash
85 lines
5.0 KiB
Bash
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
|
||
# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file, plus
|
||
# the refusal-direction adapter applied at load.
|
||
#
|
||
# ONE OWNER: shape, payload identity, eras, laws and every measured number
|
||
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
|
||
# knobs and only the reason a reader editing THIS line cannot recover
|
||
# elsewhere. Where the two disagree, this file wins on values.
|
||
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
|
||
# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
|
||
# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
|
||
# metadata dequantizes wrong, silently.
|
||
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
|
||
# Q8_0 IS MANDATORY: the publisher states the BF16 tower can crash llama-server
|
||
# under concurrent image requests, which is exactly --parallel 3.
|
||
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
|
||
# Refusal-direction adapter (a rank-1 LoRA, 3.2 MB) applied at load; it edits
|
||
# the compute graph, it is not a second model. Scale 1 is the card's "held-out
|
||
# refusals 0 %" point. LORA_FILE="" is the documented opt-out — the plain pack
|
||
# again, no rebuild — and LORA_SCALE=0 is the same result through the flag.
|
||
# PROVENANCE ASYMMETRY: the adapter's own prism.abliterate.base_gguf names
|
||
# Ternary-Bonsai-2-9B-PTQ1_0.gguf as the pack its direction was fitted against,
|
||
# which is NOT this packing and is not on this host. The tensor names match, so
|
||
# it applies; the strength was calibrated on a sibling quantization, so the
|
||
# effect is proven by the endpoint's behaviour, never by the metadata.
|
||
LORA_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-Uncensored-LoRA.gguf
|
||
LORA_SCALE=1
|
||
# Served id — the harness/family name, not a fact about the payload.
|
||
# Id-keyed consumers: OpenWebUI (model row, access_grant, api_configs), models.yml.
|
||
ALIAS=frank-zealot
|
||
BIND_ADDR=0.0.0.0
|
||
LISTEN_PORT=7771
|
||
# CTX_SIZE is the POOL llama.cpp divides across slots: 602,112 / 3 = 200,704 a
|
||
# conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
|
||
# models.yml contextWindow quotes the PER-SLOT number — a longer prompt is a
|
||
# 400, never a truncation. The pool is bought, not granted: K+V at 25,088
|
||
# B/token is 14,406.00 MiB here and the MTP draft context bills its own f16 KV
|
||
# over the WHOLE pool (4,096 B/position ⇒ 2,352.00 MiB), so the old 786,432
|
||
# pool cannot carry a drafter at any KV dtype — 18,816 + 3,072 of cache alone
|
||
# against the 24,123 MiB the driver leaves addressable.
|
||
CTX_SIZE=602112
|
||
PARALLEL=3
|
||
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch (the law
|
||
# on [[frank-zealot]]); 1024×6 keeps it a clean multiple of the ubatch.
|
||
BATCH_SIZE=6144
|
||
# Ubatch is a prefill knob (decode never moves with it) and it sets the
|
||
# compute-buffer size, which decides whether a drafter fits. 1024 is the rung
|
||
# every measured MTP arm here ran on (213.5 t/s against 109.2 at 2048). Its
|
||
# VRAM credit is SHAPE-specific: the 786,432-pool arm census showed ~1,306 MiB
|
||
# back, this boot measured 2,089.88 MiB of target compute against 2,496.13 at
|
||
# ubatch 2048 — 406 MiB back, with the split moving (CUDA0 +257, CUDA_Host
|
||
# −663). The drafter is funded by CACHE_TYPE_V below, not by this knob.
|
||
UBATCH_SIZE=1024
|
||
# K exact, V in the 8-bit class: 16,384 + 8,704 = 25,088 B/token (8,192
|
||
# elements per token per cache × 2 B for f16, × 1.0625 for q8_0 — there is no
|
||
# true fp8 KV dtype in this build). That 7,680 B/token saved against exact V is
|
||
# what funds BOTH the third conversation and the drafter. Quantized V is
|
||
# illegal without --flash-attn, which the serve script pins. Rate law and the
|
||
# arithmetic: [[frank-zealot]] → Laws.
|
||
CACHE_TYPE_K=f16
|
||
CACHE_TYPE_V=q8_0
|
||
# Speculation ON (Frank, 2026-10-19): the head is in the file
|
||
# (qwen35.nextn_predict_layers = 1, blk.32.nextn.*) and rides the MAIN model's
|
||
# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
|
||
# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
|
||
# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
|
||
# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
|
||
# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
|
||
# every --spec-draft-*); half a set has segfaulted at load.
|
||
SPEC_TYPE=draft-mtp
|
||
SPEC_DRAFT_N_MAX=2
|
||
SPEC_DRAFT_N_MIN=0
|
||
SPEC_DRAFT_P_MIN=0
|
||
# The sampler recipe lives in the checkpoint (general.sampling.*) and
|
||
# llama.cpp reads it itself. Presence penalty is the one recipe value with no
|
||
# GGUF metadata key, so it must reach argv. "" is the documented opt-out.
|
||
PRESENCE_PENALTY=1.0
|
||
NGL=99
|
||
# 4 = debug: KV layout, buffer sizes and graph capture land in the journal, and
|
||
# that is what VRAM forensics read. Above 4 floods the journal.
|
||
VERBOSITY=4
|
||
# Without --metrics llama.cpp answers 501 on /metrics, which Prometheus reads
|
||
# as DOWN while /health stays 200. 0 opts out and re-breaks the scrape.
|
||
METRICS=1
|