Frank
af5ddfc54c
Every KEY=value is byte-identical to the previous revision (diffed against git show HEAD: -- 28/25/20 keys). What left is the per-swap narration: the archon header drops 207 lines of era prose, dragoon 127, zealot 104. Each header now keeps the format lock, the digest/identity law, and one line per knob explaining what a reader editing that line cannot recover anywhere. The facts moved to their one owner: LLM/frank-archon.md, LLM/frank-dragoon.md, LLM/frank-zealot.md. One stale claim dies with the trim -- the dragoon header still said the KV pool is ONE 200,704-position budget behind 2 slots, which contradicted its own 401,408 paragraph; the page carries the correct ledger. Also kills two wrong arithmetic claims in the zealot header: the q8_0-V and CTX_SIZE=401408 rungs each free 3,840 MiB at the 524,288 pool, not 8,192 and not ~6.7 GiB.
57 lines
3.0 KiB
Bash
57 lines
3.0 KiB
Bash
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
|
|
# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file.
|
|
#
|
|
# ONE OWNER: shape, payload identity, eras, laws and every measured number
|
|
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
|
|
# knobs and only the reason a reader editing THIS line cannot recover
|
|
# elsewhere. Where the two disagree, this file wins on values.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
|
|
# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
|
|
# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
|
|
# metadata dequantizes wrong, silently.
|
|
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
|
|
# Q8_0 IS MANDATORY: the publisher states the BF16 tower can crash llama-server
|
|
# under concurrent image requests, which is exactly --parallel 2.
|
|
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
|
|
# Served id — the harness/family name, not a fact about the payload.
|
|
# Id-keyed consumers: OpenWebUI (model row, access_grant, api_configs), models.yml.
|
|
ALIAS=frank-zealot
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
# CTX_SIZE is the POOL llama.cpp divides across slots: 524,288 / 2 = 262,144 a
|
|
# conversation, each still at the trained window. MUST stay divisible by
|
|
# PARALLEL (the seam enforces it) and models.yml contextWindow quotes the
|
|
# PER-SLOT number — a longer prompt is a 400, never a truncation. The pool
|
|
# cannot go back to 786,432 under exact f16 KV: 24,576 MiB of cache against the
|
|
# 24,123 MiB the driver leaves addressable, before a weight loads.
|
|
CTX_SIZE=524288
|
|
PARALLEL=2
|
|
BATCH_SIZE=4096
|
|
# Ubatch is a prefill knob (decode never moves with it) and it sets the
|
|
# compute-buffer size — which is what decides whether a drafter fits.
|
|
# UBATCH_SIZE=1024 is the documented MTP arm.
|
|
UBATCH_SIZE=2048
|
|
# Exact K and V: the retrieval-fidelity ask, paid with one conversation of
|
|
# concurrency. Rate law and the arithmetic: [[frank-zealot]] → Laws.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
# Speculation OFF — measured, not assumed: the draft context bills its own f16
|
|
# KV over the WHOLE pool, and there is no room beside exact KV (the arm census
|
|
# is on the page). SPEC_TYPE=none strips the WHOLE set (--spec-type and every
|
|
# --spec-draft-*); half a set has segfaulted at load.
|
|
SPEC_TYPE=none
|
|
SPEC_DRAFT_N_MAX=2
|
|
SPEC_DRAFT_N_MIN=0
|
|
SPEC_DRAFT_P_MIN=0
|
|
# The sampler recipe lives in the checkpoint (general.sampling.*) and
|
|
# llama.cpp reads it itself. Presence penalty is the one recipe value with no
|
|
# GGUF metadata key, so it must reach argv. "" is the documented opt-out.
|
|
PRESENCE_PENALTY=1.0
|
|
NGL=99
|
|
# 4 = debug: KV layout, buffer sizes and graph capture land in the journal, and
|
|
# that is what VRAM forensics read. Above 4 floods the journal.
|
|
VERBOSITY=4
|
|
# Without --metrics llama.cpp answers 501 on /metrics, which Prometheus reads
|
|
# as DOWN while /health stays 200. 0 opts out and re-breaks the scrape.
|
|
METRICS=1
|