Frank
25a8b00043
Frank's order, 2026-10-19: --parallel 3 at 200,704 a conversation (pool 602,112), K at the engine default (f16), V to q8_0, batch 1024x6 / ubatch 1024, and the MTP drafter armed explicitly. The three requests are one budget equation. At 8,192 elements per token per cache the pair now bills 25,088 B/token = 14,406.00 MiB over the 602,112 pool, and the armed draft context adds its own f16 KV over that whole pool (4,096 B/position = 2,352.00 MiB) plus buffers and recurrent cells. Dropping V to the 8-bit class frees 4,410 MiB against exact V and the ubatch drop hands back ~1,306 - that is what pays for the third slot AND the drafter, neither of which exact V could fund. The serve script's written-down fallbacks move with the env so a missing env file cannot boot the retired shape, and the seam now: asserts the per-slot contract (divides evenly, <= the trained 262,144), bills an armed drafter in the VRAM fit gate, proves the gate still REFUSES both boots this shape can walk into, pins the shipped MTP set (and the absence of --spec-draft-model, which would load the trunk twice), and pins SPEC_TYPE=none as the rollback that strips the whole set.
70 lines
3.9 KiB
Bash
70 lines
3.9 KiB
Bash
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
|
||
# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file.
|
||
#
|
||
# ONE OWNER: shape, payload identity, eras, laws and every measured number
|
||
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
|
||
# knobs and only the reason a reader editing THIS line cannot recover
|
||
# elsewhere. Where the two disagree, this file wins on values.
|
||
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
|
||
# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
|
||
# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
|
||
# metadata dequantizes wrong, silently.
|
||
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
|
||
# Q8_0 IS MANDATORY: the publisher states the BF16 tower can crash llama-server
|
||
# under concurrent image requests, which is exactly --parallel 3.
|
||
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
|
||
# Served id — the harness/family name, not a fact about the payload.
|
||
# Id-keyed consumers: OpenWebUI (model row, access_grant, api_configs), models.yml.
|
||
ALIAS=frank-zealot
|
||
BIND_ADDR=0.0.0.0
|
||
LISTEN_PORT=7771
|
||
# CTX_SIZE is the POOL llama.cpp divides across slots: 602,112 / 3 = 200,704 a
|
||
# conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
|
||
# models.yml contextWindow quotes the PER-SLOT number — a longer prompt is a
|
||
# 400, never a truncation. The pool is bought, not granted: K+V at 25,088
|
||
# B/token is 14,406.00 MiB here and the MTP draft context bills its own f16 KV
|
||
# over the WHOLE pool (4,096 B/position ⇒ 2,352.00 MiB), so the old 786,432
|
||
# pool cannot carry a drafter at any KV dtype — 18,816 + 3,072 of cache alone
|
||
# against the 24,123 MiB the driver leaves addressable.
|
||
CTX_SIZE=602112
|
||
PARALLEL=3
|
||
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch (the law
|
||
# on [[frank-zealot]]); 1024×6 keeps it a clean multiple of the ubatch.
|
||
BATCH_SIZE=6144
|
||
# Ubatch is a prefill knob (decode never moves with it) and it sets the
|
||
# compute-buffer size — which is what decides whether a drafter fits: 1024
|
||
# hands back ~1,306 MiB beside the draft context, and it is the rung the
|
||
# measured MTP arm ran (213.5 t/s at 1024 against 109.2 at 2048).
|
||
UBATCH_SIZE=1024
|
||
# K exact, V in the 8-bit class: 16,384 + 8,704 = 25,088 B/token (8,192
|
||
# elements per token per cache × 2 B for f16, × 1.0625 for q8_0 — there is no
|
||
# true fp8 KV dtype in this build). That 7,680 B/token saved against exact V is
|
||
# what funds BOTH the third conversation and the drafter. Quantized V is
|
||
# illegal without --flash-attn, which the serve script pins. Rate law and the
|
||
# arithmetic: [[frank-zealot]] → Laws.
|
||
CACHE_TYPE_K=f16
|
||
CACHE_TYPE_V=q8_0
|
||
# Speculation ON (Frank, 2026-10-19): the head is in the file
|
||
# (qwen35.nextn_predict_layers = 1, blk.32.nextn.*) and rides the MAIN model's
|
||
# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
|
||
# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
|
||
# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
|
||
# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
|
||
# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
|
||
# every --spec-draft-*); half a set has segfaulted at load.
|
||
SPEC_TYPE=draft-mtp
|
||
SPEC_DRAFT_N_MAX=2
|
||
SPEC_DRAFT_N_MIN=0
|
||
SPEC_DRAFT_P_MIN=0
|
||
# The sampler recipe lives in the checkpoint (general.sampling.*) and
|
||
# llama.cpp reads it itself. Presence penalty is the one recipe value with no
|
||
# GGUF metadata key, so it must reach argv. "" is the documented opt-out.
|
||
PRESENCE_PENALTY=1.0
|
||
NGL=99
|
||
# 4 = debug: KV layout, buffer sizes and graph capture land in the journal, and
|
||
# that is what VRAM forensics read. Above 4 floods the journal.
|
||
VERBOSITY=4
|
||
# Without --metrics llama.cpp answers 501 on /metrics, which Prometheus reads
|
||
# as DOWN while /health stays 200. 0 opts out and re-breaks the scrape.
|
||
METRICS=1
|