Frank
30e8ed55a0
The boot of record for the new shape, from the verbosity-4 journal plus
settled nvidia-smi: target KV 14,406.00 (K f16 9,408.00 + V q8_0 4,998.00),
draft KV 2,352.00 over the same 200,704 cells x 3 seqs, recurrent state
452.25, compute 1,665.00 + 424.88 (draft 480.97 + 424.86), weights 2,283.29
+ 257.66, projector 595.29 + 248.10 + 24.93, settled 22,870 used / 1,255
free with n_slots = 3 / n_ctx_slot = 200704.
Two inherited numbers did not survive it, and both are corrected in the env,
the serve rationale and the page:
- The ubatch 2048 -> 1024 step was quoted at "-~1,306 MiB". That was the
786,432 / q8_0-pair arm. At THIS shape the target's compute buffers went
2,496.13 -> 2,089.88: 406 MiB, with CUDA0 going UP 257 while CUDA_Host
went down 663. q8_0 V (4,410 MiB, not the 3,840 previously written) is
what funds the drafter, alone.
- The fit gate priced 22,428 against a real 22,870: conservative by 442
MiB, so it stays honest as written.
And one finding stated plainly: this is the first boot on this card that
misses llama.cpp's own free-memory target - the engine printed "cannot meet
free memory target of 1,867 MiB, need to reduce device memory by 102 MiB"
and settled at 1,255 MiB free, under the ~2 GiB band every previous era
held. Cheapest rungs back are CTX_SIZE=401408 or SPEC_TYPE=none, both one
line.
Probes on the same boot go in too: chat turn (448 tokens, 1,755 chars of
thinking - empty content at max_tokens 64 is the documented trap), tools
(finish_reason tool_calls, {"a":24,"b":7}), vision (512 px solid red ->
"Red", 280 prompt tokens), and the drafter's own statistics (acceptance
0.706 / 0.763, mean accepted length 2.41 / 2.53).
73 lines
4.1 KiB
Bash
73 lines
4.1 KiB
Bash
# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
|
||
# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file.
|
||
#
|
||
# ONE OWNER: shape, payload identity, eras, laws and every measured number
|
||
# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
|
||
# knobs and only the reason a reader editing THIS line cannot recover
|
||
# elsewhere. Where the two disagree, this file wins on values.
|
||
LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
|
||
# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
|
||
# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
|
||
# metadata dequantizes wrong, silently.
|
||
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
|
||
# Q8_0 IS MANDATORY: the publisher states the BF16 tower can crash llama-server
|
||
# under concurrent image requests, which is exactly --parallel 3.
|
||
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
|
||
# Served id — the harness/family name, not a fact about the payload.
|
||
# Id-keyed consumers: OpenWebUI (model row, access_grant, api_configs), models.yml.
|
||
ALIAS=frank-zealot
|
||
BIND_ADDR=0.0.0.0
|
||
LISTEN_PORT=7771
|
||
# CTX_SIZE is the POOL llama.cpp divides across slots: 602,112 / 3 = 200,704 a
|
||
# conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
|
||
# models.yml contextWindow quotes the PER-SLOT number — a longer prompt is a
|
||
# 400, never a truncation. The pool is bought, not granted: K+V at 25,088
|
||
# B/token is 14,406.00 MiB here and the MTP draft context bills its own f16 KV
|
||
# over the WHOLE pool (4,096 B/position ⇒ 2,352.00 MiB), so the old 786,432
|
||
# pool cannot carry a drafter at any KV dtype — 18,816 + 3,072 of cache alone
|
||
# against the 24,123 MiB the driver leaves addressable.
|
||
CTX_SIZE=602112
|
||
PARALLEL=3
|
||
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch (the law
|
||
# on [[frank-zealot]]); 1024×6 keeps it a clean multiple of the ubatch.
|
||
BATCH_SIZE=6144
|
||
# Ubatch is a prefill knob (decode never moves with it) and it sets the
|
||
# compute-buffer size, which decides whether a drafter fits. 1024 is the rung
|
||
# every measured MTP arm here ran on (213.5 t/s against 109.2 at 2048). Its
|
||
# VRAM credit is SHAPE-specific: the 786,432-pool arm census showed ~1,306 MiB
|
||
# back, this boot measured 2,089.88 MiB of target compute against 2,496.13 at
|
||
# ubatch 2048 — 406 MiB back, with the split moving (CUDA0 +257, CUDA_Host
|
||
# −663). The drafter is funded by CACHE_TYPE_V below, not by this knob.
|
||
UBATCH_SIZE=1024
|
||
# K exact, V in the 8-bit class: 16,384 + 8,704 = 25,088 B/token (8,192
|
||
# elements per token per cache × 2 B for f16, × 1.0625 for q8_0 — there is no
|
||
# true fp8 KV dtype in this build). That 7,680 B/token saved against exact V is
|
||
# what funds BOTH the third conversation and the drafter. Quantized V is
|
||
# illegal without --flash-attn, which the serve script pins. Rate law and the
|
||
# arithmetic: [[frank-zealot]] → Laws.
|
||
CACHE_TYPE_K=f16
|
||
CACHE_TYPE_V=q8_0
|
||
# Speculation ON (Frank, 2026-10-19): the head is in the file
|
||
# (qwen35.nextn_predict_layers = 1, blk.32.nextn.*) and rides the MAIN model's
|
||
# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
|
||
# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
|
||
# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
|
||
# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
|
||
# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
|
||
# every --spec-draft-*); half a set has segfaulted at load.
|
||
SPEC_TYPE=draft-mtp
|
||
SPEC_DRAFT_N_MAX=2
|
||
SPEC_DRAFT_N_MIN=0
|
||
SPEC_DRAFT_P_MIN=0
|
||
# The sampler recipe lives in the checkpoint (general.sampling.*) and
|
||
# llama.cpp reads it itself. Presence penalty is the one recipe value with no
|
||
# GGUF metadata key, so it must reach argv. "" is the documented opt-out.
|
||
PRESENCE_PENALTY=1.0
|
||
NGL=99
|
||
# 4 = debug: KV layout, buffer sizes and graph capture land in the journal, and
|
||
# that is what VRAM forensics read. Above 4 floods the journal.
|
||
VERBOSITY=4
|
||
# Without --metrics llama.cpp answers 501 on /metrics, which Prometheus reads
|
||
# as DOWN while /health stays 200. 0 opts out and re-breaks the scrape.
|
||
METRICS=1
|