feat(frank-zealot): reslice to 3 x 200,704 with q8_0 V and MTP armed
Frank's order, 2026-10-19: --parallel 3 at 200,704 a conversation (pool 602,112), K at the engine default (f16), V to q8_0, batch 1024x6 / ubatch 1024, and the MTP drafter armed explicitly. The three requests are one budget equation. At 8,192 elements per token per cache the pair now bills 25,088 B/token = 14,406.00 MiB over the 602,112 pool, and the armed draft context adds its own f16 KV over that whole pool (4,096 B/position = 2,352.00 MiB) plus buffers and recurrent cells. Dropping V to the 8-bit class frees 4,410 MiB against exact V and the ubatch drop hands back ~1,306 - that is what pays for the third slot AND the drafter, neither of which exact V could fund. The serve script's written-down fallbacks move with the env so a missing env file cannot boot the retired shape, and the seam now: asserts the per-slot contract (divides evenly, <= the trained 262,144), bills an armed drafter in the VRAM fit gate, proves the gate still REFUSES both boots this shape can walk into, pins the shipped MTP set (and the absence of --spec-draft-model, which would load the trunk twice), and pins SPEC_TYPE=none as the rollback that strips the whole set.
This commit is contained in:
1 parent
129b953843
commit
25a8b00043
4 files changed
+198
-149
No files matched your search
@@ -1459,7 +1459,7 @@ LLM_ZEALOT_RUNTIME="${LLM_ZEALOT_RUNTIME:-/home/dank/GitPackages/llama-prism-b10
|
|||||||
# image path — a deploy that starts without it silently ships a text-only
|
# image path — a deploy that starts without it silently ships a text-only
|
||||||
# endpoint. Checked unconditionally, like frank-dragoon's. Q8_0 is MANDATORY,
|
# endpoint. Checked unconditionally, like frank-dragoon's. Q8_0 is MANDATORY,
|
||||||
# not a preference: the card says the BF16 tower can crash llama-server when
|
# not a preference: the card says the BF16 tower can crash llama-server when
|
||||||
# several image requests run at once, and this endpoint runs --parallel 2. Its
|
# several image requests run at once, and this endpoint runs --parallel 3. Its
|
||||||
# 868.32 MiB worst-case VRAM (595.29 model + 248.10 compute in the boot ledger)
|
# 868.32 MiB worst-case VRAM (595.29 model + 248.10 compute in the boot ledger)
|
||||||
# lands AFTER the engine's fit check, so it is not inside the q8_0 budget.
|
# lands AFTER the engine's fit check, so it is not inside the q8_0 budget.
|
||||||
LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}"
|
LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf}"
|
||||||
|
|||||||
+34
-21
@@ -11,35 +11,48 @@ LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
|
|||||||
# metadata dequantizes wrong, silently.
|
# metadata dequantizes wrong, silently.
|
||||||
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
|
MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
|
||||||
# Q8_0 IS MANDATORY: the publisher states the BF16 tower can crash llama-server
|
# Q8_0 IS MANDATORY: the publisher states the BF16 tower can crash llama-server
|
||||||
# under concurrent image requests, which is exactly --parallel 2.
|
# under concurrent image requests, which is exactly --parallel 3.
|
||||||
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
|
MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
|
||||||
# Served id — the harness/family name, not a fact about the payload.
|
# Served id — the harness/family name, not a fact about the payload.
|
||||||
# Id-keyed consumers: OpenWebUI (model row, access_grant, api_configs), models.yml.
|
# Id-keyed consumers: OpenWebUI (model row, access_grant, api_configs), models.yml.
|
||||||
ALIAS=frank-zealot
|
ALIAS=frank-zealot
|
||||||
BIND_ADDR=0.0.0.0
|
BIND_ADDR=0.0.0.0
|
||||||
LISTEN_PORT=7771
|
LISTEN_PORT=7771
|
||||||
# CTX_SIZE is the POOL llama.cpp divides across slots: 524,288 / 2 = 262,144 a
|
# CTX_SIZE is the POOL llama.cpp divides across slots: 602,112 / 3 = 200,704 a
|
||||||
# conversation, each still at the trained window. MUST stay divisible by
|
# conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
|
||||||
# PARALLEL (the seam enforces it) and models.yml contextWindow quotes the
|
# models.yml contextWindow quotes the PER-SLOT number — a longer prompt is a
|
||||||
# PER-SLOT number — a longer prompt is a 400, never a truncation. The pool
|
# 400, never a truncation. The pool is bought, not granted: K+V at 25,088
|
||||||
# cannot go back to 786,432 under exact f16 KV: 24,576 MiB of cache against the
|
# B/token is 14,406.00 MiB here and the MTP draft context bills its own f16 KV
|
||||||
# 24,123 MiB the driver leaves addressable, before a weight loads.
|
# over the WHOLE pool (4,096 B/position ⇒ 2,352.00 MiB), so the old 786,432
|
||||||
CTX_SIZE=524288
|
# pool cannot carry a drafter at any KV dtype — 18,816 + 3,072 of cache alone
|
||||||
PARALLEL=2
|
# against the 24,123 MiB the driver leaves addressable.
|
||||||
BATCH_SIZE=4096
|
CTX_SIZE=602112
|
||||||
|
PARALLEL=3
|
||||||
|
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch (the law
|
||||||
|
# on [[frank-zealot]]); 1024×6 keeps it a clean multiple of the ubatch.
|
||||||
|
BATCH_SIZE=6144
|
||||||
# Ubatch is a prefill knob (decode never moves with it) and it sets the
|
# Ubatch is a prefill knob (decode never moves with it) and it sets the
|
||||||
# compute-buffer size — which is what decides whether a drafter fits.
|
# compute-buffer size — which is what decides whether a drafter fits: 1024
|
||||||
# UBATCH_SIZE=1024 is the documented MTP arm.
|
# hands back ~1,306 MiB beside the draft context, and it is the rung the
|
||||||
UBATCH_SIZE=2048
|
# measured MTP arm ran (213.5 t/s at 1024 against 109.2 at 2048).
|
||||||
# Exact K and V: the retrieval-fidelity ask, paid with one conversation of
|
UBATCH_SIZE=1024
|
||||||
# concurrency. Rate law and the arithmetic: [[frank-zealot]] → Laws.
|
# K exact, V in the 8-bit class: 16,384 + 8,704 = 25,088 B/token (8,192
|
||||||
|
# elements per token per cache × 2 B for f16, × 1.0625 for q8_0 — there is no
|
||||||
|
# true fp8 KV dtype in this build). That 7,680 B/token saved against exact V is
|
||||||
|
# what funds BOTH the third conversation and the drafter. Quantized V is
|
||||||
|
# illegal without --flash-attn, which the serve script pins. Rate law and the
|
||||||
|
# arithmetic: [[frank-zealot]] → Laws.
|
||||||
CACHE_TYPE_K=f16
|
CACHE_TYPE_K=f16
|
||||||
CACHE_TYPE_V=f16
|
CACHE_TYPE_V=q8_0
|
||||||
# Speculation OFF — measured, not assumed: the draft context bills its own f16
|
# Speculation ON (Frank, 2026-10-19): the head is in the file
|
||||||
# KV over the WHOLE pool, and there is no room beside exact KV (the arm census
|
# (qwen35.nextn_predict_layers = 1, blk.32.nextn.*) and rides the MAIN model's
|
||||||
# is on the page). SPEC_TYPE=none strips the WHOLE set (--spec-type and every
|
# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
|
||||||
# --spec-draft-*); half a set has segfaulted at load.
|
# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
|
||||||
SPEC_TYPE=none
|
# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
|
||||||
|
# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
|
||||||
|
# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
|
||||||
|
# every --spec-draft-*); half a set has segfaulted at load.
|
||||||
|
SPEC_TYPE=draft-mtp
|
||||||
SPEC_DRAFT_N_MAX=2
|
SPEC_DRAFT_N_MAX=2
|
||||||
SPEC_DRAFT_N_MIN=0
|
SPEC_DRAFT_N_MIN=0
|
||||||
SPEC_DRAFT_P_MIN=0
|
SPEC_DRAFT_P_MIN=0
|
||||||
|
|||||||
@@ -5,33 +5,43 @@
|
|||||||
#
|
#
|
||||||
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
|
# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
|
||||||
# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
|
# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
|
||||||
# an UNQUANTIZED f16 KV cache. The 9B bills 8,192 elements per token per cache
|
# an UNQUANTIZED f16 KV cache.
|
||||||
# (8 full-attention layers instead of the 27B's 16), so exact K and V costs
|
#
|
||||||
# 32,768 B/token — 16,384.00 MiB at the 524,288 pool, which is what a 24,123 MiB
|
# 2026-10-19 (Frank): THREE 200,704-token conversations over a 602,112 pool
|
||||||
# addressable card can carry beside the trunk. Verbosity 4 ships so the VRAM-fit
|
# (--ctx-size 602112 --parallel 3), K left at the engine's own default (f16),
|
||||||
# forensics (KV layout, buffer sizes, graph capture) land in the journal.
|
# V dropped to q8_0, batch 1024×6 / ubatch 1024, and the MTP drafter ARMED
|
||||||
|
# (--spec-type draft-mtp). The 9B bills 8,192 elements per token per cache (8
|
||||||
|
# full-attention layers instead of the 27B's 16), so f16 K is 16,384 B/token and
|
||||||
|
# q8_0 V is 8,704: 25,088 B/token = 14,406.00 MiB at the 602,112 pool. That
|
||||||
|
# 7,680 B/token against exact V is exactly what funds the third conversation
|
||||||
|
# AND a draft context that bills f16 KV over the whole pool. Verbosity 4 ships
|
||||||
|
# so the VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in
|
||||||
|
# the journal.
|
||||||
#
|
#
|
||||||
# Architecture (engine ledger, not the marketing page): qwen35 hybrid, 32
|
# Architecture (engine ledger, not the marketing page): qwen35 hybrid, 32
|
||||||
# layers with qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated
|
# layers with qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated
|
||||||
# DeltaNet, 4 KV heads × 256 dim, native 262,144. The KV pool scales with the 8
|
# DeltaNet, 4 KV heads × 256 dim, native 262,144. The KV pool scales with the 8
|
||||||
# full layers (16,384.00 MiB of f16 at the 524,288 pool); the DeltaNet layers
|
# full layers (14,406.00 MiB of f16 K + q8_0 V at the 602,112 pool); the
|
||||||
# cost a fixed recurrent state per SLOT — ~50.25 MiB a cell, so 100.50 MiB for
|
# DeltaNet layers cost a fixed recurrent state per SLOT — ~50.25 MiB a cell, so
|
||||||
# two. The runtime is a hard lock, not a preference: custom GGML type 142 plus
|
# 150.75 MiB for three, and an armed depth-2 drafter triples that again. The
|
||||||
# prism.hadamard.* metadata, which only the vendor build registers (the b10776
|
# runtime is a hard lock, not a preference: custom GGML type 142 plus
|
||||||
# gate that moved the prebuilt 10754 → 10777 was MTP beside quantized V, which
|
# prism.hadamard.* metadata, which only the vendor build registers — and the
|
||||||
# exact KV now makes moot — the type-142 lock alone keeps the build pinned).
|
# b10776 gate that moved the prebuilt 10754 → 10777 (MTP beside a quantized V)
|
||||||
|
# is LOAD-BEARING again on this shape, not moot as it was under exact KV.
|
||||||
# Vision comes from the publisher's Q8_0 projector (the BF16 one can crash the
|
# Vision comes from the publisher's Q8_0 projector (the BF16 one can crash the
|
||||||
# server on concurrent images, and this endpoint runs two slots), so --mmproj is
|
# server on concurrent images, and this endpoint runs three slots), so --mmproj is
|
||||||
# what gives this endpoint eyes; MMPROJ_FILE="" is the documented text-only
|
# what gives this endpoint eyes; MMPROJ_FILE="" is the documented text-only
|
||||||
# opt-out.
|
# opt-out.
|
||||||
#
|
#
|
||||||
# Speculative decoding ships OFF, measured: the head loads and accepts well
|
# Speculative decoding ships ON (Frank, 2026-10-19). The head is IN the file and
|
||||||
# (0.798/0.617 per position, mean length 2.41) but the draft context bills its
|
# accepts well (0.798/0.617 per position, mean length 2.41, measured 2026-10-18);
|
||||||
# own f16 KV over the WHOLE pool — 4,096 B/position × 524,288 = 2,048 MiB plus
|
# what made it lose that evening was the bill, not the drafts — the draft context
|
||||||
# buffers — and the trunk already projects 21,234 MiB with exact KV. Exact KV
|
# is built against the target's WHOLE pool, so blk.32 adds 4,096 B/position
|
||||||
# and MTP cannot share this card; frank-zealot.env carries the q8_0-era arm table
|
# (2,352.00 MiB at 602,112) plus ~0.45 GiB of buffers at this ubatch, and the
|
||||||
# and the two doors back to the drafter (one window, or KV back to q8_0 at
|
# recurrent state triples. Dropping V to q8_0 (−3,840 MiB at this pool) and the
|
||||||
# --ubatch-size 1024 for 213.5 t/s).
|
# ubatch to 1024 (−~1,306 MiB of compute buffers) is how those two bills got
|
||||||
|
# paid; the arm census and the rates are on [[frank-zealot]]. SPEC_TYPE=none is
|
||||||
|
# the rollback and strips the whole flag set.
|
||||||
#
|
#
|
||||||
# Sampling: the checkpoint carries the publisher's thinking recipe as GGUF
|
# Sampling: the checkpoint carries the publisher's thinking recipe as GGUF
|
||||||
# metadata (general.sampling.* = temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
|
# metadata (general.sampling.* = temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
|
||||||
@@ -58,49 +68,46 @@ fi
|
|||||||
: "${ALIAS:=frank-zealot}"
|
: "${ALIAS:=frank-zealot}"
|
||||||
: "${BIND_ADDR:=0.0.0.0}"
|
: "${BIND_ADDR:=0.0.0.0}"
|
||||||
: "${LISTEN_PORT:=7771}"
|
: "${LISTEN_PORT:=7771}"
|
||||||
# ctx-size is the POOL llama.cpp divides across slots: 524,288 / 2 = 262,144
|
# ctx-size is the POOL llama.cpp divides across slots: 602,112 / 3 = 200,704 a
|
||||||
# per conversation — every slot still at the model's trained window. The harness
|
# conversation. The harness contextWindow MUST quote the PER-SLOT number — a
|
||||||
# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered
|
# longer prompt is answered with a 400, never truncated. The pool is priced by
|
||||||
# with a 400, never truncated. Under exact KV the pool cannot stay at 786,432:
|
# the KV bill, not by the trained window: at 25,088 B/token (f16 K + q8_0 V) the
|
||||||
# 32,768 B/token × 786,432 = 24,576 MiB, more than the card's whole addressable
|
# 602,112 pool costs 14,406.00 MiB and the armed drafter adds its own f16 KV over
|
||||||
# 24,123 MiB before a single weight loads.
|
# the whole pool (2,352.00 MiB). Exact V at this pool (18,816 MiB) or the old
|
||||||
: "${CTX_SIZE:=524288}"
|
# 786,432 pool at any dtype cannot boot inside the 24,123 MiB the driver leaves
|
||||||
: "${PARALLEL:=2}"
|
# addressable.
|
||||||
|
: "${CTX_SIZE:=602112}"
|
||||||
|
: "${PARALLEL:=3}"
|
||||||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
|
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
|
||||||
# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill
|
# The logical batch is VRAM- and throughput-neutral at a fixed ubatch, so 6144
|
||||||
# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and
|
# (1024×6) is free; the ubatch is what sets the compute-buffer size, and 1024 is
|
||||||
# throughput-neutral at a fixed ubatch. It ALSO sets the compute-buffer size,
|
# the rung that leaves room for the draft context (−~1,306 MiB against 2048, and
|
||||||
# which is the term that decides whether an MTP draft context fits beside a KV
|
# the measured MTP arm ran here: 213.5 t/s at 1024 against 109.2 at 2048).
|
||||||
# pool: 1024 hands back 1,306 MiB and was the arm that ran the drafter at three
|
: "${BATCH_SIZE:=6144}"
|
||||||
# q8_0 windows (frank-zealot.env carries the table).
|
: "${UBATCH_SIZE:=1024}"
|
||||||
: "${BATCH_SIZE:=4096}"
|
# Asymmetric KV: f16 K (the engine's own default, spelled out so the contract is
|
||||||
: "${UBATCH_SIZE:=2048}"
|
# visible in argv) + q8_0 V. 8,192 elements per token per cache (8 full-attention
|
||||||
# Exact KV: f16 K + f16 V — no quantization error anywhere in the attention
|
# layers × 4 KV heads × 256 dim) ⇒ 16,384 B/token for K and 8,704 for V (q8_0 is
|
||||||
# state. 32 KiB/token at this geometry (8 full-attention layers × 4 KV heads ×
|
# 1.0625 B/element — llama.cpp has no true fp8 KV dtype), 25,088 B/token and
|
||||||
# 256 dim × 2 B × 2 caches); 16,384.00 MiB at the 524,288 pool. That is the whole
|
# 14,406.00 MiB at the 602,112 pool. That is the halfway door between exact KV
|
||||||
# reason the slot count is 2: the same dtype at the old 786,432 pool is 24,576
|
# (32,768 B/token, which costs the third conversation and the drafter both) and
|
||||||
# MiB and dies at cudaMalloc. q8_0 K + q8_0 V (17,408 B/token, 13,056.00 MiB at
|
# the fully quantized pair (17,408 B/token, which costs retrieval fidelity in the
|
||||||
# 786,432) is the documented fallback rung and funds three windows; CACHE_TYPE_V
|
# keys); --flash-attn stays pinned because quantized V is illegal without it.
|
||||||
# alone is the halfway door. Explicit either way so the contract is visible in
|
|
||||||
# the argv, and --flash-attn stays pinned because quantized V — the fallback —
|
|
||||||
# is illegal without it (no true fp8 KV dtype exists in this build).
|
|
||||||
: "${CACHE_TYPE_K:=f16}"
|
: "${CACHE_TYPE_K:=f16}"
|
||||||
: "${CACHE_TYPE_V:=f16}"
|
: "${CACHE_TYPE_V:=q8_0}"
|
||||||
# Speculative decoding defaults OFF, measured on this card and shape: the head
|
# Speculative decoding defaults ON: the head IS in the checkpoint
|
||||||
# IS in the checkpoint (qwen35.nextn_predict_layers = 1 — MTP is a property of
|
# (qwen35.nextn_predict_layers = 1 — MTP is a property of the FILE, so the plain
|
||||||
# the FILE, so the plain PQ2_0 sibling could not run it at any flag value), it
|
# PQ2_0 sibling could not run it at any flag value) and the draft context rides
|
||||||
# loads, and it accepts well (mean accepted length 2.41) — but the draft context
|
# the MAIN model's weights, so there is no --spec-draft-model. Its bill is the
|
||||||
# bills its own f16 KV over the WHOLE pool (4,096 B/position, 2,048 MiB at the
|
# target's whole pool at 4,096 B/position plus ~0.45 GiB of buffers at this
|
||||||
# 524,288 pool) and at ubatch 2048 that squeeze halves decode: 167.2 → 109.2 t/s
|
# ubatch and a tripled recurrent state; CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024
|
||||||
# on the q8_0 3-slot shape. The draft cache dtype IS a knob (--spec-draft-type-k
|
# above are what pay for it. The draft cache dtype IS a further knob
|
||||||
# / --spec-draft-type-v, both default f16 — verified in this build's --help), so
|
# (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
|
||||||
# an armed drafter can halve its own bill; there is no --spec-draft-model because
|
# build's --help) and stays at its default here. Card gate: quantized V alongside
|
||||||
# the draft rides the main model's weights. Card gate: quantized V alongside MTP
|
# MTP needs fork build >= prism-b10776, which is why LLAMA_BIN is the 10777
|
||||||
# needs fork build >= prism-b10776, which is why LLAMA_BIN moved to the 10777
|
# prebuilt; the type-142 loader lock keeps it pinned regardless.
|
||||||
# prebuilt; exact KV removes that gate but NOT the type-142 loader lock.
|
# SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
|
||||||
# SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 is the documented MTP arm (213.5 t/s);
|
: "${SPEC_TYPE:=draft-mtp}"
|
||||||
# SPEC_TYPE=none strips the whole set (flags move together).
|
|
||||||
: "${SPEC_TYPE:=none}"
|
|
||||||
: "${SPEC_DRAFT_N_MAX:=2}"
|
: "${SPEC_DRAFT_N_MAX:=2}"
|
||||||
: "${SPEC_DRAFT_N_MIN:=0}"
|
: "${SPEC_DRAFT_N_MIN:=0}"
|
||||||
: "${SPEC_DRAFT_P_MIN:=0}"
|
: "${SPEC_DRAFT_P_MIN:=0}"
|
||||||
|
|||||||
@@ -4,15 +4,15 @@
|
|||||||
# qwen35 hybrid with 32 layers (8 full-attention + 24 Gated DeltaNet) and the
|
# qwen35 hybrid with 32 layers (8 full-attention + 24 Gated DeltaNet) and the
|
||||||
# MTP draft head INSIDE the file — via the PrismML fork build of llama.cpp
|
# MTP draft head INSIDE the file — via the PrismML fork build of llama.cpp
|
||||||
# (vendor build 10777), the ONLY loader that registers the payload's custom
|
# (vendor build 10777), the ONLY loader that registers the payload's custom
|
||||||
# tensor type 142 and honours its prism.hadamard.* metadata: TWO 262,144-token
|
# tensor type 142 and honours its prism.hadamard.* metadata: THREE 200,704-token
|
||||||
# conversations (--ctx-size 524288 --parallel 2) over an UNQUANTIZED f16 KV
|
# conversations (--ctx-size 602112 --parallel 3) over an f16 K + q8_0 V cache —
|
||||||
# cache. Exact KV bills 32,768 B/token here, so the pool is what pays for it —
|
# 25,088 B/token, 14,406.00 MiB at this pool — with the MTP drafter ARMED
|
||||||
# the same dtype at the old 786,432 pool is 24,576 MiB against the 24,123 MiB
|
# (--spec-type draft-mtp). The asymmetry IS the shape: exact V (32,768 B/token)
|
||||||
# the driver leaves addressable, and the slot count is the term that moved. The
|
# leaves nothing for a draft context that bills its own f16 KV over the WHOLE
|
||||||
# publisher's own Q8_0 vision projector rides along (the BF16 one can crash on
|
# pool (4,096 B/position, 2,352.00 MiB here), and the 786,432 pool this shape
|
||||||
# concurrent images), and MTP stays OFF: the head is present, but exact KV
|
# replaced cannot carry a drafter at any dtype inside the 24,123 MiB the driver
|
||||||
# leaves no room for a draft context that bills f16 KV over the whole pool. Log
|
# leaves addressable. The publisher's own Q8_0 vision projector rides along (the
|
||||||
# verbosity 4 ships so the VRAM-fit forensics land in the journal.
|
# BF16 one can crash on concurrent images). Log
|
||||||
# This seam runs the serve script against a stub engine and asserts on the
|
# This seam runs the serve script against a stub engine and asserts on the
|
||||||
# argv it actually execs, so what is pinned is the command the operator gets.
|
# argv it actually execs, so what is pinned is the command the operator gets.
|
||||||
|
|
||||||
@@ -143,7 +143,7 @@ run_stage() {
|
|||||||
done
|
done
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "two conversations share the pool at a 262144 window each" {
|
@test "three conversations share the pool at a 200704 window each" {
|
||||||
serve_argv
|
serve_argv
|
||||||
[ "$status" -eq 0 ]
|
[ "$status" -eq 0 ]
|
||||||
[ "$(argv_value --alias)" = "frank-zealot" ]
|
[ "$(argv_value --alias)" = "frank-zealot" ]
|
||||||
@@ -152,30 +152,36 @@ run_stage() {
|
|||||||
[ "$(argv_value --model)" = "$MODEL_DIR/$MODEL_STUB" ]
|
[ "$(argv_value --model)" = "$MODEL_DIR/$MODEL_STUB" ]
|
||||||
[ "$(argv_value --n-gpu-layers)" -gt 0 ]
|
[ "$(argv_value --n-gpu-layers)" -gt 0 ]
|
||||||
# llama.cpp divides --ctx-size across slots, so the observable contract is
|
# llama.cpp divides --ctx-size across slots, so the observable contract is
|
||||||
# the per-slot window. Exact KV costs the SLOT COUNT, not the window: every
|
# the per-slot window: harness contextWindow, Prometheus dashboards and every
|
||||||
# conversation still gets the model's TRAINED 262,144, and the harness
|
# consumer's compaction trigger read THAT number, and a longer prompt is a
|
||||||
# contextWindow must quote that number — the pool and the slot count are
|
# 400, never a truncation. Two invariants the pool/slot split has to keep:
|
||||||
# neither, and a longer prompt is a 400, never a truncation.
|
# it divides evenly (llama.cpp floors it otherwise) and a slot never exceeds
|
||||||
[ "$(argv_value --parallel)" -eq 2 ]
|
# the model's TRAINED window (there is no YaRN here, so the engine would be
|
||||||
[ "$(argv_value --ctx-size)" -eq 524288 ]
|
# quoting a window it cannot fill).
|
||||||
local pool slots
|
[ "$(argv_value --parallel)" -eq 3 ]
|
||||||
|
[ "$(argv_value --ctx-size)" -eq 602112 ]
|
||||||
|
local pool slots window
|
||||||
pool="$(argv_value --ctx-size)"; slots="$(argv_value --parallel)"
|
pool="$(argv_value --ctx-size)"; slots="$(argv_value --parallel)"
|
||||||
[ $((pool % slots)) -eq 0 ]
|
[ $((pool % slots)) -eq 0 ]
|
||||||
[ $((pool / slots)) -eq 262144 ]
|
window=$((pool / slots))
|
||||||
|
[ "$window" -eq 200704 ]
|
||||||
|
[ "$window" -le 262144 ]
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "the KV dtype pair follows the env file and flash-attn stays pinned on" {
|
@test "the KV dtype pair follows the env file and flash-attn stays pinned on" {
|
||||||
# Asserted against the env rather than a literal so a retune moves one
|
# Asserted against the env rather than a literal so a retune moves one
|
||||||
# file, not two. The shipped pair is f16/f16: exact attention state, and the
|
# file, not two. The shipped pair is ASYMMETRIC — f16 K (the engine's own
|
||||||
# slot count is what makes it affordable — this payload bills 8,192 elements
|
# default) plus q8_0 V: this payload bills 8,192 elements per token per cache
|
||||||
# per token per cache (8 full-attention layers × 4 KV heads × 256 dim), so
|
# (8 full-attention layers × 4 KV heads × 256 dim), so K costs 16,384 B/token
|
||||||
# exact KV is 32,768 B/token = 16,384.00 MiB at the 524,288 pool, where the
|
# and V 8,704 (q8_0 is 1.0625 B/element; there is no true fp8 KV dtype in
|
||||||
# q8_0 pair this file used to ship billed 17,408 B/token = 13,056.00 MiB at
|
# this build) = 25,088 B/token = 14,406.00 MiB at the 602,112 pool. Exact V
|
||||||
# the old 786,432 pool (ledger-exact both times). The 24 DeltaNet layers
|
# would bill 32,768 B/token and the all-q8_0 pair this file used to ship
|
||||||
|
# billed 17,408 — the halfway rung is what pays for the third slot AND the
|
||||||
|
# armed drafter (the fit gate below prices both). The 24 DeltaNet layers
|
||||||
# bill a fixed recurrent state per slot (~50.25 MiB a cell), which is why
|
# bill a fixed recurrent state per slot (~50.25 MiB a cell), which is why
|
||||||
# pool arithmetic alone under-prices the shape. flash-attn is not
|
# pool arithmetic alone under-prices the shape. flash-attn is not
|
||||||
# decoration: the quantized-V fallback is only legal with it, so it stays
|
# decoration: quantized V is only legal with it, so it stays explicit and a
|
||||||
# explicit and a dtype retune cannot silently fall back.
|
# dtype retune cannot silently fall back.
|
||||||
local env_k env_v
|
local env_k env_v
|
||||||
env_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
env_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||||
env_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
env_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||||
@@ -192,30 +198,47 @@ run_stage() {
|
|||||||
# The fit is arithmetic off the payload geometry plus the non-KV terms
|
# The fit is arithmetic off the payload geometry plus the non-KV terms
|
||||||
# measured off the boot ledger (weights 2,036.71 + compute buffers 1,409.02
|
# measured off the boot ledger (weights 2,036.71 + compute buffers 1,409.02
|
||||||
# + projector 595 + 248.10 + CUDA context ≈ 460 = 4,749 MiB, dtype- and
|
# + projector 595 + 248.10 + CUDA context ≈ 460 = 4,749 MiB, dtype- and
|
||||||
# slot-independent at this ubatch), plus one ~51 MiB recurrent-state cell
|
# slot-independent at ubatch 2048), plus one ~51 MiB recurrent-state cell per
|
||||||
# per slot. 24,123 MiB is what the driver leaves addressable (nvidia-smi
|
# slot. That constant is deliberately NOT credited with the ~1,306 MiB the
|
||||||
# reports it as used + free; its total column reads 24,576).
|
# shipped ubatch 1024 hands back: the gate stays conservative.
|
||||||
local env_k env_v pool slots rate_k rate_v kv total
|
# An ARMED drafter is billed, because it is the term this shape pays
|
||||||
|
# CACHE_TYPE_V=q8_0 to afford: the draft context is built against the
|
||||||
|
# TARGET's whole pool at its own f16 rate (measured 4,096 B/position at
|
||||||
|
# blk.32) plus buffers and its own recurrent cells — the measured spec-ON
|
||||||
|
# against spec-OFF delta at the 786,432 / ubatch-1024 arm was 3,524 MiB
|
||||||
|
# beside 3,072 MiB of draft KV, i.e. ~452 MiB, doubled here for the third
|
||||||
|
# slot's extra cells. 24,123 MiB is what the driver leaves addressable
|
||||||
|
# (nvidia-smi reports it as used + free; its total column reads 24,576).
|
||||||
|
local env_k env_v pool slots rate_k rate_v kv total spec f16 exact_pool exact_v
|
||||||
env_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
env_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||||
env_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
env_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||||
pool="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
pool="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||||
slots="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
slots="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||||
|
spec="$(sed -nE 's/^SPEC_TYPE=(.+)$/\1/p' "$ENV_FILE")"
|
||||||
rate_k="$(kv_bytes_per_token "$env_k")"
|
rate_k="$(kv_bytes_per_token "$env_k")"
|
||||||
rate_v="$(kv_bytes_per_token "$env_v")"
|
rate_v="$(kv_bytes_per_token "$env_v")"
|
||||||
|
f16="$(kv_bytes_per_token f16)"
|
||||||
kv="$(( pool * (rate_k + rate_v) / 1048576 ))"
|
kv="$(( pool * (rate_k + rate_v) / 1048576 ))"
|
||||||
total="$(( kv + 4749 + 51 * slots ))"
|
total="$(( kv + 4749 + 51 * slots ))"
|
||||||
|
if [ "$spec" != "none" ]; then
|
||||||
|
total="$(( total + pool * 4096 / 1048576 + 768 ))"
|
||||||
|
fi
|
||||||
# The shipped arm has to leave real room: the projector's worst case is
|
# The shipped arm has to leave real room: the projector's worst case is
|
||||||
# charged AFTER the engine's own fit check, so a projection that lands on
|
# charged AFTER the engine's own fit check, so a projection that lands on
|
||||||
# the ceiling is a crash waiting for the first image request.
|
# the ceiling is a crash waiting for the first image request.
|
||||||
[ "$total" -le $(( 24123 - 1024 )) ]
|
[ "$total" -le $(( 24123 - 1024 )) ]
|
||||||
|
|
||||||
# And the gate must be able to REFUSE: the same exact KV at the pool this
|
# And the gate must be able to REFUSE, or passing above means nothing. Two
|
||||||
# shape left behind is 24,576 MiB of cache alone — a boot that dies before
|
# boots this shape can walk straight into: the retired dtype pair at the
|
||||||
# /health ever answers. Proving the gate bites is what makes passing one
|
# retired pool (exact f16 at 786,432 = 24,576 MiB of cache alone, which died
|
||||||
# meaningful.
|
# at cudaMalloc), and the shipped pool with V moved back to f16 while the
|
||||||
local over
|
# drafter stays armed — 18,816 MiB of cache plus the draft's 2,352. Both die
|
||||||
over="$(( 786432 * (rate_k + rate_v) / 1048576 + 4749 + 51 * 3 ))"
|
# before /health ever answers, and no health gate can see either.
|
||||||
[ "$over" -gt 24123 ]
|
exact_pool="$(( 786432 * (rate_k + f16) / 1048576 + 4749 + 51 * 3 ))"
|
||||||
|
[ "$exact_pool" -gt 24123 ]
|
||||||
|
exact_v="$(( pool * (rate_k + f16) / 1048576 + 4749 + 51 * slots
|
||||||
|
+ pool * 4096 / 1048576 + 768 ))"
|
||||||
|
[ "$exact_v" -gt 24123 ]
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "log verbosity follows the env file" {
|
@test "log verbosity follows the env file" {
|
||||||
@@ -277,48 +300,54 @@ run_stage() {
|
|||||||
argv_value --ubatch-size | grep -qE '^[0-9]+$'
|
argv_value --ubatch-size | grep -qE '^[0-9]+$'
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "speculation is off: no drafter flag reaches the argv" {
|
@test "the armed drafter ships the whole MTP set and no second copy of the trunk" {
|
||||||
# Frank, 2026-10-18: this file DOES carry the MTP head (the boot arms prove
|
# Frank, 2026-10-19: the drafter is ON. The head is IN this file
|
||||||
# it loads and drafts, acceptance 0.798/0.617 per position, mean accepted
|
# (qwen35.nextn_predict_layers = 1, blk.32.nextn.*) and it accepted well
|
||||||
# length 2.41) and it is still off, because the draft context bills its own
|
# when it was measured off (0.798/0.617 per position, mean accepted length
|
||||||
# f16 KV over the WHOLE pool — 4,096 B/position × 524,288 = 2,048 MiB plus
|
# 2.41); what it now pays for is CACHE_TYPE_V=q8_0 plus UBATCH_SIZE=1024,
|
||||||
# buffers — on a trunk that already projects 21,234 MiB with exact KV. The
|
# which is why the dtype pair and the batch pair are pinned next door.
|
||||||
# MTP arms in frank-zealot.env are all q8_0-era measurements; the doors back
|
# The observable contract is the SET: --spec-type plus every --spec-draft-*
|
||||||
# to the drafter are one window per conversation, or KV back to q8_0 with
|
# the env names, and NO --spec-draft-model — the MTP draft rides the MAIN
|
||||||
# UBATCH_SIZE=1024 (213.5 t/s). The flags move as a set:
|
# model's weights, so a draft file would load the trunk a second time.
|
||||||
# SPEC_TYPE=none strips --spec-type, --spec-draft-n-max AND
|
local n_max
|
||||||
# --spec-draft-model together, so a no-spec boot cannot carry half a spec
|
n_max="$(sed -nE 's/^SPEC_DRAFT_N_MAX=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||||
# pair (the ROCmFPX-era baseline arm segfaulted at load doing that).
|
|
||||||
serve_argv
|
serve_argv
|
||||||
[ "$status" -eq 0 ]
|
[ "$status" -eq 0 ]
|
||||||
|
[ "$(argv_value --spec-type)" = "draft-mtp" ]
|
||||||
|
[ "$(argv_value --spec-draft-n-max)" = "$n_max" ]
|
||||||
|
[ "$(argv_value --spec-draft-n-min)" = "0" ]
|
||||||
|
[ "$(argv_value --spec-draft-p-min)" = "0" ]
|
||||||
|
! grep -qxF -- '--spec-draft-model' <<<"$output"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "the rollback knob strips the entire spec set" {
|
||||||
|
# SPEC_TYPE=none is the documented retreat (and the shape this endpoint
|
||||||
|
# shipped before 2026-10-19). The law it exists to obey: the flags move as a
|
||||||
|
# SET — a no-spec boot that keeps --spec-draft-n-max, or a baseline arm that
|
||||||
|
# hands over --model-draft with no --spec-type, has segfaulted at load with
|
||||||
|
# no diagnostic. So the rollback is proven by ABSENCE of all of them.
|
||||||
|
serve_argv
|
||||||
|
sed -i 's/^SPEC_TYPE=.*/SPEC_TYPE=none/' "$STAGE/test.env"
|
||||||
|
run_stage
|
||||||
|
[ "$status" -eq 0 ]
|
||||||
! grep -qxF -- '--spec-type' <<<"$output"
|
! grep -qxF -- '--spec-type' <<<"$output"
|
||||||
! grep -qxF -- '--spec-draft-n-max' <<<"$output"
|
! grep -qxF -- '--spec-draft-n-max' <<<"$output"
|
||||||
|
! grep -qxF -- '--spec-draft-n-min' <<<"$output"
|
||||||
|
! grep -qxF -- '--spec-draft-p-min' <<<"$output"
|
||||||
! grep -qxF -- '--spec-draft-model' <<<"$output"
|
! grep -qxF -- '--spec-draft-model' <<<"$output"
|
||||||
grep -qE '^SPEC_TYPE=none$' "$ENV_FILE"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "naming a drafter ships the MTP set, with or without a separate draft file" {
|
@test "an external drafter still reaches the engine as a complete set" {
|
||||||
# The documented MTP arm (frank-zealot.env): SPEC_TYPE=draft-mtp ALONE is
|
# The other arm the ROCmFPX-era crash was about: a named draft FILE must
|
||||||
# legal here because the head is IN the checkpoint and the draft context
|
# arrive alongside --spec-type and the depth recipe, never alone. This
|
||||||
# rides the main model's weights — so --spec-draft-model must NOT appear,
|
# endpoint does not use it (the head is in the trunk), so the knob ships
|
||||||
# or the trunk would be loaded twice.
|
# empty and only this test sets it.
|
||||||
serve_argv
|
serve_argv
|
||||||
{
|
|
||||||
echo 'SPEC_TYPE=draft-mtp'
|
|
||||||
echo 'SPEC_DRAFT_N_MAX=2'
|
|
||||||
} >> "$STAGE/test.env"
|
|
||||||
run_stage
|
|
||||||
[ "$status" -eq 0 ]
|
|
||||||
[ "$(argv_value --spec-type)" = "draft-mtp" ]
|
|
||||||
[ "$(argv_value --spec-draft-n-max)" = "2" ]
|
|
||||||
! grep -qxF -- '--spec-draft-model' <<<"$output"
|
|
||||||
|
|
||||||
# And an external drafter still reaches the engine as a complete set — the
|
|
||||||
# pair the ROCmFPX-era crash was about.
|
|
||||||
echo "SPEC_DRAFT_FILE=$MODEL_DIR/$DRAFT_STUB" >> "$STAGE/test.env"
|
echo "SPEC_DRAFT_FILE=$MODEL_DIR/$DRAFT_STUB" >> "$STAGE/test.env"
|
||||||
run_stage
|
run_stage
|
||||||
[ "$status" -eq 0 ]
|
[ "$status" -eq 0 ]
|
||||||
[ "$(argv_value --spec-draft-model)" = "$MODEL_DIR/$DRAFT_STUB" ]
|
[ "$(argv_value --spec-draft-model)" = "$MODEL_DIR/$DRAFT_STUB" ]
|
||||||
|
[ "$(argv_value --spec-type)" = "draft-mtp" ]
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "vision ships by default and MMPROJ_FILE empty opts out of the image path" {
|
@test "vision ships by default and MMPROJ_FILE empty opts out of the image path" {
|
||||||
|
|||||||
Reference in new issue
Block a user