fix(frank-zealot): 2 x 262,144 over unquantized f16 KV
Frank: "shift frank-zealot to 2 parallels of unquantized kv". Shape: CTX_SIZE 786432 -> 524288, PARALLEL 3 -> 2, CACHE_TYPE_K/V q8_0 -> f16. The per-slot window is UNCHANGED at 262,144 (the native trained window, and models.yml contextWindow), so what the exact cache buys is fidelity and what it pays is one concurrent conversation. The POOL is the term that had to move: exact KV bills 32,768 B/token at this geometry (8 full-attention layers x 4 KV heads x 256 dim, 2 B, K and V), so the old 786,432 pool asks 24,576 MiB against the 24,123 MiB the driver leaves addressable. 524,288 positions = 16,384 MiB, projected ~21,200 used / ~2,900 free from the 13:19 boot ledger (only the KV pool and one recurrent-state cell move). Also corrects a repo-wide arithmetic error: "f16 would bill 26,112 MiB" doubled the q8_0 ledger figure INCLUDING its 1/32 scale overhead. q8_0 is 1.0625 B/element, f16 exactly 2.0, so the true ratio is 1.882x. The wall stands; the number was wrong. And it corrects a false claim about the drafter: this build DOES expose --spec-draft-type-k / --spec-draft-type-v (verified against llama-server --help), so the draft cache is quantizable -- the exact-KV-plus-MTP exclusion is a measurement gap, not a law, and the 1,088 MiB quantized-draft arm is recorded as unmeasured. Seam: tests/frank_zealot_unit.bats re-pins the shape contract (2 slots, 524,288 pool, 262,144 per slot) and adds a VRAM fit gate that computes dtype x pool + the measured non-KV ledger terms against 24,123 MiB with >=1 GiB of slack -- and proves the gate REFUSES exact KV at the old pool, so passing means something. 18/18 green.
This commit is contained in:
1 parent
100b62ba22
commit
68afc6dbaf
5 files changed
+217
-141
No files matched your search
@@ -3,15 +3,16 @@
|
||||
# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 — the ternary Bonsai-2-9B, a
|
||||
# qwen35 hybrid with 32 layers (8 full-attention + 24 Gated DeltaNet) and the
|
||||
# MTP draft head INSIDE the file — via the PrismML fork build of llama.cpp
|
||||
# (vendor build 10777, the minimum that pairs MTP with a quantized V cache),
|
||||
# the ONLY loader that registers the payload's custom tensor type 142 and
|
||||
# honours its prism.hadamard.* metadata: THREE 262,144-token conversations
|
||||
# (--ctx-size 786432 --parallel 3), q8_0 K + q8_0 V under flash attention (f16
|
||||
# KV would bill 26,112 MiB of a 24,123 MiB card), the publisher's own Q8_0
|
||||
# vision projector (the BF16 one can crash on concurrent images), and MTP OFF
|
||||
# — measured, not because the head is missing: the draft context bills f16 KV
|
||||
# over the whole pool and halves decode at this shape. Log verbosity 4 ships so
|
||||
# the VRAM-fit forensics land in the journal.
|
||||
# (vendor build 10777), the ONLY loader that registers the payload's custom
|
||||
# tensor type 142 and honours its prism.hadamard.* metadata: TWO 262,144-token
|
||||
# conversations (--ctx-size 524288 --parallel 2) over an UNQUANTIZED f16 KV
|
||||
# cache. Exact KV bills 32,768 B/token here, so the pool is what pays for it —
|
||||
# the same dtype at the old 786,432 pool is 24,576 MiB against the 24,123 MiB
|
||||
# the driver leaves addressable, and the slot count is the term that moved. The
|
||||
# publisher's own Q8_0 vision projector rides along (the BF16 one can crash on
|
||||
# concurrent images), and MTP stays OFF: the head is present, but exact KV
|
||||
# leaves no room for a draft context that bills f16 KV over the whole pool. Log
|
||||
# verbosity 4 ships so the VRAM-fit forensics land in the journal.
|
||||
# This seam runs the serve script against a stub engine and asserts on the
|
||||
# argv it actually execs, so what is pinned is the command the operator gets.
|
||||
|
||||
@@ -76,6 +77,22 @@ argv_value() {
|
||||
grep -A1 -xF -- "$1" <<<"$output" | tail -1
|
||||
}
|
||||
|
||||
# kv_bytes_per_token DTYPE — bytes ONE cache costs per token at this payload's
|
||||
# geometry: 8 full-attention layers × 4 KV heads × 256 dim = 8,192 elements
|
||||
# (measured from the GGUF header: n_embd_k_gqa = n_embd_v_gqa = 1024). f16 is
|
||||
# 2 B/element; q8_0 is 1.0625 (32 codes plus one 2-byte scale per 32-block);
|
||||
# q4_0 is 0.625 (codes + scale + min per 32-block). Unknown dtype is a FAILURE,
|
||||
# never a silent zero — the fit gate below is only honest if it cannot be
|
||||
# satisfied by a rung it does not understand.
|
||||
kv_bytes_per_token() {
|
||||
case "$1" in
|
||||
f16|bf16) echo 16384 ;;
|
||||
q8_0) echo 8704 ;;
|
||||
q4_0) echo 5120 ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# run_stage — exec the CURRENT $STAGE/test.env as-is. serve_argv REWRITES
|
||||
# that file from the shipped env, so any test that mutates it (drop a line,
|
||||
# append a drafter) must exec through this, not serve_argv — which silently
|
||||
@@ -126,7 +143,7 @@ run_stage() {
|
||||
done
|
||||
}
|
||||
|
||||
@test "three conversations share the pool at a 262144 window each" {
|
||||
@test "two conversations share the pool at a 262144 window each" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --alias)" = "frank-zealot" ]
|
||||
@@ -134,15 +151,13 @@ run_stage() {
|
||||
[ "$(argv_value --host)" = "0.0.0.0" ]
|
||||
[ "$(argv_value --model)" = "$MODEL_DIR/$MODEL_STUB" ]
|
||||
[ "$(argv_value --n-gpu-layers)" -gt 0 ]
|
||||
# llama.cpp divides --ctx-size across slots, so the observable contract
|
||||
# is the per-slot window. The 9B swap exists precisely so every
|
||||
# conversation gets the model's TRAINED 262,144 (the 27B had to be cut to
|
||||
# 200,704 to fund two), and the harness contextWindow must quote 262144 —
|
||||
# the pool and the slot count are neither, and a longer prompt is a 400.
|
||||
# Three is also the ceiling: a fourth slot would push each window under the
|
||||
# fleet's worst-case compaction input, which is this endpoint's job.
|
||||
[ "$(argv_value --parallel)" -eq 3 ]
|
||||
[ "$(argv_value --ctx-size)" -eq 786432 ]
|
||||
# llama.cpp divides --ctx-size across slots, so the observable contract is
|
||||
# the per-slot window. Exact KV costs the SLOT COUNT, not the window: every
|
||||
# conversation still gets the model's TRAINED 262,144, and the harness
|
||||
# contextWindow must quote that number — the pool and the slot count are
|
||||
# neither, and a longer prompt is a 400, never a truncation.
|
||||
[ "$(argv_value --parallel)" -eq 2 ]
|
||||
[ "$(argv_value --ctx-size)" -eq 524288 ]
|
||||
local pool slots
|
||||
pool="$(argv_value --ctx-size)"; slots="$(argv_value --parallel)"
|
||||
[ $((pool % slots)) -eq 0 ]
|
||||
@@ -151,15 +166,16 @@ run_stage() {
|
||||
|
||||
@test "the KV dtype pair follows the env file and flash-attn stays pinned on" {
|
||||
# Asserted against the env rather than a literal so a retune moves one
|
||||
# file, not two. The shipped pair is q8_0/q8_0: this payload bills
|
||||
# 8 full-attention layers × 4 KV heads × 256 dim = 8,192 elements per token
|
||||
# per cache, so f16/f16 would need 26,112 MiB of the 24,123 MiB card while
|
||||
# q8_0 bills 17,408 B/token = 13,056.00 MiB at the 786,432 pool — the
|
||||
# ledger said exactly that, and it is the referee. The 24 DeltaNet layers
|
||||
# bill a fixed recurrent state per slot (452.25 MiB for three cells), which
|
||||
# is why pool arithmetic alone under-prices the shape. flash-attn is not
|
||||
# decoration: quantized V is only legal with it, and it must stay explicit
|
||||
# so a future quant swap cannot silently fall back.
|
||||
# file, not two. The shipped pair is f16/f16: exact attention state, and the
|
||||
# slot count is what makes it affordable — this payload bills 8,192 elements
|
||||
# per token per cache (8 full-attention layers × 4 KV heads × 256 dim), so
|
||||
# exact KV is 32,768 B/token = 16,384.00 MiB at the 524,288 pool, where the
|
||||
# q8_0 pair this file used to ship billed 17,408 B/token = 13,056.00 MiB at
|
||||
# the old 786,432 pool (ledger-exact both times). The 24 DeltaNet layers
|
||||
# bill a fixed recurrent state per slot (~50.25 MiB a cell), which is why
|
||||
# pool arithmetic alone under-prices the shape. flash-attn is not
|
||||
# decoration: the quantized-V fallback is only legal with it, so it stays
|
||||
# explicit and a dtype retune cannot silently fall back.
|
||||
local env_k env_v
|
||||
env_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
env_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
@@ -170,6 +186,38 @@ run_stage() {
|
||||
[ "$(argv_value --flash-attn)" = "on" ]
|
||||
}
|
||||
|
||||
@test "the shipped KV dtype and pool actually fit the card this boots on" {
|
||||
# No health gate can see this mistake: llama.cpp sizes the KV pool as
|
||||
# rate × pool-positions and cudaMalloc fails at boot ON THE LIVE ENDPOINT.
|
||||
# The fit is arithmetic off the payload geometry plus the non-KV terms
|
||||
# measured off the boot ledger (weights 2,036.71 + compute buffers 1,409.02
|
||||
# + projector 595 + 248.10 + CUDA context ≈ 460 = 4,749 MiB, dtype- and
|
||||
# slot-independent at this ubatch), plus one ~51 MiB recurrent-state cell
|
||||
# per slot. 24,123 MiB is what the driver leaves addressable (nvidia-smi
|
||||
# reports it as used + free; its total column reads 24,576).
|
||||
local env_k env_v pool slots rate_k rate_v kv total
|
||||
env_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
env_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
pool="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
slots="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
rate_k="$(kv_bytes_per_token "$env_k")"
|
||||
rate_v="$(kv_bytes_per_token "$env_v")"
|
||||
kv="$(( pool * (rate_k + rate_v) / 1048576 ))"
|
||||
total="$(( kv + 4749 + 51 * slots ))"
|
||||
# The shipped arm has to leave real room: the projector's worst case is
|
||||
# charged AFTER the engine's own fit check, so a projection that lands on
|
||||
# the ceiling is a crash waiting for the first image request.
|
||||
[ "$total" -le $(( 24123 - 1024 )) ]
|
||||
|
||||
# And the gate must be able to REFUSE: the same exact KV at the pool this
|
||||
# shape left behind is 24,576 MiB of cache alone — a boot that dies before
|
||||
# /health ever answers. Proving the gate bites is what makes passing one
|
||||
# meaningful.
|
||||
local over
|
||||
over="$(( 786432 * (rate_k + rate_v) / 1048576 + 4749 + 51 * 3 ))"
|
||||
[ "$over" -gt 24123 ]
|
||||
}
|
||||
|
||||
@test "log verbosity follows the env file" {
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
@@ -233,10 +281,11 @@ run_stage() {
|
||||
# Frank, 2026-10-18: this file DOES carry the MTP head (the boot arms prove
|
||||
# it loads and drafts, acceptance 0.798/0.617 per position, mean accepted
|
||||
# length 2.41) and it is still off, because the draft context bills its own
|
||||
# f16 KV over the WHOLE pool — 4,096 B/position × 786,432 = 3,072 MiB plus
|
||||
# buffers — and at --ubatch-size 2048 that squeeze halves decode: 167.2 →
|
||||
# 109.2 t/s. The MTP arm is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (213.5
|
||||
# t/s), documented in frank-zealot.env. The flags move as a set:
|
||||
# f16 KV over the WHOLE pool — 4,096 B/position × 524,288 = 2,048 MiB plus
|
||||
# buffers — on a trunk that already projects 21,234 MiB with exact KV. The
|
||||
# MTP arms in frank-zealot.env are all q8_0-era measurements; the doors back
|
||||
# to the drafter are one window per conversation, or KV back to q8_0 with
|
||||
# UBATCH_SIZE=1024 (213.5 t/s). The flags move as a set:
|
||||
# SPEC_TYPE=none strips --spec-type, --spec-draft-n-max AND
|
||||
# --spec-draft-model together, so a no-spec boot cannot carry half a spec
|
||||
# pair (the ROCmFPX-era baseline arm segfaulted at load doing that).
|
||||
@@ -276,7 +325,7 @@ run_stage() {
|
||||
# The Bonsai-2 text file carries no vision tensors — the publisher's Q8_0
|
||||
# projector argument IS the vision capability: no flag, no eyes. Q8_0 (not
|
||||
# BF16) is the publisher's own requirement for concurrent image requests,
|
||||
# which is what --parallel 3 means in practice.
|
||||
# which is what --parallel 2 means in practice.
|
||||
serve_argv
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$(argv_value --mmproj)" = "$MODEL_DIR/$MM_STUB" ]
|
||||
|
||||
Reference in new issue
Block a user