From 6584c059d6235901c4e31930d0a0a4634e947f12 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 3 Oct 2026 02:04:54 -0700 Subject: [PATCH] frank-small: two wide unquantized contexts (2 x 200,704) on the 3090 Frank's order, two messages: "extend the max context of frank-small to 200700 per individual stream without further quantization", then "reduce it to 2 streams for a total of 401400-ish tokens, set ubatch to 1k", then "unquantize the KV if we have extra space". CTX_SIZE 540672 -> 401408 with PARALLEL 3 -> 2 gives n_ctx_slot 200,704: above the 200,700 the harness compacts at, so a full-length request lands in one stream instead of a 400 exceed_context_size_error. The pool must divide evenly across slots and stay 256-aligned for the FA path, which is why it is 401,408 and not 401,400. CACHE_TYPE_V q8_0 -> f16: the freed third slot is exactly what pays for unquantized values (24.5 -> 32 KiB/token), and the budget lands back in the 2-3 GiB band Frank set on 2026-10-02 instead of leaving ~6 GiB of the card idle. The asymmetric-KV contract that created this endpoint is spent with it; both --cache-type flags stay explicit. --batch-size/--ubatch-size are now passed (2048/1024). The endpoint had never passed either, so it ran on engine defaults nobody wrote down. Franksdank-Runbook: serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md --- scripts/frank-small.env | 76 ++++++++++++++++++++++-------------- scripts/frank-small.service | 2 +- scripts/serve-frank-small.sh | 37 +++++++++++------- tests/frank_small_unit.bats | 69 +++++++++++++++++++------------- 4 files changed, 112 insertions(+), 72 deletions(-) diff --git a/scripts/frank-small.env b/scripts/frank-small.env index f5c2537..da5b7f3 100644 --- a/scripts/frank-small.env +++ b/scripts/frank-small.env @@ -1,32 +1,42 @@ # frank-small — llama.cpp (master, CUDA sm_86) serving # bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M on the RTX 3090, :7771. # -# 2026-10-02 (Frank): THREE contexts, vision on, and a hard instruction to -# leave 2-3 GiB of the card free. Both new demands cost memory, so the per-slot -# window shrinks — see the budget below. +# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704 +# tokens per conversation, above the 200,700 the harness compacts at — and the +# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same +# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given +# up is exactly the VRAM that pays for unquantized V. No further quantization +# anywhere on this endpoint. # -# Two measurements on this card, this runtime, price the terms: +# Three measurements on this card, this runtime, price the terms: # 2 x 262,144, no projector: 20,190 MiB used / 4,386 free # KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 + # DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB # 3 x 196,608, projector on: 22,972 MiB used / 1,153 free # KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB -# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608 -# per slot left only 1.1 GiB free — outside the band. The pool therefore is -# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted. -# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and -# the number the bats budget gate reproduces (it prices non-KV at 9 GiB). +# 3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02 +# shape) => non-KV = 8,748 MiB +# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V +# (32 KiB/token) this pool is 12,544 MiB, so the prediction is +# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02 +# shape hit, bought with precision instead of a slot. The 1024 ubatch +# re-prices the compute buffers upward, so MEASURE after any change: if the +# boot lands under 2 GiB free, step the pool down in 12,288-token +# (384 MiB at this rate) increments. # # KV arithmetic: attention KV exists only in the 8 full-attention layers -# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token -# (34 B per 32-element block) => 24.5 KiB/token, so 12,288 tokens (=4096/slot) -# is one 293 MiB step — that is the ladder to walk if this moves again. The -# native 262,144 per slot is NOT reachable three times: 3 x 262,144 x -# 24.5 KiB = 18.4 GiB of KV before anything else is loaded. +# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for +# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that +# is the ladder to walk if this moves again. Two native windows (2 x 262,144) +# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV +# against a card with ~15.7 GiB of KV room once weights, the vision tower, the +# DeltaNet states and the graphs are loaded. 200,704 per stream is what full +# precision buys; q8_0 on V would buy 262,144 instead. # -# Why V stays q8_0 and K stays f16: full-precision V would add 8 KiB/token — -# +4.5 GiB on this pool — which is what pushes the card past its 24 GiB. K is -# the tensor the asymmetric-KV contract exists to protect. +# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01 +# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit, +# and the third slot is gone. The flags stay explicit so the choice is visible +# in the argv either way. LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf @@ -41,20 +51,28 @@ ALIAS=frank-small BIND_ADDR=0.0.0.0 LISTEN_PORT=7771 -# Three independent conversations. llama.cpp divides --ctx-size across +# Two independent conversations. llama.cpp divides --ctx-size across # --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model # cannot use unified KV, so each slot gets its own cache and its own window. -# 180,224 per slot is BELOW the native 262,144, and that is the cost of the -# third slot: a longer single prompt needs compaction instead of one shot. -# Consequence worth knowing: frank-primary's compaction trigger is 183,501 -# tokens, which no longer fits a frank-small slot — see the runbook's -# "three slots vs compaction" note. -CTX_SIZE=540672 -PARALLEL=3 +# 200,704 per slot is BELOW the native 262,144 — the price of paying for the +# window in full precision — and ABOVE the harness's compaction trigger, which +# is the point: a full-length request now lands in one stream instead of a 400 +# exceed_context_size_error. Cost of the third slot going away: a third +# concurrent request queues behind a released slot instead of running. +CTX_SIZE=401408 +PARALLEL=2 -# The KV contract (Frank, 2026-10-01 night): unquantized K, q8_0 V. -# q8_0 on V is only served by the flash-attention path; the serve script -# pins --flash-attn on and must never lose it. +# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03, +# so it ran on llama.cpp's own defaults (2048/512); the fleet now asks for a +# 1024 ubatch on this shape. Both are tunables because the pair is a +# measurement, not a law: on the 27B sibling 512 was the prefill peak and every +# step up cost GiB of compute buffer for no speed. +BATCH_SIZE=2048 +UBATCH_SIZE=1024 + +# The KV contract (Frank, 2026-10-03): unquantized K AND V. --flash-attn stays +# pinned on in the serve script — it is what made the q8_0-V path cheap and it +# remains the fast path for f16 KV. CACHE_TYPE_K=f16 -CACHE_TYPE_V=q8_0 +CACHE_TYPE_V=f16 NGL=99 diff --git a/scripts/frank-small.service b/scripts/frank-small.service index f8a242c..02f7a5b 100644 --- a/scripts/frank-small.service +++ b/scripts/frank-small.service @@ -1,5 +1,5 @@ [Unit] -Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 3x180224, f16 K + q8_0 V, mmproj vision) on the RTX 3090 (dankclaw, frank-small) +Description=llama.cpp MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (qwen35 hybrid, 2x200704, unquantized f16 KV, mmproj vision) on the RTX 3090 (dankclaw, frank-small) After=network-online.target Wants=network-online.target diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index 300b55b..9069b97 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -4,10 +4,10 @@ # # 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound # safetensors to the llama.cpp GGUF path, because llama.cpp is the only -# runtime on this host that can serve the MiMo qwen35 hybrid with -# ASYMMETRIC KV: --cache-type-k f16 (unquantized keys) + --cache-type-v -# q8_0 (quantized values), which is Frank's explicit KV contract for this -# endpoint. SGLang's --kv-cache-dtype only moves K and V together. +# runtime on this host that moves K and V independently — the asymmetric +# f16-K + q8_0-V contract that endpoint was built for. 2026-10-03 (Frank): +# the pair is now f16/f16 — two wide slots freed the VRAM, so the values are +# unquantized too. The per-tensor flags stay explicit either way. # # Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated # DeltaNet, native 262144 window. KV lives only in the 8 attention layers; @@ -37,14 +37,20 @@ fi : "${ALIAS:=frank-small}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7771}" -# ctx-size is the POOL llama.cpp divides across slots: 540672 / 3 gives -# 180224 per slot — three independent conversations (the native 262144 no -# longer fits three times on a 24 GiB card once the vision tower is loaded; -# frank-small.env carries the two measurements that set this number). -: "${CTX_SIZE:=540672}" -: "${PARALLEL:=3}" +# ctx-size is the POOL llama.cpp divides across slots: 401408 / 2 gives +# 200,704 per slot — two independent conversations, each one wide enough to +# hold a whole frank-primary-sized prompt in a single shot (the native 262,144 +# no longer fits twice on a 24 GiB card once the vision tower is loaded; +# frank-small.env carries the three measurements that set this number). +: "${CTX_SIZE:=401408}" +: "${PARALLEL:=2}" +# Compute-buffer geometry. Before 2026-10-03 this endpoint never passed either +# flag, so it ran on llama.cpp's own defaults (2048/512); they are tunables now +# because the 1024 ubatch is what this shape was booted with. +: "${BATCH_SIZE:=2048}" +: "${UBATCH_SIZE:=1024}" : "${CACHE_TYPE_K:=f16}" -: "${CACHE_TYPE_V:=q8_0}" +: "${CACHE_TYPE_V:=f16}" : "${NGL:=99}" # Vision projector. No colon on purpose: an explicitly empty value means # "text-only, on purpose" and must NOT be replaced by the default path. @@ -66,9 +72,10 @@ fi # open after boot; make sure it exists before the server needs it. nvidia-modprobe -u -c=0 2>/dev/null || true -# q8_0 on V REQUIRES flash attention (the quantized-value kernels only exist -# on the FA path); an FA-less build silently falls back and the KV contract -# in frank-small.env becomes a lie, so pin the flag pair here. +# Flash attention stays pinned on: it is the fast path for f16 KV, and it is +# the only path that could serve quantized V at all — so if CACHE_TYPE_V is +# ever set back to q8_0 this flag must not be lost, or the build falls back +# silently and the KV contract in frank-small.env becomes a lie. ARGS=( --model "$MODEL_FILE" --alias "$ALIAS" @@ -76,6 +83,8 @@ ARGS=( --port "$LISTEN_PORT" --ctx-size "$CTX_SIZE" --parallel "$PARALLEL" + --batch-size "$BATCH_SIZE" + --ubatch-size "$UBATCH_SIZE" --n-gpu-layers "$NGL" --flash-attn on --cache-type-k "$CACHE_TYPE_K" diff --git a/tests/frank_small_unit.bats b/tests/frank_small_unit.bats index da65a8b..41a2156 100755 --- a/tests/frank_small_unit.bats +++ b/tests/frank_small_unit.bats @@ -2,11 +2,11 @@ # Seam: the frank-small llama.cpp root unit on dankclaw. # frank-small is the RTX 3090 endpoint; since 2026-10-01 night it serves # bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M via a CUDA (sm_86) build of -# llama.cpp master, chosen for the ASYMMETRIC KV Frank asked for: K f16 -# (unquantized) + V q8_0, which only llama.cpp exposes per-tensor. Since -# 2026-10-02 it runs THREE contexts and carries a vision projector, and the -# card must keep 2-3 GiB free. This seam runs the serve script against a stub -# engine and asserts on the argv it actually execs, so what is pinned is the +# llama.cpp master, chosen because it is the only runtime here that moves K and +# V independently. Since 2026-10-03 it runs TWO unquantized contexts of +# 200,704 tokens each and carries a vision projector, and the card must keep +# 2-3 GiB free. This seam runs the serve script against a stub engine and +# asserts on the argv it actually execs, so what is pinned is the # command the operator gets. load helpers @@ -98,7 +98,7 @@ argv_value() { done } -@test "the shipped env serves the fleet shape: three 180224-token contexts" { +@test "the shipped env serves the fleet shape: two 200704-token contexts" { serve_argv [ "$status" -eq 0 ] [ "$(argv_value --alias)" = "frank-small" ] @@ -108,43 +108,56 @@ argv_value() { [ "$(argv_value --n-gpu-layers)" -gt 0 ] # llama.cpp divides --ctx-size across slots and a mamba hybrid cannot # share one cache, so the observable contract is the per-slot window. - [ "$(argv_value --parallel)" -eq 3 ] + [ "$(argv_value --parallel)" -eq 2 ] ctx="$(argv_value --ctx-size)" par="$(argv_value --parallel)" [ $((ctx % par)) -eq 0 ] - [ $((ctx / par)) -eq 180224 ] + [ $((ctx / par)) -eq 200704 ] [ $((ctx / par)) -le 262144 ] } -@test "the KV contract is asymmetric: unquantized K, q8_0 V, flash-attn on" { +@test "the KV contract is unquantized both halves, flash-attn on" { serve_argv [ "$status" -eq 0 ] [ "$(argv_value --cache-type-k)" = "f16" ] - [ "$(argv_value --cache-type-v)" = "q8_0" ] - # q8_0 on V only exists on the flash-attention path; without this flag - # pair the KV contract in the env file is a lie the engine never sees. + # Frank, 2026-10-03: the values are unquantized too. The flag pair is what + # makes this observable — a build without the FA path silently falls back, + # so both must appear in the argv the supervisor runs. + [ "$(argv_value --cache-type-v)" = "f16" ] [ "$(argv_value --flash-attn)" = "on" ] } @test "the KV budget leaves 2-3 GiB of the 3090 free" { - # Rate: K f16 (16 KiB/token) + V q8_0 (~8.5, budgeted 25 with slack). - # The non-KV half is MEASURED on this card on 2026-10-02: with the - # projector loaded the engine reported 22,972 MiB used against 14,113 - # MiB of KV, so weights + vision tower + DeltaNet state + compute - # buffers + CUDA graphs = 8,859 MiB => 9 GiB. Two asserts, both about - # the output the operator sees: the pool must still fit with 2 GiB - # spare, and it must be large enough that 5 GiB spare is NOT reachable - # (an undersized pool is as wrong as an oversized one). Arithmetic, not - # a boot log: change CTX_SIZE and this bites before the deploy does — - # it rejects the 3 x 196,608 pool that measured only 1.1 GiB free. The - # same pool with f16 V (34 KiB/token) lands past the card, which is why - # the contract is asymmetric. + # Rate, from the GGUF geometry: 8 full-attention layers x 4 kv heads x + # 256 dim x 2 B = 16 KiB/token for K and the same for V => 32 KiB/token + # (budgeted 33 with slack). The non-KV half is MEASURED on this card on + # 2026-10-02: with the projector loaded the engine reported 22,972 MiB + # used against 14,113 MiB of KV, so weights + vision tower + DeltaNet + # state + compute buffers + CUDA graphs = 8,859 MiB => 9 GiB. Two asserts, + # both about the output the operator sees: the pool must still fit with + # 2 GiB spare, and it must be large enough that 5 GiB spare is NOT + # reachable (an undersized pool is as wrong as an oversized one). + # Arithmetic, not a boot log: change CTX_SIZE and this bites before the + # deploy does. The same pool with the OLD 24.5 KiB/token rate would leave + # ~6 GiB free — i.e. the unquantized V is what puts this shape in the band. ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" - kv_gib=$((ctx * 25 / 1048576)) + kv_gib=$((ctx * 33 / 1048576)) [ $((kv_gib + 9 + 2)) -le 24 ] [ $((kv_gib + 9 + 5)) -gt 24 ] - f16v_gib=$((ctx * 34 / 1048576)) - [ $((f16v_gib + 9)) -gt 24 ] + # And the ceiling that sets the per-slot window: two NATIVE windows cannot + # be paid for in full precision on this card. + native=$((2 * 262144)) + [ $((native * 33 / 1048576 + 9)) -gt 24 ] +} + +@test "the compute-buffer pair is explicit" { + # This endpoint ran on llama.cpp's own defaults until 2026-10-03; the + # fleet now pins 2048/1024 so the number a benchmark attributes the + # prefill rate to is a repo value, not an engine default nobody wrote down. + serve_argv + [ "$status" -eq 0 ] + [ "$(argv_value --batch-size)" = "2048" ] + [ "$(argv_value --ubatch-size)" = "1024" ] } @test "no speculative draft ships by default" { @@ -170,7 +183,7 @@ argv_value() { ! grep -qxF -- '--mmproj' <<<"$output" } -@test "the fleet reasoning contract survives the reslice to three contexts" { +@test "the fleet reasoning contract survives the reslice to two wide contexts" { serve_argv [ "$status" -eq 0 ] # deepseek format = thinking lands in reasoning_content, the field the