From 380aafa1cc6d715228a26763f1ab56c0d8ef0995 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 5 Oct 2026 18:31:50 -0700 Subject: [PATCH] =?UTF-8?q?frank-small:=20Frank=20overrules=20q8=5F0=20V?= =?UTF-8?q?=20=E2=80=94=20f16=20K+V,=20ubatch=201024=20first,=20ONE=20nati?= =?UTF-8?q?ve=20262144=20slot=20(the=20only=20f16/f16=20shape=20that=20fit?= =?UTF-8?q?s=20Q8=20weights=20on=2024=20GiB)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- scripts/frank-small.env | 87 ++++++++++++++++++++----------------- tests/frank_small_unit.bats | 59 +++++++++++++------------ 2 files changed, 78 insertions(+), 68 deletions(-) diff --git a/scripts/frank-small.env b/scripts/frank-small.env index 3cfafd1..a950f7b 100644 --- a/scripts/frank-small.env +++ b/scripts/frank-small.env @@ -9,21 +9,20 @@ # 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN # the file — capability only; no spec decode ships, same law as before). # -# THE KV CONTRACT MOVED BACK TO f16 K + q8_0 V, and here is the arithmetic -# that forces it (2026-10-05, from the header + measured MiMo numbers): +# THE KV CONTRACT STAYS f16 K + f16 V — NO KV QUANTIZATION (Frank, 2026-10-05, +# overruling a written q8_0-V proposal). The arithmetic that makes this hard: # KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again -# for V at f16 => 36 KiB/token (MiMo was 32: one fewer full-attn layer). -# Weights +1,900 MiB (9,334 vs 7,432). Measured free at the Q6_K shape was -# 2,279 MiB, so Q8_0 at f16/f16 lands ~379 MiB free — a dead boot. -# Pool-stepping cannot rescue it: the harness compaction invariant -# (0.7 x slot + 57,344 <= slot) needs slot >= 191,148, so two slots need -# >= 13.6 GiB of f16/f16 KV even at the minimum legal window. The ONLY -# lever that keeps both the 2 x 200,704 window contract and the 2-3 GiB -# floor is q8_0 on V (26.25 KiB/token -> ~10.05 GiB KV -> ~2.87 GiB free). -# The 2026-10-03 "unquantized both halves" note was bought with the VRAM -# the third slot's death freed; the Q8 rung re-spends it. K stays f16. -# MEASURE after the boot: under ~1.5 GiB free, step UBATCH_SIZE down to 1024 -# (the measured 616 MiB rung), never the window. +# for V at f16 => 36 KiB/token (MiMo was ~24.5 measured: one fewer +# full-attn layer AND one fewer KV head). +# Weights +1,900 MiB (9,334 vs Q6_K's 7,432). Measured free at the Q6_K +# 4096/2048 shape was 2,279 MiB; Q8_0 at f16/f16 does not fit that shape — +# the deficit is made up the way Frank ordered: UBATCH FIRST (2048 -> 1024, +# the measured 616 MiB rung), then POOL STEPS, then measure. Guess-and-check +# on the live card is the accepted method for llama.cpp sizing here: boot, +# read nvidia-smi free, step the pool (12,288-token increments, ~432 MiB +# each at the 36 KiB/token rate) until the 2-3 GiB floor holds. Compute +# buffers allocate at graph capture during BOOT, so a failure is loud and +# fast, not a surprise at the first long prompt. # # Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the # rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third @@ -69,11 +68,11 @@ LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server MODEL_FILE=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf -# Vision: the Q6_K GGUF ships no `v.*` tensors either, so the projector file -# IS the vision capability (sha256 ff348f31..., 918,166,048 B, fetched by -# scripts/download-frank-small.sh; hardlinked into this dir from MiMo-Q5KM/, -# same bytes — the Q6_K dir is self-contained). Empty string = text-only on -# purpose. +# Vision: the Ornith Q8_0 GGUF ships no `v.*` tensors either (dense-9B byte +# arithmetic: 9.79 GB ≈ 9B params x ~1.06 B/param leaves no room for a tower), +# so the projector file IS the vision capability (mmproj-Ornith-1.5-9B-BF16 +# .gguf, 921,704,672 B, LFS oid 626f9f90..., fetched by +# scripts/download-frank-small.sh). Empty string = text-only on purpose. MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf # Fleet endpoint: same port and alias as every frank-small before it, so the @@ -82,16 +81,18 @@ ALIAS=frank-small BIND_ADDR=0.0.0.0 LISTEN_PORT=7771 -# Two independent conversations. llama.cpp divides --ctx-size across -# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model -# cannot use unified KV, so each slot gets its own cache and its own window. -# 200,704 per slot is BELOW the native 262,144 — the price of paying for the -# window in full precision — and ABOVE the harness's compaction trigger, which -# is the point: a full-length request now lands in one stream instead of a 400 -# exceed_context_size_error. Cost of the third slot going away: a third -# concurrent request queues behind a released slot instead of running. -CTX_SIZE=401408 -PARALLEL=2 +# ONE conversation at the NATIVE 262,144 window (Frank, 2026-10-05). The +# two-slot shape died under Ornith and the NO-QUANTIZE law, by arithmetic: +# the KV rate is the ARCHITECTURE's, not the quant's — 9 full-attn layers x +# 4 KV heads x 256 dim x 2 B = 36 KiB/token at f16 K+V (MiMo measured 24.5), +# so 2 x 200,704 wants ~14.1 GiB of KV on top of +1,900 MiB of Q8 weights +# against 2,279 MiB measured free; ubatch 1024 claws back 616, and no +# pool step fits BOTH the 2-3 GiB floor and the harness compaction +# invariant (slot >= 191,148). One native slot fits with room; the window +# per conversation RISES 200,704 -> 262,144. The cost: a second concurrent +# request queues behind the released slot (subagent concurrency 2 -> 1). +CTX_SIZE=262144 +PARALLEL=1 # Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03, # so it ran on llama.cpp's own defaults (2048/512); the pair is now 4096/2048. @@ -113,19 +114,23 @@ PARALLEL=2 # graph VRAM on the 35B MoE (7,330 -> 8,574 MiB); this 9B hybrid has far # smaller compute tensors, so the delta should be well under that, but the # predecessor Q5_K_M shape recorded "ubatch 2048 failed the scratch boot" at -# ~3,571 MiB free. This shape ships with LESS free (2,815 MiB measured -# 2026-10-05). Compute buffers are allocated at graph capture during BOOT, so -# a failure is loud and fast, not a surprise at the first long prompt. MEASURE -# nvidia-smi free immediately after the deploy: under ~1.5 GiB, or a boot -# failure, step down to 4096/1536 (or 4096/1024, the last known-good rung). +# ~3,571 MiB free. The Q8_0 rung ships UBATCH 1024 (Frank, 2026-10-05: lower +# ubatch FIRST — the measured 616 MiB step is the first VRAM the Q8 weights +# claw back, before any pool move). Compute buffers are allocated at graph +# capture during BOOT, so a failure is loud and fast, not a surprise at the +# first long prompt. MEASURE nvidia-smi free immediately after the deploy: +# under ~1.5 GiB, or a boot failure, step the POOL down next (12,288-token +# increments, ~432 MiB each at the Ornith 36 KiB/token f16 KV rate). BATCH_SIZE=4096 -UBATCH_SIZE=2048 +UBATCH_SIZE=1024 -# The KV contract (2026-10-05, Q8_0 era): f16 K + q8_0 V. See the header -# arithmetic — the Q8 weights consume the VRAM that funded f16 V on -# 2026-10-03, and pool-stepping cannot restore the floor. --flash-attn stays -# pinned on in the serve script: it is the fast path for both f16 and q8_0 V -# and the only path that serves quantized V at all. +# The KV contract (2026-10-05, Frank): f16 K + f16 V — NO KV quantization. +# The q8_0-V arithmetic was written and OVERRULED. The Q8 weights eat the +# VRAM that funded f16 V on 2026-10-03, so the remedy ladder is UBATCH FIRST +# (2048 -> 1024, the measured 616 MiB rung), then POOL STEPS — and +# guess-and-check on the live card is the accepted method: boot, read +# nvidia-smi, step down until the floor holds. --flash-attn stays pinned on +# in the serve script: it is the fast path for f16 KV. CACHE_TYPE_K=f16 -CACHE_TYPE_V=q8_0 +CACHE_TYPE_V=f16 NGL=99 diff --git a/tests/frank_small_unit.bats b/tests/frank_small_unit.bats index 2ac4355..9dd0541 100755 --- a/tests/frank_small_unit.bats +++ b/tests/frank_small_unit.bats @@ -100,7 +100,7 @@ argv_value() { done } -@test "the shipped env serves the fleet shape: two 200704-token contexts" { +@test "the shipped env serves the fleet shape: one native 262144-token context" { serve_argv [ "$status" -eq 0 ] [ "$(argv_value --alias)" = "frank-small" ] @@ -110,49 +110,54 @@ argv_value() { [ "$(argv_value --n-gpu-layers)" -gt 0 ] # llama.cpp divides --ctx-size across slots and a mamba hybrid cannot # share one cache, so the observable contract is the per-slot window. - [ "$(argv_value --parallel)" -eq 2 ] + # 2026-10-05 (Frank): NO KV quantization forces ONE slot at the NATIVE + # window — Ornith's 9 full-attn layers price f16 KV at 36 KiB/token, and + # two 200,704 slots cannot fit the 2-3 GiB floor on 24 GiB with Q8 + # weights. The window per conversation ROSE 200,704 -> 262,144. + [ "$(argv_value --parallel)" -eq 1 ] ctx="$(argv_value --ctx-size)" par="$(argv_value --parallel)" [ $((ctx % par)) -eq 0 ] - [ $((ctx / par)) -eq 200704 ] - [ $((ctx / par)) -le 262144 ] + [ $((ctx / par)) -eq 262144 ] } -@test "the KV contract is f16 K + q8_0 V, flash-attn on" { +@test "the KV contract is unquantized both halves, flash-attn on" { serve_argv [ "$status" -eq 0 ] [ "$(argv_value --cache-type-k)" = "f16" ] - # Frank, 2026-10-05 (1862dc3): the Ornith payload has NINE full-attention - # layers (18 KiB/token/half) and +1,900 MiB of weights vs MiMo Q6_K, so - # f16/f16 lands ~379 MiB free — a dead boot; q8_0 V keeps 2 x 200,704 - # AND the 2-3 GiB floor. The flag pair is what makes this observable — a - # build without the FA path silently falls back, so both must appear in - # the argv the supervisor runs. - [ "$(argv_value --cache-type-v)" = "q8_0" ] + # Frank, 2026-10-05: NO KV quantization on this endpoint. The Q8_0 rung + # funds itself by UBATCH first (2048 -> 1024), then pool steps — a q8_0-V + # proposal was written and overruled. The flag pair is what makes this + # observable — a build without the FA path silently falls back, so both + # must appear in the argv the supervisor runs. + [ "$(argv_value --cache-type-v)" = "f16" ] [ "$(argv_value --flash-attn)" = "on" ] } @test "the KV budget leaves 2-3 GiB of the 3090 free" { - # Rate, from the GGUF geometry: 8 full-attention layers x 4 kv heads x - # 256 dim x 2 B = 16 KiB/token for K and the same for V => 32 KiB/token - # (budgeted 33 with slack). The non-KV half is MEASURED on this card on - # 2026-10-02: with the projector loaded the engine reported 22,972 MiB - # used against 14,113 MiB of KV, so weights + vision tower + DeltaNet - # state + compute buffers + CUDA graphs = 8,859 MiB => 9 GiB. Two asserts, - # both about the output the operator sees: the pool must still fit with - # 2 GiB spare, and it must be large enough that 5 GiB spare is NOT - # reachable (an undersized pool is as wrong as an oversized one). + # Rate, from the Ornith GGUF header (read from the Hub file, 2026-10-05): + # NINE full-attention layers (block_count 33, full_attention_interval 4) + # x 4 kv heads x 256 dim x 2 B = 18 KiB/token for K and the same for V + # => 36 KiB/token (budgeted 37 with slack; MiMo measured 24.5 — one + # fewer layer AND one fewer KV head). The non-KV half: Q8_0 weights + # 9,334 MiB (9,786,060,384 B) + vision tower ~1.3 GiB (mmproj 879 MiB at + # the measured ~1.5x cost law) + DeltaNet state + compute buffers + + # CUDA graphs => 11 GiB budgeted. Two asserts, both about the output + # the operator sees: the pool must still fit with 2 GiB spare, and it + # must be large enough that 5 GiB spare is NOT reachable (an undersized + # pool is as wrong as an oversized one). # Arithmetic, not a boot log: change CTX_SIZE and this bites before the - # deploy does. The same pool with the OLD 24.5 KiB/token rate would leave - # ~6 GiB free — i.e. the unquantized V is what puts this shape in the band. + # deploy does. At Ornith's rate the two-200704-slot pool (14.1 GiB KV) + # FAILS the first assert — which is exactly why the shipped shape is + # ONE native slot. ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" - kv_gib=$((ctx * 33 / 1048576)) - [ $((kv_gib + 9 + 2)) -le 24 ] - [ $((kv_gib + 9 + 5)) -gt 24 ] + kv_gib=$((ctx * 37 / 1048576)) + [ $((kv_gib + 11 + 2)) -le 24 ] + [ $((kv_gib + 11 + 5)) -gt 24 ] # And the ceiling that sets the per-slot window: two NATIVE windows cannot # be paid for in full precision on this card. native=$((2 * 262144)) - [ $((native * 33 / 1048576 + 9)) -gt 24 ] + [ $((native * 37 / 1048576 + 11)) -gt 24 ] } @test "the compute-buffer pair follows the env file and stays self-consistent" {