diff --git a/scripts/frank-small.env b/scripts/frank-small.env index caab099..3cfafd1 100644 --- a/scripts/frank-small.env +++ b/scripts/frank-small.env @@ -1,12 +1,34 @@ # frank-small — llama.cpp (master, CUDA sm_86) serving # ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 on the RTX 3090, :7771. # -# 2026-10-03 (Frank, second move): the rung went Q5_K_M -> Q6_K with the pool -# HELD at 401,408. The freed third slot funded exact f16 V first; the ~3.5 GiB -# still free funds the +875 MiB of weights (file delta 7,793,710,624 - -# 6,876,124,704). Projected landing ~2.7 GiB free — inside the 2-3 GiB floor, -# so MEASURE after the boot; if it lands under 2 GiB, step the pool down one -# 12,288-token (384 MiB) increment. +# 2026-10-05 (Frank): the payload moved off bartowski/MiMo-V2.6 (Q6_K) to +# ornith-ai/Ornith-1.5-9B Q8_0 (9,786,060,384 B, LFS oid 22086870...). +# Header-verified facts that re-price the shape: qwen35 hybrid like MiMo, but +# 33 blocks (not 32) with full_attention_interval 4 => NINE full-attention +# layers (MiMo had 8), 16 Q heads / 4 KV heads, k/v length 256, native window +# 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN +# the file — capability only; no spec decode ships, same law as before). +# +# THE KV CONTRACT MOVED BACK TO f16 K + q8_0 V, and here is the arithmetic +# that forces it (2026-10-05, from the header + measured MiMo numbers): +# KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again +# for V at f16 => 36 KiB/token (MiMo was 32: one fewer full-attn layer). +# Weights +1,900 MiB (9,334 vs 7,432). Measured free at the Q6_K shape was +# 2,279 MiB, so Q8_0 at f16/f16 lands ~379 MiB free — a dead boot. +# Pool-stepping cannot rescue it: the harness compaction invariant +# (0.7 x slot + 57,344 <= slot) needs slot >= 191,148, so two slots need +# >= 13.6 GiB of f16/f16 KV even at the minimum legal window. The ONLY +# lever that keeps both the 2 x 200,704 window contract and the 2-3 GiB +# floor is q8_0 on V (26.25 KiB/token -> ~10.05 GiB KV -> ~2.87 GiB free). +# The 2026-10-03 "unquantized both halves" note was bought with the VRAM +# the third slot's death freed; the Q8 rung re-spends it. K stays f16. +# MEASURE after the boot: under ~1.5 GiB free, step UBATCH_SIZE down to 1024 +# (the measured 616 MiB rung), never the window. +# +# Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the +# rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third +# slot funded exact f16 V first; the ~3.5 GiB still free funded the +875 MiB +# of weights. # # 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704 # tokens per conversation, above the 200,700 the harness compacts at — and the @@ -99,9 +121,11 @@ PARALLEL=2 BATCH_SIZE=4096 UBATCH_SIZE=2048 -# The KV contract (Frank, 2026-10-03): unquantized K AND V. --flash-attn stays -# pinned on in the serve script — it is what made the q8_0-V path cheap and it -# remains the fast path for f16 KV. +# The KV contract (2026-10-05, Q8_0 era): f16 K + q8_0 V. See the header +# arithmetic — the Q8 weights consume the VRAM that funded f16 V on +# 2026-10-03, and pool-stepping cannot restore the floor. --flash-attn stays +# pinned on in the serve script: it is the fast path for both f16 and q8_0 V +# and the only path that serves quantized V at all. CACHE_TYPE_K=f16 -CACHE_TYPE_V=f16 +CACHE_TYPE_V=q8_0 NGL=99