frank-small: env arithmetic for the Ornith Q8_0 rung (f16 K + q8_0 V)

The rogue session's swap left this env reprice uncommitted in the tree
(2026-10-05): Ornith is a 33-block qwen35 hybrid with NINE full-attn
layers (18 KiB/token/half), +1,900 MiB of weights vs MiMo Q6_K — f16/f16
lands ~379 MiB free, a dead boot; pool-stepping cannot rescue it because
the compaction invariant needs slot >= 191,148. The only lever that
keeps 2 x 200,704 AND the 2-3 GiB floor is q8_0 V (~2.87 GiB free). K
stays f16. Deployer gate + seam already retargeted in 8f53df7; fetch
script manifest already flipped in 3ee88d7. MEASURE after boot; under
1.5 GiB free, UBATCH_SIZE down to 1024, never the window.
This commit is contained in:
Frank
2026-10-05 18:18:06 -07:00
parent 2930431052
commit 1862dc3c0f
+34 -10
View File
@@ -1,12 +1,34 @@
# frank-small — llama.cpp (master, CUDA sm_86) serving
# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 on the RTX 3090, :7771.
#
# 2026-10-03 (Frank, second move): the rung went Q5_K_M -> Q6_K with the pool
# HELD at 401,408. The freed third slot funded exact f16 V first; the ~3.5 GiB
# still free funds the +875 MiB of weights (file delta 7,793,710,624 -
# 6,876,124,704). Projected landing ~2.7 GiB free — inside the 2-3 GiB floor,
# so MEASURE after the boot; if it lands under 2 GiB, step the pool down one
# 12,288-token (384 MiB) increment.
# 2026-10-05 (Frank): the payload moved off bartowski/MiMo-V2.6 (Q6_K) to
# ornith-ai/Ornith-1.5-9B Q8_0 (9,786,060,384 B, LFS oid 22086870...).
# Header-verified facts that re-price the shape: qwen35 hybrid like MiMo, but
# 33 blocks (not 32) with full_attention_interval 4 => NINE full-attention
# layers (MiMo had 8), 16 Q heads / 4 KV heads, k/v length 256, native window
# 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN
# the file — capability only; no spec decode ships, same law as before).
#
# THE KV CONTRACT MOVED BACK TO f16 K + q8_0 V, and here is the arithmetic
# that forces it (2026-10-05, from the header + measured MiMo numbers):
# KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again
# for V at f16 => 36 KiB/token (MiMo was 32: one fewer full-attn layer).
# Weights +1,900 MiB (9,334 vs 7,432). Measured free at the Q6_K shape was
# 2,279 MiB, so Q8_0 at f16/f16 lands ~379 MiB free — a dead boot.
# Pool-stepping cannot rescue it: the harness compaction invariant
# (0.7 x slot + 57,344 <= slot) needs slot >= 191,148, so two slots need
# >= 13.6 GiB of f16/f16 KV even at the minimum legal window. The ONLY
# lever that keeps both the 2 x 200,704 window contract and the 2-3 GiB
# floor is q8_0 on V (26.25 KiB/token -> ~10.05 GiB KV -> ~2.87 GiB free).
# The 2026-10-03 "unquantized both halves" note was bought with the VRAM
# the third slot's death freed; the Q8 rung re-spends it. K stays f16.
# MEASURE after the boot: under ~1.5 GiB free, step UBATCH_SIZE down to 1024
# (the measured 616 MiB rung), never the window.
#
# Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the
# rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third
# slot funded exact f16 V first; the ~3.5 GiB still free funded the +875 MiB
# of weights.
#
# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
# tokens per conversation, above the 200,700 the harness compacts at — and the
@@ -99,9 +121,11 @@ PARALLEL=2
BATCH_SIZE=4096
UBATCH_SIZE=2048
# The KV contract (Frank, 2026-10-03): unquantized K AND V. --flash-attn stays
# pinned on in the serve script — it is what made the q8_0-V path cheap and it
# remains the fast path for f16 KV.
# The KV contract (2026-10-05, Q8_0 era): f16 K + q8_0 V. See the header
# arithmetic — the Q8 weights consume the VRAM that funded f16 V on
# 2026-10-03, and pool-stepping cannot restore the floor. --flash-attn stays
# pinned on in the serve script: it is the fast path for both f16 and q8_0 V
# and the only path that serves quantized V at all.
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
CACHE_TYPE_V=q8_0
NGL=99