frank-small: env arithmetic for the Ornith Q8_0 rung (f16 K + q8_0 V)
The rogue session's swap left this env reprice uncommitted in the tree (2026-10-05): Ornith is a 33-block qwen35 hybrid with NINE full-attn layers (18 KiB/token/half), +1,900 MiB of weights vs MiMo Q6_K — f16/f16 lands ~379 MiB free, a dead boot; pool-stepping cannot rescue it because the compaction invariant needs slot >= 191,148. The only lever that keeps 2 x 200,704 AND the 2-3 GiB floor is q8_0 V (~2.87 GiB free). K stays f16. Deployer gate + seam already retargeted in 8f53df7; fetch script manifest already flipped in 3ee88d7. MEASURE after boot; under 1.5 GiB free, UBATCH_SIZE down to 1024, never the window.
This commit is contained in:
+34
-10
@@ -1,12 +1,34 @@
|
||||
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
||||
# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 on the RTX 3090, :7771.
|
||||
#
|
||||
# 2026-10-03 (Frank, second move): the rung went Q5_K_M -> Q6_K with the pool
|
||||
# HELD at 401,408. The freed third slot funded exact f16 V first; the ~3.5 GiB
|
||||
# still free funds the +875 MiB of weights (file delta 7,793,710,624 -
|
||||
# 6,876,124,704). Projected landing ~2.7 GiB free — inside the 2-3 GiB floor,
|
||||
# so MEASURE after the boot; if it lands under 2 GiB, step the pool down one
|
||||
# 12,288-token (384 MiB) increment.
|
||||
# 2026-10-05 (Frank): the payload moved off bartowski/MiMo-V2.6 (Q6_K) to
|
||||
# ornith-ai/Ornith-1.5-9B Q8_0 (9,786,060,384 B, LFS oid 22086870...).
|
||||
# Header-verified facts that re-price the shape: qwen35 hybrid like MiMo, but
|
||||
# 33 blocks (not 32) with full_attention_interval 4 => NINE full-attention
|
||||
# layers (MiMo had 8), 16 Q heads / 4 KV heads, k/v length 256, native window
|
||||
# 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN
|
||||
# the file — capability only; no spec decode ships, same law as before).
|
||||
#
|
||||
# THE KV CONTRACT MOVED BACK TO f16 K + q8_0 V, and here is the arithmetic
|
||||
# that forces it (2026-10-05, from the header + measured MiMo numbers):
|
||||
# KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again
|
||||
# for V at f16 => 36 KiB/token (MiMo was 32: one fewer full-attn layer).
|
||||
# Weights +1,900 MiB (9,334 vs 7,432). Measured free at the Q6_K shape was
|
||||
# 2,279 MiB, so Q8_0 at f16/f16 lands ~379 MiB free — a dead boot.
|
||||
# Pool-stepping cannot rescue it: the harness compaction invariant
|
||||
# (0.7 x slot + 57,344 <= slot) needs slot >= 191,148, so two slots need
|
||||
# >= 13.6 GiB of f16/f16 KV even at the minimum legal window. The ONLY
|
||||
# lever that keeps both the 2 x 200,704 window contract and the 2-3 GiB
|
||||
# floor is q8_0 on V (26.25 KiB/token -> ~10.05 GiB KV -> ~2.87 GiB free).
|
||||
# The 2026-10-03 "unquantized both halves" note was bought with the VRAM
|
||||
# the third slot's death freed; the Q8 rung re-spends it. K stays f16.
|
||||
# MEASURE after the boot: under ~1.5 GiB free, step UBATCH_SIZE down to 1024
|
||||
# (the measured 616 MiB rung), never the window.
|
||||
#
|
||||
# Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the
|
||||
# rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third
|
||||
# slot funded exact f16 V first; the ~3.5 GiB still free funded the +875 MiB
|
||||
# of weights.
|
||||
#
|
||||
# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
|
||||
# tokens per conversation, above the 200,700 the harness compacts at — and the
|
||||
@@ -99,9 +121,11 @@ PARALLEL=2
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=2048
|
||||
|
||||
# The KV contract (Frank, 2026-10-03): unquantized K AND V. --flash-attn stays
|
||||
# pinned on in the serve script — it is what made the q8_0-V path cheap and it
|
||||
# remains the fast path for f16 KV.
|
||||
# The KV contract (2026-10-05, Q8_0 era): f16 K + q8_0 V. See the header
|
||||
# arithmetic — the Q8 weights consume the VRAM that funded f16 V on
|
||||
# 2026-10-03, and pool-stepping cannot restore the floor. --flash-attn stays
|
||||
# pinned on in the serve script: it is the fast path for both f16 and q8_0 V
|
||||
# and the only path that serves quantized V at all.
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
CACHE_TYPE_V=q8_0
|
||||
NGL=99
|
||||
|
||||
Reference in New Issue
Block a user