frank-small: Frank overrules q8_0 V — f16 K+V, ubatch 1024 first, ONE native 262144 slot (the only f16/f16 shape that fits Q8 weights on 24 GiB)

scripts/frank-small.env

@@ -9,21 +9,20 @@
 # 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN
 # the file — capability only; no spec decode ships, same law as before).
 #
-# THE KV CONTRACT MOVED BACK TO f16 K + q8_0 V, and here is the arithmetic
-# that forces it (2026-10-05, from the header + measured MiMo numbers):
+# THE KV CONTRACT STAYS f16 K + f16 V — NO KV QUANTIZATION (Frank, 2026-10-05,
+# overruling a written q8_0-V proposal). The arithmetic that makes this hard:
 #   KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again
-#   for V at f16 => 36 KiB/token (MiMo was 32: one fewer full-attn layer).
-#   Weights +1,900 MiB (9,334 vs 7,432). Measured free at the Q6_K shape was
-#   2,279 MiB, so Q8_0 at f16/f16 lands ~379 MiB free — a dead boot.
-#   Pool-stepping cannot rescue it: the harness compaction invariant
-#   (0.7 x slot + 57,344 <= slot) needs slot >= 191,148, so two slots need
-#   >= 13.6 GiB of f16/f16 KV even at the minimum legal window. The ONLY
-#   lever that keeps both the 2 x 200,704 window contract and the 2-3 GiB
-#   floor is q8_0 on V (26.25 KiB/token -> ~10.05 GiB KV -> ~2.87 GiB free).
-#   The 2026-10-03 "unquantized both halves" note was bought with the VRAM
-#   the third slot's death freed; the Q8 rung re-spends it. K stays f16.
-# MEASURE after the boot: under ~1.5 GiB free, step UBATCH_SIZE down to 1024
-# (the measured 616 MiB rung), never the window.
+#   for V at f16 => 36 KiB/token (MiMo was ~24.5 measured: one fewer
+#   full-attn layer AND one fewer KV head).
+#   Weights +1,900 MiB (9,334 vs Q6_K's 7,432). Measured free at the Q6_K
+#   4096/2048 shape was 2,279 MiB; Q8_0 at f16/f16 does not fit that shape —
+#   the deficit is made up the way Frank ordered: UBATCH FIRST (2048 -> 1024,
+#   the measured 616 MiB rung), then POOL STEPS, then measure. Guess-and-check
+#   on the live card is the accepted method for llama.cpp sizing here: boot,
+#   read nvidia-smi free, step the pool (12,288-token increments, ~432 MiB
+#   each at the 36 KiB/token rate) until the 2-3 GiB floor holds. Compute
+#   buffers allocate at graph capture during BOOT, so a failure is loud and
+#   fast, not a surprise at the first long prompt.
 #
 # Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the
 # rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third
@@ -69,11 +68,11 @@
 LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
 MODEL_FILE=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf
 
-# Vision: the Q6_K GGUF ships no `v.*` tensors either, so the projector file
-# IS the vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
-# scripts/download-frank-small.sh; hardlinked into this dir from MiMo-Q5KM/,
-# same bytes — the Q6_K dir is self-contained). Empty string = text-only on
-# purpose.
+# Vision: the Ornith Q8_0 GGUF ships no `v.*` tensors either (dense-9B byte
+# arithmetic: 9.79 GB ≈ 9B params x ~1.06 B/param leaves no room for a tower),
+# so the projector file IS the vision capability (mmproj-Ornith-1.5-9B-BF16
+# .gguf, 921,704,672 B, LFS oid 626f9f90..., fetched by
+# scripts/download-frank-small.sh). Empty string = text-only on purpose.
 MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf
 
 # Fleet endpoint: same port and alias as every frank-small before it, so the
@@ -82,16 +81,18 @@ ALIAS=frank-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7771
 
-# Two independent conversations. llama.cpp divides --ctx-size across
-# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
-# cannot use unified KV, so each slot gets its own cache and its own window.
-# 200,704 per slot is BELOW the native 262,144 — the price of paying for the
-# window in full precision — and ABOVE the harness's compaction trigger, which
-# is the point: a full-length request now lands in one stream instead of a 400
-# exceed_context_size_error. Cost of the third slot going away: a third
-# concurrent request queues behind a released slot instead of running.
-CTX_SIZE=401408
-PARALLEL=2
+# ONE conversation at the NATIVE 262,144 window (Frank, 2026-10-05). The
+# two-slot shape died under Ornith and the NO-QUANTIZE law, by arithmetic:
+# the KV rate is the ARCHITECTURE's, not the quant's — 9 full-attn layers x
+# 4 KV heads x 256 dim x 2 B = 36 KiB/token at f16 K+V (MiMo measured 24.5),
+# so 2 x 200,704 wants ~14.1 GiB of KV on top of +1,900 MiB of Q8 weights
+# against 2,279 MiB measured free; ubatch 1024 claws back 616, and no
+# pool step fits BOTH the 2-3 GiB floor and the harness compaction
+# invariant (slot >= 191,148). One native slot fits with room; the window
+# per conversation RISES 200,704 -> 262,144. The cost: a second concurrent
+# request queues behind the released slot (subagent concurrency 2 -> 1).
+CTX_SIZE=262144
+PARALLEL=1
 
 # Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03,
 # so it ran on llama.cpp's own defaults (2048/512); the pair is now 4096/2048.
@@ -113,19 +114,23 @@ PARALLEL=2
 # graph VRAM on the 35B MoE (7,330 -> 8,574 MiB); this 9B hybrid has far
 # smaller compute tensors, so the delta should be well under that, but the
 # predecessor Q5_K_M shape recorded "ubatch 2048 failed the scratch boot" at
-# ~3,571 MiB free. This shape ships with LESS free (2,815 MiB measured
-# 2026-10-05). Compute buffers are allocated at graph capture during BOOT, so
-# a failure is loud and fast, not a surprise at the first long prompt. MEASURE
-# nvidia-smi free immediately after the deploy: under ~1.5 GiB, or a boot
-# failure, step down to 4096/1536 (or 4096/1024, the last known-good rung).
+# ~3,571 MiB free. The Q8_0 rung ships UBATCH 1024 (Frank, 2026-10-05: lower
+# ubatch FIRST — the measured 616 MiB step is the first VRAM the Q8 weights
+# claw back, before any pool move). Compute buffers are allocated at graph
+# capture during BOOT, so a failure is loud and fast, not a surprise at the
+# first long prompt. MEASURE nvidia-smi free immediately after the deploy:
+# under ~1.5 GiB, or a boot failure, step the POOL down next (12,288-token
+# increments, ~432 MiB each at the Ornith 36 KiB/token f16 KV rate).
 BATCH_SIZE=4096
-UBATCH_SIZE=2048
+UBATCH_SIZE=1024
 
-# The KV contract (2026-10-05, Q8_0 era): f16 K + q8_0 V. See the header
-# arithmetic — the Q8 weights consume the VRAM that funded f16 V on
-# 2026-10-03, and pool-stepping cannot restore the floor. --flash-attn stays
-# pinned on in the serve script: it is the fast path for both f16 and q8_0 V
-# and the only path that serves quantized V at all.
+# The KV contract (2026-10-05, Frank): f16 K + f16 V — NO KV quantization.
+# The q8_0-V arithmetic was written and OVERRULED. The Q8 weights eat the
+# VRAM that funded f16 V on 2026-10-03, so the remedy ladder is UBATCH FIRST
+# (2048 -> 1024, the measured 616 MiB rung), then POOL STEPS — and
+# guess-and-check on the live card is the accepted method: boot, read
+# nvidia-smi, step down until the floor holds. --flash-attn stays pinned on
+# in the serve script: it is the fast path for f16 KV.
 CACHE_TYPE_K=f16
-CACHE_TYPE_V=q8_0
+CACHE_TYPE_V=f16
 NGL=99