Frank
1862dc3c0f
The rogue session's swap left this env reprice uncommitted in the tree (2026-10-05): Ornith is a 33-block qwen35 hybrid with NINE full-attn layers (18 KiB/token/half), +1,900 MiB of weights vs MiMo Q6_K — f16/f16 lands ~379 MiB free, a dead boot; pool-stepping cannot rescue it because the compaction invariant needs slot >= 191,148. The only lever that keeps 2 x 200,704 AND the 2-3 GiB floor is q8_0 V (~2.87 GiB free). K stays f16. Deployer gate + seam already retargeted in 8f53df7; fetch script manifest already flipped in 3ee88d7. MEASURE after boot; under 1.5 GiB free, UBATCH_SIZE down to 1024, never the window.
132 lines
7.6 KiB
Bash
132 lines
7.6 KiB
Bash
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
|
# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-05 (Frank): the payload moved off bartowski/MiMo-V2.6 (Q6_K) to
|
|
# ornith-ai/Ornith-1.5-9B Q8_0 (9,786,060,384 B, LFS oid 22086870...).
|
|
# Header-verified facts that re-price the shape: qwen35 hybrid like MiMo, but
|
|
# 33 blocks (not 32) with full_attention_interval 4 => NINE full-attention
|
|
# layers (MiMo had 8), 16 Q heads / 4 KV heads, k/v length 256, native window
|
|
# 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN
|
|
# the file — capability only; no spec decode ships, same law as before).
|
|
#
|
|
# THE KV CONTRACT MOVED BACK TO f16 K + q8_0 V, and here is the arithmetic
|
|
# that forces it (2026-10-05, from the header + measured MiMo numbers):
|
|
# KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again
|
|
# for V at f16 => 36 KiB/token (MiMo was 32: one fewer full-attn layer).
|
|
# Weights +1,900 MiB (9,334 vs 7,432). Measured free at the Q6_K shape was
|
|
# 2,279 MiB, so Q8_0 at f16/f16 lands ~379 MiB free — a dead boot.
|
|
# Pool-stepping cannot rescue it: the harness compaction invariant
|
|
# (0.7 x slot + 57,344 <= slot) needs slot >= 191,148, so two slots need
|
|
# >= 13.6 GiB of f16/f16 KV even at the minimum legal window. The ONLY
|
|
# lever that keeps both the 2 x 200,704 window contract and the 2-3 GiB
|
|
# floor is q8_0 on V (26.25 KiB/token -> ~10.05 GiB KV -> ~2.87 GiB free).
|
|
# The 2026-10-03 "unquantized both halves" note was bought with the VRAM
|
|
# the third slot's death freed; the Q8 rung re-spends it. K stays f16.
|
|
# MEASURE after the boot: under ~1.5 GiB free, step UBATCH_SIZE down to 1024
|
|
# (the measured 616 MiB rung), never the window.
|
|
#
|
|
# Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the
|
|
# rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third
|
|
# slot funded exact f16 V first; the ~3.5 GiB still free funded the +875 MiB
|
|
# of weights.
|
|
#
|
|
# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
|
|
# tokens per conversation, above the 200,700 the harness compacts at — and the
|
|
# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same
|
|
# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given
|
|
# up is exactly the VRAM that pays for unquantized V. No further quantization
|
|
# anywhere on this endpoint.
|
|
#
|
|
# Three measurements on this card, this runtime, price the terms:
|
|
# 2 x 262,144, no projector: 20,190 MiB used / 4,386 free
|
|
# KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
|
|
# DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
|
|
# 3 x 196,608, projector on: 22,972 MiB used / 1,153 free
|
|
# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
|
|
# 3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02
|
|
# shape) => non-KV = 8,748 MiB
|
|
# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V
|
|
# (32 KiB/token) this pool is 12,544 MiB, so the prediction is
|
|
# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02
|
|
# shape hit, bought with precision instead of a slot. The 1024 ubatch
|
|
# re-prices the compute buffers upward, so MEASURE after any change: if the
|
|
# boot lands under 2 GiB free, step the pool down in 12,288-token
|
|
# (384 MiB at this rate) increments.
|
|
#
|
|
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
|
# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for
|
|
# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that
|
|
# is the ladder to walk if this moves again. Two native windows (2 x 262,144)
|
|
# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV
|
|
# against a card with ~15.7 GiB of KV room once weights, the vision tower, the
|
|
# DeltaNet states and the graphs are loaded. 200,704 per stream is what full
|
|
# precision buys; q8_0 on V would buy 262,144 instead.
|
|
#
|
|
# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01
|
|
# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit,
|
|
# and the third slot is gone. The flags stay explicit so the choice is visible
|
|
# in the argv either way.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf
|
|
|
|
# Vision: the Q6_K GGUF ships no `v.*` tensors either, so the projector file
|
|
# IS the vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
|
|
# scripts/download-frank-small.sh; hardlinked into this dir from MiMo-Q5KM/,
|
|
# same bytes — the Q6_K dir is self-contained). Empty string = text-only on
|
|
# purpose.
|
|
MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf
|
|
|
|
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
|
# harness models.yml id and the homepage probe do not move.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Two independent conversations. llama.cpp divides --ctx-size across
|
|
# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
|
|
# cannot use unified KV, so each slot gets its own cache and its own window.
|
|
# 200,704 per slot is BELOW the native 262,144 — the price of paying for the
|
|
# window in full precision — and ABOVE the harness's compaction trigger, which
|
|
# is the point: a full-length request now lands in one stream instead of a 400
|
|
# exceed_context_size_error. Cost of the third slot going away: a third
|
|
# concurrent request queues behind a released slot instead of running.
|
|
CTX_SIZE=401408
|
|
PARALLEL=2
|
|
|
|
# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03,
|
|
# so it ran on llama.cpp's own defaults (2048/512); the pair is now 4096/2048.
|
|
# Both are tunables because the pair is a measurement, not a law — and the
|
|
# measurement is CARD-SPECIFIC. On the 2x R9700 layer-split sibling the ubatch
|
|
# curve peaks at 512 and every step up costs GiB of compute buffer for no
|
|
# speed (see frank-medium-batch-ubatch). On THIS CUDA 3090 the curve climbs
|
|
# monotonically to 2048: 710 -> 1188 t/s prefill from ubatch 1024 to 2048 on
|
|
# the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep, with decode flat
|
|
# to under 1 t/s on every rung. Ubatch is a prefill knob; decode never moves.
|
|
#
|
|
# The logical batch is free — two independent sweeps agree that at a fixed
|
|
# ubatch, 2048/4096/8192 give identical prefill speed AND identical peak VRAM,
|
|
# because the compute buffers are sized by the ubatch alone. Doubling it costs
|
|
# nothing and buys nothing; it is here because it is the value the pair was
|
|
# measured at.
|
|
#
|
|
# THE RISK IS ALL IN THE UBATCH. The 1024 -> 2048 step cost ~1,244 MiB of peak
|
|
# graph VRAM on the 35B MoE (7,330 -> 8,574 MiB); this 9B hybrid has far
|
|
# smaller compute tensors, so the delta should be well under that, but the
|
|
# predecessor Q5_K_M shape recorded "ubatch 2048 failed the scratch boot" at
|
|
# ~3,571 MiB free. This shape ships with LESS free (2,815 MiB measured
|
|
# 2026-10-05). Compute buffers are allocated at graph capture during BOOT, so
|
|
# a failure is loud and fast, not a surprise at the first long prompt. MEASURE
|
|
# nvidia-smi free immediately after the deploy: under ~1.5 GiB, or a boot
|
|
# failure, step down to 4096/1536 (or 4096/1024, the last known-good rung).
|
|
BATCH_SIZE=4096
|
|
UBATCH_SIZE=2048
|
|
|
|
# The KV contract (2026-10-05, Q8_0 era): f16 K + q8_0 V. See the header
|
|
# arithmetic — the Q8 weights consume the VRAM that funded f16 V on
|
|
# 2026-10-03, and pool-stepping cannot restore the floor. --flash-attn stays
|
|
# pinned on in the serve script: it is the fast path for both f16 and q8_0 V
|
|
# and the only path that serves quantized V at all.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=q8_0
|
|
NGL=99
|