Frank
e5893dc30d
BATCH_SIZE 2048 -> 4096, UBATCH_SIZE 1024 -> 2048. Why the ubatch move is right on THIS card: the prefill curve is card-specific. On the 2x R9700 layer-split sibling it peaks at ubatch 512 and every step up costs GiB for no speed; on this CUDA 3090 it climbs monotonically to 2048 (710 -> 1188 t/s on the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep) with decode flat to under 1 t/s on every rung. The logical batch is free either way: two sweeps agree that at a fixed ubatch, 2048/4096/8192 give identical prefill speed and identical peak VRAM, because the compute buffers are sized by the ubatch alone. The risk is all in the ubatch, and it is documented rather than assumed: the step cost ~1,244 MiB of peak graph VRAM on the 35B MoE, the shape this replaces is 2,815 MiB free, and the Q5_K_M predecessor recorded "ubatch 2048 failed the scratch boot" with MORE free than that. Compute buffers are allocated at graph capture during boot, so a failure is loud and fast; the step-down rungs (4096/1536, then 4096/1024) are named in the env. The seam stops hard-coding a rung: serve_argv now carries BATCH_SIZE and UBATCH_SIZE the way it already carried CTX_SIZE et al., and the pair test asserts the argv follows the shipped env plus the llama.cpp invariant ubatch <= batch. Verified both ways: retuning the env to 8192/4096 keeps the seam green, inverting the pair to 2048/4096 turns it red.
108 lines
6.0 KiB
Bash
108 lines
6.0 KiB
Bash
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
|
# bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q6_K on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-03 (Frank, second move): the rung went Q5_K_M -> Q6_K with the pool
|
|
# HELD at 401,408. The freed third slot funded exact f16 V first; the ~3.5 GiB
|
|
# still free funds the +875 MiB of weights (file delta 7,793,710,624 -
|
|
# 6,876,124,704). Projected landing ~2.7 GiB free — inside the 2-3 GiB floor,
|
|
# so MEASURE after the boot; if it lands under 2 GiB, step the pool down one
|
|
# 12,288-token (384 MiB) increment.
|
|
#
|
|
# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
|
|
# tokens per conversation, above the 200,700 the harness compacts at — and the
|
|
# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same
|
|
# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given
|
|
# up is exactly the VRAM that pays for unquantized V. No further quantization
|
|
# anywhere on this endpoint.
|
|
#
|
|
# Three measurements on this card, this runtime, price the terms:
|
|
# 2 x 262,144, no projector: 20,190 MiB used / 4,386 free
|
|
# KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
|
|
# DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
|
|
# 3 x 196,608, projector on: 22,972 MiB used / 1,153 free
|
|
# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
|
|
# 3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02
|
|
# shape) => non-KV = 8,748 MiB
|
|
# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V
|
|
# (32 KiB/token) this pool is 12,544 MiB, so the prediction is
|
|
# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02
|
|
# shape hit, bought with precision instead of a slot. The 1024 ubatch
|
|
# re-prices the compute buffers upward, so MEASURE after any change: if the
|
|
# boot lands under 2 GiB free, step the pool down in 12,288-token
|
|
# (384 MiB at this rate) increments.
|
|
#
|
|
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
|
# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for
|
|
# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that
|
|
# is the ladder to walk if this moves again. Two native windows (2 x 262,144)
|
|
# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV
|
|
# against a card with ~15.7 GiB of KV room once weights, the vision tower, the
|
|
# DeltaNet states and the graphs are loaded. 200,704 per stream is what full
|
|
# precision buys; q8_0 on V would buy 262,144 instead.
|
|
#
|
|
# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01
|
|
# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit,
|
|
# and the third slot is gone. The flags stay explicit so the choice is visible
|
|
# in the argv either way.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf
|
|
|
|
# Vision: the Q6_K GGUF ships no `v.*` tensors either, so the projector file
|
|
# IS the vision capability (sha256 ff348f31..., 918,166,048 B, fetched by
|
|
# scripts/download-frank-small.sh; hardlinked into this dir from MiMo-Q5KM/,
|
|
# same bytes — the Q6_K dir is self-contained). Empty string = text-only on
|
|
# purpose.
|
|
MMPROJ_FILE=/home/dank/models/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf
|
|
|
|
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
|
# harness models.yml id and the homepage probe do not move.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Two independent conversations. llama.cpp divides --ctx-size across
|
|
# --parallel slots (n_ctx_slot = CTX_SIZE/PARALLEL), and a mamba-hybrid model
|
|
# cannot use unified KV, so each slot gets its own cache and its own window.
|
|
# 200,704 per slot is BELOW the native 262,144 — the price of paying for the
|
|
# window in full precision — and ABOVE the harness's compaction trigger, which
|
|
# is the point: a full-length request now lands in one stream instead of a 400
|
|
# exceed_context_size_error. Cost of the third slot going away: a third
|
|
# concurrent request queues behind a released slot instead of running.
|
|
CTX_SIZE=401408
|
|
PARALLEL=2
|
|
|
|
# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03,
|
|
# so it ran on llama.cpp's own defaults (2048/512); the pair is now 4096/2048.
|
|
# Both are tunables because the pair is a measurement, not a law — and the
|
|
# measurement is CARD-SPECIFIC. On the 2x R9700 layer-split sibling the ubatch
|
|
# curve peaks at 512 and every step up costs GiB of compute buffer for no
|
|
# speed (see frank-medium-batch-ubatch). On THIS CUDA 3090 the curve climbs
|
|
# monotonically to 2048: 710 -> 1188 t/s prefill from ubatch 1024 to 2048 on
|
|
# the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep, with decode flat
|
|
# to under 1 t/s on every rung. Ubatch is a prefill knob; decode never moves.
|
|
#
|
|
# The logical batch is free — two independent sweeps agree that at a fixed
|
|
# ubatch, 2048/4096/8192 give identical prefill speed AND identical peak VRAM,
|
|
# because the compute buffers are sized by the ubatch alone. Doubling it costs
|
|
# nothing and buys nothing; it is here because it is the value the pair was
|
|
# measured at.
|
|
#
|
|
# THE RISK IS ALL IN THE UBATCH. The 1024 -> 2048 step cost ~1,244 MiB of peak
|
|
# graph VRAM on the 35B MoE (7,330 -> 8,574 MiB); this 9B hybrid has far
|
|
# smaller compute tensors, so the delta should be well under that, but the
|
|
# predecessor Q5_K_M shape recorded "ubatch 2048 failed the scratch boot" at
|
|
# ~3,571 MiB free. This shape ships with LESS free (2,815 MiB measured
|
|
# 2026-10-05). Compute buffers are allocated at graph capture during BOOT, so
|
|
# a failure is loud and fast, not a surprise at the first long prompt. MEASURE
|
|
# nvidia-smi free immediately after the deploy: under ~1.5 GiB, or a boot
|
|
# failure, step down to 4096/1536 (or 4096/1024, the last known-good rung).
|
|
BATCH_SIZE=4096
|
|
UBATCH_SIZE=2048
|
|
|
|
# The KV contract (Frank, 2026-10-03): unquantized K AND V. --flash-attn stays
|
|
# pinned on in the serve script — it is what made the q8_0-V path cheap and it
|
|
# remains the fast path for f16 KV.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
NGL=99
|