Frank
e5893dc30d
BATCH_SIZE 2048 -> 4096, UBATCH_SIZE 1024 -> 2048. Why the ubatch move is right on THIS card: the prefill curve is card-specific. On the 2x R9700 layer-split sibling it peaks at ubatch 512 and every step up costs GiB for no speed; on this CUDA 3090 it climbs monotonically to 2048 (710 -> 1188 t/s on the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep) with decode flat to under 1 t/s on every rung. The logical batch is free either way: two sweeps agree that at a fixed ubatch, 2048/4096/8192 give identical prefill speed and identical peak VRAM, because the compute buffers are sized by the ubatch alone. The risk is all in the ubatch, and it is documented rather than assumed: the step cost ~1,244 MiB of peak graph VRAM on the 35B MoE, the shape this replaces is 2,815 MiB free, and the Q5_K_M predecessor recorded "ubatch 2048 failed the scratch boot" with MORE free than that. Compute buffers are allocated at graph capture during boot, so a failure is loud and fast; the step-down rungs (4096/1536, then 4096/1024) are named in the env. The seam stops hard-coding a rung: serve_argv now carries BATCH_SIZE and UBATCH_SIZE the way it already carried CTX_SIZE et al., and the pair test asserts the argv follows the shipped env plus the llama.cpp invariant ubatch <= batch. Verified both ways: retuning the env to 8192/4096 keeps the seam green, inverting the pair to 2048/4096 turns it red.
100 lines
4.3 KiB
Bash
Executable File
100 lines
4.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Serve bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF (Q6_K) as frank-small on
|
|
# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
|
|
#
|
|
# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound
|
|
# safetensors to the llama.cpp GGUF path, because llama.cpp is the only
|
|
# runtime on this host that moves K and V independently — the asymmetric
|
|
# f16-K + q8_0-V contract that endpoint was built for. 2026-10-03 (Frank):
|
|
# the pair is now f16/f16 — two wide slots freed the VRAM, so the values are
|
|
# unquantized too. The per-tensor flags stay explicit either way.
|
|
#
|
|
# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
|
|
# DeltaNet, native 262144 window. KV lives only in the 8 attention layers;
|
|
# the DeltaNet state is per-slot fixed size. Vision comes from the repo's
|
|
# separate projector (the Q6_K GGUF carries no `v.*` tensors at all), so
|
|
# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented
|
|
# text-only opt-out.
|
|
#
|
|
# Sampling: llama.cpp takes it per request (the fleet harness sends its
|
|
# recipe from models.yml); greedy is the harness default.
|
|
#
|
|
# Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides.
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
|
ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-small.env}"
|
|
if [[ -f "$ENV_FILE" ]]; then
|
|
set -a
|
|
# shellcheck disable=SC1090
|
|
source "$ENV_FILE"
|
|
set +a
|
|
fi
|
|
|
|
# --- defaults (used when the env file does not set them) -------------------
|
|
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
|
: "${MODEL_FILE:=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf}"
|
|
: "${ALIAS:=frank-small}"
|
|
: "${BIND_ADDR:=0.0.0.0}"
|
|
: "${LISTEN_PORT:=7771}"
|
|
# ctx-size is the POOL llama.cpp divides across slots: 401408 / 2 gives
|
|
# 200,704 per slot — two independent conversations, each one wide enough to
|
|
# hold a whole frank-primary-sized prompt in a single shot (the native 262,144
|
|
# no longer fits twice on a 24 GiB card once the vision tower is loaded;
|
|
# frank-small.env carries the three measurements that set this number).
|
|
: "${CTX_SIZE:=401408}"
|
|
: "${PARALLEL:=2}"
|
|
# Compute-buffer geometry. Before 2026-10-03 this endpoint never passed either
|
|
# flag, so it ran on llama.cpp's own defaults (2048/512). The pair is 4096/2048
|
|
# since 2026-10-05: on this CUDA card the prefill curve climbs all the way to
|
|
# ubatch 2048 (unlike the R9700 layer-split sibling, where 512 is the peak),
|
|
# and the logical batch is VRAM- and throughput-neutral. frank-small.env
|
|
# carries the sweep tables and the VRAM risk note.
|
|
: "${BATCH_SIZE:=4096}"
|
|
: "${UBATCH_SIZE:=2048}"
|
|
: "${CACHE_TYPE_K:=f16}"
|
|
: "${CACHE_TYPE_V:=f16}"
|
|
: "${NGL:=99}"
|
|
# Vision projector. No colon on purpose: an explicitly empty value means
|
|
# "text-only, on purpose" and must NOT be replaced by the default path.
|
|
: "${MMPROJ_FILE=/home/dank/models/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf}"
|
|
|
|
# Fail loudly rather than half-loading: a missing file is a download that
|
|
# has not finished, and Restart=always would turn it into a crash loop that
|
|
# reads like a runtime fault.
|
|
for path in "$LLAMA_BIN" "$MODEL_FILE"; do
|
|
[[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
|
|
done
|
|
EXTRA_ARGS=()
|
|
if [[ -n "$MMPROJ_FILE" ]]; then
|
|
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-small: missing $MMPROJ_FILE" >&2; exit 1; }
|
|
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
|
|
fi
|
|
|
|
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
|
# open after boot; make sure it exists before the server needs it.
|
|
nvidia-modprobe -u -c=0 2>/dev/null || true
|
|
|
|
# Flash attention stays pinned on: it is the fast path for f16 KV, and it is
|
|
# the only path that could serve quantized V at all — so if CACHE_TYPE_V is
|
|
# ever set back to q8_0 this flag must not be lost, or the build falls back
|
|
# silently and the KV contract in frank-small.env becomes a lie.
|
|
ARGS=(
|
|
--model "$MODEL_FILE"
|
|
--alias "$ALIAS"
|
|
--host "$BIND_ADDR"
|
|
--port "$LISTEN_PORT"
|
|
--ctx-size "$CTX_SIZE"
|
|
--parallel "$PARALLEL"
|
|
--batch-size "$BATCH_SIZE"
|
|
--ubatch-size "$UBATCH_SIZE"
|
|
--n-gpu-layers "$NGL"
|
|
--flash-attn on
|
|
--cache-type-k "$CACHE_TYPE_K"
|
|
--cache-type-v "$CACHE_TYPE_V"
|
|
--jinja
|
|
--reasoning auto
|
|
--reasoning-format deepseek
|
|
)
|
|
exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
|