# frank-small — llama.cpp (master, CUDA sm_86) serving
# ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S **-mtp** on the RTX 3090,
# :7771.
#
# 2026-10-05 (Frank, evening): the payload moved to Swift-1.5 Qwen3.8-27B
# GSQ-RCO IQ3_S WITH the MTP head in the file, each conversation capped at
# 131,072 tokens, KV at FP8 class, vision on. Header-verified facts that
# re-price the shape (read from the Hub file 2026-10-05, HTTP range + struct
# parse): qwen35 hybrid like MiMo/Ornith but bigger — block_count 65 (64 real
# layers + the blk.64 MTP head), full_attention_interval 4 => SIXTEEN
# full-attention layers (MiMo had 8, Ornith 9), 24 Q heads / 4 KV heads,
# k/v length 256, native window 262144, dense, nextn_predict_layers=1 with
# blk.64.nextn.{enorm,hnorm,shared_head_norm} (the -mtp file's head tensors
# are Q6_K/F32 per the publisher RECIPE, "MTP_OK" verified).
#
# THE KV CONTRACT IS NOW q8_0 K + q8_0 V — Frank's order for THIS swap
# ("k/v at FP8", 2026-10-05), superseding the same-day f16/f16 rule, which
# was written for the Ornith 9B fit ("no further quantization anywhere on
# this endpoint"). llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit
# class (1.0625 B/elem => 8.5 effective bits) and the only FP8-class lever
# this runtime moves. Arithmetic that makes the order load-bearing:
#   KV rate = 16 attn layers x 4 heads x 256 x 1.0625 B, K and V again
#   => 34 KiB/token (f16 would be 64 KiB/token — halogen's 27B card quotes
#   the same f16 number; at f16 the 131,072 slot alone would cost 8.0 GiB).
#
# Why ONE slot at 131,072 is the maximum (Frank's cap per conversation):
#   card 24,576 MiB. Weights 11,561 MiB (12,120,016,896 B, the IQ3_S-mtp
#   file; the GSQ "IQ3_S" tier is a per-tensor ALLOCATION mix — histogram
#   IQ2_XS 144 / IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 ...
#   all stock types, stock llama.cpp loads it). Projector ~1,350 MiB
#   (927,607,488 B F16 file at the measured ~1.45x card law). Compute
#   buffers at ubatch 1024 ~1.6 GiB. CUDA ctx ~500 MiB. DeltaNet state
#   ~150 MiB/slot (48 linear layers x 48 value heads x 128x128 x 4 B).
#   MTP draft context ~0.3-0.5 GiB (one nextn layer over the slot window;
#   the boot ledger is the truth). Non-KV ~= 15.5 GiB. One q8_0 slot =
#   4,352 MiB => free ~= 4.5 GiB. A SECOND slot wants +4,352 MiB => free
#   ~0.2 GiB, far below the 2-3 GiB floor. "As many 128k contexts as fit"
#   = 1. MEASURE nvidia-smi free at boot (quote the free field): under
#   ~1.5 GiB or a boot failure, step the pool down in 12,288-token
#   increments (~410 MiB each at this rate). Compute buffers allocate at
#   graph capture during BOOT, so a failure is loud and fast.
#
# Payload history (for the rollback trail): the Ornith-1.5-9B Q8_0 swap of
# earlier on 2026-10-05 (commits 77decb5..380aafa) was committed but never
# deployed — the live service was still MiMo-V2.6 Q6_K (2 x 200,704, f16
# KV) when Swift cut over. Rollback payloads on disk: Ornith-Q8/ and
# MiMo-Q6K/ (one MODEL_FILE + shape edit back; see the runbook).
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
MODEL_FILE=/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf

# Vision: no Swift repo ships a projector (ukisai's release is
# language-model-only by their own README), so the projector comes from the
# SHARED BASE — unsloth's mmproj-F16.gguf built from Qwen/Qwen3.8-27B,
# stored as mmproj-Qwen3.8-27B-F16.gguf (927,607,488 B, LFS sha256
# cbb841a9..., fetched by scripts/download-frank-small.sh). The substitution
# is proven, not assumed: Swift-1.5's vision stack (tower + merger) was
# hash-compared against the base on 2026-10-05 — 333/333 visual tensor names
# and 6/6 sampled tensors byte-identical — so the base projector IS the
# model's own projector (Swift's RL/OPD post-training touched text only).
# Empty string = text-only on purpose.
MMPROJ_FILE=/home/dank/models/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf

# Fleet endpoint: same port and alias as every frank-small before it, so the
# harness models.yml id and the homepage probe do not move.
ALIAS=frank-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771

# ONE conversation at Frank's 128k cap: --ctx-size is the POOL llama.cpp
# divides across slots; 131072 / 1 = 131,072 per conversation. The cap is a
# product decision (Swift's own benchmark protocol ran at 131,072), and a
# second 131,072 slot does not fit the 2-3 GiB floor (arithmetic above).
CTX_SIZE=131072
PARALLEL=1

# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves);
# on this CUDA 3090 the curve climbs monotonically to 2048, but Frank's
# standing law for a resize is LOWER UBATCH FIRST (the measured 616 MiB rung
# on the 9B; the 27B's buffers are larger, so the claw-back is worth more).
# The pair is a measurement, not a law: if boot lands well above the floor,
# ubatch 2048 is the first thing to A/B for prefill (loud failure at graph
# capture if it does not fit).
BATCH_SIZE=4096
UBATCH_SIZE=1024

# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V —
# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in
# the argv either way, and --flash-attn stays pinned on in the serve script:
# it is required for quantized V (the build carries
# GGML_CUDA_FA_ALL_QUANTS=ON since 2026-09-17, so q8_0 KV is a real path,
# not a silent f16 fallback).
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q8_0

# Speculative decoding: the MTP head is IN the -mtp file (blk.64.nextn.*,
# nextn_predict_layers=1), so draft-mtp needs no second model and no second
# KV pool — the head shares the main model's weights. Depth 3 matches the
# fleet's radiance MTP depth; measure draft acceptance from the response
# timings object and retune here if the head earns more.
SPEC_TYPE=draft-mtp
SPEC_DRAFT_N_MAX=3

NGL=99