Frank
812d9ebff5
CTX 262144 / PARALLEL 2 = two 131,072 conversations; SPEC_TYPE=none strips both speculative flags in the serve script (ngram-mod skipped on purpose — on a qwen35 hybrid it disables recurrent-state rollback); ubatch 1024->512 claws back ~0.3 GiB. Budget: non-KV ~15.5 GiB + two q8_0 slots 8,704 MiB lands AT the card — the 2-3 GiB floor is knowingly violated by explicit order; nvidia-smi free at boot is the measured truth. Seam retargeted: shape (2 slots, 131072/slot), budget (two fit, three don't), spec-off (absence of both flags), unit Description.
113 lines
6.1 KiB
Bash
113 lines
6.1 KiB
Bash
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
|
# ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S **-mtp** on the RTX 3090,
|
|
# :7771.
|
|
#
|
|
# 2026-10-05 (Frank, evening): the payload moved to Swift-1.5 Qwen3.8-27B
|
|
# GSQ-RCO IQ3_S WITH the MTP head in the file, each conversation capped at
|
|
# 131,072 tokens, KV at FP8 class, vision on. Header-verified facts that
|
|
# re-price the shape (read from the Hub file 2026-10-05, HTTP range + struct
|
|
# parse): qwen35 hybrid like MiMo/Ornith but bigger — block_count 65 (64 real
|
|
# layers + the blk.64 MTP head), full_attention_interval 4 => SIXTEEN
|
|
# full-attention layers (MiMo had 8, Ornith 9), 24 Q heads / 4 KV heads,
|
|
# k/v length 256, native window 262144, dense, nextn_predict_layers=1 with
|
|
# blk.64.nextn.{enorm,hnorm,shared_head_norm} (the -mtp file's head tensors
|
|
# are Q6_K/F32 per the publisher RECIPE, "MTP_OK" verified).
|
|
#
|
|
# THE KV CONTRACT IS NOW q8_0 K + q8_0 V — Frank's order for THIS swap
|
|
# ("k/v at FP8", 2026-10-05), superseding the same-day f16/f16 rule, which
|
|
# was written for the Ornith 9B fit ("no further quantization anywhere on
|
|
# this endpoint"). llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit
|
|
# class (1.0625 B/elem => 8.5 effective bits) and the only FP8-class lever
|
|
# this runtime moves. Arithmetic that makes the order load-bearing:
|
|
# KV rate = 16 attn layers x 4 heads x 256 x 1.0625 B, K and V again
|
|
# => 34 KiB/token (f16 would be 64 KiB/token — halogen's 27B card quotes
|
|
# the same f16 number; at f16 the 131,072 slot alone would cost 8.0 GiB).
|
|
#
|
|
# 2026-10-06 (Frank): drop the MTP drafter (skip ngram too — on a qwen35
|
|
# hybrid, ngram speculation disables recurrent-state rollback: fleet lesson
|
|
# from the frank-large chain measurement), drop ubatch to 512, and take TWO
|
|
# conversations at 131,072 — "should be a very tight fit", accepted
|
|
# explicitly, i.e. the 2-3 GiB floor is knowingly violated. Measured at the
|
|
# 1x131072 MTP boot (2026-10-05): 19,404 used / 4,721 MiB free.
|
|
#
|
|
# Why TWO slots at 131,072 fits, by arithmetic (pool = 2 x 131,072 =
|
|
# 262,144; the per-conversation cap stays Frank's 128k):
|
|
#
|
|
# card 24,576 MiB. Weights 11,561 MiB (the IQ3_S-mtp file; the GSQ
|
|
# "IQ3_S" tier is a per-tensor ALLOCATION mix — histogram IQ2_XS 144 /
|
|
# IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 ... all stock
|
|
# types, stock llama.cpp loads it). Projector ~1,350 MiB (927,607,488 B
|
|
# F16 at the measured ~1.45x card law). Compute buffers at ubatch 512
|
|
# ~1.3 GiB (inferred from the measured 616 MiB/doubling rung on the 9B).
|
|
# CUDA ctx ~500 MiB. DeltaNet state ~150 MiB/slot x 2. NO draft context
|
|
# (the MTP draft ctx cost ~0.3-0.5 GiB and is gone with speculation).
|
|
# Non-KV ~15.5 GiB. Two q8_0 slots = 8,704 MiB (34 KiB/token x 131,072
|
|
# each). Predicted free at boot ~0.3-1.3 GiB. Compute buffers allocate
|
|
# at graph capture during BOOT, so a miss is loud and fast; MEASURE
|
|
# nvidia-smi free at boot (quote the free field) and let the runbook
|
|
# carry the number.
|
|
#
|
|
# Payload history (for the rollback trail): the Ornith-1.5-9B Q8_0 swap of
|
|
# earlier on 2026-10-05 (commits 77decb5..380aafa) was committed but never
|
|
# deployed — the live service was still MiMo-V2.6 Q6_K (2 x 200,704, f16
|
|
# KV) when Swift cut over. Rollback payloads on disk: Ornith-Q8/ and
|
|
# MiMo-Q6K/ (one MODEL_FILE + shape edit back; see the runbook).
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
|
|
|
|
# Vision: no Swift repo ships a projector (ukisai's release is
|
|
# language-model-only by their own README), so the projector comes from the
|
|
# SHARED BASE — unsloth's mmproj-F16.gguf built from Qwen/Qwen3.8-27B,
|
|
# stored as mmproj-Qwen3.8-27B-F16.gguf (927,607,488 B, LFS sha256
|
|
# cbb841a9..., fetched by scripts/download-frank-small.sh). The substitution
|
|
# is proven, not assumed: Swift-1.5's vision stack (tower + merger) was
|
|
# hash-compared against the base on 2026-10-05 — 333/333 visual tensor names
|
|
# and 6/6 sampled tensors byte-identical — so the base projector IS the
|
|
# model's own projector (Swift's RL/OPD post-training touched text only).
|
|
# Empty string = text-only on purpose.
|
|
MMPROJ_FILE=/home/dank/models/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf
|
|
|
|
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
|
# harness models.yml id and the homepage probe do not move.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# TWO conversations at Frank's 128k cap (2026-10-06): --ctx-size is the
|
|
# POOL llama.cpp divides across slots; 262144 / 2 = 131,072 per
|
|
# conversation. Frank accepted the tight fit explicitly (arithmetic above);
|
|
# the per-slot window is the product contract, so models.yml stays at
|
|
# 131,072.
|
|
CTX_SIZE=262144
|
|
PARALLEL=2
|
|
|
|
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves);
|
|
# on the 9B the 1024->2048 rung measured +4.6 % prefill for 616 MiB, and
|
|
# Frank's 2026-10-06 order spends the other direction: ubatch 512 claws
|
|
# back ~0.3 GiB toward the second slot at some prefill cost. The pair is a
|
|
# measurement, not a law: ubatch 1024 is the first A/B back if a future
|
|
# boot lands with headroom (loud failure at graph capture if it does not
|
|
# fit).
|
|
BATCH_SIZE=4096
|
|
UBATCH_SIZE=512
|
|
|
|
# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V —
|
|
# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in
|
|
# the argv either way, and --flash-attn stays pinned on in the serve script:
|
|
# it is required for quantized V (the build carries
|
|
# GGML_CUDA_FA_ALL_QUANTS=ON since 2026-09-17, so q8_0 KV is a real path,
|
|
# not a silent f16 fallback).
|
|
CACHE_TYPE_K=q8_0
|
|
CACHE_TYPE_V=q8_0
|
|
|
|
# Speculative decoding is OFF (Frank, 2026-10-06): no draft-mtp, and
|
|
# ngram-mod is skipped too — on a qwen35 hybrid ngram disables
|
|
# recurrent-state rollback, so a rejected draft would corrupt the DeltaNet
|
|
# state. SPEC_TYPE=none strips BOTH speculative flags in the serve script
|
|
# (flags move as a set). The -mtp file still carries the head; it is simply
|
|
# not drafted with. Measured MTP reference (2026-10-05): 44.8 % acceptance
|
|
# at the recipe temperature => 58.9 t/s, 52.1 % at temp 0.8 => 66.1 t/s.
|
|
SPEC_TYPE=none
|
|
|
|
NGL=99
|