98 lines
4.9 KiB
Bash
98 lines
4.9 KiB
Bash
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
|
# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF:RCO-IQ3_XXS
|
|
# **with the MTP head in the file** on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-06 (Frank, evening): the payload moved to ajgazin's UNCELERSED
|
|
# Swift-1.5-Qwen3.8-27B at RCO-IQ3_XXS (10,442,827,392 B) — 1.56 GiB
|
|
# lighter than the IQ3_S-mtp it replaces — with MTP BACK ON (draft-mtp,
|
|
# depth 3), TWO 131,072 slots, and ubatch 512. The uncensored finetune
|
|
# changes behavior by design (publisher: 23/100 refusals vs Swift 1.5's
|
|
# 98/100, KL 0.0884 from Swift 1.5) — Frank's order. The RCO tier is the
|
|
# same per-tensor ALLOCATION recipe as the IQ3_S file (histogram: iq3_s
|
|
# 97 / bf16 96 / iq3_xxs 83 / iq2_s 72 / iq4_xs 33 / iq2_xs 32 / iq2_xxs
|
|
# 30 / q2_k 28 / q4_k 19 / q6_k 8 / iq1_s 4 / iq1_m 4 — all stock types,
|
|
# stock llama.cpp loads it); publisher KLD: Mean 0.002631, 90 % 0.0421
|
|
# (kld_results.csv). RCO files are "more experimental" per the README —
|
|
# the quality floor is the IQ3_S rung, not a promise.
|
|
#
|
|
# The architecture is unchanged (same model family): qwen35 hybrid —
|
|
# block_count 65 (64 real layers + the blk.64 MTP head),
|
|
# full_attention_interval 4 => SIXTEEN full-attention layers, 24 Q heads
|
|
# / 4 KV heads, k/v length 256, native window 262144, dense,
|
|
# nextn_predict_layers=1. The MTP head ships in every main GGUF of this
|
|
# repo (README); --spec-type draft-mtp loads it against the main model's
|
|
# own weights.
|
|
#
|
|
# THE KV CONTRACT STAYS q8_0 K + q8_0 V (Frank, 2026-10-05): 34 KiB/token
|
|
# (16 attn layers x 4 heads x 256 x 1.0625 B, K and V).
|
|
#
|
|
# Why TWO slots + MTP fits now (it did not under IQ3_S):
|
|
# card 24,576 MiB. Weights 9,963 MiB (10,442,827,392 B; IQ3_S was
|
|
# 11,561). Projector ~1,290 MiB (the PUBLISHER's OWN mmproj-BF16.gguf,
|
|
# 931,145,888 B, at the measured ~1.45x card law — no substitution).
|
|
# Compute buffers at ubatch 512 ~1.3 GiB. CUDA ctx ~500 MiB. DeltaNet
|
|
# state ~150 MiB/slot x 2. MTP draft ctx back: ~0.4 GiB. Non-KV ~13.9
|
|
# GiB. Two q8_0 slots = 8,704 MiB. Predicted free ~2.6-3.2 GiB — back
|
|
# over the old 2-3 GiB floor that the IQ3_S two-slot boot violated
|
|
# (measured there: 1,397 MiB free). MEASURE nvidia-smi free at boot
|
|
# (quote the free field) and let the runbook carry the number.
|
|
#
|
|
# Payload history (rollback trail): /home/dank/models/Swift15-IQ3S/ (the
|
|
# IQ3_S-mtp that was live until this swap), Ornith-Q8/, MiMo-Q6K/ all
|
|
# stay on disk; one MODEL_FILE + shape edit back each.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf
|
|
|
|
# Vision: the publisher SHIPS the projector — mmproj-BF16.gguf
|
|
# (931,145,888 B) from the same repo, fetched by
|
|
# scripts/download-frank-small.sh. No substitution needed for this
|
|
# payload (the unsloth base-tower proof of 2026-10-05 remains the
|
|
# fallback argument for the IQ3_S rollback payload).
|
|
# Empty string = text-only on purpose.
|
|
MMPROJ_FILE=/home/dank/models/Swift15-UncIQ3XXS/mmproj-BF16.gguf
|
|
|
|
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
|
# harness models.yml id and the homepage probe do not move.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# TWO conversations at Frank's 128k cap (2026-10-06): --ctx-size is the
|
|
# POOL llama.cpp divides across slots; 262144 / 2 = 131,072 per
|
|
# conversation. Frank accepted the tight fit explicitly (arithmetic above);
|
|
# the per-slot window is the product contract, so models.yml stays at
|
|
# 131,072.
|
|
CTX_SIZE=262144
|
|
PARALLEL=2
|
|
|
|
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves);
|
|
# on the 9B the 1024->2048 rung measured +4.6 % prefill for 616 MiB, and
|
|
# Frank's 2026-10-06 order spends the other direction: ubatch 512 claws
|
|
# back ~0.3 GiB toward the second slot at some prefill cost. The pair is a
|
|
# measurement, not a law: ubatch 1024 is the first A/B back if a future
|
|
# boot lands with headroom (loud failure at graph capture if it does not
|
|
# fit).
|
|
BATCH_SIZE=4096
|
|
UBATCH_SIZE=512
|
|
|
|
# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V —
|
|
# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in
|
|
# the argv either way, and --flash-attn stays pinned on in the serve script:
|
|
# it is required for quantized V (the build carries
|
|
# GGML_CUDA_FA_ALL_QUANTS=ON since 2026-09-17, so q8_0 KV is a real path,
|
|
# not a silent f16 fallback).
|
|
CACHE_TYPE_K=q8_0
|
|
CACHE_TYPE_V=q8_0
|
|
|
|
# Speculative decoding is ON (Frank, 2026-10-06 evening): draft-mtp at
|
|
# depth 3 against the head IN the file (the -mtp variant ships the
|
|
# blk.64.nextn.* tensors). ngram-mod stays skipped — on a qwen35 hybrid
|
|
# it disables recurrent-state rollback, so a rejected draft would
|
|
# corrupt the DeltaNet state; MTP needs no such bypass (its draft ctx
|
|
# re-derives state from the target's own weights). The flags move as a
|
|
# set: the serve script emits BOTH only when SPEC_TYPE names a drafter.
|
|
SPEC_TYPE=draft-mtp
|
|
SPEC_DRAFT_N_MAX=3
|
|
|
|
NGL=99
|