frank-small: Swift-1.5 IQ3_S-mtp swap (parallel session) + fleet docs index
The parallel session's uncommitted swap, committed at Frank's direction (commit whatever's in the working tree): - llama.cpp master on the RTX 3090 now serves ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE 131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to rollback target. - env/serve/download + seam twins updated together (deployer payload gates rode the previous commit's deploy-remote.sh). - docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135 seam green). - comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/ workflows/ (12 KiB).
This commit is contained in:
Frank
+83
-116
@@ -1,79 +1,64 @@
|
||||
# frank-small — llama.cpp (master, CUDA sm_86) serving
|
||||
# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 on the RTX 3090, :7771.
|
||||
# ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S **-mtp** on the RTX 3090,
|
||||
# :7771.
|
||||
#
|
||||
# 2026-10-05 (Frank): the payload moved off bartowski/MiMo-V2.6 (Q6_K) to
|
||||
# ornith-ai/Ornith-1.5-9B Q8_0 (9,786,060,384 B, LFS oid 22086870...).
|
||||
# Header-verified facts that re-price the shape: qwen35 hybrid like MiMo, but
|
||||
# 33 blocks (not 32) with full_attention_interval 4 => NINE full-attention
|
||||
# layers (MiMo had 8), 16 Q heads / 4 KV heads, k/v length 256, native window
|
||||
# 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN
|
||||
# the file — capability only; no spec decode ships, same law as before).
|
||||
# 2026-10-05 (Frank, evening): the payload moved to Swift-1.5 Qwen3.8-27B
|
||||
# GSQ-RCO IQ3_S WITH the MTP head in the file, each conversation capped at
|
||||
# 131,072 tokens, KV at FP8 class, vision on. Header-verified facts that
|
||||
# re-price the shape (read from the Hub file 2026-10-05, HTTP range + struct
|
||||
# parse): qwen35 hybrid like MiMo/Ornith but bigger — block_count 65 (64 real
|
||||
# layers + the blk.64 MTP head), full_attention_interval 4 => SIXTEEN
|
||||
# full-attention layers (MiMo had 8, Ornith 9), 24 Q heads / 4 KV heads,
|
||||
# k/v length 256, native window 262144, dense, nextn_predict_layers=1 with
|
||||
# blk.64.nextn.{enorm,hnorm,shared_head_norm} (the -mtp file's head tensors
|
||||
# are Q6_K/F32 per the publisher RECIPE, "MTP_OK" verified).
|
||||
#
|
||||
# THE KV CONTRACT STAYS f16 K + f16 V — NO KV QUANTIZATION (Frank, 2026-10-05,
|
||||
# overruling a written q8_0-V proposal). The arithmetic that makes this hard:
|
||||
# KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again
|
||||
# for V at f16 => 36 KiB/token (MiMo was ~24.5 measured: one fewer
|
||||
# full-attn layer AND one fewer KV head).
|
||||
# Weights +1,900 MiB (9,334 vs Q6_K's 7,432). Measured free at the Q6_K
|
||||
# 4096/2048 shape was 2,279 MiB; Q8_0 at f16/f16 does not fit that shape —
|
||||
# the deficit is made up the way Frank ordered: UBATCH FIRST (2048 -> 1024,
|
||||
# the measured 616 MiB rung), then POOL STEPS, then measure. Guess-and-check
|
||||
# on the live card is the accepted method for llama.cpp sizing here: boot,
|
||||
# read nvidia-smi free, step the pool (12,288-token increments, ~432 MiB
|
||||
# each at the 36 KiB/token rate) until the 2-3 GiB floor holds. Compute
|
||||
# buffers allocate at graph capture during BOOT, so a failure is loud and
|
||||
# fast, not a surprise at the first long prompt.
|
||||
# THE KV CONTRACT IS NOW q8_0 K + q8_0 V — Frank's order for THIS swap
|
||||
# ("k/v at FP8", 2026-10-05), superseding the same-day f16/f16 rule, which
|
||||
# was written for the Ornith 9B fit ("no further quantization anywhere on
|
||||
# this endpoint"). llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit
|
||||
# class (1.0625 B/elem => 8.5 effective bits) and the only FP8-class lever
|
||||
# this runtime moves. Arithmetic that makes the order load-bearing:
|
||||
# KV rate = 16 attn layers x 4 heads x 256 x 1.0625 B, K and V again
|
||||
# => 34 KiB/token (f16 would be 64 KiB/token — halogen's 27B card quotes
|
||||
# the same f16 number; at f16 the 131,072 slot alone would cost 8.0 GiB).
|
||||
#
|
||||
# Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the
|
||||
# rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third
|
||||
# slot funded exact f16 V first; the ~3.5 GiB still free funded the +875 MiB
|
||||
# of weights.
|
||||
# Why ONE slot at 131,072 is the maximum (Frank's cap per conversation):
|
||||
# card 24,576 MiB. Weights 11,561 MiB (12,120,016,896 B, the IQ3_S-mtp
|
||||
# file; the GSQ "IQ3_S" tier is a per-tensor ALLOCATION mix — histogram
|
||||
# IQ2_XS 144 / IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 ...
|
||||
# all stock types, stock llama.cpp loads it). Projector ~1,350 MiB
|
||||
# (927,607,488 B F16 file at the measured ~1.45x card law). Compute
|
||||
# buffers at ubatch 1024 ~1.6 GiB. CUDA ctx ~500 MiB. DeltaNet state
|
||||
# ~150 MiB/slot (48 linear layers x 48 value heads x 128x128 x 4 B).
|
||||
# MTP draft context ~0.3-0.5 GiB (one nextn layer over the slot window;
|
||||
# the boot ledger is the truth). Non-KV ~= 15.5 GiB. One q8_0 slot =
|
||||
# 4,352 MiB => free ~= 4.5 GiB. A SECOND slot wants +4,352 MiB => free
|
||||
# ~0.2 GiB, far below the 2-3 GiB floor. "As many 128k contexts as fit"
|
||||
# = 1. MEASURE nvidia-smi free at boot (quote the free field): under
|
||||
# ~1.5 GiB or a boot failure, step the pool down in 12,288-token
|
||||
# increments (~410 MiB each at this rate). Compute buffers allocate at
|
||||
# graph capture during BOOT, so a failure is loud and fast.
|
||||
#
|
||||
# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704
|
||||
# tokens per conversation, above the 200,700 the harness compacts at — and the
|
||||
# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same
|
||||
# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given
|
||||
# up is exactly the VRAM that pays for unquantized V. No further quantization
|
||||
# anywhere on this endpoint.
|
||||
#
|
||||
# Three measurements on this card, this runtime, price the terms:
|
||||
# 2 x 262,144, no projector: 20,190 MiB used / 4,386 free
|
||||
# KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 +
|
||||
# DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB
|
||||
# 3 x 196,608, projector on: 22,972 MiB used / 1,153 free
|
||||
# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
|
||||
# 3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02
|
||||
# shape) => non-KV = 8,748 MiB
|
||||
# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V
|
||||
# (32 KiB/token) this pool is 12,544 MiB, so the prediction is
|
||||
# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02
|
||||
# shape hit, bought with precision instead of a slot. The 1024 ubatch
|
||||
# re-prices the compute buffers upward, so MEASURE after any change: if the
|
||||
# boot lands under 2 GiB free, step the pool down in 12,288-token
|
||||
# (384 MiB at this rate) increments.
|
||||
#
|
||||
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
||||
# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for
|
||||
# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that
|
||||
# is the ladder to walk if this moves again. Two native windows (2 x 262,144)
|
||||
# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV
|
||||
# against a card with ~15.7 GiB of KV room once weights, the vision tower, the
|
||||
# DeltaNet states and the graphs are loaded. 200,704 per stream is what full
|
||||
# precision buys; q8_0 on V would buy 262,144 instead.
|
||||
#
|
||||
# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01
|
||||
# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit,
|
||||
# and the third slot is gone. The flags stay explicit so the choice is visible
|
||||
# in the argv either way.
|
||||
# Payload history (for the rollback trail): the Ornith-1.5-9B Q8_0 swap of
|
||||
# earlier on 2026-10-05 (commits 77decb5..380aafa) was committed but never
|
||||
# deployed — the live service was still MiMo-V2.6 Q6_K (2 x 200,704, f16
|
||||
# KV) when Swift cut over. Rollback payloads on disk: Ornith-Q8/ and
|
||||
# MiMo-Q6K/ (one MODEL_FILE + shape edit back; see the runbook).
|
||||
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
||||
MODEL_FILE=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf
|
||||
MODEL_FILE=/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
|
||||
|
||||
# Vision: the Ornith Q8_0 GGUF ships no `v.*` tensors either (dense-9B byte
|
||||
# arithmetic: 9.79 GB ≈ 9B params x ~1.06 B/param leaves no room for a tower),
|
||||
# so the projector file IS the vision capability (mmproj-Ornith-1.5-9B-BF16
|
||||
# .gguf, 921,704,672 B, LFS oid 626f9f90..., fetched by
|
||||
# scripts/download-frank-small.sh). Empty string = text-only on purpose.
|
||||
MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf
|
||||
# Vision: no Swift repo ships a projector (ukisai's release is
|
||||
# language-model-only by their own README), so the projector comes from the
|
||||
# SHARED BASE — unsloth's mmproj-F16.gguf built from Qwen/Qwen3.8-27B,
|
||||
# stored as mmproj-Qwen3.8-27B-F16.gguf (927,607,488 B, LFS sha256
|
||||
# cbb841a9..., fetched by scripts/download-frank-small.sh). The substitution
|
||||
# is proven, not assumed: Swift-1.5's vision stack (tower + merger) was
|
||||
# hash-compared against the base on 2026-10-05 — 333/333 visual tensor names
|
||||
# and 6/6 sampled tensors byte-identical — so the base projector IS the
|
||||
# model's own projector (Swift's RL/OPD post-training touched text only).
|
||||
# Empty string = text-only on purpose.
|
||||
MMPROJ_FILE=/home/dank/models/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf
|
||||
|
||||
# Fleet endpoint: same port and alias as every frank-small before it, so the
|
||||
# harness models.yml id and the homepage probe do not move.
|
||||
@@ -81,56 +66,38 @@ ALIAS=frank-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7771
|
||||
|
||||
# ONE conversation at the NATIVE 262,144 window (Frank, 2026-10-05). The
|
||||
# two-slot shape died under Ornith and the NO-QUANTIZE law, by arithmetic:
|
||||
# the KV rate is the ARCHITECTURE's, not the quant's — 9 full-attn layers x
|
||||
# 4 KV heads x 256 dim x 2 B = 36 KiB/token at f16 K+V (MiMo measured 24.5),
|
||||
# so 2 x 200,704 wants ~14.1 GiB of KV on top of +1,900 MiB of Q8 weights
|
||||
# against 2,279 MiB measured free; ubatch 1024 claws back 616, and no
|
||||
# pool step fits BOTH the 2-3 GiB floor and the harness compaction
|
||||
# invariant (slot >= 191,148). One native slot fits with room; the window
|
||||
# per conversation RISES 200,704 -> 262,144. The cost: a second concurrent
|
||||
# request queues behind the released slot (subagent concurrency 2 -> 1).
|
||||
CTX_SIZE=262144
|
||||
# ONE conversation at Frank's 128k cap: --ctx-size is the POOL llama.cpp
|
||||
# divides across slots; 131072 / 1 = 131,072 per conversation. The cap is a
|
||||
# product decision (Swift's own benchmark protocol ran at 131,072), and a
|
||||
# second 131,072 slot does not fit the 2-3 GiB floor (arithmetic above).
|
||||
CTX_SIZE=131072
|
||||
PARALLEL=1
|
||||
|
||||
# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03,
|
||||
# so it ran on llama.cpp's own defaults (2048/512); the pair is now 4096/2048.
|
||||
# Both are tunables because the pair is a measurement, not a law — and the
|
||||
# measurement is CARD-SPECIFIC. On the 2x R9700 layer-split sibling the ubatch
|
||||
# curve peaks at 512 and every step up costs GiB of compute buffer for no
|
||||
# speed (see frank-medium-batch-ubatch). On THIS CUDA 3090 the curve climbs
|
||||
# monotonically to 2048: 710 -> 1188 t/s prefill from ubatch 1024 to 2048 on
|
||||
# the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep, with decode flat
|
||||
# to under 1 t/s on every rung. Ubatch is a prefill knob; decode never moves.
|
||||
#
|
||||
# The logical batch is free — two independent sweeps agree that at a fixed
|
||||
# ubatch, 2048/4096/8192 give identical prefill speed AND identical peak VRAM,
|
||||
# because the compute buffers are sized by the ubatch alone. Doubling it costs
|
||||
# nothing and buys nothing; it is here because it is the value the pair was
|
||||
# measured at.
|
||||
#
|
||||
# THE RISK IS ALL IN THE UBATCH. The 1024 -> 2048 step cost ~1,244 MiB of peak
|
||||
# graph VRAM on the 35B MoE (7,330 -> 8,574 MiB); this 9B hybrid has far
|
||||
# smaller compute tensors, so the delta should be well under that, but the
|
||||
# predecessor Q5_K_M shape recorded "ubatch 2048 failed the scratch boot" at
|
||||
# ~3,571 MiB free. The Q8_0 rung ships UBATCH 1024 (Frank, 2026-10-05: lower
|
||||
# ubatch FIRST — the measured 616 MiB step is the first VRAM the Q8 weights
|
||||
# claw back, before any pool move). Compute buffers are allocated at graph
|
||||
# capture during BOOT, so a failure is loud and fast, not a surprise at the
|
||||
# first long prompt. MEASURE nvidia-smi free immediately after the deploy:
|
||||
# under ~1.5 GiB, or a boot failure, step the POOL down next (12,288-token
|
||||
# increments, ~432 MiB each at the Ornith 36 KiB/token f16 KV rate).
|
||||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves);
|
||||
# on this CUDA 3090 the curve climbs monotonically to 2048, but Frank's
|
||||
# standing law for a resize is LOWER UBATCH FIRST (the measured 616 MiB rung
|
||||
# on the 9B; the 27B's buffers are larger, so the claw-back is worth more).
|
||||
# The pair is a measurement, not a law: if boot lands well above the floor,
|
||||
# ubatch 2048 is the first thing to A/B for prefill (loud failure at graph
|
||||
# capture if it does not fit).
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=1024
|
||||
|
||||
# The KV contract (2026-10-05, Frank): f16 K + f16 V — NO KV quantization.
|
||||
# The q8_0-V arithmetic was written and OVERRULED. The Q8 weights eat the
|
||||
# VRAM that funded f16 V on 2026-10-03, so the remedy ladder is UBATCH FIRST
|
||||
# (2048 -> 1024, the measured 616 MiB rung), then POOL STEPS — and
|
||||
# guess-and-check on the live card is the accepted method: boot, read
|
||||
# nvidia-smi, step down until the floor holds. --flash-attn stays pinned on
|
||||
# in the serve script: it is the fast path for f16 KV.
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V —
|
||||
# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in
|
||||
# the argv either way, and --flash-attn stays pinned on in the serve script:
|
||||
# it is required for quantized V (the build carries
|
||||
# GGML_CUDA_FA_ALL_QUANTS=ON since 2026-09-17, so q8_0 KV is a real path,
|
||||
# not a silent f16 fallback).
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=q8_0
|
||||
|
||||
# Speculative decoding: the MTP head is IN the -mtp file (blk.64.nextn.*,
|
||||
# nextn_predict_layers=1), so draft-mtp needs no second model and no second
|
||||
# KV pool — the head shares the main model's weights. Depth 3 matches the
|
||||
# fleet's radiance MTP depth; measure draft acceptance from the response
|
||||
# timings object and retune here if the head earns more.
|
||||
SPEC_TYPE=draft-mtp
|
||||
SPEC_DRAFT_N_MAX=3
|
||||
|
||||
NGL=99
|
||||
|
||||
Reference in New Issue
Block a user