serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 1 x 131072, q8_0 KV	The active frank-small runbook (2026-10-05 evening) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on the RTX 3090: one 131,072-token conversation at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the arithmetic that makes one slot the maximum.	llm frank-small dankclaw llama-cpp gguf rtx3090 qwen35 mtp vision	2026-10-05

frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 1 × 131,072, q8_0 KV

ACTIVE since 2026-10-05 (evening). A 27B replaces the 9B on the same 24 GiB card, bought by Frank's explicit contract for this swap: context capped at 128k per conversation, KV at FP8 class, vision on, "as many of those contexts as you can" — the answer turned out to be one, by arithmetic, not by choice (see Sizing). This supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K shape (frank-small-llama-mimo-q6k-2x200704, now retired) and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed.

Identity

Host / device	Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB
Endpoint	http://192.168.0.143:7771 — alias frank-small, /v1 OpenAI surface
Runtime	llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build …/llama-master/build-cuda/bin/llama-server (GGML_CUDA_FA_ALL_QUANTS=ON — required for quantized V)
Unit	root systemd frank-small.service (scripts/frank-small.service), User=dank, Restart=on-failure, LimitMEMLOCK=infinity
Serve script	scripts/serve-frank-small.sh, tunables scripts/frank-small.env
Weights	/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf — ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF IQ3_S -mtp, 12,120,016,896 B, sha256 9aecf1cd… (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; MTP head in the file: blk.64.nextn.*, nextn_predict_layers=1)
Projector	…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf — unsloth/Qwen3.8-27B-GGUF mmproj-F16.gguf, 927,607,488 B, sha256 cbb841a9… (27-layer ViT, 1152 hidden, qwen3vl_merger → 5120)
Fetch	scripts/download-frank-small.sh — -f, byte count + LFS sha256 verified per file; both verified on the host before deploy
Seam	tests/frank_small_unit.bats (argv assertions against a stub engine: payload path, slot geometry, KV pair, MTP pair, batch pair, projector opt-out)

Why a base-model projector is legal here

No Swift repo ships a projector — ukisai's GSQ-RCO release is language-model-only by their own README ("A Swift 27B vision projector has not been verified for this release"). The substitution is proven, not assumed: Swift-1.5 is an RL/OPD post-train of Qwen/Qwen3.8-27B, and its vision stack was hash-compared against the base on 2026-10-05 over HTTP range reads of the two BF16 repos — 333/333 visual tensor names match and 6/6 sampled tensors are byte-identical (merger linear_fc1/linear_fc2, blocks 0/13/26 attn/mlp). Swift's post-training touched text only, so the base projector IS the model's own projector. The mmproj header matches Swift's vision_config field-for-field (depth 27, hidden 1152, patch 16, merge 2, ffn 4304, 16 heads, out 5120).

Shape and KV contract

--ctx-size 131072 --parallel 1 — ONE conversation at Frank's 128k cap (the GSQ publisher's own benchmark protocol ran at 131,072). CTX_SIZE stays divisible by PARALLEL; the seam enforces it.

KV at FP8 class: q8_0 K + q8_0 V (Frank, 2026-10-05 evening). llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem ⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves. The order supersedes the same-day f16/f16 rule, which was written for the Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would cost 8.0 GiB of KV. At q8_0 the rate is 34 KiB/token (16 attn layers × 4 KV heads × 256 dim × 1.0625 B, K and again V). --flash-attn on is pinned: quantized V requires it, and the build's GGML_CUDA_FA_ALL_QUANTS=ON (2026-09-17) makes the path real rather than a silent f16 fallback.

One slot is the maximum, by arithmetic. Card 24,576 MiB; weights 11,561 MiB; projector ~1,350 MiB (927,607,488 B at the measured ~1.45× card law); compute buffers at ubatch 1024 ~1.6 GiB; CUDA ~500 MiB; DeltaNet state ~150 MiB/slot (48 linear layers × 48 value heads × 128×128 × 4 B); MTP draft context ~0.3–0.5 GiB. Non-KV ≈ 15.5 GiB. One q8_0 slot = 4,352 MiB ⇒ free ≈ 4.5 GiB. A second slot wants +4,352 MiB ⇒ ~0.2 GiB free, far under the 2–3 GiB floor. The pool is capped by the 128k law, not by the card — leftover headroom is honest slack, not an undersized pool.

MTP (speculative decoding)

The MTP head ships in the file — the -mtp GGUF carries blk.64.nextn.{enorm,hnorm,shared_head_norm} + the head layer's tensors at Q6_K/F32 (publisher RECIPE: "MTP_OK verified"), qwen35.nextn_predict_layers=1. --spec-type draft-mtp loads it against the main model's own weights — no --model-draft, no second draft KV pool (an MTP head can NEVER load as a separate draft model on this stack; two context-backed drafters cannot share a process). Depth 3 (--spec-draft-n-max 3) matches the fleet's radiance MTP depth. The flags move as a set: a no-spec baseline must strip BOTH. Acceptance reads from the response timings object (draft_n, draft_n_accepted) — measure on real traffic, never on a synthetic repeat (greedy-text acceptance lies).

Vision contract

--mmproj is the entire vision path (the GGUF carries no vision tensors). Probes: a solid-color PNG ≥ 256 px answers its exact color — a tiny probe undersupplies image tokens and the model invents ("video"); the legacy top-level images field is IGNORED, only the OpenAI image_url content part reaches the model. Same traps the MiMo eras documented.

Client contract

• Reasoning in reasoning_content (--reasoning-format deepseek); thinking tokens COUNT against max_tokens.
• The embedded chat template gates reasoning_effort to {xhigh, medium, low} — anything else is a raise/HTTP 500, so models.yml compat.reasoningEffortMap folds the harness levels onto that set (mandatory, same as every Qwen3.8-template endpoint).
• Sampling rides the wire (extraBody): Swift's publisher recipe temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 (the card was evaluated at exactly this recipe at ctx 131,072).
• Swift 1.5 is a reasoning-efficient post-train: −58.5 % median thinking tokens vs base at equal-or-better accuracy. Token budgets go further; the 32,768 maxTokens is unchanged from the MiMo era and still clears the compaction invariant (91,750 + 32,768 ≤ 131,072).
• The GSQ-RCO "IQ3_S" tier is a per-tensor allocation mix (IQ2_XS 144 / IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …), all stock quant types — stock llama.cpp loads it. Publisher KLD vs Swift BF16: 0.0513 (development), held-out 0.024–0.052 per domain; the MTP packaging is NOT separately evaluated by the publisher.

Rollback

Two payloads on disk, both fetchable by the same script's git-history recipes:

1. MiMo-V2.6 Q6_K 2×200,704 f16 KV — the payload that was LIVE until this cutover: /home/dank/models/MiMo-Q6K/; revert frank-small-llama-mimo-q6k-2x200704 (env trio at commit e5893dc or 380aafa's parent), one MODEL_FILE + shape edit.
2. Ornith-1.5-9B Q8_0 — the committed-but-never-deployed swap: /home/dank/models/Ornith-Q8/; revert to 380aafa's tree (ONE native 262,144 slot, f16/f16 KV).

Trail

• The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was committed and pushed but its deploy never completed (blocked at the whole-tree docs gate); the live service was still MiMo Q6_K when Swift cut over. Both Ornith payloads were verified on disk before Swift.
• Swift header facts (block_count 65, interval 4, KV geometry, nextn) were read from the Hub file by HTTP range + struct parse — no download needed — before the payload was fetched.