--- type: reference title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 2 x 131072, no spec, q8_0 KV" description: "The active frank-small runbook (2026-10-05 evening; 2 × 131,072 no-spec since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S (the -mtp file's head present but NOT drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap (the tight fit he ordered — 1,397 MiB measured free), q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the full slot-2 arithmetic." tags: - llm - frank-small - dankclaw - llama-cpp - gguf - rtx3090 - qwen35 - mtp - vision timestamp: 2026-10-05 --- # frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 2 × 131,072, no spec, q8_0 KV > **ACTIVE since 2026-10-05 (evening); TWO slots, spec OFF since > 2026-10-06.** A 27B replaces the 9B on the same 24 GiB card, bought by > Frank's explicit contract for the swap: context capped at 128k per > conversation, KV at FP8 class, vision on, "as many of those contexts as > you can" — the 2026-10-05 arithmetic answered **one**; Frank's 2026-10-06 > order ("drop mtp, skip ngram, drop ubatch to 512, let's get 2 contexts at > 128k — should be a very tight fit") bought the second slot by spending > the drafter (~0.4 GiB) and ubatch headroom (~0.3 GiB). It fits: > **1,397 MiB measured free**, the old 2-3 GiB floor knowingly violated. > This supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K > shape ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the > Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never > deployed. ## Identity | | | |---|---| | Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB | | Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface | | Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) | | Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` | | Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) | | Weights | `/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` — ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp**, 12,120,016,896 B, sha256 `9aecf1cd…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`) | | Projector | `…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf` — unsloth/Qwen3.8-27B-GGUF `mmproj-F16.gguf`, 927,607,488 B, sha256 `cbb841a9…` (27-layer ViT, 1152 hidden, `qwen3vl_merger` → 5120) | | Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified per file; both verified on the host before deploy | | Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, no-spec pair (absence of both flags), batch pair, projector opt-out) | ## Why a base-model projector is legal here No Swift repo ships a projector — ukisai's GSQ-RCO release is language-model-only by their own README ("A Swift 27B vision projector has not been verified for this release"). The substitution is proven, not assumed: Swift-1.5 is an RL/OPD post-train of `Qwen/Qwen3.8-27B`, and its vision stack was hash-compared against the base on 2026-10-05 over HTTP range reads of the two BF16 repos — **333/333 visual tensor names match and 6/6 sampled tensors are byte-identical** (merger `linear_fc1`/`linear_fc2`, blocks 0/13/26 attn/mlp). Swift's post-training touched text only, so the base projector IS the model's own projector. The mmproj header matches Swift's `vision_config` field-for-field (depth 27, hidden 1152, patch 16, merge 2, ffn 4304, 16 heads, out 5120). ## Shape and KV contract `--ctx-size 262144 --parallel 2` — TWO conversations at Frank's 128k cap (the GSQ publisher's own benchmark protocol ran at 131,072 per stream). `CTX_SIZE` stays divisible by `PARALLEL`; the seam enforces it, and the per-slot number (`262144 / 2 = 131,072`) is the product contract — models.yml stays at 131,072. **KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).** llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem ⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves. The order supersedes the same-day f16/f16 rule, which was written for the Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers × 4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is pinned: quantized V requires it, and the build's `GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than a silent f16 fallback. **Two slots, by arithmetic and by order (2026-10-06).** The 2026-10-05 arithmetic made ONE 131,072 slot the maximum under the 2-3 GiB floor (a second slot wanted +4,352 MiB of KV against 4,721 free — ~0.2 GiB left, far under the floor). Frank then ordered the budget spent differently: drop the MTP drafter (~0.4 GiB: one nextn layer of draft KV + its compute buffers), drop ubatch 1024→512 (~0.3 GiB, inferred from the measured 616 MiB/doubling rung on the 9B), and take the second slot anyway — "should be a very tight fit", i.e. the 2-3 GiB floor is KNOWINGLY VIOLATED. The measured boot confirms: **22,728 used / 1,397 MiB free** (nvidia-smi `free` field), and a 512×512 image still answers `Red` afterwards (1,363 MiB after the probe), so the fit carries the vision transients. Compute buffers allocate at graph capture during BOOT, so a miss is loud and fast — the failure mode is a crash loop, not silent corruption. ## Speculative decoding — OFF since 2026-10-06 **Frank's order: drop MTP, skip ngram-mod too.** The MTP head ships IN the file — the `-mtp` GGUF carries `blk.64.nextn.{enorm,hnorm,shared_head_norm}` (`qwen35.nextn_predict_layers=1`, publisher RECIPE "MTP_OK verified") — and `--spec-type draft-mtp` loads it against the main model's own weights (no `--model-draft`, no second draft model — two context-backed drafters cannot share a process). It simply is NOT drafted with now: `SPEC_TYPE=none` strips BOTH speculative flags in the serve script (flags move as a set). **Why ngram-mod was skipped, not swapped in:** on a qwen35 hybrid, ngram speculation logs *"recurrent state rollback is not compatible with 'ngram-mod' — disabling rollback support"* (fleet lesson from frank-large's chain measurement, 2026-10-03). A rejected draft on Gated DeltaNet layers cannot be rewound by KV truncation — without the rollback checkpoints the DeltaNet state carries phantom transitions and output degrades silently. MTP measured fine here (below); it was spent for the second slot, not abandoned for ngram. **Measured MTP references (2026-10-05, for the day the head returns)**, draft counts from the live `timings` on real-ish traffic: temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s, 166/318; deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s, 159/355 (44.8 %). Acceptance reads from the response `timings` object (`draft_n`, `draft_n_accepted`) — never on a synthetic repeat (greedy-text acceptance lies). ## Vision contract `--mmproj` is the entire vision path (the GGUF carries no vision tensors). Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny probe undersupplies image tokens and the model invents ("video"); the legacy top-level `images` field is IGNORED, only the OpenAI `image_url` content part reaches the model. Same traps the MiMo eras documented. ## Measured: the 1 × 131,072 MTP boot (2026-10-05, historical) - **Boot ledger**: `creating MTP draft context against the target model '…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf'`, `n_slots = 1, n_ctx_slot = 131072` — every expected line, no q8→f16 fallback. - **VRAM: 19,404 used / 4,721 MiB free** (nvidia-smi `free` field). The 4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free number doubles as the q8_0 proof. - **Decode + MTP acceptance** (draft counts from the live `timings`): temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s, 166/318; **deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s, 159/355 (44.8 %)** — temperature buys fluency, costs acceptance and rate. For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this 27B at IQ3_S runs ~2/3 the rate with ~3× the capability class. - **Vision**: 512×512 solid-red PNG → `Red`, `finish_reason: stop`. - **Template gate**: `reasoning_effort: xhigh` → 200 with `reasoning_content` populated; `high` → HTTP 500 (Jinja raise) — the models.yml fold is mandatory, exactly as the MiMo eras found. - Streaming completion: 200. No first-request silence window (the halogen-era 45–60 s stall has no analogue here). ## Measured: the 2 × 131,072 no-spec boot (2026-10-06, live) - **Boot ledger**: `loaded multimodal model, '…/Swift15-IQ3S/ mmproj-Qwen3.8-27B-F16.gguf'` and NO draft-context line (speculation off). `/slots`: **2 slots, n_ctx 131,072 each** (pool 262,144). - **VRAM: 22,728 used / 1,397 MiB free** (nvidia-smi `free` field) — the predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly violated by Frank's order. After a 512×512 image probe: 1,363 MiB — the vision transients fit. - **Decode without the drafter**: 48.0 tok/s (temp 0) / 47.8 (deployed recipe temp 1.0). `draft_n` absent from `timings` (no drafter). The MTP era's 58.9–66.9 tok/s is the honest price of the second context: −18–28 %. - **Vision**: 512×512 solid-red PNG → `Red` again on this shape. - Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet. ## Client contract - Reasoning in `reasoning_content` (`--reasoning-format deepseek`); thinking tokens COUNT against `max_tokens`. - The embedded chat template gates `reasoning_effort` to `{xhigh, medium, low}` — anything else is a raise/HTTP 500, so `models.yml` `compat.reasoningEffortMap` folds the harness levels onto that set (mandatory, same as every Qwen3.8-template endpoint). - Sampling rides the wire (`extraBody`): Swift's publisher recipe temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 (the card was evaluated at exactly this recipe at ctx 131,072). - Swift 1.5 is a reasoning-efficient post-train: −58.5 % median thinking tokens vs base at equal-or-better accuracy. Token budgets go further; the 32,768 `maxTokens` is unchanged from the MiMo era and still clears the compaction invariant (91,750 + 32,768 ≤ 131,072). - The GSQ-RCO "IQ3_S" tier is a **per-tensor allocation mix** (IQ2_XS 144 / IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …), all stock quant types — stock llama.cpp loads it. Publisher KLD vs Swift BF16: 0.0513 (development), held-out 0.024–0.052 per domain; the MTP packaging is NOT separately evaluated by the publisher. ## Rollback The 1 × 131,072 MTP shape is ONE commit away — revert 812d9eb (the two-slot order) and redeploy: the same env trio then ships `SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`, `CTX_SIZE=131072`, `PARALLEL=1` (the measured 58.9–66.9 tok/s with the head drafting). Two payloads on disk, both fetchable by the same script's git-history recipes: 1. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until this cutover: `/home/dank/models/MiMo-Q6K/`; revert [[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or 380aafa's parent), one `MODEL_FILE` + shape edit. 2. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap: `/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native 262,144 slot, f16/f16 KV). ## Trail - The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was committed and pushed but its deploy never completed (blocked at the whole-tree docs gate); the live service was still MiMo Q6_K when Swift cut over. Both Ornith payloads were verified on disk before Swift. - Swift header facts (block_count 65, interval 4, KV geometry, nextn) were read from the Hub file by HTTP range + struct parse — no download needed — before the payload was fetched. - Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel) label- committed both sets at Frank's direction — 1919ad7 (frank-primary radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs). Deploy ran `deploy.sh dankclaw --service frank-small`; the service dispatch and health gate went green; the first e2e run failed ONE spec (12-crawl, `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient, unrelated to frank-small), green on rerun. - The unit's `Description=` (MiMo era) was stale at cutover time and was updated in-repo afterward — the relinked unit picks it up at the next deploy/restart, not retroactively. - 2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts (812d9eb). The tight fit landed: 1,397 MiB free, vision `Red`, decode 48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch ~0.3 GiB bought the second 4,352 MiB KV slot.