frank-small — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 131,072, draft-mtp, q8_0 KV

ACTIVE since 2026-10-05 (evening); reshaped twice on 2026-10-06. A 27B replaces the 9B on the same 24 GiB card, bought by Frank's explicit contract for the swap: context capped at 128k per conversation, KV at FP8 class, vision on, "as many of those contexts as you can". The morning order ("drop mtp, skip ngram, drop ubatch to 512, let's get 2 contexts at 128k — should be a very tight fit") bought the second slot by spending the drafter (~0.4 GiB assumed) and ubatch headroom (~0.3 GiB): 1,397 MiB measured free. The evening order — swap to the UNCELERSED Swift-1.5 at RCO-IQ3_XXS and turn MTP back on — landed 489 MiB free: the lighter quant (−1.56 GiB of weights) funds the second slot, but the MTP draft ctx costs ~2.4 GiB measured, not the ~0.4 GiB assumed — the drafter's KV is built against the target's full context (the 262,144 draft window), not the 3-token proposal depth. The 2–3 GiB floor is knowingly violated, as ordered. This supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K shape (frank-small-llama-mimo-q6k-2x200704, now retired) and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed.

Identity


Host / device	Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB
Endpoint	`http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface
Runtime	llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V)
Unit	root systemd `frank-small.service` (`scripts/frank-small.service`), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity`
Serve script	`scripts/serve-frank-small.sh`, tunables `scripts/frank-small.env`
Weights	`/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF RCO-IQ3_XXS -mtp, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; MTP head in the file: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it)
Projector	`…/Swift15-UncIQ3XXS/mmproj-BF16.gguf` — PUBLISHER-SHIPPED in the same repo, 931,145,888 B, sha256 `19acd7fc…` (27-layer ViT → 5120)
Fetch	`scripts/download-frank-small.sh` — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes)
Seam	`tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out)

Why this payload (2026-10-06 evening)

Frank asked for the UNCELERSED Swift-1.5 with MTP back on. The RCO-IQ3_XXS is 1.56 GiB lighter than the IQ3_S-mtp it replaced — and that lighter weight is exactly what funds the MTP draft context (~2.4 GiB measured) PLUS the second 131,072 slot, while keeping the q8_0 KV contract and the two-slot shape Frank ordered that morning.

Uncensored, intentionally: the publisher reports 23/100 refusals vs Swift 1.5's 98/100 (KL 0.0884 against the censored base). The refusal escape hatch is the same as every payload here: revert and redeploy.

Shape and KV contract

--ctx-size 262144 --parallel 2 — TWO conversations at Frank's 128k cap (the GSQ publisher's own benchmark protocol ran at 131,072 per stream). CTX_SIZE stays divisible by PARALLEL; the seam enforces it, and the per-slot number (262144 / 2 = 131,072) is the product contract — models.yml stays at 131,072.

KV at FP8 class: q8_0 K + q8_0 V (Frank, 2026-10-05 evening). llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem ⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves. The order supersedes the same-day f16/f16 rule, which was written for the Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would cost 8.0 GiB of KV. At q8_0 the rate is 34 KiB/token (16 attn layers × 4 KV heads × 256 dim × 1.0625 B, K and again V). --flash-attn on is pinned: quantized V requires it, and the build's GGML_CUDA_FA_ALL_QUANTS=ON (2026-09-17) makes the path real rather than a silent f16 fallback.

Two slots + drafter, by arithmetic and by order (2026-10-06). The morning order bought the second slot by spending the drafter and ubatch headroom (1,397 MiB free). The evening swap put the drafter BACK and changed the arithmetic: the draft ctx is built against the target's FULL context (262,144 — the draft window, not the 3-token proposal depth), so it costs ~2.4 GiB measured (23,636 − 22,728 + 1,561), not the ~0.4 GiB the one-slot era assumed. The lighter quant pays for it: 23,636 used / 489 MiB free (nvidia-smi free field) — the tightest fit in frank-small's history, the 2–3 GiB floor knowingly violated, and the 512×512 vision transient (34 MiB) still fits and recovers. Compute buffers allocate at graph capture during BOOT, so a miss is loud and fast — the failure mode is a crash loop, not silent corruption.

Speculative decoding — ON since 2026-10-06 evening (draft-mtp, depth 3)

Frank's evening order: MTP back on. The MTP head ships IN the file — the -mtp GGUF carries blk.64.nextn.* (qwen35.nextn_predict_layers=1) — and --spec-type draft-mtp loads it against the main model's own weights (no --model-draft, no second draft model — two context-backed drafters cannot share a process). SPEC_DRAFT_N_MAX=3 is the proposal depth. ngram-mod stays skipped, not swapped in: on a qwen35 hybrid, ngram speculation logs "recurrent state rollback is not compatible with 'ngram-mod' — disabling rollback support" (fleet lesson from frank-large's chain measurement, 2026-10-03). A rejected draft on Gated DeltaNet layers cannot be rewound by KV truncation — without the rollback checkpoints the DeltaNet state carries phantom transitions and output degrades silently. The flags move as a set: the serve script emits BOTH --spec-type and --spec-draft-n-max together or neither (SPEC_TYPE=none strips both).

Vision contract

--mmproj is the entire vision path (the GGUF carries no vision tensors). Probes: a solid-color PNG ≥ 256 px answers its exact color — a tiny probe undersupplies image tokens and the model invents ("video"); the legacy top-level images field is IGNORED, only the OpenAI image_url content part reaches the model. New on this payload: the uncensored model THINKS more than the censored Swift 1.5 — a 64-token budget was eaten entirely by reasoning_content (finish_reason: length, empty content); 128 tokens clears it. Token budgets for vision probes must leave room for thinking.

Measured: the 1 × 131,072 MTP boot (2026-10-05, historical)

Boot ledger: creating MTP draft context against the target model '…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf', loaded multimodal model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf', n_slots = 1, n_ctx_slot = 131072 — every expected line, no q8→f16 fallback.
VRAM: 19,404 used / 4,721 MiB free (nvidia-smi free field). The 4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free number doubles as the q8_0 proof.
Decode + MTP acceptance (draft counts from the live timings): temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s, 166/318; deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s, 159/355 (44.8 %) — temperature buys fluency, costs acceptance and rate. For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this 27B at IQ3_S runs ~2/3 the rate with ~3× the capability class.
Vision: 512×512 solid-red PNG → Red, finish_reason: stop.
Template gate: reasoning_effort: xhigh → 200 with reasoning_content populated; high → HTTP 500 (Jinja raise) — the models.yml fold is mandatory, exactly as the MiMo eras found.
Streaming completion: 200. No first-request silence window (the halogen-era 45–60 s stall has no analogue here).

Measured: the 2 × 131,072 no-spec boot (2026-10-06 morning, historical)

Boot ledger: loaded multimodal model, '…/Swift15-IQ3S/ mmproj-Qwen3.8-27B-F16.gguf' and NO draft-context line (speculation off). /slots: 2 slots, n_ctx 131,072 each (pool 262,144).
VRAM: 22,728 used / 1,397 MiB free (nvidia-smi free field) — the predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly violated by Frank's order. After a 512×512 image probe: 1,363 MiB — the vision transients fit.
Decode without the drafter: 48.0 tok/s (temp 0) / 47.8 (deployed recipe temp 1.0). draft_n absent from timings (no drafter). The MTP era's 58.9–66.9 tok/s is the honest price of the second context: −18–28 %.
Vision: 512×512 solid-red PNG → Red again on this shape.
Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet.

Measured: the 2 × 131,072 draft-mtp boot (2026-10-06 evening, live)

Boot ledger: creating MTP draft context against the target model '…/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf', loaded multimodal model, '…/Swift15-UncIQ3XXS/mmproj-BF16.gguf', n_slots = 2, n_ctx_slot = 131072 — every expected line, no q8→f16 fallback.
VRAM: 23,636 used / 489 MiB free (nvidia-smi free field) — the draft ctx measured ~2.4 GiB, ~6× the ~0.4 GiB the one-slot era assumed; the lighter quant (−1.56 GiB) plus the morning's no-spec base still fund both slots. After a 512×512 image probe: 489 MiB — the vision transient (34 MiB) fits and recovers.
Decode + MTP acceptance (draft counts from the live timings, 512-token essay): temp 0 → 72.9 tok/s, 333/534 accepted (62.4 %); deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 63.3 tok/s, 305/612 (49.8 %). e2e traffic in the boot window read 0.52–0.67 acceptance. Against the no-spec shape (48.0 tok/s): +31 % at the recipe temp — the drafter earned its 2.4 GiB.
Vision: 512×512 solid-red PNG → Red at a 128-token budget (64 tokens: all thinking, empty content). prompt_tokens: 321 confirms the image part reached the model.
Prompt cache: cached_tokens: 317 of 321 on the second probe — the slot prefix cache is live.
Deploy: commit da36fa3, e2e gate 33/33 green on the live fleet.

Client contract

Reasoning in reasoning_content (--reasoning-format deepseek); thinking tokens COUNT against max_tokens. This payload thinks more than the censored Swift 1.5 (uncensored finetune) — short max_tokens budgets can be eaten entirely by thinking (finish_reason: length, empty content); the 32,768 maxTokens in models.yml is comfortably above.
The embedded chat template gates reasoning_effort to {xhigh, medium, low} — anything else is a raise/HTTP 500, so models.yml compat.reasoningEffortMap folds the harness levels onto that set (mandatory, same as every Qwen3.8-template endpoint).
Sampling rides the wire (extraBody): the publisher recipe temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 (evaluated at ctx 131,072).
The RCO-IQ3_XXS tier is a per-tensor allocation mix (IQ3_S 97 / BF16 96 / IQ3_XXS 83 / IQ2_S 72 / IQ4_XS 33 / IQ2_XS 32 / IQ2_XXS 30 / Q2_K 28 / Q4_K 19 / Q6_K 8 / IQ1_S 4 / IQ1_M 4) — all stock quant types; stock llama.cpp loads it. Publisher KLD vs Swift BF16: mean 0.002631, 90th 0.042136, max 34.49 — RCO is the publisher's "more experimental" family.

Rollback

The 2 × 131,072 no-spec IQ3_S-mtp shape is ONE commit away — revert da36fa3 and redeploy: the env trio then ships SPEC_TYPE=none, CTX_SIZE=262144, PARALLEL=2, UBATCH_SIZE=512 against /home/dank/models/Swift15-IQ3S/ (the IQ3_S-mtp payload, still on disk, measured 48.0 tok/s no-spec). The 1 × 131,072 MTP shape is two reverts back (812d9eb's parent): the same IQ3_S-mtp file with SPEC_TYPE=draft-mtp, UBATCH_SIZE=1024, CTX_SIZE=131072, PARALLEL=1 (measured 58.9–66.9 tok/s with the head drafting).

Three payloads on disk, all fetchable by the same script's git-history recipes:

Swift-1.5 IQ3_S-mtp — the payload live until this cutover: /home/dank/models/Swift15-IQ3S/; the IQ3_XXS swap above.
MiMo-V2.6 Q6_K 2×200,704 f16 KV — the payload that was LIVE until the 2026-10-05 cutover: /home/dank/models/MiMo-Q6K/; revert frank-small-llama-mimo-q6k-2x200704 (env trio at commit e5893dc or 380aafa's parent), one MODEL_FILE + shape edit.
Ornith-1.5-9B Q8_0 — the committed-but-never-deployed swap: /home/dank/models/Ornith-Q8/; revert to 380aafa's tree (ONE native 262,144 slot, f16/f16 KV).

Trail

The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was committed and pushed but its deploy never completed (blocked at the whole-tree docs gate); the live service was still MiMo Q6_K when Swift cut over. Both Ornith payloads were verified on disk before Swift.
Swift header facts (block_count 65, interval 4, KV geometry, nextn) were read from the Hub file by HTTP range + struct parse — no download needed — before the payload was fetched.
Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel) label-committed both sets at Frank's direction — 1919ad7 (frank-primary radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs). Deploy ran deploy.sh dankclaw --service frank-small; the service dispatch and health gate went green; the first e2e run failed ONE spec (12-crawl, net::ERR_ABORTED on www.franksdank.cloud — a transient, unrelated to frank-small), green on rerun.
The unit's Description= (MiMo era) was stale at cutover time and was updated in-repo afterward — the relinked unit picks it up at the next deploy/restart, not retroactively.
2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts (812d9eb). The tight fit landed: 1,397 MiB free, vision Red, decode 48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch ~0.3 GiB bought the second 4,352 MiB KV slot.
2026-10-06 (Frank, evening): swap to the UNCELERSED Swift-1.5 at RCO-IQ3_XXS, MTP back on (da36fa3). The lighter quant (−1.56 GiB) funds the draft ctx (~2.4 GiB measured — the drafter's KV scales with the target's full 262,144 context, not the 3-token proposal depth) plus the second 131,072 slot: 489 MiB free, vision Red at a 128-token budget, decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec).

15 KiB Raw Blame History Unescape Escape