Frank
13112a93f5
Boot ledger lines, 19,404 used / 4,721 MiB free (the q8_0 proof), decode arms (66.1 t/s @0.8, 66.9 @0, 58.9 @recipe 1.0 with acceptance 52.1%/52.2%/44.8%), vision Red probe, reasoning_effort xhigh/high contract, e2e 12-crawl transient + full-gate green rerun. Unit Description updated in-repo; the relinked unit picks it up at the next deploy/restart (no bounce for cosmetics). 360/360 suite green.
180 lines
10 KiB
Markdown
180 lines
10 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 1 x 131072, q8_0 KV"
|
||
description: "The active frank-small runbook (2026-10-05 evening) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on the RTX 3090: one 131,072-token conversation at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the arithmetic that makes one slot the maximum."
|
||
tags:
|
||
- llm
|
||
- frank-small
|
||
- dankclaw
|
||
- llama-cpp
|
||
- gguf
|
||
- rtx3090
|
||
- qwen35
|
||
- mtp
|
||
- vision
|
||
timestamp: 2026-10-05
|
||
---
|
||
|
||
# frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 1 × 131,072, q8_0 KV
|
||
|
||
> **ACTIVE since 2026-10-05 (evening).** A 27B replaces the 9B on the same
|
||
> 24 GiB card, bought by Frank's explicit contract for this swap: context
|
||
> capped at 128k per conversation, KV at FP8 class, vision on, "as many of
|
||
> those contexts as you can" — the answer turned out to be **one**, by
|
||
> arithmetic, not by choice (see *Sizing*). This supersedes BOTH earlier
|
||
> 2026-10-05 payloads: the live MiMo-V2.6 Q6_K shape
|
||
> ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the Ornith-1.5-9B
|
||
> Q8_0 swap that was committed (77decb5..380aafa) but never deployed.
|
||
|
||
## Identity
|
||
|
||
| | |
|
||
|---|---|
|
||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
|
||
| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) |
|
||
| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||
| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
|
||
| Weights | `/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` — ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp**, 12,120,016,896 B, sha256 `9aecf1cd…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`) |
|
||
| Projector | `…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf` — unsloth/Qwen3.8-27B-GGUF `mmproj-F16.gguf`, 927,607,488 B, sha256 `cbb841a9…` (27-layer ViT, 1152 hidden, `qwen3vl_merger` → 5120) |
|
||
| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified per file; both verified on the host before deploy |
|
||
| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, MTP pair, batch pair, projector opt-out) |
|
||
|
||
## Why a base-model projector is legal here
|
||
|
||
No Swift repo ships a projector — ukisai's GSQ-RCO release is
|
||
language-model-only by their own README ("A Swift 27B vision projector has
|
||
not been verified for this release"). The substitution is proven, not
|
||
assumed: Swift-1.5 is an RL/OPD post-train of `Qwen/Qwen3.8-27B`, and its
|
||
vision stack was hash-compared against the base on 2026-10-05 over HTTP
|
||
range reads of the two BF16 repos — **333/333 visual tensor names match and
|
||
6/6 sampled tensors are byte-identical** (merger `linear_fc1`/`linear_fc2`,
|
||
blocks 0/13/26 attn/mlp). Swift's post-training touched text only, so the
|
||
base projector IS the model's own projector. The mmproj header matches
|
||
Swift's `vision_config` field-for-field (depth 27, hidden 1152, patch 16,
|
||
merge 2, ffn 4304, 16 heads, out 5120).
|
||
|
||
## Shape and KV contract
|
||
|
||
`--ctx-size 131072 --parallel 1` — ONE conversation at Frank's 128k cap
|
||
(the GSQ publisher's own benchmark protocol ran at 131,072). `CTX_SIZE`
|
||
stays divisible by `PARALLEL`; the seam enforces it.
|
||
|
||
**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).**
|
||
llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem
|
||
⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves. The
|
||
order supersedes the same-day f16/f16 rule, which was written for the
|
||
Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would
|
||
cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers ×
|
||
4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is
|
||
pinned: quantized V requires it, and the build's
|
||
`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than a
|
||
silent f16 fallback.
|
||
|
||
**One slot is the maximum, by arithmetic.** Card 24,576 MiB; weights
|
||
11,561 MiB; projector ~1,350 MiB (927,607,488 B at the measured ~1.45× card
|
||
law); compute buffers at ubatch 1024 ~1.6 GiB; CUDA ~500 MiB; DeltaNet
|
||
state ~150 MiB/slot (48 linear layers × 48 value heads × 128×128 × 4 B); MTP
|
||
draft context ~0.3–0.5 GiB. Non-KV ≈ 15.5 GiB. One q8_0 slot = 4,352 MiB ⇒
|
||
free ≈ 4.5 GiB. A second slot wants +4,352 MiB ⇒ ~0.2 GiB free, far under
|
||
the 2–3 GiB floor. The pool is capped by the 128k law, not by the card —
|
||
leftover headroom is honest slack, not an undersized pool.
|
||
|
||
## MTP (speculative decoding)
|
||
|
||
The MTP head ships **in the file** — the `-mtp` GGUF carries
|
||
`blk.64.nextn.{enorm,hnorm,shared_head_norm}` + the head layer's tensors at
|
||
Q6_K/F32 (publisher RECIPE: "MTP_OK verified"),
|
||
`qwen35.nextn_predict_layers=1`. `--spec-type draft-mtp` loads it against
|
||
the main model's own weights — no `--model-draft`, no second draft KV pool
|
||
(an MTP head can NEVER load as a separate draft model on this stack; two
|
||
context-backed drafters cannot share a process). Depth 3
|
||
(`--spec-draft-n-max 3`) matches the fleet's radiance MTP depth. The flags
|
||
move as a set: a no-spec baseline must strip BOTH. Acceptance reads from
|
||
the response `timings` object (`draft_n`, `draft_n_accepted`) — measure on
|
||
real traffic, never on a synthetic repeat (greedy-text acceptance lies).
|
||
|
||
## Vision contract
|
||
|
||
`--mmproj` is the entire vision path (the GGUF carries no vision tensors).
|
||
Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny
|
||
probe undersupplies image tokens and the model invents ("video"); the
|
||
legacy top-level `images` field is IGNORED, only the OpenAI `image_url`
|
||
content part reaches the model. Same traps the MiMo eras documented.
|
||
|
||
## Measured at cutover (2026-10-05, live)
|
||
|
||
- **Boot ledger**: `creating MTP draft context against the target model
|
||
'…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal
|
||
model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf'`, `n_slots = 1,
|
||
n_ctx_slot = 131072` — every expected line, no q8→f16 fallback.
|
||
- **VRAM: 19,404 used / 4,721 MiB free** (nvidia-smi `free` field). The
|
||
4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free
|
||
number doubles as the q8_0 proof.
|
||
- **Decode + MTP acceptance** (draft counts from the live `timings`):
|
||
temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s,
|
||
166/318; **deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s,
|
||
159/355 (44.8 %)** — temperature buys fluency, costs acceptance and rate.
|
||
For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this 27B at
|
||
IQ3_S runs ~2/3 the rate with ~3× the capability class.
|
||
- **Vision**: 512×512 solid-red PNG → `Red`, `finish_reason: stop`.
|
||
- **Template gate**: `reasoning_effort: xhigh` → 200 with
|
||
`reasoning_content` populated; `high` → HTTP 500 (Jinja raise) — the
|
||
models.yml fold is mandatory, exactly as the MiMo eras found.
|
||
- Streaming completion: 200. No first-request silence window (the
|
||
halogen-era 45–60 s stall has no analogue here).
|
||
|
||
## Client contract
|
||
|
||
- Reasoning in `reasoning_content` (`--reasoning-format deepseek`); thinking
|
||
tokens COUNT against `max_tokens`.
|
||
- The embedded chat template gates `reasoning_effort` to
|
||
`{xhigh, medium, low}` — anything else is a raise/HTTP 500, so
|
||
`models.yml` `compat.reasoningEffortMap` folds the harness levels onto
|
||
that set (mandatory, same as every Qwen3.8-template endpoint).
|
||
- Sampling rides the wire (`extraBody`): Swift's publisher recipe temp 1.0
|
||
/ top_p 0.95 / top_k 20 / min_p 0 (the card was evaluated at exactly
|
||
this recipe at ctx 131,072).
|
||
- Swift 1.5 is a reasoning-efficient post-train: −58.5 % median thinking
|
||
tokens vs base at equal-or-better accuracy. Token budgets go further;
|
||
the 32,768 `maxTokens` is unchanged from the MiMo era and still clears
|
||
the compaction invariant (91,750 + 32,768 ≤ 131,072).
|
||
- The GSQ-RCO "IQ3_S" tier is a **per-tensor allocation mix** (IQ2_XS 144 /
|
||
IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …), all stock quant
|
||
types — stock llama.cpp loads it. Publisher KLD vs Swift BF16: 0.0513
|
||
(development), held-out 0.024–0.052 per domain; the MTP packaging is NOT
|
||
separately evaluated by the publisher.
|
||
|
||
## Rollback
|
||
|
||
Two payloads on disk, both fetchable by the same script's git-history
|
||
recipes:
|
||
|
||
1. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
|
||
this cutover: `/home/dank/models/MiMo-Q6K/`; revert
|
||
[[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
|
||
380aafa's parent), one `MODEL_FILE` + shape edit.
|
||
2. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
|
||
`/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
|
||
262,144 slot, f16/f16 KV).
|
||
|
||
## Trail
|
||
|
||
- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was
|
||
committed and pushed but its deploy never completed (blocked at the
|
||
whole-tree docs gate); the live service was still MiMo Q6_K when Swift
|
||
cut over. Both Ornith payloads were verified on disk before Swift.
|
||
- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were
|
||
read from the Hub file by HTTP range + struct parse — no download needed —
|
||
before the payload was fetched.
|
||
- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel) label-
|
||
committed both sets at Frank's direction — 1919ad7 (frank-primary radiance)
|
||
then baf33d0 (this swap: trio, seam, runbooks, fleet docs). Deploy ran
|
||
`deploy.sh dankclaw --service frank-small`; the service dispatch and health
|
||
gate went green; the first e2e run failed ONE spec (12-crawl,
|
||
`net::ERR_ABORTED` on `www.franksdank.cloud` — a transient, unrelated to
|
||
frank-small), green on rerun.
|
||
- The unit's `Description=` (MiMo era) was stale at cutover time and was
|
||
updated in-repo afterward — the relinked unit picks it up at the next
|
||
deploy/restart, not retroactively.
|