docs: record the two-slot no-spec frank-small shape (measured 1,397 MiB free, 48.0 t/s)
Runbook reshaped: 2 x 131072 no-spec identity, slot-2 arithmetic (drafter ~0.4 GiB + ubatch ~0.3 GiB bought the second 4,352 MiB KV slot), ngram-mod skipped on the recurrent-rollback law, MTP references kept (44.8-52.1% acceptance, 58.9-66.9 t/s), rollback = one revert of 812d9eb. AGENTS fleet row + section, LLM/index.md intro + entry, root index entry all flipped to the 2026-10-06 shape. 360/360 suite green; e2e gate 33/33 green at deploy.
This commit is contained in:
Frank
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 1 x 131072, q8_0 KV"
|
||||
description: "The active frank-small runbook (2026-10-05 evening) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on the RTX 3090: one 131,072-token conversation at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the arithmetic that makes one slot the maximum."
|
||||
title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 2 x 131072, no spec, q8_0 KV"
|
||||
description: "The active frank-small runbook (2026-10-05 evening; 2 × 131,072 no-spec since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S (the -mtp file's head present but NOT drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap (the tight fit he ordered — 1,397 MiB measured free), q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the full slot-2 arithmetic."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
@@ -15,16 +15,21 @@ tags:
|
||||
timestamp: 2026-10-05
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 1 × 131,072, q8_0 KV
|
||||
# frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 2 × 131,072, no spec, q8_0 KV
|
||||
|
||||
> **ACTIVE since 2026-10-05 (evening).** A 27B replaces the 9B on the same
|
||||
> 24 GiB card, bought by Frank's explicit contract for this swap: context
|
||||
> capped at 128k per conversation, KV at FP8 class, vision on, "as many of
|
||||
> those contexts as you can" — the answer turned out to be **one**, by
|
||||
> arithmetic, not by choice (see *Sizing*). This supersedes BOTH earlier
|
||||
> 2026-10-05 payloads: the live MiMo-V2.6 Q6_K shape
|
||||
> ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the Ornith-1.5-9B
|
||||
> Q8_0 swap that was committed (77decb5..380aafa) but never deployed.
|
||||
> **ACTIVE since 2026-10-05 (evening); TWO slots, spec OFF since
|
||||
> 2026-10-06.** A 27B replaces the 9B on the same 24 GiB card, bought by
|
||||
> Frank's explicit contract for the swap: context capped at 128k per
|
||||
> conversation, KV at FP8 class, vision on, "as many of those contexts as
|
||||
> you can" — the 2026-10-05 arithmetic answered **one**; Frank's 2026-10-06
|
||||
> order ("drop mtp, skip ngram, drop ubatch to 512, let's get 2 contexts at
|
||||
> 128k — should be a very tight fit") bought the second slot by spending
|
||||
> the drafter (~0.4 GiB) and ubatch headroom (~0.3 GiB). It fits:
|
||||
> **1,397 MiB measured free**, the old 2-3 GiB floor knowingly violated.
|
||||
> This supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K
|
||||
> shape ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the
|
||||
> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never
|
||||
> deployed.
|
||||
|
||||
## Identity
|
||||
|
||||
@@ -38,7 +43,7 @@ timestamp: 2026-10-05
|
||||
| Weights | `/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` — ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp**, 12,120,016,896 B, sha256 `9aecf1cd…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`) |
|
||||
| Projector | `…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf` — unsloth/Qwen3.8-27B-GGUF `mmproj-F16.gguf`, 927,607,488 B, sha256 `cbb841a9…` (27-layer ViT, 1152 hidden, `qwen3vl_merger` → 5120) |
|
||||
| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified per file; both verified on the host before deploy |
|
||||
| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, MTP pair, batch pair, projector opt-out) |
|
||||
| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, no-spec pair (absence of both flags), batch pair, projector opt-out) |
|
||||
|
||||
## Why a base-model projector is legal here
|
||||
|
||||
@@ -56,9 +61,11 @@ merge 2, ffn 4304, 16 heads, out 5120).
|
||||
|
||||
## Shape and KV contract
|
||||
|
||||
`--ctx-size 131072 --parallel 1` — ONE conversation at Frank's 128k cap
|
||||
(the GSQ publisher's own benchmark protocol ran at 131,072). `CTX_SIZE`
|
||||
stays divisible by `PARALLEL`; the seam enforces it.
|
||||
`--ctx-size 262144 --parallel 2` — TWO conversations at Frank's 128k cap
|
||||
(the GSQ publisher's own benchmark protocol ran at 131,072 per stream).
|
||||
`CTX_SIZE` stays divisible by `PARALLEL`; the seam enforces it, and the
|
||||
per-slot number (`262144 / 2 = 131,072`) is the product contract —
|
||||
models.yml stays at 131,072.
|
||||
|
||||
**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).**
|
||||
llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem
|
||||
@@ -71,28 +78,46 @@ pinned: quantized V requires it, and the build's
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than a
|
||||
silent f16 fallback.
|
||||
|
||||
**One slot is the maximum, by arithmetic.** Card 24,576 MiB; weights
|
||||
11,561 MiB; projector ~1,350 MiB (927,607,488 B at the measured ~1.45× card
|
||||
law); compute buffers at ubatch 1024 ~1.6 GiB; CUDA ~500 MiB; DeltaNet
|
||||
state ~150 MiB/slot (48 linear layers × 48 value heads × 128×128 × 4 B); MTP
|
||||
draft context ~0.3–0.5 GiB. Non-KV ≈ 15.5 GiB. One q8_0 slot = 4,352 MiB ⇒
|
||||
free ≈ 4.5 GiB. A second slot wants +4,352 MiB ⇒ ~0.2 GiB free, far under
|
||||
the 2–3 GiB floor. The pool is capped by the 128k law, not by the card —
|
||||
leftover headroom is honest slack, not an undersized pool.
|
||||
**Two slots, by arithmetic and by order (2026-10-06).** The 2026-10-05
|
||||
arithmetic made ONE 131,072 slot the maximum under the 2-3 GiB floor (a
|
||||
second slot wanted +4,352 MiB of KV against 4,721 free — ~0.2 GiB left,
|
||||
far under the floor). Frank then ordered the budget spent differently:
|
||||
drop the MTP drafter (~0.4 GiB: one nextn layer of draft KV + its compute
|
||||
buffers), drop ubatch 1024→512 (~0.3 GiB, inferred from the measured 616
|
||||
MiB/doubling rung on the 9B), and take the second slot anyway — "should
|
||||
be a very tight fit", i.e. the 2-3 GiB floor is KNOWINGLY VIOLATED. The
|
||||
measured boot confirms: **22,728 used / 1,397 MiB free** (nvidia-smi
|
||||
`free` field), and a 512×512 image still answers `Red` afterwards
|
||||
(1,363 MiB after the probe), so the fit carries the vision transients.
|
||||
Compute buffers allocate at graph capture during BOOT, so a miss is loud
|
||||
and fast — the failure mode is a crash loop, not silent corruption.
|
||||
|
||||
## MTP (speculative decoding)
|
||||
## Speculative decoding — OFF since 2026-10-06
|
||||
|
||||
The MTP head ships **in the file** — the `-mtp` GGUF carries
|
||||
`blk.64.nextn.{enorm,hnorm,shared_head_norm}` + the head layer's tensors at
|
||||
Q6_K/F32 (publisher RECIPE: "MTP_OK verified"),
|
||||
`qwen35.nextn_predict_layers=1`. `--spec-type draft-mtp` loads it against
|
||||
the main model's own weights — no `--model-draft`, no second draft KV pool
|
||||
(an MTP head can NEVER load as a separate draft model on this stack; two
|
||||
context-backed drafters cannot share a process). Depth 3
|
||||
(`--spec-draft-n-max 3`) matches the fleet's radiance MTP depth. The flags
|
||||
move as a set: a no-spec baseline must strip BOTH. Acceptance reads from
|
||||
the response `timings` object (`draft_n`, `draft_n_accepted`) — measure on
|
||||
real traffic, never on a synthetic repeat (greedy-text acceptance lies).
|
||||
**Frank's order: drop MTP, skip ngram-mod too.** The MTP head ships IN the
|
||||
file — the `-mtp` GGUF carries `blk.64.nextn.{enorm,hnorm,shared_head_norm}`
|
||||
(`qwen35.nextn_predict_layers=1`, publisher RECIPE "MTP_OK verified") — and
|
||||
`--spec-type draft-mtp` loads it against the main model's own weights (no
|
||||
`--model-draft`, no second draft model — two context-backed drafters cannot
|
||||
share a process). It simply is NOT drafted with now: `SPEC_TYPE=none`
|
||||
strips BOTH speculative flags in the serve script (flags move as a set).
|
||||
|
||||
**Why ngram-mod was skipped, not swapped in:** on a qwen35 hybrid, ngram
|
||||
speculation logs *"recurrent state rollback is not compatible with
|
||||
'ngram-mod' — disabling rollback support"* (fleet lesson from frank-large's
|
||||
chain measurement, 2026-10-03). A rejected draft on Gated DeltaNet layers
|
||||
cannot be rewound by KV truncation — without the rollback checkpoints the
|
||||
DeltaNet state carries phantom transitions and output degrades silently.
|
||||
MTP measured fine here (below); it was spent for the second slot, not
|
||||
abandoned for ngram.
|
||||
|
||||
**Measured MTP references (2026-10-05, for the day the head returns)**,
|
||||
draft counts from the live `timings` on real-ish traffic: temp 0.8 →
|
||||
66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s, 166/318;
|
||||
deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s, 159/355
|
||||
(44.8 %). Acceptance reads from the response `timings` object
|
||||
(`draft_n`, `draft_n_accepted`) — never on a synthetic repeat (greedy-text
|
||||
acceptance lies).
|
||||
|
||||
## Vision contract
|
||||
|
||||
@@ -102,7 +127,7 @@ probe undersupplies image tokens and the model invents ("video"); the
|
||||
legacy top-level `images` field is IGNORED, only the OpenAI `image_url`
|
||||
content part reaches the model. Same traps the MiMo eras documented.
|
||||
|
||||
## Measured at cutover (2026-10-05, live)
|
||||
## Measured: the 1 × 131,072 MTP boot (2026-10-05, historical)
|
||||
|
||||
- **Boot ledger**: `creating MTP draft context against the target model
|
||||
'…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal
|
||||
@@ -124,6 +149,22 @@ content part reaches the model. Same traps the MiMo eras documented.
|
||||
- Streaming completion: 200. No first-request silence window (the
|
||||
halogen-era 45–60 s stall has no analogue here).
|
||||
|
||||
## Measured: the 2 × 131,072 no-spec boot (2026-10-06, live)
|
||||
|
||||
- **Boot ledger**: `loaded multimodal model, '…/Swift15-IQ3S/
|
||||
mmproj-Qwen3.8-27B-F16.gguf'` and NO draft-context line (speculation
|
||||
off). `/slots`: **2 slots, n_ctx 131,072 each** (pool 262,144).
|
||||
- **VRAM: 22,728 used / 1,397 MiB free** (nvidia-smi `free` field) — the
|
||||
predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly
|
||||
violated by Frank's order. After a 512×512 image probe: 1,363 MiB —
|
||||
the vision transients fit.
|
||||
- **Decode without the drafter**: 48.0 tok/s (temp 0) / 47.8 (deployed
|
||||
recipe temp 1.0). `draft_n` absent from `timings` (no drafter). The MTP
|
||||
era's 58.9–66.9 tok/s is the honest price of the second context:
|
||||
−18–28 %.
|
||||
- **Vision**: 512×512 solid-red PNG → `Red` again on this shape.
|
||||
- Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet.
|
||||
|
||||
## Client contract
|
||||
|
||||
- Reasoning in `reasoning_content` (`--reasoning-format deepseek`); thinking
|
||||
@@ -147,6 +188,11 @@ content part reaches the model. Same traps the MiMo eras documented.
|
||||
|
||||
## Rollback
|
||||
|
||||
The 1 × 131,072 MTP shape is ONE commit away — revert 812d9eb (the
|
||||
two-slot order) and redeploy: the same env trio then ships
|
||||
`SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`, `CTX_SIZE=131072`,
|
||||
`PARALLEL=1` (the measured 58.9–66.9 tok/s with the head drafting).
|
||||
|
||||
Two payloads on disk, both fetchable by the same script's git-history
|
||||
recipes:
|
||||
|
||||
@@ -177,3 +223,7 @@ recipes:
|
||||
- The unit's `Description=` (MiMo era) was stale at cutover time and was
|
||||
updated in-repo afterward — the relinked unit picks it up at the next
|
||||
deploy/restart, not retroactively.
|
||||
- 2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts
|
||||
(812d9eb). The tight fit landed: 1,397 MiB free, vision `Red`, decode
|
||||
48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch
|
||||
~0.3 GiB bought the second 4,352 MiB KV slot.
|
||||
|
||||
@@ -16,9 +16,10 @@ ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
|
||||
engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision
|
||||
lives on frank-small)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: ukisai Swift-1.5-Qwen3.8-27B
|
||||
GSQ-RCO IQ3_S-mtp GGUF — 27B with the MTP head IN the file, ONE 131,072
|
||||
conversation at Frank's 128k cap, q8_0 K + q8_0 V (FP8 class), vision via
|
||||
the base-model projector, on the RTX 3090)**.
|
||||
GSQ-RCO IQ3_S-mtp GGUF — 27B (head present, NOT drafted), TWO 131,072
|
||||
conversations at Frank's 128k cap (tight fit, spec off since 2026-10-06),
|
||||
q8_0 K + q8_0 V (FP8 class), vision via the base-model projector, on the
|
||||
RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -81,14 +82,18 @@ frank-small, port 7771]
|
||||
Rollback: revert the swap commit; the ROCmFPX payload + fork build +
|
||||
`build-rocmfpx.sh` pin all remain on disk
|
||||
- [[frank-small-llama-swift15-iq3s-mtp-131072]] — **ACTIVE since 2026-10-05
|
||||
(evening)** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86)
|
||||
serving ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S-mtp** (27B, the
|
||||
checkpoint's own MTP head IN the file — `--spec-type draft-mtp`, depth 3)
|
||||
on the RTX 3090, ONE 131,072-token conversation at Frank's 128k cap (a
|
||||
second 131,072 slot misses the 2-3 GiB floor) with q8_0 K + q8_0 V (FP8
|
||||
class, 34 KiB/token vs 64 at f16) and vision via unsloth's
|
||||
mmproj-Qwen3.8-27B-F16.gguf — no Swift projector ships, and the base
|
||||
vision stack was hash-proven byte-identical to Swift's (6/6 tensors).
|
||||
(evening); 2 × 131,072 no-spec since 2026-10-06** (root systemd unit,
|
||||
port 7771): llama.cpp master (CUDA sm_86) serving ukisai/Swift-1.5-
|
||||
Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S-mtp** (27B; the -mtp file's head is
|
||||
present but NOT drafted — speculation OFF by Frank's order, ngram skipped
|
||||
on the recurrent-rollback law) on the RTX 3090, TWO 131,072-token
|
||||
conversations at Frank's 128k cap (pool 262144 / parallel 2 — the tight
|
||||
fit he ordered: **1,397 MiB measured free**, the 2-3 GiB floor knowingly
|
||||
violated) with q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16),
|
||||
ubatch 512, and vision via unsloth's mmproj-Qwen3.8-27B-F16.gguf — no
|
||||
Swift projector ships, and the base vision stack was hash-proven
|
||||
byte-identical to Swift's (6/6 tensors). Decode 48.0 t/s without the
|
||||
drafter (MTP measured 58.9–66.9; references kept in frank-small.env).
|
||||
The GSQ-RCO "IQ3_S" tier is a per-tensor allocation mix of stock types.
|
||||
Replaces BOTH same-day payloads: the live MiMo Q6_K shape (below) and
|
||||
the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
|
||||
|
||||
@@ -370,11 +370,13 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-llama-swift15-iq3s-mtp-131072]] — the ACTIVE frank-small
|
||||
runbook since 2026-10-05 (evening): llama.cpp master (CUDA sm_86) serving
|
||||
ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on
|
||||
the RTX 3090 — ONE 131,072-token conversation at Frank's 128k cap, q8_0
|
||||
K + q8_0 V, vision through the base-model projector; the arithmetic that
|
||||
makes one slot the maximum
|
||||
runbook since 2026-10-05 (evening; **2 × 131,072 no-spec since
|
||||
2026-10-06**): llama.cpp master (CUDA sm_86) serving ukisai's
|
||||
Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S (the -mtp file's head present but NOT
|
||||
drafted — speculation off) on the RTX 3090 — TWO 131,072-token
|
||||
conversations at Frank's 128k cap, the tight fit he ordered (1,397 MiB
|
||||
measured free), q8_0 K + q8_0 V, vision through the base-model
|
||||
projector, ~48 t/s without the drafter
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
|
||||
superseded by swift15): llama.cpp master serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at
|
||||
|
||||
Reference in New Issue
Block a user