frank-small docs: runbook renamed + rewritten for the RCO-IQ3_XXS MTP-on shape (489 MiB free, 63.3 t/s recipe, draft ctx ~2.4 GiB measured); AGENTS, vault indexes, models.yml comment, MiMo cross-link retargeted
This commit is contained in:
Frank
@@ -15,8 +15,8 @@ timestamp: 2026-10-03
|
||||
# frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
|
||||
|
||||
> **RETIRED 2026-10-05 (evening).** Superseded by
|
||||
> [[frank-small-llama-swift15-iq3s-mtp-131072]] — the Swift-1.5-27B GSQ-RCO
|
||||
> IQ3_S-mtp swap. Two facts for the trail: (1) the Ornith-1.5-9B Q8_0 swap
|
||||
> [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — the Swift-1.5-27B
|
||||
> RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). Two facts for the trail:
|
||||
> committed earlier the same day (77decb5..380aafa) was **never deployed** —
|
||||
> THIS MiMo shape was the live payload when Swift cut over; (2) the KV
|
||||
> contract Frank set here (f16 K + f16 V, "no further quantization on this
|
||||
|
||||
@@ -1,229 +0,0 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 2 x 131072, no spec, q8_0 KV"
|
||||
description: "The active frank-small runbook (2026-10-05 evening; 2 × 131,072 no-spec since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S (the -mtp file's head present but NOT drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap (the tight fit he ordered — 1,397 MiB measured free), q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the full slot-2 arithmetic."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
- rtx3090
|
||||
- qwen35
|
||||
- mtp
|
||||
- vision
|
||||
timestamp: 2026-10-05
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 2 × 131,072, no spec, q8_0 KV
|
||||
|
||||
> **ACTIVE since 2026-10-05 (evening); TWO slots, spec OFF since
|
||||
> 2026-10-06.** A 27B replaces the 9B on the same 24 GiB card, bought by
|
||||
> Frank's explicit contract for the swap: context capped at 128k per
|
||||
> conversation, KV at FP8 class, vision on, "as many of those contexts as
|
||||
> you can" — the 2026-10-05 arithmetic answered **one**; Frank's 2026-10-06
|
||||
> order ("drop mtp, skip ngram, drop ubatch to 512, let's get 2 contexts at
|
||||
> 128k — should be a very tight fit") bought the second slot by spending
|
||||
> the drafter (~0.4 GiB) and ubatch headroom (~0.3 GiB). It fits:
|
||||
> **1,397 MiB measured free**, the old 2-3 GiB floor knowingly violated.
|
||||
> This supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K
|
||||
> shape ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the
|
||||
> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never
|
||||
> deployed.
|
||||
|
||||
## Identity
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) |
|
||||
| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
|
||||
| Weights | `/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` — ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp**, 12,120,016,896 B, sha256 `9aecf1cd…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`) |
|
||||
| Projector | `…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf` — unsloth/Qwen3.8-27B-GGUF `mmproj-F16.gguf`, 927,607,488 B, sha256 `cbb841a9…` (27-layer ViT, 1152 hidden, `qwen3vl_merger` → 5120) |
|
||||
| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified per file; both verified on the host before deploy |
|
||||
| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, no-spec pair (absence of both flags), batch pair, projector opt-out) |
|
||||
|
||||
## Why a base-model projector is legal here
|
||||
|
||||
No Swift repo ships a projector — ukisai's GSQ-RCO release is
|
||||
language-model-only by their own README ("A Swift 27B vision projector has
|
||||
not been verified for this release"). The substitution is proven, not
|
||||
assumed: Swift-1.5 is an RL/OPD post-train of `Qwen/Qwen3.8-27B`, and its
|
||||
vision stack was hash-compared against the base on 2026-10-05 over HTTP
|
||||
range reads of the two BF16 repos — **333/333 visual tensor names match and
|
||||
6/6 sampled tensors are byte-identical** (merger `linear_fc1`/`linear_fc2`,
|
||||
blocks 0/13/26 attn/mlp). Swift's post-training touched text only, so the
|
||||
base projector IS the model's own projector. The mmproj header matches
|
||||
Swift's `vision_config` field-for-field (depth 27, hidden 1152, patch 16,
|
||||
merge 2, ffn 4304, 16 heads, out 5120).
|
||||
|
||||
## Shape and KV contract
|
||||
|
||||
`--ctx-size 262144 --parallel 2` — TWO conversations at Frank's 128k cap
|
||||
(the GSQ publisher's own benchmark protocol ran at 131,072 per stream).
|
||||
`CTX_SIZE` stays divisible by `PARALLEL`; the seam enforces it, and the
|
||||
per-slot number (`262144 / 2 = 131,072`) is the product contract —
|
||||
models.yml stays at 131,072.
|
||||
|
||||
**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).**
|
||||
llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem
|
||||
⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves. The
|
||||
order supersedes the same-day f16/f16 rule, which was written for the
|
||||
Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would
|
||||
cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers ×
|
||||
4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is
|
||||
pinned: quantized V requires it, and the build's
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than a
|
||||
silent f16 fallback.
|
||||
|
||||
**Two slots, by arithmetic and by order (2026-10-06).** The 2026-10-05
|
||||
arithmetic made ONE 131,072 slot the maximum under the 2-3 GiB floor (a
|
||||
second slot wanted +4,352 MiB of KV against 4,721 free — ~0.2 GiB left,
|
||||
far under the floor). Frank then ordered the budget spent differently:
|
||||
drop the MTP drafter (~0.4 GiB: one nextn layer of draft KV + its compute
|
||||
buffers), drop ubatch 1024→512 (~0.3 GiB, inferred from the measured 616
|
||||
MiB/doubling rung on the 9B), and take the second slot anyway — "should
|
||||
be a very tight fit", i.e. the 2-3 GiB floor is KNOWINGLY VIOLATED. The
|
||||
measured boot confirms: **22,728 used / 1,397 MiB free** (nvidia-smi
|
||||
`free` field), and a 512×512 image still answers `Red` afterwards
|
||||
(1,363 MiB after the probe), so the fit carries the vision transients.
|
||||
Compute buffers allocate at graph capture during BOOT, so a miss is loud
|
||||
and fast — the failure mode is a crash loop, not silent corruption.
|
||||
|
||||
## Speculative decoding — OFF since 2026-10-06
|
||||
|
||||
**Frank's order: drop MTP, skip ngram-mod too.** The MTP head ships IN the
|
||||
file — the `-mtp` GGUF carries `blk.64.nextn.{enorm,hnorm,shared_head_norm}`
|
||||
(`qwen35.nextn_predict_layers=1`, publisher RECIPE "MTP_OK verified") — and
|
||||
`--spec-type draft-mtp` loads it against the main model's own weights (no
|
||||
`--model-draft`, no second draft model — two context-backed drafters cannot
|
||||
share a process). It simply is NOT drafted with now: `SPEC_TYPE=none`
|
||||
strips BOTH speculative flags in the serve script (flags move as a set).
|
||||
|
||||
**Why ngram-mod was skipped, not swapped in:** on a qwen35 hybrid, ngram
|
||||
speculation logs *"recurrent state rollback is not compatible with
|
||||
'ngram-mod' — disabling rollback support"* (fleet lesson from frank-large's
|
||||
chain measurement, 2026-10-03). A rejected draft on Gated DeltaNet layers
|
||||
cannot be rewound by KV truncation — without the rollback checkpoints the
|
||||
DeltaNet state carries phantom transitions and output degrades silently.
|
||||
MTP measured fine here (below); it was spent for the second slot, not
|
||||
abandoned for ngram.
|
||||
|
||||
**Measured MTP references (2026-10-05, for the day the head returns)**,
|
||||
draft counts from the live `timings` on real-ish traffic: temp 0.8 →
|
||||
66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s, 166/318;
|
||||
deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s, 159/355
|
||||
(44.8 %). Acceptance reads from the response `timings` object
|
||||
(`draft_n`, `draft_n_accepted`) — never on a synthetic repeat (greedy-text
|
||||
acceptance lies).
|
||||
|
||||
## Vision contract
|
||||
|
||||
`--mmproj` is the entire vision path (the GGUF carries no vision tensors).
|
||||
Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny
|
||||
probe undersupplies image tokens and the model invents ("video"); the
|
||||
legacy top-level `images` field is IGNORED, only the OpenAI `image_url`
|
||||
content part reaches the model. Same traps the MiMo eras documented.
|
||||
|
||||
## Measured: the 1 × 131,072 MTP boot (2026-10-05, historical)
|
||||
|
||||
- **Boot ledger**: `creating MTP draft context against the target model
|
||||
'…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal
|
||||
model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf'`, `n_slots = 1,
|
||||
n_ctx_slot = 131072` — every expected line, no q8→f16 fallback.
|
||||
- **VRAM: 19,404 used / 4,721 MiB free** (nvidia-smi `free` field). The
|
||||
4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free
|
||||
number doubles as the q8_0 proof.
|
||||
- **Decode + MTP acceptance** (draft counts from the live `timings`):
|
||||
temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s,
|
||||
166/318; **deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s,
|
||||
159/355 (44.8 %)** — temperature buys fluency, costs acceptance and rate.
|
||||
For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this 27B at
|
||||
IQ3_S runs ~2/3 the rate with ~3× the capability class.
|
||||
- **Vision**: 512×512 solid-red PNG → `Red`, `finish_reason: stop`.
|
||||
- **Template gate**: `reasoning_effort: xhigh` → 200 with
|
||||
`reasoning_content` populated; `high` → HTTP 500 (Jinja raise) — the
|
||||
models.yml fold is mandatory, exactly as the MiMo eras found.
|
||||
- Streaming completion: 200. No first-request silence window (the
|
||||
halogen-era 45–60 s stall has no analogue here).
|
||||
|
||||
## Measured: the 2 × 131,072 no-spec boot (2026-10-06, live)
|
||||
|
||||
- **Boot ledger**: `loaded multimodal model, '…/Swift15-IQ3S/
|
||||
mmproj-Qwen3.8-27B-F16.gguf'` and NO draft-context line (speculation
|
||||
off). `/slots`: **2 slots, n_ctx 131,072 each** (pool 262,144).
|
||||
- **VRAM: 22,728 used / 1,397 MiB free** (nvidia-smi `free` field) — the
|
||||
predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly
|
||||
violated by Frank's order. After a 512×512 image probe: 1,363 MiB —
|
||||
the vision transients fit.
|
||||
- **Decode without the drafter**: 48.0 tok/s (temp 0) / 47.8 (deployed
|
||||
recipe temp 1.0). `draft_n` absent from `timings` (no drafter). The MTP
|
||||
era's 58.9–66.9 tok/s is the honest price of the second context:
|
||||
−18–28 %.
|
||||
- **Vision**: 512×512 solid-red PNG → `Red` again on this shape.
|
||||
- Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet.
|
||||
|
||||
## Client contract
|
||||
|
||||
- Reasoning in `reasoning_content` (`--reasoning-format deepseek`); thinking
|
||||
tokens COUNT against `max_tokens`.
|
||||
- The embedded chat template gates `reasoning_effort` to
|
||||
`{xhigh, medium, low}` — anything else is a raise/HTTP 500, so
|
||||
`models.yml` `compat.reasoningEffortMap` folds the harness levels onto
|
||||
that set (mandatory, same as every Qwen3.8-template endpoint).
|
||||
- Sampling rides the wire (`extraBody`): Swift's publisher recipe temp 1.0
|
||||
/ top_p 0.95 / top_k 20 / min_p 0 (the card was evaluated at exactly
|
||||
this recipe at ctx 131,072).
|
||||
- Swift 1.5 is a reasoning-efficient post-train: −58.5 % median thinking
|
||||
tokens vs base at equal-or-better accuracy. Token budgets go further;
|
||||
the 32,768 `maxTokens` is unchanged from the MiMo era and still clears
|
||||
the compaction invariant (91,750 + 32,768 ≤ 131,072).
|
||||
- The GSQ-RCO "IQ3_S" tier is a **per-tensor allocation mix** (IQ2_XS 144 /
|
||||
IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …), all stock quant
|
||||
types — stock llama.cpp loads it. Publisher KLD vs Swift BF16: 0.0513
|
||||
(development), held-out 0.024–0.052 per domain; the MTP packaging is NOT
|
||||
separately evaluated by the publisher.
|
||||
|
||||
## Rollback
|
||||
|
||||
The 1 × 131,072 MTP shape is ONE commit away — revert 812d9eb (the
|
||||
two-slot order) and redeploy: the same env trio then ships
|
||||
`SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`, `CTX_SIZE=131072`,
|
||||
`PARALLEL=1` (the measured 58.9–66.9 tok/s with the head drafting).
|
||||
|
||||
Two payloads on disk, both fetchable by the same script's git-history
|
||||
recipes:
|
||||
|
||||
1. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
|
||||
this cutover: `/home/dank/models/MiMo-Q6K/`; revert
|
||||
[[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
|
||||
380aafa's parent), one `MODEL_FILE` + shape edit.
|
||||
2. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
|
||||
`/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
|
||||
262,144 slot, f16/f16 KV).
|
||||
|
||||
## Trail
|
||||
|
||||
- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was
|
||||
committed and pushed but its deploy never completed (blocked at the
|
||||
whole-tree docs gate); the live service was still MiMo Q6_K when Swift
|
||||
cut over. Both Ornith payloads were verified on disk before Swift.
|
||||
- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were
|
||||
read from the Hub file by HTTP range + struct parse — no download needed —
|
||||
before the payload was fetched.
|
||||
- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel) label-
|
||||
committed both sets at Frank's direction — 1919ad7 (frank-primary radiance)
|
||||
then baf33d0 (this swap: trio, seam, runbooks, fleet docs). Deploy ran
|
||||
`deploy.sh dankclaw --service frank-small`; the service dispatch and health
|
||||
gate went green; the first e2e run failed ONE spec (12-crawl,
|
||||
`net::ERR_ABORTED` on `www.franksdank.cloud` — a transient, unrelated to
|
||||
frank-small), green on rerun.
|
||||
- The unit's `Description=` (MiMo era) was stale at cutover time and was
|
||||
updated in-repo afterward — the relinked unit picks it up at the next
|
||||
deploy/restart, not retroactively.
|
||||
- 2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts
|
||||
(812d9eb). The tight fit landed: 1,397 MiB free, vision `Red`, decode
|
||||
48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch
|
||||
~0.3 GiB bought the second 4,352 MiB KV slot.
|
||||
@@ -0,0 +1,262 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 131072, draft-mtp, q8_0 KV"
|
||||
description: "The active frank-small runbook (2026-10-05 evening; MTP back ON on the lighter RCO-IQ3_XXS since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- gguf
|
||||
- rtx3090
|
||||
- qwen35
|
||||
- mtp
|
||||
- vision
|
||||
timestamp: 2026-10-06
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 131,072, draft-mtp, q8_0 KV
|
||||
|
||||
> **ACTIVE since 2026-10-05 (evening); reshaped twice on 2026-10-06.** A
|
||||
> 27B replaces the 9B on the same 24 GiB card, bought by Frank's explicit
|
||||
> contract for the swap: context capped at 128k per conversation, KV at
|
||||
> FP8 class, vision on, "as many of those contexts as you can". The
|
||||
> morning order ("drop mtp, skip ngram, drop ubatch to 512, let's get 2
|
||||
> contexts at 128k — should be a very tight fit") bought the second slot
|
||||
> by spending the drafter (~0.4 GiB assumed) and ubatch headroom (~0.3
|
||||
> GiB): 1,397 MiB measured free. The evening order — swap to the
|
||||
> UNCELERSED Swift-1.5 at RCO-IQ3_XXS and turn MTP back on — landed
|
||||
> **489 MiB free**: the lighter quant (−1.56 GiB of weights) funds the
|
||||
> second slot, but the MTP draft ctx costs **~2.4 GiB measured, not the
|
||||
> ~0.4 GiB assumed** — the drafter's KV is built against the target's
|
||||
> full context (the 262,144 draft window), not the 3-token proposal
|
||||
> depth. The 2–3 GiB floor is knowingly violated, as ordered. This
|
||||
> supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K
|
||||
> shape ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the
|
||||
> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never
|
||||
> deployed.
|
||||
|
||||
## Identity
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
|
||||
| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
|
||||
| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) |
|
||||
| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
|
||||
| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
|
||||
| Weights | `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp**, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it) |
|
||||
| Projector | `…/Swift15-UncIQ3XXS/mmproj-BF16.gguf` — PUBLISHER-SHIPPED in the same repo, 931,145,888 B, sha256 `19acd7fc…` (27-layer ViT → 5120) |
|
||||
| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) |
|
||||
| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) |
|
||||
|
||||
## Why this payload (2026-10-06 evening)
|
||||
|
||||
Frank asked for the UNCELERSED Swift-1.5 with MTP back on. The RCO-IQ3_XXS
|
||||
is 1.56 GiB lighter than the IQ3_S-mtp it replaced — and that lighter
|
||||
weight is exactly what funds the MTP draft context (~2.4 GiB measured)
|
||||
PLUS the second 131,072 slot, while keeping the q8_0 KV contract and the
|
||||
two-slot shape Frank ordered that morning.
|
||||
|
||||
**Uncensored, intentionally:** the publisher reports 23/100 refusals vs
|
||||
Swift 1.5's 98/100 (KL 0.0884 against the censored base). The refusal
|
||||
escape hatch is the same as every payload here: revert and redeploy.
|
||||
|
||||
## Shape and KV contract
|
||||
|
||||
`--ctx-size 262144 --parallel 2` — TWO conversations at Frank's 128k cap
|
||||
(the GSQ publisher's own benchmark protocol ran at 131,072 per stream).
|
||||
`CTX_SIZE` stays divisible by `PARALLEL`; the seam enforces it, and the
|
||||
per-slot number (`262144 / 2 = 131,072`) is the product contract —
|
||||
models.yml stays at 131,072.
|
||||
|
||||
**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).**
|
||||
llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem
|
||||
⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves.
|
||||
The order supersedes the same-day f16/f16 rule, which was written for the
|
||||
Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would
|
||||
cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers ×
|
||||
4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is
|
||||
pinned: quantized V requires it, and the build's
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than
|
||||
a silent f16 fallback.
|
||||
|
||||
**Two slots + drafter, by arithmetic and by order (2026-10-06).** The
|
||||
morning order bought the second slot by spending the drafter and ubatch
|
||||
headroom (1,397 MiB free). The evening swap put the drafter BACK and
|
||||
changed the arithmetic: the draft ctx is built against the target's FULL
|
||||
context (262,144 — the draft window, not the 3-token proposal depth), so
|
||||
it costs **~2.4 GiB measured** (23,636 − 22,728 + 1,561), not the ~0.4 GiB
|
||||
the one-slot era assumed. The lighter quant pays for it: **23,636 used /
|
||||
489 MiB free** (nvidia-smi `free` field) — the tightest fit in
|
||||
frank-small's history, the 2–3 GiB floor knowingly violated, and the
|
||||
512×512 vision transient (34 MiB) still fits and recovers. Compute buffers
|
||||
allocate at graph capture during BOOT, so a miss is loud and fast — the
|
||||
failure mode is a crash loop, not silent corruption.
|
||||
|
||||
## Speculative decoding — ON since 2026-10-06 evening (draft-mtp, depth 3)
|
||||
|
||||
**Frank's evening order: MTP back on.** The MTP head ships IN the file —
|
||||
the `-mtp` GGUF carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`)
|
||||
— and `--spec-type draft-mtp` loads it against the main model's own
|
||||
weights (no `--model-draft`, no second draft model — two context-backed
|
||||
drafters cannot share a process). `SPEC_DRAFT_N_MAX=3` is the proposal
|
||||
depth. **ngram-mod stays skipped, not swapped in:** on a qwen35 hybrid,
|
||||
ngram speculation logs *"recurrent state rollback is not compatible with
|
||||
'ngram-mod' — disabling rollback support"* (fleet lesson from
|
||||
frank-large's chain measurement, 2026-10-03). A rejected draft on Gated
|
||||
DeltaNet layers cannot be rewound by KV truncation — without the rollback
|
||||
checkpoints the DeltaNet state carries phantom transitions and output
|
||||
degrades silently. The flags move as a set: the serve script emits BOTH
|
||||
`--spec-type` and `--spec-draft-n-max` together or neither
|
||||
(`SPEC_TYPE=none` strips both).
|
||||
|
||||
## Vision contract
|
||||
|
||||
`--mmproj` is the entire vision path (the GGUF carries no vision tensors).
|
||||
Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny
|
||||
probe undersupplies image tokens and the model invents ("video"); the
|
||||
legacy top-level `images` field is IGNORED, only the OpenAI `image_url`
|
||||
content part reaches the model. **New on this payload: the uncensored
|
||||
model THINKS more than the censored Swift 1.5** — a 64-token budget was
|
||||
eaten entirely by `reasoning_content` (`finish_reason: length`, empty
|
||||
`content`); 128 tokens clears it. Token budgets for vision probes must
|
||||
leave room for thinking.
|
||||
|
||||
## Measured: the 1 × 131,072 MTP boot (2026-10-05, historical)
|
||||
|
||||
- **Boot ledger**: `creating MTP draft context against the target model
|
||||
'…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal
|
||||
model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf'`, `n_slots = 1,
|
||||
n_ctx_slot = 131072` — every expected line, no q8→f16 fallback.
|
||||
- **VRAM: 19,404 used / 4,721 MiB free** (nvidia-smi `free` field). The
|
||||
4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free
|
||||
number doubles as the q8_0 proof.
|
||||
- **Decode + MTP acceptance** (draft counts from the live `timings`):
|
||||
temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s,
|
||||
166/318; **deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s,
|
||||
159/355 (44.8 %)** — temperature buys fluency, costs acceptance and
|
||||
rate. For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this
|
||||
27B at IQ3_S runs ~2/3 the rate with ~3× the capability class.
|
||||
- **Vision**: 512×512 solid-red PNG → `Red`, `finish_reason: stop`.
|
||||
- **Template gate**: `reasoning_effort: xhigh` → 200 with
|
||||
`reasoning_content` populated; `high` → HTTP 500 (Jinja raise) — the
|
||||
models.yml fold is mandatory, exactly as the MiMo eras found.
|
||||
- Streaming completion: 200. No first-request silence window (the
|
||||
halogen-era 45–60 s stall has no analogue here).
|
||||
|
||||
## Measured: the 2 × 131,072 no-spec boot (2026-10-06 morning, historical)
|
||||
|
||||
- **Boot ledger**: `loaded multimodal model, '…/Swift15-IQ3S/
|
||||
mmproj-Qwen3.8-27B-F16.gguf'` and NO draft-context line (speculation
|
||||
off). `/slots`: **2 slots, n_ctx 131,072 each** (pool 262,144).
|
||||
- **VRAM: 22,728 used / 1,397 MiB free** (nvidia-smi `free` field) — the
|
||||
predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly
|
||||
violated by Frank's order. After a 512×512 image probe: 1,363 MiB —
|
||||
the vision transients fit.
|
||||
- **Decode without the drafter**: 48.0 tok/s (temp 0) / 47.8 (deployed
|
||||
recipe temp 1.0). `draft_n` absent from `timings` (no drafter). The MTP
|
||||
era's 58.9–66.9 tok/s is the honest price of the second context:
|
||||
−18–28 %.
|
||||
- **Vision**: 512×512 solid-red PNG → `Red` again on this shape.
|
||||
- Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet.
|
||||
|
||||
## Measured: the 2 × 131,072 draft-mtp boot (2026-10-06 evening, live)
|
||||
|
||||
- **Boot ledger**: `creating MTP draft context against the target model
|
||||
'…/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf'`,
|
||||
`loaded multimodal model, '…/Swift15-UncIQ3XXS/mmproj-BF16.gguf'`,
|
||||
`n_slots = 2, n_ctx_slot = 131072` — every expected line, no q8→f16
|
||||
fallback.
|
||||
- **VRAM: 23,636 used / 489 MiB free** (nvidia-smi `free` field) — the
|
||||
draft ctx measured ~2.4 GiB, ~6× the ~0.4 GiB the one-slot era
|
||||
assumed; the lighter quant (−1.56 GiB) plus the morning's no-spec base
|
||||
still fund both slots. After a 512×512 image probe: 489 MiB — the vision
|
||||
transient (34 MiB) fits and recovers.
|
||||
- **Decode + MTP acceptance** (draft counts from the live `timings`,
|
||||
512-token essay): temp 0 → **72.9 tok/s**, 333/534 accepted (62.4 %);
|
||||
deployed recipe (temp 1.0/top_p 0.95/top_k 20) → **63.3 tok/s**,
|
||||
305/612 (49.8 %). e2e traffic in the boot window read 0.52–0.67
|
||||
acceptance. Against the no-spec shape (48.0 tok/s): **+31 % at the
|
||||
recipe temp** — the drafter earned its 2.4 GiB.
|
||||
- **Vision**: 512×512 solid-red PNG → `Red` at a 128-token budget (64
|
||||
tokens: all thinking, empty content). `prompt_tokens: 321` confirms the
|
||||
image part reached the model.
|
||||
- **Prompt cache**: `cached_tokens: 317` of 321 on the second probe — the
|
||||
slot prefix cache is live.
|
||||
- Deploy: commit da36fa3, e2e gate 33/33 green on the live fleet.
|
||||
|
||||
## Client contract
|
||||
|
||||
- Reasoning in `reasoning_content` (`--reasoning-format deepseek`);
|
||||
thinking tokens COUNT against `max_tokens`. **This payload thinks more
|
||||
than the censored Swift 1.5** (uncensored finetune) — short `max_tokens`
|
||||
budgets can be eaten entirely by thinking (`finish_reason: length`,
|
||||
empty content); the 32,768 `maxTokens` in models.yml is comfortably
|
||||
above.
|
||||
- The embedded chat template gates `reasoning_effort` to
|
||||
`{xhigh, medium, low}` — anything else is a raise/HTTP 500, so
|
||||
`models.yml` `compat.reasoningEffortMap` folds the harness levels onto
|
||||
that set (mandatory, same as every Qwen3.8-template endpoint).
|
||||
- Sampling rides the wire (`extraBody`): the publisher recipe temp 1.0 /
|
||||
top_p 0.95 / top_k 20 / min_p 0 (evaluated at ctx 131,072).
|
||||
- The RCO-IQ3_XXS tier is a **per-tensor allocation mix** (IQ3_S 97 /
|
||||
BF16 96 / IQ3_XXS 83 / IQ2_S 72 / IQ4_XS 33 / IQ2_XS 32 / IQ2_XXS 30 /
|
||||
Q2_K 28 / Q4_K 19 / Q6_K 8 / IQ1_S 4 / IQ1_M 4) — all stock quant types;
|
||||
stock llama.cpp loads it. Publisher KLD vs Swift BF16: mean 0.002631,
|
||||
90th 0.042136, max 34.49 — RCO is the publisher's "more experimental"
|
||||
family.
|
||||
|
||||
## Rollback
|
||||
|
||||
The 2 × 131,072 no-spec IQ3_S-mtp shape is ONE commit away — revert
|
||||
da36fa3 and redeploy: the env trio then ships `SPEC_TYPE=none`,
|
||||
`CTX_SIZE=262144`, `PARALLEL=2`, `UBATCH_SIZE=512` against
|
||||
`/home/dank/models/Swift15-IQ3S/` (the IQ3_S-mtp payload, still on disk,
|
||||
measured 48.0 tok/s no-spec). The 1 × 131,072 MTP shape is two reverts
|
||||
back (812d9eb's parent): the same IQ3_S-mtp file with
|
||||
`SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`, `CTX_SIZE=131072`, `PARALLEL=1`
|
||||
(measured 58.9–66.9 tok/s with the head drafting).
|
||||
|
||||
Three payloads on disk, all fetchable by the same script's git-history
|
||||
recipes:
|
||||
|
||||
1. **Swift-1.5 IQ3_S-mtp** — the payload live until this cutover:
|
||||
`/home/dank/models/Swift15-IQ3S/`; the IQ3_XXS swap above.
|
||||
2. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
|
||||
the 2026-10-05 cutover: `/home/dank/models/MiMo-Q6K/`; revert
|
||||
[[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
|
||||
380aafa's parent), one `MODEL_FILE` + shape edit.
|
||||
3. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
|
||||
`/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
|
||||
262,144 slot, f16/f16 KV).
|
||||
|
||||
## Trail
|
||||
|
||||
- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was
|
||||
committed and pushed but its deploy never completed (blocked at the
|
||||
whole-tree docs gate); the live service was still MiMo Q6_K when Swift
|
||||
cut over. Both Ornith payloads were verified on disk before Swift.
|
||||
- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were
|
||||
read from the Hub file by HTTP range + struct parse — no download needed
|
||||
— before the payload was fetched.
|
||||
- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel)
|
||||
label-committed both sets at Frank's direction — 1919ad7 (frank-primary
|
||||
radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs).
|
||||
Deploy ran `deploy.sh dankclaw --service frank-small`; the service
|
||||
dispatch and health gate went green; the first e2e run failed ONE spec
|
||||
(12-crawl, `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient,
|
||||
unrelated to frank-small), green on rerun.
|
||||
- The unit's `Description=` (MiMo era) was stale at cutover time and was
|
||||
updated in-repo afterward — the relinked unit picks it up at the next
|
||||
deploy/restart, not retroactively.
|
||||
- 2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts
|
||||
(812d9eb). The tight fit landed: 1,397 MiB free, vision `Red`, decode
|
||||
48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch
|
||||
~0.3 GiB bought the second 4,352 MiB KV slot.
|
||||
- 2026-10-06 (Frank, evening): swap to the UNCELERSED Swift-1.5 at
|
||||
RCO-IQ3_XXS, MTP back on (da36fa3). The lighter quant (−1.56 GiB) funds
|
||||
the draft ctx (~2.4 GiB measured — the drafter's KV scales with the
|
||||
target's full 262,144 context, not the 3-token proposal depth) plus the
|
||||
second 131,072 slot: 489 MiB free, vision `Red` at a 128-token budget,
|
||||
decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec).
|
||||
@@ -15,11 +15,12 @@ ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
|
||||
/halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other
|
||||
engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision
|
||||
lives on frank-small)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: ukisai Swift-1.5-Qwen3.8-27B
|
||||
GSQ-RCO IQ3_S-mtp GGUF — 27B (head present, NOT drafted), TWO 131,072
|
||||
conversations at Frank's 128k cap (tight fit, spec off since 2026-10-06),
|
||||
q8_0 K + q8_0 V (FP8 class), vision via the base-model projector, on the
|
||||
RTX 3090)**.
|
||||
**llama.cpp CUDA (frank-small, port 7771: ajgazin Swift-1.5-Qwen3.8-27B
|
||||
Uncensored Dynamic-MTP RCO-IQ3_XXS GGUF — 27B (the MTP head in the file
|
||||
IS drafted, draft-mtp depth 3), TWO 131,072 conversations at Frank's
|
||||
128k cap (the tightest fit in the endpoint's history — 489 MiB free),
|
||||
q8_0 K + q8_0 V (FP8 class), vision via the publisher's own BF16
|
||||
mmproj, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -50,8 +51,8 @@ frank-large, port 7770]
|
||||
1 x 200700 ctx, d2 KV, text-only]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[Swift-1.5-27B GSQ-RCO IQ3_S-mtp GGUF
|
||||
1 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision]
|
||||
H[Swift-1.5-27B Uncensored RCO-IQ3_XXS-mtp GGUF
|
||||
2 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -94,23 +95,23 @@ frank-small, port 7771]
|
||||
on this topology — the default tile stands. The deep harness roles
|
||||
(`slow`, `advisor`) rerouted to frank-primary; frank-large keeps
|
||||
OpenWebUI and shallow callers.
|
||||
- [[frank-small-llama-swift15-iq3s-mtp-131072]] — **ACTIVE since 2026-10-05
|
||||
(evening); 2 × 131,072 no-spec since 2026-10-06** (root systemd unit,
|
||||
port 7771): llama.cpp master (CUDA sm_86) serving ukisai/Swift-1.5-
|
||||
Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S-mtp** (27B; the -mtp file's head is
|
||||
present but NOT drafted — speculation OFF by Frank's order, ngram skipped
|
||||
on the recurrent-rollback law) on the RTX 3090, TWO 131,072-token
|
||||
conversations at Frank's 128k cap (pool 262144 / parallel 2 — the tight
|
||||
fit he ordered: **1,397 MiB measured free**, the 2-3 GiB floor knowingly
|
||||
violated) with q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16),
|
||||
ubatch 512, and vision via unsloth's mmproj-Qwen3.8-27B-F16.gguf — no
|
||||
Swift projector ships, and the base vision stack was hash-proven
|
||||
byte-identical to Swift's (6/6 tensors). Decode 48.0 t/s without the
|
||||
drafter (MTP measured 58.9–66.9; references kept in frank-small.env).
|
||||
The GSQ-RCO "IQ3_S" tier is a per-tensor allocation mix of stock types.
|
||||
Replaces BOTH same-day payloads: the live MiMo Q6_K shape (below) and
|
||||
the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
|
||||
never deployed
|
||||
- [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — **ACTIVE since
|
||||
2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since
|
||||
2026-10-06 evening** (root systemd unit, port 7771): llama.cpp master
|
||||
(CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-
|
||||
GGUF **RCO-IQ3_XXS-mtp** (27B; the MTP head in the file is DRAFTED —
|
||||
draft-mtp depth 3, ngram skipped on the recurrent-rollback law) on the
|
||||
RTX 3090, TWO 131,072-token conversations at Frank's 128k cap (pool
|
||||
262144 / parallel 2 — the tightest fit in the endpoint's history:
|
||||
**489 MiB measured free**, the 2-3 GiB floor knowingly violated) with
|
||||
q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16), ubatch 512, and
|
||||
vision through the publisher's own mmproj-BF16.gguf. Decode 63.3 t/s
|
||||
recipe / 72.9 temp 0 (MTP acceptance 49.8–62.4 %; 48.0 no-spec). The
|
||||
RCO-IQ3_XXS tier is a per-tensor allocation mix of stock types; the
|
||||
payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's
|
||||
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
|
||||
(below) and the Ornith-1.5-9B Q8_0 swap that was committed
|
||||
(77decb5..380aafa) but never deployed
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
|
||||
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
|
||||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
|
||||
|
||||
@@ -369,14 +369,14 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-llama-swift15-iq3s-mtp-131072]] — the ACTIVE frank-small
|
||||
runbook since 2026-10-05 (evening; **2 × 131,072 no-spec since
|
||||
2026-10-06**): llama.cpp master (CUDA sm_86) serving ukisai's
|
||||
Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S (the -mtp file's head present but NOT
|
||||
drafted — speculation off) on the RTX 3090 — TWO 131,072-token
|
||||
conversations at Frank's 128k cap, the tight fit he ordered (1,397 MiB
|
||||
measured free), q8_0 K + q8_0 V, vision through the base-model
|
||||
projector, ~48 t/s without the drafter
|
||||
- [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — the ACTIVE
|
||||
frank-small runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS
|
||||
with MTP back on since 2026-10-06 evening**): llama.cpp master (CUDA
|
||||
sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS
|
||||
(the MTP head in the file is drafted) on the RTX 3090 — TWO
|
||||
131,072-token conversations at Frank's 128k cap, the tightest fit in
|
||||
the endpoint's history (489 MiB measured free), q8_0 K + q8_0 V, vision
|
||||
through the publisher's own projector, ~63 t/s recipe decode with MTP
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
|
||||
superseded by swift15): llama.cpp master serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at
|
||||
|
||||
Reference in New Issue
Block a user