frank-small docs: runbook renamed + rewritten for the RCO-IQ3_XXS MTP-on shape (489 MiB free, 63.3 t/s recipe, draft ctx ~2.4 GiB measured); AGENTS, vault indexes, models.yml comment, MiMo cross-link retargeted

serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md

@@ -0,0 +1,262 @@
+---
+type: reference
+title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 131072, draft-mtp, q8_0 KV"
+description: "The active frank-small runbook (2026-10-05 evening; MTP back ON on the lighter RCO-IQ3_XXS since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - llama-cpp
+  - gguf
+  - rtx3090
+  - qwen35
+  - mtp
+  - vision
+timestamp: 2026-10-06
+---
+
+# frank-small — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 131,072, draft-mtp, q8_0 KV
+
+> **ACTIVE since 2026-10-05 (evening); reshaped twice on 2026-10-06.** A
+> 27B replaces the 9B on the same 24 GiB card, bought by Frank's explicit
+> contract for the swap: context capped at 128k per conversation, KV at
+> FP8 class, vision on, "as many of those contexts as you can". The
+> morning order ("drop mtp, skip ngram, drop ubatch to 512, let's get 2
+> contexts at 128k — should be a very tight fit") bought the second slot
+> by spending the drafter (~0.4 GiB assumed) and ubatch headroom (~0.3
+> GiB): 1,397 MiB measured free. The evening order — swap to the
+> UNCELERSED Swift-1.5 at RCO-IQ3_XXS and turn MTP back on — landed
+> **489 MiB free**: the lighter quant (−1.56 GiB of weights) funds the
+> second slot, but the MTP draft ctx costs **~2.4 GiB measured, not the
+> ~0.4 GiB assumed** — the drafter's KV is built against the target's
+> full context (the 262,144 draft window), not the 3-token proposal
+> depth. The 2–3 GiB floor is knowingly violated, as ordered. This
+> supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K
+> shape ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the
+> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never
+> deployed.
+
+## Identity
+
+| | |
+|---|---|
+| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
+| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface |
+| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) |
+| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
+| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) |
+| Weights | `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp**, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it) |
+| Projector | `…/Swift15-UncIQ3XXS/mmproj-BF16.gguf` — PUBLISHER-SHIPPED in the same repo, 931,145,888 B, sha256 `19acd7fc…` (27-layer ViT → 5120) |
+| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) |
+| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) |
+
+## Why this payload (2026-10-06 evening)
+
+Frank asked for the UNCELERSED Swift-1.5 with MTP back on. The RCO-IQ3_XXS
+is 1.56 GiB lighter than the IQ3_S-mtp it replaced — and that lighter
+weight is exactly what funds the MTP draft context (~2.4 GiB measured)
+PLUS the second 131,072 slot, while keeping the q8_0 KV contract and the
+two-slot shape Frank ordered that morning.
+
+**Uncensored, intentionally:** the publisher reports 23/100 refusals vs
+Swift 1.5's 98/100 (KL 0.0884 against the censored base). The refusal
+escape hatch is the same as every payload here: revert and redeploy.
+
+## Shape and KV contract
+
+`--ctx-size 262144 --parallel 2` — TWO conversations at Frank's 128k cap
+(the GSQ publisher's own benchmark protocol ran at 131,072 per stream).
+`CTX_SIZE` stays divisible by `PARALLEL`; the seam enforces it, and the
+per-slot number (`262144 / 2 = 131,072`) is the product contract —
+models.yml stays at 131,072.
+
+**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).**
+llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem
+⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves.
+The order supersedes the same-day f16/f16 rule, which was written for the
+Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would
+cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers ×
+4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is
+pinned: quantized V requires it, and the build's
+`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than
+a silent f16 fallback.
+
+**Two slots + drafter, by arithmetic and by order (2026-10-06).** The
+morning order bought the second slot by spending the drafter and ubatch
+headroom (1,397 MiB free). The evening swap put the drafter BACK and
+changed the arithmetic: the draft ctx is built against the target's FULL
+context (262,144 — the draft window, not the 3-token proposal depth), so
+it costs **~2.4 GiB measured** (23,636 − 22,728 + 1,561), not the ~0.4 GiB
+the one-slot era assumed. The lighter quant pays for it: **23,636 used /
+489 MiB free** (nvidia-smi `free` field) — the tightest fit in
+frank-small's history, the 2–3 GiB floor knowingly violated, and the
+512×512 vision transient (34 MiB) still fits and recovers. Compute buffers
+allocate at graph capture during BOOT, so a miss is loud and fast — the
+failure mode is a crash loop, not silent corruption.
+
+## Speculative decoding — ON since 2026-10-06 evening (draft-mtp, depth 3)
+
+**Frank's evening order: MTP back on.** The MTP head ships IN the file —
+the `-mtp` GGUF carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`)
+— and `--spec-type draft-mtp` loads it against the main model's own
+weights (no `--model-draft`, no second draft model — two context-backed
+drafters cannot share a process). `SPEC_DRAFT_N_MAX=3` is the proposal
+depth. **ngram-mod stays skipped, not swapped in:** on a qwen35 hybrid,
+ngram speculation logs *"recurrent state rollback is not compatible with
+'ngram-mod' — disabling rollback support"* (fleet lesson from
+frank-large's chain measurement, 2026-10-03). A rejected draft on Gated
+DeltaNet layers cannot be rewound by KV truncation — without the rollback
+checkpoints the DeltaNet state carries phantom transitions and output
+degrades silently. The flags move as a set: the serve script emits BOTH
+`--spec-type` and `--spec-draft-n-max` together or neither
+(`SPEC_TYPE=none` strips both).
+
+## Vision contract
+
+`--mmproj` is the entire vision path (the GGUF carries no vision tensors).
+Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny
+probe undersupplies image tokens and the model invents ("video"); the
+legacy top-level `images` field is IGNORED, only the OpenAI `image_url`
+content part reaches the model. **New on this payload: the uncensored
+model THINKS more than the censored Swift 1.5** — a 64-token budget was
+eaten entirely by `reasoning_content` (`finish_reason: length`, empty
+`content`); 128 tokens clears it. Token budgets for vision probes must
+leave room for thinking.
+
+## Measured: the 1 × 131,072 MTP boot (2026-10-05, historical)
+
+- **Boot ledger**: `creating MTP draft context against the target model
+  '…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal
+  model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf'`, `n_slots = 1,
+  n_ctx_slot = 131072` — every expected line, no q8→f16 fallback.
+- **VRAM: 19,404 used / 4,721 MiB free** (nvidia-smi `free` field). The
+  4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free
+  number doubles as the q8_0 proof.
+- **Decode + MTP acceptance** (draft counts from the live `timings`):
+  temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s,
+  166/318; **deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s,
+  159/355 (44.8 %)** — temperature buys fluency, costs acceptance and
+  rate. For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this
+  27B at IQ3_S runs ~2/3 the rate with ~3× the capability class.
+- **Vision**: 512×512 solid-red PNG → `Red`, `finish_reason: stop`.
+- **Template gate**: `reasoning_effort: xhigh` → 200 with
+  `reasoning_content` populated; `high` → HTTP 500 (Jinja raise) — the
+  models.yml fold is mandatory, exactly as the MiMo eras found.
+- Streaming completion: 200. No first-request silence window (the
+  halogen-era 45–60 s stall has no analogue here).
+
+## Measured: the 2 × 131,072 no-spec boot (2026-10-06 morning, historical)
+
+- **Boot ledger**: `loaded multimodal model, '…/Swift15-IQ3S/
+  mmproj-Qwen3.8-27B-F16.gguf'` and NO draft-context line (speculation
+  off). `/slots`: **2 slots, n_ctx 131,072 each** (pool 262,144).
+- **VRAM: 22,728 used / 1,397 MiB free** (nvidia-smi `free` field) — the
+  predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly
+  violated by Frank's order. After a 512×512 image probe: 1,363 MiB —
+  the vision transients fit.
+- **Decode without the drafter**: 48.0 tok/s (temp 0) / 47.8 (deployed
+  recipe temp 1.0). `draft_n` absent from `timings` (no drafter). The MTP
+  era's 58.9–66.9 tok/s is the honest price of the second context:
+  −18–28 %.
+- **Vision**: 512×512 solid-red PNG → `Red` again on this shape.
+- Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet.
+
+## Measured: the 2 × 131,072 draft-mtp boot (2026-10-06 evening, live)
+
+- **Boot ledger**: `creating MTP draft context against the target model
+  '…/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf'`,
+  `loaded multimodal model, '…/Swift15-UncIQ3XXS/mmproj-BF16.gguf'`,
+  `n_slots = 2, n_ctx_slot = 131072` — every expected line, no q8→f16
+  fallback.
+- **VRAM: 23,636 used / 489 MiB free** (nvidia-smi `free` field) — the
+  draft ctx measured ~2.4 GiB, ~6× the ~0.4 GiB the one-slot era
+  assumed; the lighter quant (−1.56 GiB) plus the morning's no-spec base
+  still fund both slots. After a 512×512 image probe: 489 MiB — the vision
+  transient (34 MiB) fits and recovers.
+- **Decode + MTP acceptance** (draft counts from the live `timings`,
+  512-token essay): temp 0 → **72.9 tok/s**, 333/534 accepted (62.4 %);
+  deployed recipe (temp 1.0/top_p 0.95/top_k 20) → **63.3 tok/s**,
+  305/612 (49.8 %). e2e traffic in the boot window read 0.52–0.67
+  acceptance. Against the no-spec shape (48.0 tok/s): **+31 % at the
+  recipe temp** — the drafter earned its 2.4 GiB.
+- **Vision**: 512×512 solid-red PNG → `Red` at a 128-token budget (64
+  tokens: all thinking, empty content). `prompt_tokens: 321` confirms the
+  image part reached the model.
+- **Prompt cache**: `cached_tokens: 317` of 321 on the second probe — the
+  slot prefix cache is live.
+- Deploy: commit da36fa3, e2e gate 33/33 green on the live fleet.
+
+## Client contract
+
+- Reasoning in `reasoning_content` (`--reasoning-format deepseek`);
+  thinking tokens COUNT against `max_tokens`. **This payload thinks more
+  than the censored Swift 1.5** (uncensored finetune) — short `max_tokens`
+  budgets can be eaten entirely by thinking (`finish_reason: length`,
+  empty content); the 32,768 `maxTokens` in models.yml is comfortably
+  above.
+- The embedded chat template gates `reasoning_effort` to
+  `{xhigh, medium, low}` — anything else is a raise/HTTP 500, so
+  `models.yml` `compat.reasoningEffortMap` folds the harness levels onto
+  that set (mandatory, same as every Qwen3.8-template endpoint).
+- Sampling rides the wire (`extraBody`): the publisher recipe temp 1.0 /
+  top_p 0.95 / top_k 20 / min_p 0 (evaluated at ctx 131,072).
+- The RCO-IQ3_XXS tier is a **per-tensor allocation mix** (IQ3_S 97 /
+  BF16 96 / IQ3_XXS 83 / IQ2_S 72 / IQ4_XS 33 / IQ2_XS 32 / IQ2_XXS 30 /
+  Q2_K 28 / Q4_K 19 / Q6_K 8 / IQ1_S 4 / IQ1_M 4) — all stock quant types;
+  stock llama.cpp loads it. Publisher KLD vs Swift BF16: mean 0.002631,
+  90th 0.042136, max 34.49 — RCO is the publisher's "more experimental"
+  family.
+
+## Rollback
+
+The 2 × 131,072 no-spec IQ3_S-mtp shape is ONE commit away — revert
+da36fa3 and redeploy: the env trio then ships `SPEC_TYPE=none`,
+`CTX_SIZE=262144`, `PARALLEL=2`, `UBATCH_SIZE=512` against
+`/home/dank/models/Swift15-IQ3S/` (the IQ3_S-mtp payload, still on disk,
+measured 48.0 tok/s no-spec). The 1 × 131,072 MTP shape is two reverts
+back (812d9eb's parent): the same IQ3_S-mtp file with
+`SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`, `CTX_SIZE=131072`, `PARALLEL=1`
+(measured 58.9–66.9 tok/s with the head drafting).
+
+Three payloads on disk, all fetchable by the same script's git-history
+recipes:
+
+1. **Swift-1.5 IQ3_S-mtp** — the payload live until this cutover:
+   `/home/dank/models/Swift15-IQ3S/`; the IQ3_XXS swap above.
+2. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
+   the 2026-10-05 cutover: `/home/dank/models/MiMo-Q6K/`; revert
+   [[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
+   380aafa's parent), one `MODEL_FILE` + shape edit.
+3. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
+   `/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
+   262,144 slot, f16/f16 KV).
+
+## Trail
+
+- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was
+  committed and pushed but its deploy never completed (blocked at the
+  whole-tree docs gate); the live service was still MiMo Q6_K when Swift
+  cut over. Both Ornith payloads were verified on disk before Swift.
+- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were
+  read from the Hub file by HTTP range + struct parse — no download needed
+  — before the payload was fetched.
+- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel)
+  label-committed both sets at Frank's direction — 1919ad7 (frank-primary
+  radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs).
+  Deploy ran `deploy.sh dankclaw --service frank-small`; the service
+  dispatch and health gate went green; the first e2e run failed ONE spec
+  (12-crawl, `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient,
+  unrelated to frank-small), green on rerun.
+- The unit's `Description=` (MiMo era) was stale at cutover time and was
+  updated in-repo afterward — the relinked unit picks it up at the next
+  deploy/restart, not retroactively.
+- 2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts
+  (812d9eb). The tight fit landed: 1,397 MiB free, vision `Red`, decode
+  48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch
+  ~0.3 GiB bought the second 4,352 MiB KV slot.
+- 2026-10-06 (Frank, evening): swap to the UNCELERSED Swift-1.5 at
+  RCO-IQ3_XXS, MTP back on (da36fa3). The lighter quant (−1.56 GiB) funds
+  the draft ctx (~2.4 GiB measured — the drafter's KV scales with the
+  target's full 262,144 context, not the 3-token proposal depth) plus the
+  second 131,072 slot: 489 MiB free, vision `Red` at a 128-token budget,
+  decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec).