docs: frank-secondary at 2x114688/ubatch 1024 — runbook renamed + rewritten, fleet docs updated
- runbook git mv ...-mtp-131072.md -> ...-mtp-114688.md; new Measured section (22,374 used / 1,751 MiB free, decode 58.2/66.9 t/s, confounded vs 2x131072 noted), rollback ladder, trail entry - AGENTS.md: topology row, ACTIVE section, contextWindow 114688, runbook index; mmproj name corrected to the publisher's mmproj-BF16.gguf - LLM/index.md, index.md, mimo runbook link, log.md entry + path fix
This commit is contained in:
Frank
@@ -15,7 +15,7 @@ timestamp: 2026-10-03
|
||||
# frank-secondary — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
|
||||
|
||||
> **RETIRED 2026-10-05 (evening).** Superseded by
|
||||
> [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072]] — the Swift-1.5-27B
|
||||
> [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — the Swift-1.5-27B
|
||||
> RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). Two facts for the trail:
|
||||
> committed earlier the same day (77decb5..380aafa) was **never deployed** —
|
||||
> THIS MiMo shape was the live payload when Swift cut over; (2) the KV
|
||||
|
||||
+90
-47
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 131072, draft-mtp, q8_0 KV"
|
||||
description: "The active frank-secondary runbook (2026-10-05 evening; MTP back ON on the lighter RCO-IQ3_XXS since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
|
||||
title: "frank-secondary on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 114688, draft-mtp, q8_0 KV"
|
||||
description: "The active frank-secondary runbook (live since 2026-10-05 evening; renamed from frank-small 2026-10-06, per-slot lowered to 112k and ubatch raised to 1024 the same evening) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 114,688-token conversations at Frank's 112k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
@@ -15,26 +15,25 @@ tags:
|
||||
timestamp: 2026-10-06
|
||||
---
|
||||
|
||||
# frank-secondary — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 131,072, draft-mtp, q8_0 KV
|
||||
# frank-secondary — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 114,688, draft-mtp, q8_0 KV
|
||||
|
||||
> **ACTIVE since 2026-10-05 (evening); reshaped twice on 2026-10-06.** A
|
||||
> 27B replaces the 9B on the same 24 GiB card, bought by Frank's explicit
|
||||
> contract for the swap: context capped at 128k per conversation, KV at
|
||||
> FP8 class, vision on, "as many of those contexts as you can". The
|
||||
> morning order ("drop mtp, skip ngram, drop ubatch to 512, let's get 2
|
||||
> contexts at 128k — should be a very tight fit") bought the second slot
|
||||
> by spending the drafter (~0.4 GiB assumed) and ubatch headroom (~0.3
|
||||
> GiB): 1,397 MiB measured free. The evening order — swap to the
|
||||
> UNCELERSED Swift-1.5 at RCO-IQ3_XXS and turn MTP back on — landed
|
||||
> **489 MiB free**: the lighter quant (−1.56 GiB of weights) funds the
|
||||
> second slot, but the MTP draft ctx costs **~2.4 GiB measured, not the
|
||||
> ~0.4 GiB assumed** — the drafter's KV is built against the target's
|
||||
> full context (the 262,144 draft window), not the 3-token proposal
|
||||
> depth. The 2–3 GiB floor is knowingly violated, as ordered. This
|
||||
> supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K
|
||||
> shape ([[frank-secondary-llama-mimo-q6k-2x200704]], now retired) and the
|
||||
> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never
|
||||
> deployed.
|
||||
> **ACTIVE since 2026-10-05 (evening); reshaped three times on
|
||||
> 2026-10-06, renamed from frank-small the same night.** A 27B replaces
|
||||
> the 9B on the same 24 GiB card: KV at FP8 class, vision on. The
|
||||
> morning order (drop MTP, ubatch 512, TWO 128k contexts) bought the
|
||||
> second slot — 1,397 MiB free. The evening order swapped to the
|
||||
> UNCELERSED Swift-1.5 at RCO-IQ3_XXS and put MTP back ON — landing
|
||||
> **489 MiB free**: the MTP draft ctx costs **~2.4 GiB, not the ~0.4
|
||||
> GiB assumed** (the drafter's KV is built against the target's full
|
||||
> context, not the 3-token proposal depth), paid for by the lighter
|
||||
> quant (−1.56 GiB). The late order dropped the per-slot window
|
||||
> 131,072 → **114,688** (Frank's 112k cap) and raised ubatch 512 →
|
||||
> 1024 — landing **1,751 MiB free**, back above the morning's number
|
||||
> with the prefill knob raised. This supersedes BOTH earlier 2026-10-05
|
||||
> payloads: the live MiMo-V2.6 Q6_K shape
|
||||
> ([[frank-secondary-llama-mimo-q6k-2x200704]], now retired) and the
|
||||
> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
|
||||
> never deployed.
|
||||
|
||||
## Identity
|
||||
|
||||
@@ -64,11 +63,12 @@ escape hatch is the same as every payload here: revert and redeploy.
|
||||
|
||||
## Shape and KV contract
|
||||
|
||||
`--ctx-size 262144 --parallel 2` — TWO conversations at Frank's 128k cap
|
||||
(the GSQ publisher's own benchmark protocol ran at 131,072 per stream).
|
||||
`CTX_SIZE` stays divisible by `PARALLEL`; the seam enforces it, and the
|
||||
per-slot number (`262144 / 2 = 131,072`) is the product contract —
|
||||
models.yml stays at 131,072.
|
||||
`--ctx-size 229376 --parallel 2` — TWO conversations at Frank's 112k
|
||||
cap (2026-10-06 late: cut from the 128k the morning/evening shapes ran
|
||||
at, to fund ubatch 1024). `CTX_SIZE` stays divisible by `PARALLEL`; the
|
||||
seam enforces it, and the per-slot number (`229376 / 2 = 114,688`) is
|
||||
the product contract — models.yml follows at 114,688 (a longer prompt
|
||||
is a 400, never a truncation).
|
||||
|
||||
**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).**
|
||||
llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem
|
||||
@@ -81,18 +81,25 @@ pinned: quantized V requires it, and the build's
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than
|
||||
a silent f16 fallback.
|
||||
|
||||
**Two slots + drafter, by arithmetic and by order (2026-10-06).** The
|
||||
morning order bought the second slot by spending the drafter and ubatch
|
||||
headroom (1,397 MiB free). The evening swap put the drafter BACK and
|
||||
changed the arithmetic: the draft ctx is built against the target's FULL
|
||||
context (262,144 — the draft window, not the 3-token proposal depth), so
|
||||
it costs **~2.4 GiB measured** (23,636 − 22,728 + 1,561), not the ~0.4 GiB
|
||||
the one-slot era assumed. The lighter quant pays for it: **23,636 used /
|
||||
489 MiB free** (nvidia-smi `free` field) — the tightest fit in
|
||||
frank-secondary's history, the 2–3 GiB floor knowingly violated, and the
|
||||
512×512 vision transient (34 MiB) still fits and recovers. Compute buffers
|
||||
allocate at graph capture during BOOT, so a miss is loud and fast — the
|
||||
failure mode is a crash loop, not silent corruption.
|
||||
**Two slots + drafter, by arithmetic and by order (2026-10-06 morning /
|
||||
evening, the 2 × 131,072 era).** The morning order bought the second slot
|
||||
by spending the drafter and ubatch headroom (1,397 MiB free). The evening
|
||||
swap put the drafter BACK and changed the arithmetic: the draft ctx is
|
||||
built against the target's FULL context (262,144 — the draft window, not
|
||||
the 3-token proposal depth), so it costs **~2.4 GiB measured** (23,636 −
|
||||
22,728 + 1,561), not the ~0.4 GiB the one-slot era assumed. The lighter
|
||||
quant pays for it: **23,636 used / 489 MiB free** (nvidia-smi `free`
|
||||
field) — the tightest fit in frank-secondary's history, the 2–3 GiB floor
|
||||
knowingly violated, and the 512×512 vision transient (34 MiB) still fits
|
||||
and recovers.
|
||||
|
||||
**The late reshape (2 × 114,688, ubatch 1024) is the current fit.** The
|
||||
per-slot cut frees 2 × 16,384 tokens × 34 KiB ≈ 1.05 GiB of KV; ubatch
|
||||
512 → 1024 spends part of it on compute buffers (a prefill knob — decode
|
||||
never moves with it). Measured boot: **22,374 used / 1,751 MiB free**.
|
||||
The drafter's own KV shrank with its target (draft window 229,376).
|
||||
Compute buffers allocate at graph capture during BOOT, so a miss is loud
|
||||
and fast — the failure mode is a crash loop, not silent corruption.
|
||||
|
||||
## Speculative decoding — ON since 2026-10-06 evening (draft-mtp, depth 3)
|
||||
|
||||
@@ -161,7 +168,7 @@ leave room for thinking.
|
||||
- **Vision**: 512×512 solid-red PNG → `Red` again on this shape.
|
||||
- Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet.
|
||||
|
||||
## Measured: the 2 × 131,072 draft-mtp boot (2026-10-06 evening, live)
|
||||
## Measured: the 2 × 131,072 draft-mtp boot (2026-10-06 evening, historical)
|
||||
|
||||
- **Boot ledger**: `creating MTP draft context against the target model
|
||||
'…/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf'`,
|
||||
@@ -186,6 +193,29 @@ leave room for thinking.
|
||||
slot prefix cache is live.
|
||||
- Deploy: commit da36fa3, e2e gate 33/33 green on the live fleet.
|
||||
|
||||
## Measured: the 2 × 114,688 / ubatch-1024 boot (2026-10-06 late, live)
|
||||
|
||||
- **argv** (from `/proc/<MainPID>/cmdline`): `--ctx-size 229376
|
||||
--parallel 2` (per-slot 114,688), `--batch-size 4096 --ubatch-size
|
||||
1024`, `--spec-type draft-mtp`. Deploys 87721ab (the rename — old
|
||||
`frank-small.service` stopped + disabled first, by hand: `disable` on
|
||||
a relinked unit removes only its symlink) then f2575a1 (the shape);
|
||||
health gate + MainPID check green, e2e 32 passed / 1 flaky-retry.
|
||||
- **VRAM: 22,374 used / 1,751 MiB free** (nvidia-smi `free` field) —
|
||||
the per-slot cut (≈1.05 GiB of KV) paid for the ubatch-1024 buffers
|
||||
and netted back ~1.25 GiB against the evening's 489 MiB; back above
|
||||
the morning's 1,397 MiB.
|
||||
- **Decode + MTP acceptance** (live `timings`): recipe (temp 1.0) →
|
||||
**58.2 t/s** (430/1005, 42.8 %); temp 0 → **66.9 t/s** (475/873,
|
||||
54.4 %). The 2 × 131,072 boot measured 63.3 / 72.9 — −8–10 %, but
|
||||
acceptance is text-dependent and these prompts differ from that
|
||||
measurement's, so the comparison is CONFOUNDED, not a shape verdict;
|
||||
a clean A/B needs stop + mask (bench policy).
|
||||
- **Smoke**: `READY` completion + `reasoning_content` populated; the
|
||||
harness reached :7771 under the new alias (models.yml provider/id
|
||||
`frank-secondary`, config.yml modelRoles rewritten; per-slot
|
||||
`contextWindow: 114688`).
|
||||
|
||||
## Client contract
|
||||
|
||||
- Reasoning in `reasoning_content` (`--reasoning-format deepseek`);
|
||||
@@ -209,14 +239,17 @@ leave room for thinking.
|
||||
|
||||
## Rollback
|
||||
|
||||
The 2 × 131,072 no-spec IQ3_S-mtp shape is ONE commit away — revert
|
||||
da36fa3 and redeploy: the env trio then ships `SPEC_TYPE=none`,
|
||||
`CTX_SIZE=262144`, `PARALLEL=2`, `UBATCH_SIZE=512` against
|
||||
`/home/dank/models/Swift15-IQ3S/` (the IQ3_S-mtp payload, still on disk,
|
||||
measured 48.0 tok/s no-spec). The 1 × 131,072 MTP shape is two reverts
|
||||
back (812d9eb's parent): the same IQ3_S-mtp file with
|
||||
`SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`, `CTX_SIZE=131072`, `PARALLEL=1`
|
||||
(measured 58.9–66.9 tok/s with the head drafting).
|
||||
**The immediate predecessor shape (2 × 131,072, ubatch 512, same
|
||||
payload) is ONE revert away** — revert f2575a1 and redeploy
|
||||
(`CTX_SIZE=262144`, `UBATCH_SIZE=512`). The 2 × 131,072 no-spec
|
||||
IQ3_S-mtp shape is one revert further — revert da36fa3 next: the env
|
||||
trio then ships `SPEC_TYPE=none`, `CTX_SIZE=262144`, `PARALLEL=2`,
|
||||
`UBATCH_SIZE=512` against `/home/dank/models/Swift15-IQ3S/` (the
|
||||
IQ3_S-mtp payload, still on disk, measured 48.0 tok/s no-spec). The
|
||||
1 × 131,072 MTP shape is two reverts further back (812d9eb's parent):
|
||||
the same IQ3_S-mtp file with `SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`,
|
||||
`CTX_SIZE=131072`, `PARALLEL=1` (measured 58.9–66.9 tok/s with the head
|
||||
drafting).
|
||||
|
||||
Three payloads on disk, all fetchable by the same script's git-history
|
||||
recipes:
|
||||
@@ -260,3 +293,13 @@ recipes:
|
||||
target's full 262,144 context, not the 3-token proposal depth) plus the
|
||||
second 131,072 slot: 489 MiB free, vision `Red` at a 128-token budget,
|
||||
decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec).
|
||||
- 2026-10-06 (late, Frank): the endpoint renamed frank-small →
|
||||
**frank-secondary** (87721ab — scripts, unit, seam, runbooks,
|
||||
identifiers; historical references to the OLD FreeToken/CUDA
|
||||
frank-small, and to the Dankserver service that became frank-medium,
|
||||
stay frank-small) and the per-slot window cut 131,072 → 114,688 to
|
||||
fund ubatch 512 → 1024 (f2575a1). Live unit transition done by hand
|
||||
before the deploy: `systemctl stop frank-small` + `systemctl disable
|
||||
frank-small`, then `deploy.sh dankclaw --service frank-secondary`.
|
||||
1,751 MiB free; decode 58.2 t/s recipe / 66.9 temp 0; models.yml +
|
||||
config.yml retargeted.
|
||||
@@ -103,18 +103,20 @@ frank-secondary, port 7771]
|
||||
(`slow`/`advisor` → frank-primary) carried forward. Superseded by the
|
||||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||||
engine moved with the model
|
||||
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-131072]] — **ACTIVE since
|
||||
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **ACTIVE since
|
||||
2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since
|
||||
2026-10-06 evening** (root systemd unit, port 7771): llama.cpp master
|
||||
2026-10-06 evening; RENAMED from frank-small and reshaped to 2 ×
|
||||
114,688 / ubatch 1024 late 2026-10-06** (root systemd unit, port
|
||||
7771): llama.cpp master
|
||||
(CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-
|
||||
GGUF **RCO-IQ3_XXS-mtp** (27B; the MTP head in the file is DRAFTED —
|
||||
draft-mtp depth 3, ngram skipped on the recurrent-rollback law) on the
|
||||
RTX 3090, TWO 131,072-token conversations at Frank's 128k cap (pool
|
||||
262144 / parallel 2 — the tightest fit in the endpoint's history:
|
||||
**489 MiB measured free**, the 2-3 GiB floor knowingly violated) with
|
||||
q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16), ubatch 512, and
|
||||
vision through the publisher's own mmproj-BF16.gguf. Decode 63.3 t/s
|
||||
recipe / 72.9 temp 0 (MTP acceptance 49.8–62.4 %; 48.0 no-spec). The
|
||||
RTX 3090, TWO 114,688-token conversations at Frank's 112k cap (pool
|
||||
229376 / parallel 2 — cut from 131,072 to fund ubatch 1024; **1,751
|
||||
MiB measured free**) with
|
||||
q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16) and
|
||||
vision through the publisher's own mmproj-BF16.gguf. Decode 58.2 t/s
|
||||
recipe / 66.9 temp 0 (MTP acceptance 42.8–54.4 %).
|
||||
RCO-IQ3_XXS tier is a per-tensor allocation mix of stock types; the
|
||||
payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's
|
||||
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
|
||||
|
||||
Reference in New Issue
Block a user