diff --git a/AGENTS.md b/AGENTS.md index 34105bc..e1a078b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *… | Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770| -| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp** (27B dense qwen3_5 hybrid, 12,120,016,896 B; a per-tensor allocation mix of stock types; the -mtp file's MTP head present but NOT drafted — **spec OFF** since 2026-10-06, ngram skipped on the recurrent-rollback law) + unsloth's Qwen3.8-27B **F16 mmproj** for **vision** (no Swift projector ships; the base vision stack was hash-proven byte-identical to Swift's), on the RTX 3090, **2 × 131,072** (Frank's 128k cap each; the tight fit he ordered 2026-10-06 — **1,397 MiB measured free**, the 2-3 GiB floor knowingly violated) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 512; **48.0 t/s** no-spec decode (MTP reference 58.9-66.9). Supersedes BOTH same-day payloads: the live MiMo-V2.6 Q6_K 2×200,704 f16-KV shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed. | 7771 | +| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 131,072** (Frank's 128k cap each; the tightest fit in the endpoint's history — **489 MiB measured free**, the 2-3 GiB floor knowingly violated) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 512; **63.3 t/s recipe decode** (MTP acceptance 49.8 %; 48.0 no-spec). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771| Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -401,16 +401,18 @@ rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays). `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`, `GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into -`/etc/systemd/system/`), serving `ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF` -at **IQ3_S -mtp** (12,120,016,896 B, sha256 `9aecf1cd…`) **plus -`mmproj-Qwen3.8-27B-F16.gguf`** (927,607,488 B, sha256 `cbb841a9…`), both -`/home/dank/models/Swift15-IQ3S/`, on port 7771. Cutover 2026-10-05 -(evening) on Frank's explicit contract: **128k context cap, KV at FP8, -vision on, as many 128k contexts as fit** — the answer is ONE, by -arithmetic. The same-day payloads it supersedes: the LIVE MiMo-V2.6 Q6_K -shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) -but never deployed (blocked at the docs gate; the docs debt is what this -section settles). +`/etc/systemd/system/`), serving `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` +at **RCO-IQ3_XXS -mtp** (10,442,827,392 B, sha256 `e13d3e74…`) **plus +the publisher's own `mmproj-BF16.gguf`** (931,145,888 B, sha256 +`19acd7fc…`), both `/home/dank/models/Swift15-UncIQ3XXS/`, on port 7771. +Cutover 2026-10-05 (evening) on Frank's explicit contract: **128k +context cap, KV at FP8, vision on, as many 128k contexts as fit**; +reshaped twice on 2026-10-06 — morning to two 131,072 slots no-spec, +evening to this UNCERSED RCO-IQ3_XXS payload with MTP back on. The +same-day payloads it supersedes: the LIVE MiMo-V2.6 Q6_K shape and the +Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never +deployed (blocked at the docs gate; the docs debt is what this section +settles). **Architecture: qwen3_5 hybrid, bigger** — block_count 65 (64 real layers + the blk.64 MTP head), `full_attention_interval 4` ⇒ **16 full-attention + @@ -420,51 +422,59 @@ at q8_0**. The DeltaNet state is a fixed per-slot buffer (~150 MiB/slot at 48 linear layers × 48 value heads). Header facts were read from the Hub file by HTTP range + struct parse before anything was downloaded. -**The shape (2026-10-06, Frank's tight-fit order):** `--ctx-size 262144 ---parallel 2` = **TWO 131,072-token conversations** (Frank's 128k cap each; -`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` pinned, **q8_0 K + -q8_0 V** (llama.cpp's FP8 class — no true fp8 KV dtype exists; 8.5 effective -bits; supersedes the same-day f16/f16 rule written for the Ornith 9B fit), -all layers on the GPU, `--mmproj` for vision, batch 4096 / ubatch 512. -**Speculative decoding is OFF** (same order): the -mtp file's head stays but -is not drafted with; ngram-mod skipped on purpose — on a qwen35 hybrid -ngram disables recurrent-state rollback, so a rejected draft corrupts the -DeltaNet state. +**The shape (2026-10-06, Frank's tight-fit order, MTP back on in the +evening):** `--ctx-size 262144 --parallel 2` = **TWO 131,072-token +conversations** (Frank's 128k cap each; `n_ctx_slot = CTX_SIZE / +PARALLEL`), `--flash-attn on` pinned, **q8_0 K + q8_0 V** (llama.cpp's +FP8 class — no true fp8 KV dtype exists; 8.5 effective bits; supersedes +the same-day f16/f16 rule written for the Ornith 9B fit), all layers on +the GPU, `--mmproj` for vision, batch 4096 / ubatch 512. **Speculative +decoding is ON** (draft-mtp, depth 3): the -mtp file's head is drafted +against the main model's own weights; ngram-mod skipped on purpose — on +a qwen35 hybrid ngram disables recurrent-state rollback, so a rejected +draft corrupts the DeltaNet state. -**Measured at the two-slot boot (2026-10-06): 22,728 used / 1,397 MiB -free** (`nvidia-smi` free field) — under the old 2-3 GiB floor BY ORDER ("a -very tight fit"); a 512×512 solid-red image still answers `Red` afterwards -(1,363 MiB free after the probe), so the fit carries the vision transients. -The slot-2 pricing came from the measured 1×131,072 MTP boot (2026-10-05: -19,404 used / 4,721 free — also the q8_0 proof, f16 would have left ~700): -the second q8_0 slot costs 4,352 MiB KV + ~150 MiB DeltaNet, funded by -dropping the MTP draft ctx (~0.4 GiB) and ubatch 1024→512 (~0.3 GiB, -inferred from the measured 616 MiB/doubling rung on the 9B). **Decode -without the drafter: 48.0 tok/s (temp 0) / 47.8 (recipe temp 1.0)** — the -MTP references stay in frank-small.env for the day the head returns (44.8 % -acceptance at the recipe temp ⇒ 58.9 tok/s; 52.1 % at temp 0.8 ⇒ 66.1). -`reasoning_effort` probe unchanged: `xhigh` 200, `high` → 500 Jinja raise -(the models.yml fold stays mandatory). +**Measured at the two-slot MTP boot (2026-10-06 evening): 23,636 used / +489 MiB free** (`nvidia-smi` free field) — the tightest fit in the +endpoint's history, under the old 2-3 GiB floor BY ORDER; a 512×512 +solid-red image still answers `Red` afterwards (489 MiB free after the +probe), so the fit carries the vision transients. The MTP draft ctx +costs **~2.4 GiB measured** (23,636 − 22,728 + 1,561) — ~6× the ~0.4 GiB +the one-slot era assumed, because the drafter's KV is built against the +target's full 262,144 context (the draft window), not the 3-token +proposal depth; the lighter RCO-IQ3_XXS quant (−1.56 GiB vs IQ3_S-mtp) +funds it. **Decode: 63.3 tok/s (recipe temp 1.0) / 72.9 (temp 0)** with +MTP acceptance 49.8–62.4 % (48.0 no-spec — the drafter earned its +2.4 GiB). `reasoning_effort` probe unchanged: `xhigh` 200, `high` → 500 +Jinja raise (the models.yml fold stays mandatory). -**Vision without a Swift projector**: no Swift repo ships one (ukisai's -README is explicit), so the projector is unsloth's mmproj built from the -SHARED BASE `Qwen/Qwen3.8-27B` — legal because Swift-1.5's vision stack was -hash-proven byte-identical to the base (333/333 tensor names, 6/6 sampled -tensors: merger fc1/fc2, blocks 0/13/26) on 2026-10-05. The probe traps -carry over: ≥256 px probe images or the model answers "video"; only the -OpenAI `image_url` content part reaches the model. +**Vision with the publisher's own projector**: ajgazin's repo SHIPS +`mmproj-BF16.gguf` in the same repo (the IQ3_S era's unsloth mmproj was +built from the SHARED BASE `Qwen/Qwen3.8-27B`, legal because Swift-1.5's +vision stack was hash-proven byte-identical to the base). The probe +traps carry over: ≥256 px probe images or the model answers "video"; +only the OpenAI `image_url` content part reaches the model. **New on +this payload: the uncensored model thinks more than the censored +Swift 1.5** — a 64-token budget is eaten entirely by +`reasoning_content` (empty `content`); 128 tokens clears the vision +probe. -The GSQ-RCO "IQ3_S" tier is a **per-tensor allocation mix** of stock quant -types (IQ2_XS 144 / IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …) -— stock llama.cpp loads it; the tensor-type histogram is the law (see Key -Lessons). Publisher KLD vs Swift BF16: 0.0513 development / 0.024–0.052 -held-out; the MTP packaging is NOT separately evaluated by the publisher. +The RCO-IQ3_XXS tier is a **per-tensor allocation mix** of stock quant +types (IQ3_S 97 / BF16 96 / IQ3_XXS 83 / IQ2_S 72 / IQ4_XS 33 / IQ2_XS 32 +/ IQ2_XXS 30 / Q2_K 28 / Q4_K 19 / Q6_K 8 / IQ1_S 4 / IQ1_M 4) — stock +llama.cpp loads it; the tensor-type histogram is the law (see Key +Lessons). Publisher KLD vs Swift BF16: mean 0.002631 / 90th 0.042136 / +max 34.49 — RCO is the publisher's "more experimental" family, and the +payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's +98/100, KL 0.0884). Runbook: -`serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md`. -Rollback: `/home/dank/models/MiMo-Q6K/` (the LIVE payload until the cutover; -revert to the 2×200,704 f16 shape — [[frank-small-llama-mimo-q6k-2x200704]]) -or `/home/dank/models/Ornith-Q8/` (the committed-never-deployed Q8_0; +`serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md`. +Rollback: `/home/dank/models/Swift15-IQ3S/` (the IQ3_S-mtp payload live +until the 2026-10-06 evening swap — revert da36fa3 for the two-slot +no-spec shape) or `/home/dank/models/MiMo-Q6K/` (revert to the +2×200,704 f16 shape — [[frank-small-llama-mimo-q6k-2x200704]]) or +`/home/dank/models/Ornith-Q8/` (the committed-never-deployed Q8_0; revert to 380aafa's tree). ### frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next FP8 native, TP=2 (Dankserver, ACTIVE) @@ -988,7 +998,7 @@ Active: | Service | Runbook | |---------|---------| | frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` | -| frank-small (llama.cpp CUDA, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 1×131,072 q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md` | +| frank-small (llama.cpp CUDA, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×131,072 draft-mtp q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md` | | frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -1001,7 +1011,7 @@ Retired (rollback targets and history): | tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` | | FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` | | llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-small-llama-mimo-q6k-2x200704.md` | -| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-small`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-small-llama-swift15-iq3s-mtp-131072.md` | +| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-small`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md` | | ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` | | gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` | | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` | diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md index c0c4c27..e5f0840 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md @@ -15,8 +15,8 @@ timestamp: 2026-10-03 # frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV > **RETIRED 2026-10-05 (evening).** Superseded by -> [[frank-small-llama-swift15-iq3s-mtp-131072]] — the Swift-1.5-27B GSQ-RCO -> IQ3_S-mtp swap. Two facts for the trail: (1) the Ornith-1.5-9B Q8_0 swap +> [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — the Swift-1.5-27B +> RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). Two facts for the trail: > committed earlier the same day (77decb5..380aafa) was **never deployed** — > THIS MiMo shape was the live payload when Swift cut over; (2) the KV > contract Frank set here (f16 K + f16 V, "no further quantization on this diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md b/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md deleted file mode 100644 index ad0748c..0000000 --- a/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md +++ /dev/null @@ -1,229 +0,0 @@ ---- -type: reference -title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 2 x 131072, no spec, q8_0 KV" -description: "The active frank-small runbook (2026-10-05 evening; 2 × 131,072 no-spec since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S (the -mtp file's head present but NOT drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap (the tight fit he ordered — 1,397 MiB measured free), q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the full slot-2 arithmetic." -tags: - - llm - - frank-small - - dankclaw - - llama-cpp - - gguf - - rtx3090 - - qwen35 - - mtp - - vision -timestamp: 2026-10-05 ---- - -# frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 2 × 131,072, no spec, q8_0 KV - -> **ACTIVE since 2026-10-05 (evening); TWO slots, spec OFF since -> 2026-10-06.** A 27B replaces the 9B on the same 24 GiB card, bought by -> Frank's explicit contract for the swap: context capped at 128k per -> conversation, KV at FP8 class, vision on, "as many of those contexts as -> you can" — the 2026-10-05 arithmetic answered **one**; Frank's 2026-10-06 -> order ("drop mtp, skip ngram, drop ubatch to 512, let's get 2 contexts at -> 128k — should be a very tight fit") bought the second slot by spending -> the drafter (~0.4 GiB) and ubatch headroom (~0.3 GiB). It fits: -> **1,397 MiB measured free**, the old 2-3 GiB floor knowingly violated. -> This supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K -> shape ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the -> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never -> deployed. - -## Identity - -| | | -|---|---| -| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB | -| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface | -| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) | -| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` | -| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) | -| Weights | `/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` — ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp**, 12,120,016,896 B, sha256 `9aecf1cd…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`) | -| Projector | `…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf` — unsloth/Qwen3.8-27B-GGUF `mmproj-F16.gguf`, 927,607,488 B, sha256 `cbb841a9…` (27-layer ViT, 1152 hidden, `qwen3vl_merger` → 5120) | -| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified per file; both verified on the host before deploy | -| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, no-spec pair (absence of both flags), batch pair, projector opt-out) | - -## Why a base-model projector is legal here - -No Swift repo ships a projector — ukisai's GSQ-RCO release is -language-model-only by their own README ("A Swift 27B vision projector has -not been verified for this release"). The substitution is proven, not -assumed: Swift-1.5 is an RL/OPD post-train of `Qwen/Qwen3.8-27B`, and its -vision stack was hash-compared against the base on 2026-10-05 over HTTP -range reads of the two BF16 repos — **333/333 visual tensor names match and -6/6 sampled tensors are byte-identical** (merger `linear_fc1`/`linear_fc2`, -blocks 0/13/26 attn/mlp). Swift's post-training touched text only, so the -base projector IS the model's own projector. The mmproj header matches -Swift's `vision_config` field-for-field (depth 27, hidden 1152, patch 16, -merge 2, ffn 4304, 16 heads, out 5120). - -## Shape and KV contract - -`--ctx-size 262144 --parallel 2` — TWO conversations at Frank's 128k cap -(the GSQ publisher's own benchmark protocol ran at 131,072 per stream). -`CTX_SIZE` stays divisible by `PARALLEL`; the seam enforces it, and the -per-slot number (`262144 / 2 = 131,072`) is the product contract — -models.yml stays at 131,072. - -**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).** -llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem -⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves. The -order supersedes the same-day f16/f16 rule, which was written for the -Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would -cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers × -4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is -pinned: quantized V requires it, and the build's -`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than a -silent f16 fallback. - -**Two slots, by arithmetic and by order (2026-10-06).** The 2026-10-05 -arithmetic made ONE 131,072 slot the maximum under the 2-3 GiB floor (a -second slot wanted +4,352 MiB of KV against 4,721 free — ~0.2 GiB left, -far under the floor). Frank then ordered the budget spent differently: -drop the MTP drafter (~0.4 GiB: one nextn layer of draft KV + its compute -buffers), drop ubatch 1024→512 (~0.3 GiB, inferred from the measured 616 -MiB/doubling rung on the 9B), and take the second slot anyway — "should -be a very tight fit", i.e. the 2-3 GiB floor is KNOWINGLY VIOLATED. The -measured boot confirms: **22,728 used / 1,397 MiB free** (nvidia-smi -`free` field), and a 512×512 image still answers `Red` afterwards -(1,363 MiB after the probe), so the fit carries the vision transients. -Compute buffers allocate at graph capture during BOOT, so a miss is loud -and fast — the failure mode is a crash loop, not silent corruption. - -## Speculative decoding — OFF since 2026-10-06 - -**Frank's order: drop MTP, skip ngram-mod too.** The MTP head ships IN the -file — the `-mtp` GGUF carries `blk.64.nextn.{enorm,hnorm,shared_head_norm}` -(`qwen35.nextn_predict_layers=1`, publisher RECIPE "MTP_OK verified") — and -`--spec-type draft-mtp` loads it against the main model's own weights (no -`--model-draft`, no second draft model — two context-backed drafters cannot -share a process). It simply is NOT drafted with now: `SPEC_TYPE=none` -strips BOTH speculative flags in the serve script (flags move as a set). - -**Why ngram-mod was skipped, not swapped in:** on a qwen35 hybrid, ngram -speculation logs *"recurrent state rollback is not compatible with -'ngram-mod' — disabling rollback support"* (fleet lesson from frank-large's -chain measurement, 2026-10-03). A rejected draft on Gated DeltaNet layers -cannot be rewound by KV truncation — without the rollback checkpoints the -DeltaNet state carries phantom transitions and output degrades silently. -MTP measured fine here (below); it was spent for the second slot, not -abandoned for ngram. - -**Measured MTP references (2026-10-05, for the day the head returns)**, -draft counts from the live `timings` on real-ish traffic: temp 0.8 → -66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s, 166/318; -deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s, 159/355 -(44.8 %). Acceptance reads from the response `timings` object -(`draft_n`, `draft_n_accepted`) — never on a synthetic repeat (greedy-text -acceptance lies). - -## Vision contract - -`--mmproj` is the entire vision path (the GGUF carries no vision tensors). -Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny -probe undersupplies image tokens and the model invents ("video"); the -legacy top-level `images` field is IGNORED, only the OpenAI `image_url` -content part reaches the model. Same traps the MiMo eras documented. - -## Measured: the 1 × 131,072 MTP boot (2026-10-05, historical) - -- **Boot ledger**: `creating MTP draft context against the target model - '…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal - model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf'`, `n_slots = 1, - n_ctx_slot = 131072` — every expected line, no q8→f16 fallback. -- **VRAM: 19,404 used / 4,721 MiB free** (nvidia-smi `free` field). The - 4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free - number doubles as the q8_0 proof. -- **Decode + MTP acceptance** (draft counts from the live `timings`): - temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s, - 166/318; **deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s, - 159/355 (44.8 %)** — temperature buys fluency, costs acceptance and rate. - For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this 27B at - IQ3_S runs ~2/3 the rate with ~3× the capability class. -- **Vision**: 512×512 solid-red PNG → `Red`, `finish_reason: stop`. -- **Template gate**: `reasoning_effort: xhigh` → 200 with - `reasoning_content` populated; `high` → HTTP 500 (Jinja raise) — the - models.yml fold is mandatory, exactly as the MiMo eras found. -- Streaming completion: 200. No first-request silence window (the - halogen-era 45–60 s stall has no analogue here). - -## Measured: the 2 × 131,072 no-spec boot (2026-10-06, live) - -- **Boot ledger**: `loaded multimodal model, '…/Swift15-IQ3S/ - mmproj-Qwen3.8-27B-F16.gguf'` and NO draft-context line (speculation - off). `/slots`: **2 slots, n_ctx 131,072 each** (pool 262,144). -- **VRAM: 22,728 used / 1,397 MiB free** (nvidia-smi `free` field) — the - predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly - violated by Frank's order. After a 512×512 image probe: 1,363 MiB — - the vision transients fit. -- **Decode without the drafter**: 48.0 tok/s (temp 0) / 47.8 (deployed - recipe temp 1.0). `draft_n` absent from `timings` (no drafter). The MTP - era's 58.9–66.9 tok/s is the honest price of the second context: - −18–28 %. -- **Vision**: 512×512 solid-red PNG → `Red` again on this shape. -- Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet. - -## Client contract - -- Reasoning in `reasoning_content` (`--reasoning-format deepseek`); thinking - tokens COUNT against `max_tokens`. -- The embedded chat template gates `reasoning_effort` to - `{xhigh, medium, low}` — anything else is a raise/HTTP 500, so - `models.yml` `compat.reasoningEffortMap` folds the harness levels onto - that set (mandatory, same as every Qwen3.8-template endpoint). -- Sampling rides the wire (`extraBody`): Swift's publisher recipe temp 1.0 - / top_p 0.95 / top_k 20 / min_p 0 (the card was evaluated at exactly - this recipe at ctx 131,072). -- Swift 1.5 is a reasoning-efficient post-train: −58.5 % median thinking - tokens vs base at equal-or-better accuracy. Token budgets go further; - the 32,768 `maxTokens` is unchanged from the MiMo era and still clears - the compaction invariant (91,750 + 32,768 ≤ 131,072). -- The GSQ-RCO "IQ3_S" tier is a **per-tensor allocation mix** (IQ2_XS 144 / - IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …), all stock quant - types — stock llama.cpp loads it. Publisher KLD vs Swift BF16: 0.0513 - (development), held-out 0.024–0.052 per domain; the MTP packaging is NOT - separately evaluated by the publisher. - -## Rollback - -The 1 × 131,072 MTP shape is ONE commit away — revert 812d9eb (the -two-slot order) and redeploy: the same env trio then ships -`SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`, `CTX_SIZE=131072`, -`PARALLEL=1` (the measured 58.9–66.9 tok/s with the head drafting). - -Two payloads on disk, both fetchable by the same script's git-history -recipes: - -1. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until - this cutover: `/home/dank/models/MiMo-Q6K/`; revert - [[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or - 380aafa's parent), one `MODEL_FILE` + shape edit. -2. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap: - `/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native - 262,144 slot, f16/f16 KV). - -## Trail - -- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was - committed and pushed but its deploy never completed (blocked at the - whole-tree docs gate); the live service was still MiMo Q6_K when Swift - cut over. Both Ornith payloads were verified on disk before Swift. -- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were - read from the Hub file by HTTP range + struct parse — no download needed — - before the payload was fetched. -- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel) label- - committed both sets at Frank's direction — 1919ad7 (frank-primary radiance) - then baf33d0 (this swap: trio, seam, runbooks, fleet docs). Deploy ran - `deploy.sh dankclaw --service frank-small`; the service dispatch and health - gate went green; the first e2e run failed ONE spec (12-crawl, - `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient, unrelated to - frank-small), green on rerun. -- The unit's `Description=` (MiMo era) was stale at cutover time and was - updated in-repo afterward — the relinked unit picks it up at the next - deploy/restart, not retroactively. -- 2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts - (812d9eb). The tight fit landed: 1,397 MiB free, vision `Red`, decode - 48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch - ~0.3 GiB bought the second 4,352 MiB KV slot. diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md b/serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md new file mode 100644 index 0000000..045d7e4 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md @@ -0,0 +1,262 @@ +--- +type: reference +title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 131072, draft-mtp, q8_0 KV" +description: "The active frank-small runbook (2026-10-05 evening; MTP back ON on the lighter RCO-IQ3_XXS since 2026-10-06) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 131,072-token conversations at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic." +tags: + - llm + - frank-small + - dankclaw + - llama-cpp + - gguf + - rtx3090 + - qwen35 + - mtp + - vision +timestamp: 2026-10-06 +--- + +# frank-small — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 131,072, draft-mtp, q8_0 KV + +> **ACTIVE since 2026-10-05 (evening); reshaped twice on 2026-10-06.** A +> 27B replaces the 9B on the same 24 GiB card, bought by Frank's explicit +> contract for the swap: context capped at 128k per conversation, KV at +> FP8 class, vision on, "as many of those contexts as you can". The +> morning order ("drop mtp, skip ngram, drop ubatch to 512, let's get 2 +> contexts at 128k — should be a very tight fit") bought the second slot +> by spending the drafter (~0.4 GiB assumed) and ubatch headroom (~0.3 +> GiB): 1,397 MiB measured free. The evening order — swap to the +> UNCELERSED Swift-1.5 at RCO-IQ3_XXS and turn MTP back on — landed +> **489 MiB free**: the lighter quant (−1.56 GiB of weights) funds the +> second slot, but the MTP draft ctx costs **~2.4 GiB measured, not the +> ~0.4 GiB assumed** — the drafter's KV is built against the target's +> full context (the 262,144 draft window), not the 3-token proposal +> depth. The 2–3 GiB floor is knowingly violated, as ordered. This +> supersedes BOTH earlier 2026-10-05 payloads: the live MiMo-V2.6 Q6_K +> shape ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the +> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never +> deployed. + +## Identity + +| | | +|---|---| +| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB | +| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface | +| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) | +| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` | +| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) | +| Weights | `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp**, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it) | +| Projector | `…/Swift15-UncIQ3XXS/mmproj-BF16.gguf` — PUBLISHER-SHIPPED in the same repo, 931,145,888 B, sha256 `19acd7fc…` (27-layer ViT → 5120) | +| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) | +| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) | + +## Why this payload (2026-10-06 evening) + +Frank asked for the UNCELERSED Swift-1.5 with MTP back on. The RCO-IQ3_XXS +is 1.56 GiB lighter than the IQ3_S-mtp it replaced — and that lighter +weight is exactly what funds the MTP draft context (~2.4 GiB measured) +PLUS the second 131,072 slot, while keeping the q8_0 KV contract and the +two-slot shape Frank ordered that morning. + +**Uncensored, intentionally:** the publisher reports 23/100 refusals vs +Swift 1.5's 98/100 (KL 0.0884 against the censored base). The refusal +escape hatch is the same as every payload here: revert and redeploy. + +## Shape and KV contract + +`--ctx-size 262144 --parallel 2` — TWO conversations at Frank's 128k cap +(the GSQ publisher's own benchmark protocol ran at 131,072 per stream). +`CTX_SIZE` stays divisible by `PARALLEL`; the seam enforces it, and the +per-slot number (`262144 / 2 = 131,072`) is the product contract — +models.yml stays at 131,072. + +**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).** +llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem +⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves. +The order supersedes the same-day f16/f16 rule, which was written for the +Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would +cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers × +4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is +pinned: quantized V requires it, and the build's +`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than +a silent f16 fallback. + +**Two slots + drafter, by arithmetic and by order (2026-10-06).** The +morning order bought the second slot by spending the drafter and ubatch +headroom (1,397 MiB free). The evening swap put the drafter BACK and +changed the arithmetic: the draft ctx is built against the target's FULL +context (262,144 — the draft window, not the 3-token proposal depth), so +it costs **~2.4 GiB measured** (23,636 − 22,728 + 1,561), not the ~0.4 GiB +the one-slot era assumed. The lighter quant pays for it: **23,636 used / +489 MiB free** (nvidia-smi `free` field) — the tightest fit in +frank-small's history, the 2–3 GiB floor knowingly violated, and the +512×512 vision transient (34 MiB) still fits and recovers. Compute buffers +allocate at graph capture during BOOT, so a miss is loud and fast — the +failure mode is a crash loop, not silent corruption. + +## Speculative decoding — ON since 2026-10-06 evening (draft-mtp, depth 3) + +**Frank's evening order: MTP back on.** The MTP head ships IN the file — +the `-mtp` GGUF carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`) +— and `--spec-type draft-mtp` loads it against the main model's own +weights (no `--model-draft`, no second draft model — two context-backed +drafters cannot share a process). `SPEC_DRAFT_N_MAX=3` is the proposal +depth. **ngram-mod stays skipped, not swapped in:** on a qwen35 hybrid, +ngram speculation logs *"recurrent state rollback is not compatible with +'ngram-mod' — disabling rollback support"* (fleet lesson from +frank-large's chain measurement, 2026-10-03). A rejected draft on Gated +DeltaNet layers cannot be rewound by KV truncation — without the rollback +checkpoints the DeltaNet state carries phantom transitions and output +degrades silently. The flags move as a set: the serve script emits BOTH +`--spec-type` and `--spec-draft-n-max` together or neither +(`SPEC_TYPE=none` strips both). + +## Vision contract + +`--mmproj` is the entire vision path (the GGUF carries no vision tensors). +Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny +probe undersupplies image tokens and the model invents ("video"); the +legacy top-level `images` field is IGNORED, only the OpenAI `image_url` +content part reaches the model. **New on this payload: the uncensored +model THINKS more than the censored Swift 1.5** — a 64-token budget was +eaten entirely by `reasoning_content` (`finish_reason: length`, empty +`content`); 128 tokens clears it. Token budgets for vision probes must +leave room for thinking. + +## Measured: the 1 × 131,072 MTP boot (2026-10-05, historical) + +- **Boot ledger**: `creating MTP draft context against the target model + '…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal + model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf'`, `n_slots = 1, + n_ctx_slot = 131072` — every expected line, no q8→f16 fallback. +- **VRAM: 19,404 used / 4,721 MiB free** (nvidia-smi `free` field). The + 4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free + number doubles as the q8_0 proof. +- **Decode + MTP acceptance** (draft counts from the live `timings`): + temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s, + 166/318; **deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s, + 159/355 (44.8 %)** — temperature buys fluency, costs acceptance and + rate. For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this + 27B at IQ3_S runs ~2/3 the rate with ~3× the capability class. +- **Vision**: 512×512 solid-red PNG → `Red`, `finish_reason: stop`. +- **Template gate**: `reasoning_effort: xhigh` → 200 with + `reasoning_content` populated; `high` → HTTP 500 (Jinja raise) — the + models.yml fold is mandatory, exactly as the MiMo eras found. +- Streaming completion: 200. No first-request silence window (the + halogen-era 45–60 s stall has no analogue here). + +## Measured: the 2 × 131,072 no-spec boot (2026-10-06 morning, historical) + +- **Boot ledger**: `loaded multimodal model, '…/Swift15-IQ3S/ + mmproj-Qwen3.8-27B-F16.gguf'` and NO draft-context line (speculation + off). `/slots`: **2 slots, n_ctx 131,072 each** (pool 262,144). +- **VRAM: 22,728 used / 1,397 MiB free** (nvidia-smi `free` field) — the + predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly + violated by Frank's order. After a 512×512 image probe: 1,363 MiB — + the vision transients fit. +- **Decode without the drafter**: 48.0 tok/s (temp 0) / 47.8 (deployed + recipe temp 1.0). `draft_n` absent from `timings` (no drafter). The MTP + era's 58.9–66.9 tok/s is the honest price of the second context: + −18–28 %. +- **Vision**: 512×512 solid-red PNG → `Red` again on this shape. +- Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet. + +## Measured: the 2 × 131,072 draft-mtp boot (2026-10-06 evening, live) + +- **Boot ledger**: `creating MTP draft context against the target model + '…/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf'`, + `loaded multimodal model, '…/Swift15-UncIQ3XXS/mmproj-BF16.gguf'`, + `n_slots = 2, n_ctx_slot = 131072` — every expected line, no q8→f16 + fallback. +- **VRAM: 23,636 used / 489 MiB free** (nvidia-smi `free` field) — the + draft ctx measured ~2.4 GiB, ~6× the ~0.4 GiB the one-slot era + assumed; the lighter quant (−1.56 GiB) plus the morning's no-spec base + still fund both slots. After a 512×512 image probe: 489 MiB — the vision + transient (34 MiB) fits and recovers. +- **Decode + MTP acceptance** (draft counts from the live `timings`, + 512-token essay): temp 0 → **72.9 tok/s**, 333/534 accepted (62.4 %); + deployed recipe (temp 1.0/top_p 0.95/top_k 20) → **63.3 tok/s**, + 305/612 (49.8 %). e2e traffic in the boot window read 0.52–0.67 + acceptance. Against the no-spec shape (48.0 tok/s): **+31 % at the + recipe temp** — the drafter earned its 2.4 GiB. +- **Vision**: 512×512 solid-red PNG → `Red` at a 128-token budget (64 + tokens: all thinking, empty content). `prompt_tokens: 321` confirms the + image part reached the model. +- **Prompt cache**: `cached_tokens: 317` of 321 on the second probe — the + slot prefix cache is live. +- Deploy: commit da36fa3, e2e gate 33/33 green on the live fleet. + +## Client contract + +- Reasoning in `reasoning_content` (`--reasoning-format deepseek`); + thinking tokens COUNT against `max_tokens`. **This payload thinks more + than the censored Swift 1.5** (uncensored finetune) — short `max_tokens` + budgets can be eaten entirely by thinking (`finish_reason: length`, + empty content); the 32,768 `maxTokens` in models.yml is comfortably + above. +- The embedded chat template gates `reasoning_effort` to + `{xhigh, medium, low}` — anything else is a raise/HTTP 500, so + `models.yml` `compat.reasoningEffortMap` folds the harness levels onto + that set (mandatory, same as every Qwen3.8-template endpoint). +- Sampling rides the wire (`extraBody`): the publisher recipe temp 1.0 / + top_p 0.95 / top_k 20 / min_p 0 (evaluated at ctx 131,072). +- The RCO-IQ3_XXS tier is a **per-tensor allocation mix** (IQ3_S 97 / + BF16 96 / IQ3_XXS 83 / IQ2_S 72 / IQ4_XS 33 / IQ2_XS 32 / IQ2_XXS 30 / + Q2_K 28 / Q4_K 19 / Q6_K 8 / IQ1_S 4 / IQ1_M 4) — all stock quant types; + stock llama.cpp loads it. Publisher KLD vs Swift BF16: mean 0.002631, + 90th 0.042136, max 34.49 — RCO is the publisher's "more experimental" + family. + +## Rollback + +The 2 × 131,072 no-spec IQ3_S-mtp shape is ONE commit away — revert +da36fa3 and redeploy: the env trio then ships `SPEC_TYPE=none`, +`CTX_SIZE=262144`, `PARALLEL=2`, `UBATCH_SIZE=512` against +`/home/dank/models/Swift15-IQ3S/` (the IQ3_S-mtp payload, still on disk, +measured 48.0 tok/s no-spec). The 1 × 131,072 MTP shape is two reverts +back (812d9eb's parent): the same IQ3_S-mtp file with +`SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`, `CTX_SIZE=131072`, `PARALLEL=1` +(measured 58.9–66.9 tok/s with the head drafting). + +Three payloads on disk, all fetchable by the same script's git-history +recipes: + +1. **Swift-1.5 IQ3_S-mtp** — the payload live until this cutover: + `/home/dank/models/Swift15-IQ3S/`; the IQ3_XXS swap above. +2. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until + the 2026-10-05 cutover: `/home/dank/models/MiMo-Q6K/`; revert + [[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or + 380aafa's parent), one `MODEL_FILE` + shape edit. +3. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap: + `/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native + 262,144 slot, f16/f16 KV). + +## Trail + +- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was + committed and pushed but its deploy never completed (blocked at the + whole-tree docs gate); the live service was still MiMo Q6_K when Swift + cut over. Both Ornith payloads were verified on disk before Swift. +- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were + read from the Hub file by HTTP range + struct parse — no download needed + — before the payload was fetched. +- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel) + label-committed both sets at Frank's direction — 1919ad7 (frank-primary + radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs). + Deploy ran `deploy.sh dankclaw --service frank-small`; the service + dispatch and health gate went green; the first e2e run failed ONE spec + (12-crawl, `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient, + unrelated to frank-small), green on rerun. +- The unit's `Description=` (MiMo era) was stale at cutover time and was + updated in-repo afterward — the relinked unit picks it up at the next + deploy/restart, not retroactively. +- 2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts + (812d9eb). The tight fit landed: 1,397 MiB free, vision `Red`, decode + 48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch + ~0.3 GiB bought the second 4,352 MiB KV slot. +- 2026-10-06 (Frank, evening): swap to the UNCELERSED Swift-1.5 at + RCO-IQ3_XXS, MTP back on (da36fa3). The lighter quant (−1.56 GiB) funds + the draft ctx (~2.4 GiB measured — the drafter's KV scales with the + target's full 262,144 context, not the 3-token proposal depth) plus the + second 131,072 slot: 489 MiB free, vision `Red` at a 128-token budget, + decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec). diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index b53b904..50afb99 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -15,11 +15,12 @@ ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai /halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision lives on frank-small)** and -**llama.cpp CUDA (frank-small, port 7771: ukisai Swift-1.5-Qwen3.8-27B -GSQ-RCO IQ3_S-mtp GGUF — 27B (head present, NOT drafted), TWO 131,072 -conversations at Frank's 128k cap (tight fit, spec off since 2026-10-06), -q8_0 K + q8_0 V (FP8 class), vision via the base-model projector, on the -RTX 3090)**. +**llama.cpp CUDA (frank-small, port 7771: ajgazin Swift-1.5-Qwen3.8-27B +Uncensored Dynamic-MTP RCO-IQ3_XXS GGUF — 27B (the MTP head in the file +IS drafted, draft-mtp depth 3), TWO 131,072 conversations at Frank's +128k cap (the tightest fit in the endpoint's history — 489 MiB free), +q8_0 K + q8_0 V (FP8 class), vision via the publisher's own BF16 +mmproj, on the RTX 3090)**. > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -50,8 +51,8 @@ frank-large, port 7770] 1 x 200700 ctx, d2 KV, text-only] G[llama.cpp master CUDA sm_86 frank-small, port 7771] - H[Swift-1.5-27B GSQ-RCO IQ3_S-mtp GGUF -1 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision] + H[Swift-1.5-27B Uncensored RCO-IQ3_XXS-mtp GGUF +2 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision] E2 --> E --> F E2 --> G --> H end @@ -94,23 +95,23 @@ frank-small, port 7771] on this topology — the default tile stands. The deep harness roles (`slow`, `advisor`) rerouted to frank-primary; frank-large keeps OpenWebUI and shallow callers. -- [[frank-small-llama-swift15-iq3s-mtp-131072]] — **ACTIVE since 2026-10-05 - (evening); 2 × 131,072 no-spec since 2026-10-06** (root systemd unit, - port 7771): llama.cpp master (CUDA sm_86) serving ukisai/Swift-1.5- - Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S-mtp** (27B; the -mtp file's head is - present but NOT drafted — speculation OFF by Frank's order, ngram skipped - on the recurrent-rollback law) on the RTX 3090, TWO 131,072-token - conversations at Frank's 128k cap (pool 262144 / parallel 2 — the tight - fit he ordered: **1,397 MiB measured free**, the 2-3 GiB floor knowingly - violated) with q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16), - ubatch 512, and vision via unsloth's mmproj-Qwen3.8-27B-F16.gguf — no - Swift projector ships, and the base vision stack was hash-proven - byte-identical to Swift's (6/6 tensors). Decode 48.0 t/s without the - drafter (MTP measured 58.9–66.9; references kept in frank-small.env). - The GSQ-RCO "IQ3_S" tier is a per-tensor allocation mix of stock types. - Replaces BOTH same-day payloads: the live MiMo Q6_K shape (below) and - the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but - never deployed +- [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — **ACTIVE since + 2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since + 2026-10-06 evening** (root systemd unit, port 7771): llama.cpp master + (CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP- + GGUF **RCO-IQ3_XXS-mtp** (27B; the MTP head in the file is DRAFTED — + draft-mtp depth 3, ngram skipped on the recurrent-rollback law) on the + RTX 3090, TWO 131,072-token conversations at Frank's 128k cap (pool + 262144 / parallel 2 — the tightest fit in the endpoint's history: + **489 MiB measured free**, the 2-3 GiB floor knowingly violated) with + q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16), ubatch 512, and + vision through the publisher's own mmproj-BF16.gguf. Decode 63.3 t/s + recipe / 72.9 temp 0 (MTP acceptance 49.8–62.4 %; 48.0 no-spec). The + RCO-IQ3_XXS tier is a per-tensor allocation mix of stock types; the + payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's + 98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape + (below) and the Ornith-1.5-9B Q8_0 swap that was committed + (77decb5..380aafa) but never deployed - [[frank-small-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening; rollback target; payload on disk)**: llama.cpp master (CUDA sm_86) serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 4742880..fb03bd7 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -369,14 +369,14 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-small-llama-swift15-iq3s-mtp-131072]] — the ACTIVE frank-small - runbook since 2026-10-05 (evening; **2 × 131,072 no-spec since - 2026-10-06**): llama.cpp master (CUDA sm_86) serving ukisai's - Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S (the -mtp file's head present but NOT - drafted — speculation off) on the RTX 3090 — TWO 131,072-token - conversations at Frank's 128k cap, the tight fit he ordered (1,397 MiB - measured free), q8_0 K + q8_0 V, vision through the base-model - projector, ~48 t/s without the drafter +- [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — the ACTIVE + frank-small runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS + with MTP back on since 2026-10-06 evening**): llama.cpp master (CUDA + sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS + (the MTP head in the file is drafted) on the RTX 3090 — TWO + 131,072-token conversations at Frank's 128k cap, the tightest fit in + the endpoint's history (489 MiB measured free), q8_0 K + q8_0 V, vision + through the publisher's own projector, ~63 t/s recipe decode with MTP - [[frank-small-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening, superseded by swift15): llama.cpp master serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at