From baf33d016fb6d938ea35dabc7272d021a2645a7d Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 5 Oct 2026 23:26:59 -0700 Subject: [PATCH] frank-small: Swift-1.5 IQ3_S-mtp swap (parallel session) + fleet docs index The parallel session's uncommitted swap, committed at Frank's direction (commit whatever's in the working tree): - llama.cpp master on the RTX 3090 now serves ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE 131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to rollback target. - env/serve/download + seam twins updated together (deployer payload gates rode the previous commit's deploy-remote.sh). - docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135 seam green). - comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/ workflows/ (12 KiB). --- AGENTS.md | 261 ++++++++++-------- .../workflows/qwen-image-2.1-t2i-art.json | 64 +++++ .../qwen-image-2.1-t2i-legible-text.json | 64 +++++ scripts/download-frank-small.sh | 67 ++--- scripts/frank-small.env | 199 ++++++------- scripts/serve-frank-small.sh | 88 +++--- .../frank-small-llama-mimo-q6k-2x200704.md | 20 +- ...ank-small-llama-swift15-iq3s-mtp-131072.md | 147 ++++++++++ serverAppConfigs-docs/LLM/index.md | 104 ++++--- serverAppConfigs-docs/index.md | 39 ++- tests/frank_small_unit.bats | 100 +++---- 11 files changed, 741 insertions(+), 412 deletions(-) create mode 100644 dankclaw/comfyui/workflows/qwen-image-2.1-t2i-art.json create mode 100644 dankclaw/comfyui/workflows/qwen-image-2.1-t2i-legible-text.json create mode 100644 serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md diff --git a/AGENTS.md b/AGENTS.md index 6647568..aef8dbd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | **FreeToken PR #587** (branch `rocm714_torch214`, head `77a7201`; local image `frank-primary-freetoken:pr587-77a7201` built on the host from `scripts/docker/frank-primary-freetoken.Dockerfile` — digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards, modelopt block-FP8, loaded straight from safetensors), **ONE R9700 (second card idle)**, all experts in **pinned host RAM** (~120 GiB) behind a GPU LRU (`--moe-cache-rate 0.10`), PLE table 47.7 GiB pinned (root unit + `LimitMEMLOCK=infinity`), **KV pool `--num-tokens 299968`** (24 KiB/token, multiple-of-64 law of the `qsa_sparse` auto backend), 2 slots, `--max-output-tokens 36864`, **parsers AUTO** (thinking → `reasoning_content`, tools qwen3.8 dialect), vision tower host-streamed. Measured: boot ~2.3 min cold / 92 s warm, decode **17.1 tok/s** single-stream; **TP=2 does not boot on ROCm** (pynccl JIT); `/metrics` 404 (Prometheus scrape reads down). Deploy gate reads the `/health` BODY (`status:"ok"`) — FreeToken 200s while loading. Rollback tag `frank-primary-vllm-shapeG-20261003` |7770| +| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (~161 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, **2 × 200,000** (first boot; native 262,144), froggeric v22.5 template, vision + tools + `reasoning_content` auto, `/metrics` scraped. Cutover 2026-10-05 (rollback `frank-primary-freetoken-pr587-20261005`, FreeToken one-card image + payload on disk) | 7770 | | Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770| -| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 | +| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp** (27B dense qwen3_5 hybrid, 12,120,016,896 B; a per-tensor allocation mix of stock types; **MTP head IN the file** — `--spec-type draft-mtp`, depth 3) + unsloth's Qwen3.8-27B **F16 mmproj** for **vision** (no Swift projector ships; the base vision stack was hash-proven byte-identical to Swift's), on the RTX 3090, **1 × 131,072** (Frank's 128k cap; a second 131,072 slot misses the 2-3 GiB floor) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 1024. Supersedes BOTH same-day payloads: the live MiMo-V2.6 Q6_K 2×200,704 f16-KV shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed. | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -264,7 +264,7 @@ merely wedged kernel usually still answers ARP. `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` - - Dankserver `frank-primary` (root unit, FreeToken on one R9700): + - Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2): `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary` 2. Bench. NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in @@ -299,8 +299,9 @@ a role pointing at a provider the file does not define fails at call time. As of 2026-10-05 **two of the three endpoints accept images** (`input: [text, image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while -`frank-primary` takes them natively (FreeToken builds the qwen3.8 vision -tower and streams its block weights from host RAM two at a time by default), +`frank-primary` takes them natively (radiance runs the checkpoint's +27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary +are the engine core; `--mm-max-patches` available), and `frank-small` through `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is @@ -378,132 +379,145 @@ rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays). ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) -**llama.cpp master** (CUDA sm_86 build -`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a -**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into -`/etc/systemd/system/`), serving -`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q6_K** (7,793,710,624 B, -sha256 `ef96d05a…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`** -(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q6K/` (the -projector is a hardlink of the Q5KM copy — the dir is self-contained), on -port 7771. The rung went Q5_K_M → Q6_K on 2026-10-03 the same morning as the -reslice, funded by the freed slots. The runtime arrived 2026-10-01 night for -one reason: -**asymmetric KV** — `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's -only per-tensor KV lever, and SGLang's `--kv-cache-dtype` moves K and V -together. On 2026-10-03 Frank spent the headroom that lever bought: the reslice -from three slots to two made room for **fully unquantized V** -(`CACHE_TYPE_V=f16`); both flags stay explicit in the serve script. Vision came -2026-10-02 via `--mmproj`: the GGUF carries no `v.*` tensors, so the projector -file is the entire vision path (and ~1.4 GiB of the card). +**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build +`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`, +`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** +(`scripts/frank-small.service`, `relink_unit`ed into +`/etc/systemd/system/`), serving `ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF` +at **IQ3_S -mtp** (12,120,016,896 B, sha256 `9aecf1cd…`) **plus +`mmproj-Qwen3.8-27B-F16.gguf`** (927,607,488 B, sha256 `cbb841a9…`), both +`/home/dank/models/Swift15-IQ3S/`, on port 7771. Cutover 2026-10-05 +(evening) on Frank's explicit contract: **128k context cap, KV at FP8, +vision on, as many 128k contexts as fit** — the answer is ONE, by +arithmetic. The same-day payloads it supersedes: the LIVE MiMo-V2.6 Q6_K +shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) +but never deployed (blocked at the docs gate; the docs debt is what this +section settles). -**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated -DeltaNet, native 262,144 window. KV exists only in the 8 attention layers -(4 KV heads × 256 dim × 2 B ≈ 2 KiB/token/layer, ~16 KiB/token for f16 K+V); -the DeltaNet state is a fixed per-slot buffer, so slot count — not context — -drives that term. +**Architecture: qwen3_5 hybrid, bigger** — block_count 65 (64 real layers + +the blk.64 MTP head), `full_attention_interval 4` ⇒ **16 full-attention + +48 Gated DeltaNet**, native 262,144 window, dense. KV exists only in the 16 +attention layers (4 KV heads × 256 dim): 64 KiB/token at f16, **34 KiB/token +at q8_0**. The DeltaNet state is a fixed per-slot buffer (~150 MiB/slot at +48 linear layers × 48 value heads). Header facts were read from the Hub +file by HTTP range + struct parse before anything was downloaded. -**The shape (2026-10-03):** `--ctx-size 401408 --parallel 2` = **TWO -conversations of 200,704 tokens each** (llama.cpp divides the pool: -`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it), -**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, and since -2026-10-05 **batch 4096 / ubatch 2048** (doubled; the boot-pair A/B is in the -runbook). Measured trade at 8,928 prompt tokens: prefill 4,024 → **4,209 t/s, -+4.6 %**, decode flat (91.9–93.0 vs 93.8–94.3 — noise), and a 279-token prompt, -which sits below either ubatch, did not move at all. The old "ubatch 2048 -failed the scratch boot" note belongs to the Q5_K_M predecessor and does NOT -generalize: this shape boots 2048 clean. Measured VRAM **21,846 used / 2,279 -free** of 24,576 MiB against **21,276 / 2,849** for the 1024 arm, reproduced -byte-identically the same hour — the doubling costs **616 MiB** and lands at -the BOTTOM of the 2–3 GiB floor Frank asked for (`4096/1536` is the middle rung -if that margin is ever wanted; non-KV 9,302 MiB, up from 8,732 by exactly the -compute buffers; f16 K+V is 32 KiB/token across the 8 attention layers; -quote `nvidia-smi`'s `free` field, never `total − used` — the driver holds -~450 MiB of reserve the subtraction hides). The 2026-10-02 3 × 180,224 + -q8_0-V shape stays one env edit away (`CTX_SIZE=540672 PARALLEL=3 -CACHE_TYPE_V=q8_0`, and it must pair with Q5_K_M — Q6_K breaks the floor -there). Three native windows need 18.4 GiB of KV alone and were never on the -table. +**The shape (2026-10-05):** `--ctx-size 131072 --parallel 1` = ONE +conversation at Frank's 128k cap (Swift's own benchmark protocol ran at +131,072), `--flash-attn on` pinned, **q8_0 K + q8_0 V** (llama.cpp's FP8 +class — no true fp8 KV dtype exists; 8.5 effective bits; the order +SUPERSEDES the same-day f16/f16 rule written for the Ornith 9B fit), all +layers on the GPU, `--mmproj` for vision, batch 4096 / ubatch 1024 (lower +ubatch first, Frank's standing law; 2048 is the first A/B if boot lands +high). Non-KV ≈ 15.5 GiB (weights 11,561 + projector ~1,350 + buffers +~1.6k + CUDA ~500 + DeltaNet ~150/slot + MTP draft ctx ~0.3-0.5k); one q8_0 +slot = 4,352 MiB ⇒ free ≈ 4.5 GiB; a SECOND slot wants +4,352 ⇒ ~0.2 GiB — +far under the 2-3 GiB floor. The pool is capped by the 128k law, not the +card; leftover headroom is honest slack. -Verified at deploy (2026-10-03): boot logs `n_slots = 2, n_ctx_slot = 200704` -and `loaded multimodal model`; decode **94.44 tok/s** at temp 0 on the probe -that measured 104.84 on Q5_K_M — **the Q6_K rung costs a measured −10 %**: -decode on this card is weight-bandwidth-bound and Q6_K streams ~10 % more -bytes/token; health 200, e2e 33/33; native `tool_calls`; thinking in -`reasoning_content`; a **512×512** solid-red PNG answers `Red` through an -`image_url` content part (two traps: the legacy top-level `images` field is -ignored, and a TINY probe image undersupplies image tokens — a 10×10 PNG made -the model claim "it's a video"; probe with ≥256 px). A prompt over the slot -is a **400 `exceed_context_size_error`**, which is why `models.yml` sets -`contextWindow: 200704`. +**Speculative decoding is the file's own MTP head**: the `-mtp` GGUF +carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`), so +`--spec-type draft-mtp --spec-draft-n-max 3` loads it against the main +model's weights — no `--model-draft`, no second draft KV pool. Flags move +as a set (a no-spec baseline strips BOTH). Acceptance reads from the +response `timings` (`draft_n`/`draft_n_accepted`) on real traffic, never a +synthetic repeat. -Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` -(the boot-pair table that prices pool changes lives in the retired -[[frank-small-llama-mimo-q5km]]). -Rollback: `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` -buys back 875 MiB and the −10 % decode rate (3,571 free, 104.84 tok/s, both -measured); the older shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` -pairs with Q5 only; SGLang rollback in -[[frank-small-sglang-mimo-v26]] (venv on disk at -`/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to -`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`). +**Vision without a Swift projector**: no Swift repo ships one (ukisai's +README is explicit), so the projector is unsloth's mmproj built from the +SHARED BASE `Qwen/Qwen3.8-27B` — legal because Swift-1.5's vision stack was +hash-proven byte-identical to the base (333/333 tensor names, 6/6 sampled +tensors: merger fc1/fc2, blocks 0/13/26) on 2026-10-05. The probe traps +carry over: ≥256 px probe images or the model answers "video"; only the +OpenAI `image_url` content part reaches the model. -### frank-primary — FreeToken (PR #587 ROCm), Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE) +The GSQ-RCO "IQ3_S" tier is a **per-tensor allocation mix** of stock quant +types (IQ2_XS 144 / IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …) +— stock llama.cpp loads it; the tensor-type histogram is the law (see Key +Lessons). Publisher KLD vs Swift BF16: 0.0513 development / 0.024–0.052 +held-out; the MTP packaging is NOT separately evaluated by the publisher. -The **FreeToken engine at PR #587 head `77a7201`** (branch -`rocm714_torch214`, built on the host into the local image -`localhost/frank-primary-freetoken:pr587-77a7201` by -`scripts/docker/frank-primary-freetoken.Dockerfile` — the PR's own -docs/install_amd.md stack: digest-pinned -`rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0` base = -torch 2.11.0+rocm7.14/triton 3.7, `--no-build-isolation --no-deps -e .`, -curated portable deps, `flashlib==0.3.0 --no-deps`, and a `libhiprtc.so` -dev-symlink the pip-ROCm wheel lacks) serving -**`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** (~172.8 GiB, 131 shards, -standard modelopt block-FP8 — 75,264 `weight_scale_inv` tensors, no -`input_scale`, loaded directly from HF safetensors, no FTW conversion) on -**ONE R9700 (gfx1201)** — the second card sits idle. Plain `podman run` -through `scripts/serve-frank-primary.sh` under the root systemd unit +Runbook: +`serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md`. +Rollback: `/home/dank/models/MiMo-Q6K/` (the LIVE payload until the cutover; +revert to the 2×200,704 f16 shape — [[frank-small-llama-mimo-q6k-2x200704]]) +or `/home/dank/models/Ornith-Q8/` (the committed-never-deployed Q8_0; +revert to 380aafa's tree). + +### frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next FP8 native, TP=2 (Dankserver, ACTIVE) + +The **radiance engine** (StillDeadcode, Apache-2.0; image +`stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…` digest-pinned, +287 MiB, `libr4d@0.5.0` kernels prebuilt for gfx1201, engine as the image +entrypoint) serving the **official `Qwen/Qwen3.8-Flash-Next-FP8` +checkpoint at NATIVE precision** — modelopt block-FP8 throughout, no +requantisation, no calibration. Plain `podman run` through +`scripts/serve-frank-primary.sh` under the root systemd unit (`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the -container name is the unit name (`frank-primary`; the vLLM-era -`frank-primary-qwen` is in the legacy cleanup list). Cutover 2026-10-05; -rollback tag `frank-primary-vllm-shapeG-20261003` (the tcclaviger vLLM trio -and its 362-line env lore live in that tag; the tcclaviger image stays in -dank's rootless storage). +container name is the unit name (`frank-primary`). Cutover 2026-10-05; +rollback tag `frank-primary-freetoken-pr587-20261005` (the FreeToken trio ++ Dockerfile are in the tag; the image and the orcarouter payload stay on +disk until the window closes), older rollback `frank-primary-vllm-shapeG-20261003`. -**Shape**: all expert weights in **pinned host RAM** (~120 GiB) behind a -GPU LRU (`--moe-cache-rate 0.10`, ~12 GiB — the single-card ceiling given -the KV contract), the ~47.7 GiB PLE ngram table pinned in host RAM -(→ root unit + memlock law), **KV pool `--num-tokens 299968`** -(24 KiB/token f16 across the 12 full-attention layers ≈ 7.0 GiB; **must be -a multiple of 64** — the auto attention backend is `qsa_sparse`, which -overrides page size to 64; 300,000 was refused at first boot), 2 decode -slots, `--max-output-tokens 36864` (the harness maxTokens; requests that -omit one get this default). **Parsers are AUTO**: thinking lands in -`reasoning_content` natively, tool calls in the qwen3.8 dialect — models.yml -flips `reasoningContentField` to `reasoning_content` on the harness side. -Vision works (models.yml keeps `input: [text, image]`); the vision tower's -blocks stream from host RAM two at a time by default. +**Payload chain**: official checkpoint → staging dir whose `config.json` +drops `quantization_config` → `stack-qwen38-official-fp8.py` (two +value-preserving reshapes: experts byte-stacked into the plugin's stacked +`gate_up_proj`/`down_proj` layout with `_scale_inv` at the same index — 49 +stacks × 512 experts, no requantisation; PLE's 128 scaleless fp8 shards +decoded to one bf16 shard with the checkpoint's own bf16 scale +1.9931793212890625e-04) → `rad-convert` with ONE `--quant` rule — pure +copy for every weight, and the rtn rule re-encodes the bf16 PLE at that +same fixed scale, an exact round-trip that restores the original codes and +lands the plugin's one-element scale plane → +`/home/dank/models/rad/qwen4exp-fp8-native.rad` (~161 GiB, floor +`FRANK_PRIMARY_RAD_MIN_MIB=158000`). The staging drop is the plugin +SELECTION law (spec §2.4): the descriptor is `(arch_id, quant)` matched +exactly, no shipped plugin claims `(qwen4exp, "fp8")`, the shipped +`qwen4exp_fp8` plugin claims `(qwen4exp, "")` — and the descriptor is only +the selection key, per-weight encodings are data, so nothing else is +requantised. +Raw serve AND unmodified convert both refuse; that is the design. The +checkpoint itself is sha256-verified against the Hub (131/131 shards, +2026-10-05). Built by `scripts/build-frank-primary-rad.sh` (`--plan-only` +first — its declare phase is the engine's own, so a resolving plan boots). -**Measured at cutover**: boot ~2.3 min cold-ish / 92 s warm → /health -status ok; decode **17.1 tok/s** single-stream temp 0 (the vLLM incumbent -measured ~30 tok/s on TWO cards; the KV contract caps the single-card LRU, -and **TP=2 exists but does not boot on ROCm** — the TP allreduce JIT -(`pynccl.cu`) can't resolve `cuda_runtime.h` for gfx1201; probed -2026-10-06, signature in the runbook). `/metrics` is a 404 — the Dankclaw -Prometheus llm-fleet scrape of :7770 reads down until repointed. -`ft ctl cache --kv/--moe` resizes pools live (grow only; shrink needs a -restart); `ft ctl stats` reports occupancy. +**Shape (both R9700s, TP=2)**: `--tp-wire wht6` (fused lossy all-reduce, ++5.6–6.3 % prefill / ~−1.3 % 8-stream decode at TP2), `--max-num-seqs 2` +(Frank's first boot; raise from the ledger + slot occupancy), `--max-model-len +200000` (native 262,144; the harness contextWindow must quote this — a longer +prompt is refused, never truncated), `--placement expert_tiered +--host-pool-mib 40960` (native FP8 experts are 112.8 GiB total, 56.4 GiB +per rank slice — the pinned pool is the price of native precision), +`--expert-vs-cache-ratio 0.82` (the measured 99 %-hit point; 0.87 collapses +to 0 %), `--kv-cache-dtype fp8` (**7.32 KiB/token**; a 200K sequence is +1.43 GiB), `--num-speculative-tokens 3` (the checkpoint's own MTP head — +3,101 `mtp.*` tensors, probe depth 3), `--max-num-batched-tokens 2048`, +prefix cache 4 GiB pinned host + 64 GiB disk at +`/home/dank/radiance-state`, and **`--ngram-placement ram`**: the 47.7 GiB +PLE n-gram table (51.2 GB = 29 % of all parameters) serves from pinned +host RAM. Froggeric v22.5 chat template vendored +(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`). -**Deploy gate law (changed for FreeToken)**: `/health` returns **200 with -`{"status":"loading"}`** while the engine loads — the FastAPI front end -binds before the scheduler is up, so a bare `curl` false-greens (measured: -"healthy after 6 s" mid-page-in). The deployer loops until the body says -`"status":"ok"`; the MainPID stray-port guard is unchanged. The image gate -is `podman image exists` (a local build carries no RepoDigests — the old -digest grep refused exactly this shape; pinning lives in the Dockerfile). +**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB pools + +47.7 GiB PLE + 4 GiB prefix host ≈ **134 GiB** charged against +`RLIMIT_MEMLOCK`. `ttm.pages_limit`/`pages_max` cap GPU-mappable pinned +pages (pre-cutover cap 123 GiB — BELOW the plan); the serve script's +TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to start until +both knobs are raised in the stop window (sysfs write first, kernel +cmdline + reboot only if read-only). -Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`. +**Contract facts** (source-verified): the HTTP listener binds AFTER the +full load, so a bare `curl /health` gate is honest (`/health` == `/ping`, +200 serving / 503 stopping); sampling params temperature…preserve_thinking +accepted, **unknown params 400** (models.yml's extraBody set is fully +inside it); `response_format` supported; `logprobs` REFUSED; reasoning in +`reasoning_content` and tool calls in the qwen3.8 dialect via the auto +chat format — no parser flags; vision native (the 27-layer Qwen-VL tower, +`--mm-max-patches`); `/metrics` carries vLLM-compatible names + +`radiance:*` families — the Dankclaw Prometheus scrape is re-added. + +Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`. +Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`. Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`. Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`. @@ -950,8 +964,8 @@ Active: | Service | Runbook | |---------|---------| | frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` | -| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` | -| frank-primary (FreeToken PR #587 ROCm, Qwen3.8-Flash-Next FP8, 1×R9700, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md` | +| frank-small (llama.cpp CUDA, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 1×131,072 q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md` | +| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -961,6 +975,9 @@ Retired (rollback targets and history): |---------|------|--------|---------| | ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` | | tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` | +| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` | +| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-small-llama-mimo-q6k-2x200704.md` | +| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-small`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-small-llama-swift15-iq3s-mtp-131072.md` | | ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` | | gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` | | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` | diff --git a/dankclaw/comfyui/workflows/qwen-image-2.1-t2i-art.json b/dankclaw/comfyui/workflows/qwen-image-2.1-t2i-art.json new file mode 100644 index 0000000..4049ad2 --- /dev/null +++ b/dankclaw/comfyui/workflows/qwen-image-2.1-t2i-art.json @@ -0,0 +1,64 @@ +{ + "3": { + "class_type": "CLIPTextEncode", + "_meta": { "title": "Positive prompt" }, + "inputs": { + "text": "A detailed illustration of a 1930s Soviet industrial design bureau at night: a well-measured engineer at a drafting table, a warm desk lamp, rolled blueprints, gears and calipers on the desk, deep red and cream palette, gouache painterly style, intricate mechanical detail, soft shadows", + "clip": [ "6", 0 ] + } + }, + "4": { + "class_type": "CLIPTextEncode", + "_meta": { "title": "Negative prompt" }, + "inputs": { + "text": "lowres, blurry, deformed, extra limbs, watermark, signature, jpeg artifacts", + "clip": [ "6", 0 ] + } + }, + "5": { + "class_type": "UNETLoader", + "_meta": { "title": "Qwen Image 2.1 (int8)" }, + "inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" } + }, + "6": { + "class_type": "CLIPLoader", + "_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" }, + "inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" } + }, + "7": { + "class_type": "VAELoader", + "_meta": { "title": "VAE" }, + "inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" } + }, + "8": { + "class_type": "EmptySD3LatentImage", + "_meta": { "title": "Empty Latent 1024x1024" }, + "inputs": { "width": 1024, "height": 1024, "batch_size": 1 } + }, + "9": { + "class_type": "KSampler", + "_meta": { "title": "KSampler" }, + "inputs": { + "model": [ "5", 0 ], + "seed": 1930, + "steps": 20, + "cfg": 1.0, + "sampler_name": "euler", + "scheduler": "simple", + "positive": [ "3", 0 ], + "negative": [ "4", 0 ], + "latent_image": [ "8", 0 ], + "denoise": 1.0 + } + }, + "10": { + "class_type": "VAEDecode", + "_meta": { "title": "VAE Decode" }, + "inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] } + }, + "11": { + "class_type": "SaveImage", + "_meta": { "title": "Save Image (art)" }, + "inputs": { "filename_prefix": "qwen21-art", "images": [ "10", 0 ] } + } +} diff --git a/dankclaw/comfyui/workflows/qwen-image-2.1-t2i-legible-text.json b/dankclaw/comfyui/workflows/qwen-image-2.1-t2i-legible-text.json new file mode 100644 index 0000000..9022cc3 --- /dev/null +++ b/dankclaw/comfyui/workflows/qwen-image-2.1-t2i-legible-text.json @@ -0,0 +1,64 @@ +{ + "3": { + "class_type": "CLIPTextEncode", + "_meta": { "title": "Positive prompt (legible text)" }, + "inputs": { + "text": "Clean minimalist movie-poster design, dark charcoal background with a subtle vintage grain. Centered large bold white sans-serif text reading \"SOLIDARITY OF THE SOVIET COMPUTER SYSTEMS\", perfectly legible, crisp letterforms, even letter spacing. Below it, a smaller line of yellow text reading \"DANKCLAW DANKSERVER EST 2026\", equally legible. Flat graphic design, high contrast, generous margins, no other words, no watermark", + "clip": [ "6", 0 ] + } + }, + "4": { + "class_type": "CLIPTextEncode", + "_meta": { "title": "Negative prompt" }, + "inputs": { + "text": "blurry, illegible, misspelled, warped letters, extra text, watermark, signature, lowres, jpeg artifacts", + "clip": [ "6", 0 ] + } + }, + "5": { + "class_type": "UNETLoader", + "_meta": { "title": "Qwen Image 2.1 (int8)" }, + "inputs": { "unet_name": "qwen_image_2.1_int8_convrot.safetensors", "weight_dtype": "default" } + }, + "6": { + "class_type": "CLIPLoader", + "_meta": { "title": "Qwen3-VL 8B text encoder (w4a8)" }, + "inputs": { "clip_name": "qwen3vl_8b_w4a8.safetensors", "type": "qwen_image" } + }, + "7": { + "class_type": "VAELoader", + "_meta": { "title": "VAE" }, + "inputs": { "vae_name": "qwen_image_2.1_vae_bf16.safetensors" } + }, + "8": { + "class_type": "EmptySD3LatentImage", + "_meta": { "title": "Empty Latent 832x1216 poster" }, + "inputs": { "width": 832, "height": 1216, "batch_size": 1 } + }, + "9": { + "class_type": "KSampler", + "_meta": { "title": "KSampler" }, + "inputs": { + "model": [ "5", 0 ], + "seed": 2026, + "steps": 25, + "cfg": 1.0, + "sampler_name": "euler", + "scheduler": "simple", + "positive": [ "3", 0 ], + "negative": [ "4", 0 ], + "latent_image": [ "8", 0 ], + "denoise": 1.0 + } + }, + "10": { + "class_type": "VAEDecode", + "_meta": { "title": "VAE Decode" }, + "inputs": { "samples": [ "9", 0 ], "vae": [ "7", 0 ] } + }, + "11": { + "class_type": "SaveImage", + "_meta": { "title": "Save Image (text poster)" }, + "inputs": { "filename_prefix": "qwen21-text", "images": [ "10", 0 ] } + } +} diff --git a/scripts/download-frank-small.sh b/scripts/download-frank-small.sh index e8ca5bf..d35cf33 100755 --- a/scripts/download-frank-small.sh +++ b/scripts/download-frank-small.sh @@ -1,32 +1,36 @@ #!/usr/bin/env bash # Download the frank-small payload for llama.cpp on the RTX 3090: -# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 plus its vision projector -# (the GGUF carries no `v.*` tensors — the tower lives only in the mmproj, so -# without this file the endpoint is text-only by construction). Public repo — -# no auth header, and never a token on the command line (it is world-readable -# in /proc//cmdline). +# ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S **with the MTP head in the +# file** (the -mtp variant), plus a vision projector. No Swift repo ships a +# projector — ukisai's release is language-model-only ("A Swift 27B vision +# projector has not been verified for this release") — so this script fetches +# unsloth's mmproj-F16.gguf built from the shared base Qwen/Qwen3.8-27B. +# That substitution is proven, not assumed: on 2026-10-05 Swift-1.5's vision +# stack was hash-compared against the base over HTTP range reads — 333/333 +# visual tensor names and 6/6 sampled tensors byte-identical (merger fc1/fc2, +# blocks 0/13/26) — so the base-model projector IS the model's own projector. +# Public repos — no auth header, and never a token on the command line (it is +# world-readable in /proc//cmdline). # # Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized file -# on a 31 GiB payload); -f/--fail-with-body so an HTTP 401 is not a success, +# on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a success, # then byte count and the Hub LFS sha256. Idempotent: complete files are -# skipped. +# skipped. The projector is stored under a self-documenting dest name; the +# URL name and sha stay pinned in the table below. # -# 2026-10-05 (Frank): the rung is Ornith-1.5-9B Q8_0. The payload moved off -# MiMo-V2.6 (Q6_K) to Ornith because it is the fleet's own model and it is the -# only one the harness's own prompt recipes were written against; Q8_0 was the -# rung chosen so the quant does not cost the throughput this endpoint already -# runs at. The Q5_K_M file this script served from 2026-10-02 stays on disk at -# /home/dank/models/MiMo-Q5KM/ (6876124704 B, LFS oid -# 04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f) as the -# rollback target; re-fetch it from the same repo with the same recipe. +# 2026-10-05 (Frank, evening): the payload moved to Swift-1.5-27B GSQ-RCO +# IQ3_S-mtp — a 27B on the 24 GiB card: ONE 131,072-token slot, q8_0 K + q8_0 +# V, the in-file MTP head drafting 3. This supersedes the Ornith-1.5-9B Q8_0 +# swap (commits 77decb5..380aafa, same day), which was committed but never +# deployed — the live service was still MiMo-V2.6 Q6_K when Swift cut over. +# Rollback payloads stay on disk: /home/dank/models/Ornith-Q8/ and +# /home/dank/models/MiMo-Q6K/ (the payload that was live until this swap). +# Their re-fetch recipes are in git history (predecessors of this script). # # Usage: # scripts/download-frank-small.sh [DEST_DIR] [--verify-only] set -euo pipefail -REPO="ornith-ai/Ornith-1.5-9B-GGUF" -BASE="https://huggingface.co/${REPO}/resolve/main" - DEST="" VERIFY_ONLY=0 for arg in "$@"; do @@ -35,19 +39,20 @@ for arg in "$@"; do *) DEST="$arg" ;; esac done -DEST="${DEST:-/home/dank/models/Ornith-Q8}" +DEST="${DEST:-/home/dank/models/Swift15-IQ3S}" -# name bytes sha256 (the sha256 is the LFS oid in the repo's pointer file) +# repo | hub-name | dest-name | bytes | sha256 (the Hub LFS oid) FILES=( - "Ornith-1.5-9B-Q8_0.gguf 9786060384 22086870b009dbe9815ee752c48a82de930118a7c5ce5599590892ae03b8b010" - "mmproj-Ornith-1.5-9B-BF16.gguf 921704672 626f9f90627402a6bf4a999111d0fbd69b5fcca7aa8ba089d69e5f10e8858e1d" + "ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF|Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf|Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf|12120016896|9aecf1cd41b2cb2f32a74e0d889e33855ebef43b26f43b43feb5720239e677e5" + "unsloth/Qwen3.8-27B-GGUF|mmproj-F16.gguf|mmproj-Qwen3.8-27B-F16.gguf|927607488|cbb841a9ee0636b2ec172f5bb8df2ea8dfeb01e90fe7c6126581d662a0b4e43e" ) mkdir -p "$DEST" status=0 for entry in "${FILES[@]}"; do - read -r name size sha <<<"$entry" - f="$DEST/$name" + IFS='|' read -r repo hubname destname size sha <<<"$entry" + url="https://huggingface.co/${repo}/resolve/main/${hubname}" + f="$DEST/$destname" have=0 if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then @@ -55,29 +60,29 @@ for entry in "${FILES[@]}"; do fi fi if (( have )); then - echo "OK $name" + echo "OK $destname" continue fi if (( VERIFY_ONLY )); then - echo "MISSING $name" + echo "MISSING $destname" status=1 continue fi - echo "FETCHING $name" + echo "FETCHING $destname (from $repo)" tmp="$f.part" - curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name" + curl -fsSL --fail-with-body -o "$tmp" "$url" mv "$tmp" "$f" if [[ "$(stat -c%s "$f")" != "$size" ]]; then - echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)" + echo "SIZE FAIL $destname ($(stat -c%s "$f") != $size)" status=1 continue fi if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then - echo "SHA FAIL $name" + echo "SHA FAIL $destname" status=1 continue fi - echo "VERIFIED $name" + echo "VERIFIED $destname" done if (( status == 0 )); then echo "DOWNLOAD_DONE $DEST" diff --git a/scripts/frank-small.env b/scripts/frank-small.env index a950f7b..71b7d0e 100644 --- a/scripts/frank-small.env +++ b/scripts/frank-small.env @@ -1,79 +1,64 @@ # frank-small — llama.cpp (master, CUDA sm_86) serving -# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 on the RTX 3090, :7771. +# ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S **-mtp** on the RTX 3090, +# :7771. # -# 2026-10-05 (Frank): the payload moved off bartowski/MiMo-V2.6 (Q6_K) to -# ornith-ai/Ornith-1.5-9B Q8_0 (9,786,060,384 B, LFS oid 22086870...). -# Header-verified facts that re-price the shape: qwen35 hybrid like MiMo, but -# 33 blocks (not 32) with full_attention_interval 4 => NINE full-attention -# layers (MiMo had 8), 16 Q heads / 4 KV heads, k/v length 256, native window -# 262144, dense (no expert keys), nextn_predict_layers=1 (an MTP head is IN -# the file — capability only; no spec decode ships, same law as before). +# 2026-10-05 (Frank, evening): the payload moved to Swift-1.5 Qwen3.8-27B +# GSQ-RCO IQ3_S WITH the MTP head in the file, each conversation capped at +# 131,072 tokens, KV at FP8 class, vision on. Header-verified facts that +# re-price the shape (read from the Hub file 2026-10-05, HTTP range + struct +# parse): qwen35 hybrid like MiMo/Ornith but bigger — block_count 65 (64 real +# layers + the blk.64 MTP head), full_attention_interval 4 => SIXTEEN +# full-attention layers (MiMo had 8, Ornith 9), 24 Q heads / 4 KV heads, +# k/v length 256, native window 262144, dense, nextn_predict_layers=1 with +# blk.64.nextn.{enorm,hnorm,shared_head_norm} (the -mtp file's head tensors +# are Q6_K/F32 per the publisher RECIPE, "MTP_OK" verified). # -# THE KV CONTRACT STAYS f16 K + f16 V — NO KV QUANTIZATION (Frank, 2026-10-05, -# overruling a written q8_0-V proposal). The arithmetic that makes this hard: -# KV rate 9 attn layers x 4 heads x 256 x 2 B = 18 KiB/token K, same again -# for V at f16 => 36 KiB/token (MiMo was ~24.5 measured: one fewer -# full-attn layer AND one fewer KV head). -# Weights +1,900 MiB (9,334 vs Q6_K's 7,432). Measured free at the Q6_K -# 4096/2048 shape was 2,279 MiB; Q8_0 at f16/f16 does not fit that shape — -# the deficit is made up the way Frank ordered: UBATCH FIRST (2048 -> 1024, -# the measured 616 MiB rung), then POOL STEPS, then measure. Guess-and-check -# on the live card is the accepted method for llama.cpp sizing here: boot, -# read nvidia-smi free, step the pool (12,288-token increments, ~432 MiB -# each at the 36 KiB/token rate) until the 2-3 GiB floor holds. Compute -# buffers allocate at graph capture during BOOT, so a failure is loud and -# fast, not a surprise at the first long prompt. +# THE KV CONTRACT IS NOW q8_0 K + q8_0 V — Frank's order for THIS swap +# ("k/v at FP8", 2026-10-05), superseding the same-day f16/f16 rule, which +# was written for the Ornith 9B fit ("no further quantization anywhere on +# this endpoint"). llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit +# class (1.0625 B/elem => 8.5 effective bits) and the only FP8-class lever +# this runtime moves. Arithmetic that makes the order load-bearing: +# KV rate = 16 attn layers x 4 heads x 256 x 1.0625 B, K and V again +# => 34 KiB/token (f16 would be 64 KiB/token — halogen's 27B card quotes +# the same f16 number; at f16 the 131,072 slot alone would cost 8.0 GiB). # -# Previous shape history (MiMo era, for the rollback trail): 2026-10-03 the -# rung went Q5_K_M -> Q6_K with the pool HELD at 401,408; the freed third -# slot funded exact f16 V first; the ~3.5 GiB still free funded the +875 MiB -# of weights. +# Why ONE slot at 131,072 is the maximum (Frank's cap per conversation): +# card 24,576 MiB. Weights 11,561 MiB (12,120,016,896 B, the IQ3_S-mtp +# file; the GSQ "IQ3_S" tier is a per-tensor ALLOCATION mix — histogram +# IQ2_XS 144 / IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 ... +# all stock types, stock llama.cpp loads it). Projector ~1,350 MiB +# (927,607,488 B F16 file at the measured ~1.45x card law). Compute +# buffers at ubatch 1024 ~1.6 GiB. CUDA ctx ~500 MiB. DeltaNet state +# ~150 MiB/slot (48 linear layers x 48 value heads x 128x128 x 4 B). +# MTP draft context ~0.3-0.5 GiB (one nextn layer over the slot window; +# the boot ledger is the truth). Non-KV ~= 15.5 GiB. One q8_0 slot = +# 4,352 MiB => free ~= 4.5 GiB. A SECOND slot wants +4,352 MiB => free +# ~0.2 GiB, far below the 2-3 GiB floor. "As many 128k contexts as fit" +# = 1. MEASURE nvidia-smi free at boot (quote the free field): under +# ~1.5 GiB or a boot failure, step the pool down in 12,288-token +# increments (~410 MiB each at this rate). Compute buffers allocate at +# graph capture during BOOT, so a failure is loud and fast. # -# 2026-10-03 (Frank): TWO streams instead of three, each one WIDE — 200,704 -# tokens per conversation, above the 200,700 the harness compacts at — and the -# KV cache is now FULLY UNQUANTIZED (f16 K, f16 V). The two moves are the same -# move: llama.cpp divides --ctx-size across --parallel slots, so the slot given -# up is exactly the VRAM that pays for unquantized V. No further quantization -# anywhere on this endpoint. -# -# Three measurements on this card, this runtime, price the terms: -# 2 x 262,144, no projector: 20,190 MiB used / 4,386 free -# KV = 524,288 tok x 24.5 KiB = 12,800 MiB => non-KV (weights 6,557 + -# DeltaNet state ~25/slot + compute buffers + CUDA graphs) = 7,390 MiB -# 3 x 196,608, projector on: 22,972 MiB used / 1,153 free -# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB -# 3 x 180,224, projector on: 21,684 MiB used / 2,441 free (the 2026-10-02 -# shape) => non-KV = 8,748 MiB -# Two slots give back one DeltaNet state (~25 MiB). Priced at f16 K+V -# (32 KiB/token) this pool is 12,544 MiB, so the prediction is -# 12,544 + ~8,800 = ~20.9 GiB used, ~3.1 GiB free — the band the 2026-10-02 -# shape hit, bought with precision instead of a slot. The 1024 ubatch -# re-prices the compute buffers upward, so MEASURE after any change: if the -# boot lands under 2 GiB free, step the pool down in 12,288-token -# (384 MiB at this rate) increments. -# -# KV arithmetic: attention KV exists only in the 8 full-attention layers -# (4 KV heads x 256 head_dim x 2 B) = 16 KiB/token for K and the same again for -# V => 32 KiB/token, so 12,288 tokens (= 6144/slot) is one 384 MiB step — that -# is the ladder to walk if this moves again. Two native windows (2 x 262,144) -# are NOT reachable with full-precision V: 524,288 x 32 KiB = 16.1 GiB of KV -# against a card with ~15.7 GiB of KV room once weights, the vision tower, the -# DeltaNet states and the graphs are loaded. 200,704 per stream is what full -# precision buys; q8_0 on V would buy 262,144 instead. -# -# The asymmetric-KV contract this endpoint was created for (Frank, 2026-10-01 -# night: f16 K + q8_0 V) is now spent — it existed to make a third slot fit, -# and the third slot is gone. The flags stay explicit so the choice is visible -# in the argv either way. +# Payload history (for the rollback trail): the Ornith-1.5-9B Q8_0 swap of +# earlier on 2026-10-05 (commits 77decb5..380aafa) was committed but never +# deployed — the live service was still MiMo-V2.6 Q6_K (2 x 200,704, f16 +# KV) when Swift cut over. Rollback payloads on disk: Ornith-Q8/ and +# MiMo-Q6K/ (one MODEL_FILE + shape edit back; see the runbook). LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server -MODEL_FILE=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf +MODEL_FILE=/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf -# Vision: the Ornith Q8_0 GGUF ships no `v.*` tensors either (dense-9B byte -# arithmetic: 9.79 GB ≈ 9B params x ~1.06 B/param leaves no room for a tower), -# so the projector file IS the vision capability (mmproj-Ornith-1.5-9B-BF16 -# .gguf, 921,704,672 B, LFS oid 626f9f90..., fetched by -# scripts/download-frank-small.sh). Empty string = text-only on purpose. -MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf +# Vision: no Swift repo ships a projector (ukisai's release is +# language-model-only by their own README), so the projector comes from the +# SHARED BASE — unsloth's mmproj-F16.gguf built from Qwen/Qwen3.8-27B, +# stored as mmproj-Qwen3.8-27B-F16.gguf (927,607,488 B, LFS sha256 +# cbb841a9..., fetched by scripts/download-frank-small.sh). The substitution +# is proven, not assumed: Swift-1.5's vision stack (tower + merger) was +# hash-compared against the base on 2026-10-05 — 333/333 visual tensor names +# and 6/6 sampled tensors byte-identical — so the base projector IS the +# model's own projector (Swift's RL/OPD post-training touched text only). +# Empty string = text-only on purpose. +MMPROJ_FILE=/home/dank/models/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf # Fleet endpoint: same port and alias as every frank-small before it, so the # harness models.yml id and the homepage probe do not move. @@ -81,56 +66,38 @@ ALIAS=frank-small BIND_ADDR=0.0.0.0 LISTEN_PORT=7771 -# ONE conversation at the NATIVE 262,144 window (Frank, 2026-10-05). The -# two-slot shape died under Ornith and the NO-QUANTIZE law, by arithmetic: -# the KV rate is the ARCHITECTURE's, not the quant's — 9 full-attn layers x -# 4 KV heads x 256 dim x 2 B = 36 KiB/token at f16 K+V (MiMo measured 24.5), -# so 2 x 200,704 wants ~14.1 GiB of KV on top of +1,900 MiB of Q8 weights -# against 2,279 MiB measured free; ubatch 1024 claws back 616, and no -# pool step fits BOTH the 2-3 GiB floor and the harness compaction -# invariant (slot >= 191,148). One native slot fits with room; the window -# per conversation RISES 200,704 -> 262,144. The cost: a second concurrent -# request queues behind the released slot (subagent concurrency 2 -> 1). -CTX_SIZE=262144 +# ONE conversation at Frank's 128k cap: --ctx-size is the POOL llama.cpp +# divides across slots; 131072 / 1 = 131,072 per conversation. The cap is a +# product decision (Swift's own benchmark protocol ran at 131,072), and a +# second 131,072 slot does not fit the 2-3 GiB floor (arithmetic above). +CTX_SIZE=131072 PARALLEL=1 -# Compute-buffer geometry. This endpoint passed neither flag until 2026-10-03, -# so it ran on llama.cpp's own defaults (2048/512); the pair is now 4096/2048. -# Both are tunables because the pair is a measurement, not a law — and the -# measurement is CARD-SPECIFIC. On the 2x R9700 layer-split sibling the ubatch -# curve peaks at 512 and every step up costs GiB of compute buffer for no -# speed (see frank-medium-batch-ubatch). On THIS CUDA 3090 the curve climbs -# monotonically to 2048: 710 -> 1188 t/s prefill from ubatch 1024 to 2048 on -# the 35B MoE sweep, 1944 -> 2423 t/s on the Ornith 8k sweep, with decode flat -# to under 1 t/s on every rung. Ubatch is a prefill knob; decode never moves. -# -# The logical batch is free — two independent sweeps agree that at a fixed -# ubatch, 2048/4096/8192 give identical prefill speed AND identical peak VRAM, -# because the compute buffers are sized by the ubatch alone. Doubling it costs -# nothing and buys nothing; it is here because it is the value the pair was -# measured at. -# -# THE RISK IS ALL IN THE UBATCH. The 1024 -> 2048 step cost ~1,244 MiB of peak -# graph VRAM on the 35B MoE (7,330 -> 8,574 MiB); this 9B hybrid has far -# smaller compute tensors, so the delta should be well under that, but the -# predecessor Q5_K_M shape recorded "ubatch 2048 failed the scratch boot" at -# ~3,571 MiB free. The Q8_0 rung ships UBATCH 1024 (Frank, 2026-10-05: lower -# ubatch FIRST — the measured 616 MiB step is the first VRAM the Q8 weights -# claw back, before any pool move). Compute buffers are allocated at graph -# capture during BOOT, so a failure is loud and fast, not a surprise at the -# first long prompt. MEASURE nvidia-smi free immediately after the deploy: -# under ~1.5 GiB, or a boot failure, step the POOL down next (12,288-token -# increments, ~432 MiB each at the Ornith 36 KiB/token f16 KV rate). +# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves); +# on this CUDA 3090 the curve climbs monotonically to 2048, but Frank's +# standing law for a resize is LOWER UBATCH FIRST (the measured 616 MiB rung +# on the 9B; the 27B's buffers are larger, so the claw-back is worth more). +# The pair is a measurement, not a law: if boot lands well above the floor, +# ubatch 2048 is the first thing to A/B for prefill (loud failure at graph +# capture if it does not fit). BATCH_SIZE=4096 UBATCH_SIZE=1024 -# The KV contract (2026-10-05, Frank): f16 K + f16 V — NO KV quantization. -# The q8_0-V arithmetic was written and OVERRULED. The Q8 weights eat the -# VRAM that funded f16 V on 2026-10-03, so the remedy ladder is UBATCH FIRST -# (2048 -> 1024, the measured 616 MiB rung), then POOL STEPS — and -# guess-and-check on the live card is the accepted method: boot, read -# nvidia-smi, step down until the floor holds. --flash-attn stays pinned on -# in the serve script: it is the fast path for f16 KV. -CACHE_TYPE_K=f16 -CACHE_TYPE_V=f16 +# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V — +# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in +# the argv either way, and --flash-attn stays pinned on in the serve script: +# it is required for quantized V (the build carries +# GGML_CUDA_FA_ALL_QUANTS=ON since 2026-09-17, so q8_0 KV is a real path, +# not a silent f16 fallback). +CACHE_TYPE_K=q8_0 +CACHE_TYPE_V=q8_0 + +# Speculative decoding: the MTP head is IN the -mtp file (blk.64.nextn.*, +# nextn_predict_layers=1), so draft-mtp needs no second model and no second +# KV pool — the head shares the main model's weights. Depth 3 matches the +# fleet's radiance MTP depth; measure draft acceptance from the response +# timings object and retune here if the head earns more. +SPEC_TYPE=draft-mtp +SPEC_DRAFT_N_MAX=3 + NGL=99 diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index 6c200e7..892f29f 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -1,20 +1,23 @@ #!/usr/bin/env bash -# Serve ornith-ai/Ornith-1.5-9B-GGUF (Q8_0) as frank-small on -# Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86). +# Serve ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF (IQ3_S **-mtp**) as +# frank-small on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA +# sm_86). # -# 2026-10-01 night (Frank): swapped from SGLang + INT8-W8A16-AutoRound -# safetensors to the llama.cpp GGUF path, because llama.cpp is the only -# runtime on this host that moves K and V independently — the asymmetric -# f16-K + q8_0-V contract that endpoint was built for. 2026-10-03 (Frank): -# the pair is now f16/f16 — two wide slots freed the VRAM, so the values are -# unquantized too. The per-tensor flags stay explicit either way. +# 2026-10-05 evening (Frank): a 27B on the 24 GiB card — ONE 131,072-token +# conversation (Frank's cap), KV at FP8 class (q8_0 both halves), the MTP +# head that lives IN the -mtp file drafting 3, vision on. The q8_0 KV +# contract supersedes the same-day f16/f16 rule, which was written for the +# Ornith 9B fit; the per-tensor flags stay explicit either way. # -# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated -# DeltaNet, native 262144 window. KV lives only in the 8 attention layers; -# the DeltaNet state is per-slot fixed size. Vision comes from the repo's -# separate projector (the Q6_K GGUF carries no `v.*` tensors at all), so -# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the documented -# text-only opt-out. +# Architecture: qwen35 hybrid — 64 real layers, 16 full-attention + 48 +# Gated DeltaNet (full_attention_interval 4), native 262144 window, plus +# the blk.64 MTP head (nextn_predict_layers=1). KV lives only in the 16 +# attention layers (34 KiB/token at q8_0); the DeltaNet state is per-slot +# fixed size. Vision comes from the projector file (the GGUF carries no +# vision tensors; unsloth's mmproj built from the shared Qwen3.8-27B base — +# proven byte-identical to Swift's own tower, see download-frank-small.sh), +# so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the +# documented text-only opt-out. # # Sampling: llama.cpp takes it per request (the fleet harness sends its # recipe from models.yml); greedy is the harness default. @@ -33,31 +36,37 @@ fi # --- defaults (used when the env file does not set them) ------------------- : "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}" -: "${MODEL_FILE:=/home/dank/models/Ornith-Q8/Ornith-1.5-9B-Q8_0.gguf}" +: "${MODEL_FILE:=/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf}" : "${ALIAS:=frank-small}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7771}" -# ctx-size is the POOL llama.cpp divides across slots: 401408 / 2 gives -# 200,704 per slot — two independent conversations, each one wide enough to -# hold a whole frank-primary-sized prompt in a single shot (the native 262,144 -# no longer fits twice on a 24 GiB card once the vision tower is loaded; -# frank-small.env carries the three measurements that set this number). -: "${CTX_SIZE:=401408}" -: "${PARALLEL:=2}" -# Compute-buffer geometry. Before 2026-10-03 this endpoint never passed either -# flag, so it ran on llama.cpp's own defaults (2048/512). The pair is 4096/2048 -# since 2026-10-05: on this CUDA card the prefill curve climbs all the way to -# ubatch 2048 (unlike the R9700 layer-split sibling, where 512 is the peak), -# and the logical batch is VRAM- and throughput-neutral. frank-small.env -# carries the sweep tables and the VRAM risk note. +# ctx-size is the POOL llama.cpp divides across slots: 131072 / 1 gives +# 131,072 per conversation — Frank's 128k cap. A second 131,072 slot does +# not fit the 2-3 GiB floor on the 24 GiB card with the 27B payload +# (frank-small.env carries the arithmetic); "as many 128k contexts as fit" +# = 1. +: "${CTX_SIZE:=131072}" +: "${PARALLEL:=1}" +# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves), +# and Frank's standing law for a resize is LOWER UBATCH FIRST; the shipped +# pair is 4096/1024. If boot lands well above the 2-3 GiB floor, ubatch +# 2048 is the first A/B for prefill (a miss fails loud at graph capture). +# The logical batch is VRAM- and throughput-neutral at a fixed ubatch. : "${BATCH_SIZE:=4096}" -: "${UBATCH_SIZE:=2048}" -: "${CACHE_TYPE_K:=f16}" -: "${CACHE_TYPE_V:=f16}" +: "${UBATCH_SIZE:=1024}" +# KV at FP8 class = q8_0 on llama.cpp (1.0625 B/elem => 8.5 effective bits; +# there is no true fp8 KV dtype). Explicit either way so the contract is +# visible in the argv. +: "${CACHE_TYPE_K:=q8_0}" +: "${CACHE_TYPE_V:=q8_0}" +# Speculative decoding: draft-mtp loads the MTP head that is IN the -mtp +# GGUF; the flags move as a set. +: "${SPEC_TYPE:=draft-mtp}" +: "${SPEC_DRAFT_N_MAX:=3}" : "${NGL:=99}" # Vision projector. No colon on purpose: an explicitly empty value means # "text-only, on purpose" and must NOT be replaced by the default path. -: "${MMPROJ_FILE=/home/dank/models/Ornith-Q8/mmproj-Ornith-1.5-9B-BF16.gguf}" +: "${MMPROJ_FILE=/home/dank/models/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf}" # Fail loudly rather than half-loading: a missing file is a download that # has not finished, and Restart=always would turn it into a crash loop that @@ -71,14 +80,21 @@ if [[ -n "$MMPROJ_FILE" ]]; then EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE") fi +# The MTP head ships IN the -mtp GGUF (blk.64.nextn.*, +# nextn_predict_layers=1): draft-mtp loads it against the main model's own +# weights — no --model-draft and no second KV pool. Speculative flags move +# as a set: a no-spec baseline must strip BOTH of these, not just one. +EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX") + # The UVM device node is created lazily by nvidia-modprobe on the first CUDA # open after boot; make sure it exists before the server needs it. nvidia-modprobe -u -c=0 2>/dev/null || true -# Flash attention stays pinned on: it is the fast path for f16 KV, and it is -# the only path that could serve quantized V at all — so if CACHE_TYPE_V is -# ever set back to q8_0 this flag must not be lost, or the build falls back -# silently and the KV contract in frank-small.env becomes a lie. +# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0), +# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so +# the q8_0 KV contract in frank-small.env is a real path — without this +# flag or that build option the engine silently falls back and the +# contract becomes a lie. ARGS=( --model "$MODEL_FILE" --alias "$ALIAS" diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md index 7f19eca..c0c4c27 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md @@ -1,7 +1,7 @@ --- type: reference -title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV" -description: "The active frank-small runbook — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change." +title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)" +description: "The RETIRED frank-small runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap." tags: - llm - frank-small @@ -14,13 +14,15 @@ timestamp: 2026-10-03 # frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV -> **ACTIVE since 2026-10-03.** The third shape Frank bought in two days on -> this card: the freed third slot paid for **exact KV values** -> ([[frank-small-llama-mimo-q5km]] documents that step and remains the -> rollback), and the ~3.5 GiB still free then paid for the **Q6_K quant rung** -> — the same file this endpoint served on 2026-10-01, come back as an -> upgrade rather than a compromise. The pool held at 401,408 (2 × 200,704); -> only the weights moved up. +> **RETIRED 2026-10-05 (evening).** Superseded by +> [[frank-small-llama-swift15-iq3s-mtp-131072]] — the Swift-1.5-27B GSQ-RCO +> IQ3_S-mtp swap. Two facts for the trail: (1) the Ornith-1.5-9B Q8_0 swap +> committed earlier the same day (77decb5..380aafa) was **never deployed** — +> THIS MiMo shape was the live payload when Swift cut over; (2) the KV +> contract Frank set here (f16 K + f16 V, "no further quantization on this +> endpoint") was superseded the same evening by the Swift swap's explicit +> q8_0/q8_0 order. This page stays as the rollback and the measured +> boot-pair record. Payload on disk: `/home/dank/models/MiMo-Q6K/`. ## Identity diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md b/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md new file mode 100644 index 0000000..6c2faa6 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-small-llama-swift15-iq3s-mtp-131072.md @@ -0,0 +1,147 @@ +--- +type: reference +title: "frank-small on Dankclaw — llama.cpp, Swift-1.5-27B GSQ-RCO IQ3_S-mtp, 1 x 131072, q8_0 KV" +description: "The active frank-small runbook (2026-10-05 evening) — llama.cpp master (CUDA sm_86) serving ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on the RTX 3090: one 131,072-token conversation at Frank's 128k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the base-model projector (proven byte-identical tower), and the arithmetic that makes one slot the maximum." +tags: + - llm + - frank-small + - dankclaw + - llama-cpp + - gguf + - rtx3090 + - qwen35 + - mtp + - vision +timestamp: 2026-10-05 +--- + +# frank-small — llama.cpp Swift-1.5-27B IQ3_S-mtp, 1 × 131,072, q8_0 KV + +> **ACTIVE since 2026-10-05 (evening).** A 27B replaces the 9B on the same +> 24 GiB card, bought by Frank's explicit contract for this swap: context +> capped at 128k per conversation, KV at FP8 class, vision on, "as many of +> those contexts as you can" — the answer turned out to be **one**, by +> arithmetic, not by choice (see *Sizing*). This supersedes BOTH earlier +> 2026-10-05 payloads: the live MiMo-V2.6 Q6_K shape +> ([[frank-small-llama-mimo-q6k-2x200704]], now retired) and the Ornith-1.5-9B +> Q8_0 swap that was committed (77decb5..380aafa) but never deployed. + +## Identity + +| | | +|---|---| +| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB | +| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface | +| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON` — required for quantized V) | +| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` | +| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) | +| Weights | `/home/dank/models/Swift15-IQ3S/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` — ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp**, 12,120,016,896 B, sha256 `9aecf1cd…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`) | +| Projector | `…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf` — unsloth/Qwen3.8-27B-GGUF `mmproj-F16.gguf`, 927,607,488 B, sha256 `cbb841a9…` (27-layer ViT, 1152 hidden, `qwen3vl_merger` → 5120) | +| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified per file; both verified on the host before deploy | +| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, MTP pair, batch pair, projector opt-out) | + +## Why a base-model projector is legal here + +No Swift repo ships a projector — ukisai's GSQ-RCO release is +language-model-only by their own README ("A Swift 27B vision projector has +not been verified for this release"). The substitution is proven, not +assumed: Swift-1.5 is an RL/OPD post-train of `Qwen/Qwen3.8-27B`, and its +vision stack was hash-compared against the base on 2026-10-05 over HTTP +range reads of the two BF16 repos — **333/333 visual tensor names match and +6/6 sampled tensors are byte-identical** (merger `linear_fc1`/`linear_fc2`, +blocks 0/13/26 attn/mlp). Swift's post-training touched text only, so the +base projector IS the model's own projector. The mmproj header matches +Swift's `vision_config` field-for-field (depth 27, hidden 1152, patch 16, +merge 2, ffn 4304, 16 heads, out 5120). + +## Shape and KV contract + +`--ctx-size 131072 --parallel 1` — ONE conversation at Frank's 128k cap +(the GSQ publisher's own benchmark protocol ran at 131,072). `CTX_SIZE` +stays divisible by `PARALLEL`; the seam enforces it. + +**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).** +llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem +⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves. The +order supersedes the same-day f16/f16 rule, which was written for the +Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would +cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers × +4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is +pinned: quantized V requires it, and the build's +`GGML_CUDA_FA_ALL_QUANTS=ON` (2026-09-17) makes the path real rather than a +silent f16 fallback. + +**One slot is the maximum, by arithmetic.** Card 24,576 MiB; weights +11,561 MiB; projector ~1,350 MiB (927,607,488 B at the measured ~1.45× card +law); compute buffers at ubatch 1024 ~1.6 GiB; CUDA ~500 MiB; DeltaNet +state ~150 MiB/slot (48 linear layers × 48 value heads × 128×128 × 4 B); MTP +draft context ~0.3–0.5 GiB. Non-KV ≈ 15.5 GiB. One q8_0 slot = 4,352 MiB ⇒ +free ≈ 4.5 GiB. A second slot wants +4,352 MiB ⇒ ~0.2 GiB free, far under +the 2–3 GiB floor. The pool is capped by the 128k law, not by the card — +leftover headroom is honest slack, not an undersized pool. + +## MTP (speculative decoding) + +The MTP head ships **in the file** — the `-mtp` GGUF carries +`blk.64.nextn.{enorm,hnorm,shared_head_norm}` + the head layer's tensors at +Q6_K/F32 (publisher RECIPE: "MTP_OK verified"), +`qwen35.nextn_predict_layers=1`. `--spec-type draft-mtp` loads it against +the main model's own weights — no `--model-draft`, no second draft KV pool +(an MTP head can NEVER load as a separate draft model on this stack; two +context-backed drafters cannot share a process). Depth 3 +(`--spec-draft-n-max 3`) matches the fleet's radiance MTP depth. The flags +move as a set: a no-spec baseline must strip BOTH. Acceptance reads from +the response `timings` object (`draft_n`, `draft_n_accepted`) — measure on +real traffic, never on a synthetic repeat (greedy-text acceptance lies). + +## Vision contract + +`--mmproj` is the entire vision path (the GGUF carries no vision tensors). +Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny +probe undersupplies image tokens and the model invents ("video"); the +legacy top-level `images` field is IGNORED, only the OpenAI `image_url` +content part reaches the model. Same traps the MiMo eras documented. + +## Client contract + +- Reasoning in `reasoning_content` (`--reasoning-format deepseek`); thinking + tokens COUNT against `max_tokens`. +- The embedded chat template gates `reasoning_effort` to + `{xhigh, medium, low}` — anything else is a raise/HTTP 500, so + `models.yml` `compat.reasoningEffortMap` folds the harness levels onto + that set (mandatory, same as every Qwen3.8-template endpoint). +- Sampling rides the wire (`extraBody`): Swift's publisher recipe temp 1.0 + / top_p 0.95 / top_k 20 / min_p 0 (the card was evaluated at exactly + this recipe at ctx 131,072). +- Swift 1.5 is a reasoning-efficient post-train: −58.5 % median thinking + tokens vs base at equal-or-better accuracy. Token budgets go further; + the 32,768 `maxTokens` is unchanged from the MiMo era and still clears + the compaction invariant (91,750 + 32,768 ≤ 131,072). +- The GSQ-RCO "IQ3_S" tier is a **per-tensor allocation mix** (IQ2_XS 144 / + IQ3_XXS 96 / Q6_K 78 / F32 360 / BF16 96 / Q3_K_M 39 …), all stock quant + types — stock llama.cpp loads it. Publisher KLD vs Swift BF16: 0.0513 + (development), held-out 0.024–0.052 per domain; the MTP packaging is NOT + separately evaluated by the publisher. + +## Rollback + +Two payloads on disk, both fetchable by the same script's git-history +recipes: + +1. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until + this cutover: `/home/dank/models/MiMo-Q6K/`; revert + [[frank-small-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or + 380aafa's parent), one `MODEL_FILE` + shape edit. +2. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap: + `/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native + 262,144 slot, f16/f16 KV). + +## Trail + +- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was + committed and pushed but its deploy never completed (blocked at the + whole-tree docs gate); the live service was still MiMo Q6_K when Swift + cut over. Both Ornith payloads were verified on disk before Swift. +- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were + read from the Hub file by HTTP range + struct parse — no download + needed — before the payload was fetched. diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 866f20a..2ea61ee 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,25 +1,24 @@ # LLM Fleet — Knowledge Base -Documentation for the LLM inference fleet on the two hosts (2026-10-05): -**Dankserver runs FreeToken (PR #587 ROCm) on frank-primary** (the local -image `frank-primary-freetoken:pr587-77a7201` built on the host from -`scripts/docker/frank-primary-freetoken.Dockerfile` — FreeToken at PR head -`77a7201` in the PR's own ROCm 7.14 base image, torch 2.11.0+rocm7.14 — -driving `podman run` directly, serving orcarouter -/Qwen3.8-Flash-Next-Uncensored-FP8 on ONE R9700, :7770: all experts in -pinned host RAM behind a GPU LRU (`--moe-cache-rate 0.10`), KV pool -`--num-tokens 299968` (multiple-of-64 law of the qsa_sparse backend), 2 -decode slots, parsers AUTO (thinking → `reasoning_content` natively; -measured 17.1 tok/s decode, TP=2 exists but does not boot on ROCm — see the -runbook). Rollback tag frank-primary-vllm-shapeG-20261003), -and Dankclaw -runs two endpoints — **halogen container (frank-large, port 7770: +**Dankserver runs the radiance engine (StillDeadcode) on frank-primary** +(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint, +driving `podman run` directly, root systemd unit, :7770): the OFFICIAL +`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision +(converted no-recipe to `qwen4exp-fp8-native.rad` — see the runbook for +the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`), +experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the +47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3, +2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005 +and frank-primary-vllm-shapeG-20261003), +and Dankclaw runs two endpoints — **halogen container (frank-large, port 7770: ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai /halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision lives on frank-small)** and -**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K -GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**. +**llama.cpp CUDA (frank-small, port 7771: ukisai Swift-1.5-Qwen3.8-27B +GSQ-RCO IQ3_S-mtp GGUF — 27B with the MTP head IN the file, ONE 131,072 +conversation at Frank's 128k cap, q8_0 K + q8_0 V (FP8 class), vision via +the base-model projector, on the RTX 3090)**. > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -33,12 +32,12 @@ graph LR subgraph Dankserver[AMD Dankserver - Ubuntu 26.04] A[2x Radeon AI PRO R9700 gfx1201, 32 GiB each] - B2[FreeToken PR587 ROCm, local image + B2[radiance StillDeadcode, digest-pinned image podman run, frank-primary, port 7770] - C[Qwen3.8-Flash-Next FP8 -1x R9700, experts in pinned host RAM + GPU LRU 10%, -KV pool 299,968 tokens, 2 slots, parsers AUTO, -TP=2 untested on ROCm; second R9700 idle] + C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8 +2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM, +PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3, +2 x 200000, froggeric v22.5] end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] @@ -50,8 +49,8 @@ frank-large, port 7770] 1 x 200700 ctx, d2 KV, text-only] G[llama.cpp master CUDA sm_86 frank-small, port 7771] - H[MiMo-V2.6 Q5_K_M GGUF -2 x 200704 ctx, f16 K + f16 V, flash-attn, mmproj vision] + H[Swift-1.5-27B GSQ-RCO IQ3_S-mtp GGUF +1 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision] E2 --> E --> F E2 --> G --> H end @@ -81,22 +80,52 @@ frank-small, port 7771] No server-side sampler — the thinking recipe rides the wire in models.yml. Rollback: revert the swap commit; the ROCmFPX payload + fork build + `build-rocmfpx.sh` pin all remain on disk -- [[frank-small-llama-mimo-q6k-2x200704]] — **ACTIVE since 2026-10-03** - (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving - bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090, TWO - slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K + - f16 V) and the mmproj for vision — 94.4 tok/s decode (the rung costs a - measured -10 % vs Q5_K_M: weight-bandwidth-bound), batch 4096 / ubatch 2048 - since 2026-10-05 (+4.6 % prefill, measured), 2,279 MiB of the card - left free. Probe-size trap (>=256 px images or the model says "video") - documented. Rollback: one MODEL_FILE line back to Q5_K_M +- [[frank-small-llama-swift15-iq3s-mtp-131072]] — **ACTIVE since 2026-10-05 + (evening)** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) + serving ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S-mtp** (27B, the + checkpoint's own MTP head IN the file — `--spec-type draft-mtp`, depth 3) + on the RTX 3090, ONE 131,072-token conversation at Frank's 128k cap (a + second 131,072 slot misses the 2-3 GiB floor) with q8_0 K + q8_0 V (FP8 + class, 34 KiB/token vs 64 at f16) and vision via unsloth's + mmproj-Qwen3.8-27B-F16.gguf — no Swift projector ships, and the base + vision stack was hash-proven byte-identical to Swift's (6/6 tensors). + The GSQ-RCO "IQ3_S" tier is a per-tensor allocation mix of stock types. + Replaces BOTH same-day payloads: the live MiMo Q6_K shape (below) and + the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but + never deployed +- [[frank-small-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening; + rollback target; payload on disk)**: llama.cpp master (CUDA sm_86) + serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of + 200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for + vision — 94.4 tok/s decode (the rung costs a measured -10 % vs Q5_K_M: + weight-bandwidth-bound), batch 4096 / ubatch 2048 since 2026-10-05 + (+4.6 % prefill, measured), 2,279 MiB of the card left free. This was + the LIVE payload when Swift cut over. The ubatch A/B table and the + probe-size trap (>=256 px images or the model says "video") live inside - [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback target; payload on disk; owns the measured boot-pair table)**: the same runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower — 104.8 tok/s, 3,571 MiB true free -- [[frank-primary-freetoken-pr587]] — **ACTIVE since 2026-10-05** - (root systemd unit, port 7770): FreeToken at PR #587 head `77a7201` - (branch rocm714_torch214) built on the host into the local image +- [[frank-primary-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05** + (root systemd unit, port 7770): the StillDeadcode **radiance** engine + (C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as + the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8` + checkpoint at NATIVE precision — converted no-recipe to + `qwen4exp-fp8-native.rad` (~161 GiB) after dropping `quantization_config` + in a staging dir (the plugin descriptor-selection law: nothing claims + `(qwen4exp, "fp8")`, the shipped plugin claims `(qwen4exp, "")`; + per-weight encodings are data, not the descriptor) — on BOTH R9700s + **TP=2** (`--tp-wire wht6` fused all-reduce), experts in a 40 GiB/rank + pinned host pool + VRAM cache (ratio 0.82 = the measured 99 %-hit point), + the 47.7 GiB PLE n-gram table PINNED to host RAM (`--ngram-placement + ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first + boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto, + `/metrics` scraped. Rollback: git tag + `frank-primary-freetoken-pr587-20261005` (FreeToken image + orcarouter + payload on disk); older rollback `frank-primary-vllm-shapeG-20261003` +- [[frank-primary-freetoken-pr587]] — **RETIRED 2026-10-05 (same day; + rollback target)**: FreeToken at PR #587 head `77a7201` (branch + rocm714_torch214) built on the host into the local image `frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law, curated --no-deps install per docs/install_amd.md), serving the SAME @@ -105,8 +134,9 @@ frank-small, port 7771] (multiple-of-64 law), 2 slots, parsers AUTO (thinking → `reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s single-stream, vision "Red" probe passes, native tool_calls; TP=2 does - not boot on ROCm (pynccl JIT) — second card idle. Rollback: git tag - frank-primary-vllm-shapeG-20261003 + not boot on ROCm (pynccl JIT) — second card idle. Superseded by the + radiance swap for TP=2; the image + payload stay on disk until the + rollback window closes - [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05 (rollback target; the vLLM trio + env lore live in the tag)**: the tcclaviger RDNA4 diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index d6384b1..8c942e2 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -181,11 +181,20 @@ The LLM inference fleet on the two hosts. Full index: ### Deployment and Topology - [[deployment]] — unified deploy and update process for both servers -- [[frank-primary-freetoken-pr587]] — the ACTIVE frank-primary runbook - (2026-10-05): FreeToken built at PR #587 head `77a7201` on the host into +- [[frank-primary-radiance-qwen38-native-fp8]] — the ACTIVE frank-primary + runbook (2026-10-05): the StillDeadcode radiance engine (digest-pinned + 287 MiB image, libr4d gfx1201 kernels) serving the OFFICIAL + `Qwen/Qwen3.8-Flash-Next-FP8` at native block-FP8 (no-recipe convert + after the descriptor-staging drop) on BOTH R9700s TP=2 `wht6`, experts + in a 40 GiB/rank pinned host pool, PLE 47.7 GiB pinned to RAM, fp8 KV + 7.32 KiB/token, MTP depth 3, 2 × 200,000 first boot; rollback is the + FreeToken one-card era (tag `frank-primary-freetoken-pr587-20261005`), + then the tcclaviger shape-G era (tag `frank-primary-vllm-shapeG-20261003`) +- [[frank-primary-freetoken-pr587]] — RETIRED 2026-10-05 (superseded the + same day by the radiance cutover above; kept as the rollback runbook): + FreeToken built at PR #587 head `77a7201` on the host into `frank-primary-freetoken:pr587-77a7201`, ONE R9700 with experts in pinned - host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO; rollback is the - tcclaviger shape-G era (tag `frank-primary-vllm-shapeG-20261003`) + host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change - [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps back; rollback target): the R9V radiance-vLLM fork (docker-pinned bundle @@ -359,14 +368,20 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-small-llama-mimo-q6k-2x200704]] — the ACTIVE frank-small runbook - since 2026-10-03: llama.cpp master (CUDA sm_86) serving - bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on Dankclaw's RTX 3090, - port 7771 — TWO conversations at 200,704 tokens each with fully - unquantized f16 KV and mmproj vision; 94.4 tok/s decode (the Q6_K rung - costs a measured −10 % vs Q5_K_M — weight-bandwidth), batch 4096 / ubatch - 2048 (+4.6 % prefill, measured 2026-10-05), 2,279 MiB free, and - the ≥256 px image-probe trap (tiny images answer "video") +- [[frank-small-llama-swift15-iq3s-mtp-131072]] — the ACTIVE frank-small + runbook since 2026-10-05 (evening): llama.cpp master (CUDA sm_86) serving + ukisai's Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head on + the RTX 3090 — ONE 131,072-token conversation at Frank's 128k cap, q8_0 + K + q8_0 V, vision through the base-model projector; the arithmetic that + makes one slot the maximum +- [[frank-small-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening, + superseded by swift15): llama.cpp master serving + bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at + 200,704 tokens each with fully unquantized f16 KV and mmproj vision; + 94.4 tok/s decode (the Q6_K rung costs a measured −10 % vs Q5_K_M — + weight-bandwidth), batch 4096 / ubatch 2048 (+4.6 % prefill, measured + 2026-10-05), 2,279 MiB free, and the ≥256 px image-probe trap (tiny + images answer "video") - [[frank-small-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback target, payload on disk; owns the measured boot-pair table): same runtime and shape one rung lower — 104.8 tok/s, 3,571 MiB true free diff --git a/tests/frank_small_unit.bats b/tests/frank_small_unit.bats index 9dd0541..3284fd5 100755 --- a/tests/frank_small_unit.bats +++ b/tests/frank_small_unit.bats @@ -1,10 +1,10 @@ #!/usr/bin/env bats # Seam: the frank-small llama.cpp root unit on dankclaw. -# frank-small is the RTX 3090 endpoint; since 2026-10-05 it serves -# ornith-ai/Ornith-1.5-9B-GGUF:Q8_0 via a CUDA (sm_86) build of -# llama.cpp master, chosen because it is the only runtime here that moves K and -# V independently. Since 2026-10-03 it runs TWO unquantized contexts of -# 200,704 tokens each and carries a vision projector, and the card must keep +# frank-small is the RTX 3090 endpoint; since 2026-10-05 evening it serves +# ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S-mtp (27B, MTP head in the +# file) via a CUDA (sm_86) build of llama.cpp master: ONE 131,072-token +# conversation at Frank's 128k cap, q8_0 K + q8_0 V (FP8 class), the +# in-file MTP head drafting 3, and a vision projector. The card must keep # 2-3 GiB free. This seam runs the serve script against a stub engine and # asserts on the argv it actually execs, so what is pinned is the # command the operator gets. @@ -16,9 +16,9 @@ SERVE="$REPO_ROOT/scripts/serve-frank-small.sh" ENV_FILE="$REPO_ROOT/scripts/frank-small.env" REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" ENV_LARGE="$REPO_ROOT/scripts/frank-large.env" -TARGET_STUB="Ornith-1.5-9B-Q8_0.gguf" +TARGET_STUB="Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf" # Same basename as the shipped projector: the test env repoints it into $STAGE. -MM_STUB="mmproj-Ornith-1.5-9B-BF16.gguf" +MM_STUB="mmproj-Qwen3.8-27B-F16.gguf" setup() { STAGE="$(mktemp -d)" @@ -81,9 +81,9 @@ argv_value() { grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT" } -@test "the shipped env points at the llama.cpp runtime and the Ornith-1.5-9B Q8_0 GGUF" { +@test "the shipped env points at the llama.cpp runtime and the Swift-1.5-27B IQ3_S-mtp GGUF" { grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE" - grep -qE "^MODEL_FILE=/home/dank/models/Ornith-Q8/$TARGET_STUB$" "$ENV_FILE" + grep -qE "^MODEL_FILE=/home/dank/models/Swift15-IQ3S/$TARGET_STUB$" "$ENV_FILE" } @test "the deployer gates on exactly the payload paths the env file ships" { @@ -100,7 +100,7 @@ argv_value() { done } -@test "the shipped env serves the fleet shape: one native 262144-token context" { +@test "the shipped env serves the fleet shape: one conversation at the 128k cap" { serve_argv [ "$status" -eq 0 ] [ "$(argv_value --alias)" = "frank-small" ] @@ -110,54 +110,50 @@ argv_value() { [ "$(argv_value --n-gpu-layers)" -gt 0 ] # llama.cpp divides --ctx-size across slots and a mamba hybrid cannot # share one cache, so the observable contract is the per-slot window. - # 2026-10-05 (Frank): NO KV quantization forces ONE slot at the NATIVE - # window — Ornith's 9 full-attn layers price f16 KV at 36 KiB/token, and - # two 200,704 slots cannot fit the 2-3 GiB floor on 24 GiB with Q8 - # weights. The window per conversation ROSE 200,704 -> 262,144. + # 2026-10-05 evening (Frank): the 27B Swift-1.5 swap caps each + # conversation at 131,072, and ONE slot is the maximum — a second + # 131,072 slot does not fit the 2-3 GiB floor (KV budget test below). [ "$(argv_value --parallel)" -eq 1 ] ctx="$(argv_value --ctx-size)" par="$(argv_value --parallel)" [ $((ctx % par)) -eq 0 ] - [ $((ctx / par)) -eq 262144 ] + [ $((ctx / par)) -eq 131072 ] } -@test "the KV contract is unquantized both halves, flash-attn on" { +@test "the KV contract is q8_0 both halves (FP8 class), flash-attn on" { serve_argv [ "$status" -eq 0 ] - [ "$(argv_value --cache-type-k)" = "f16" ] - # Frank, 2026-10-05: NO KV quantization on this endpoint. The Q8_0 rung - # funds itself by UBATCH first (2048 -> 1024), then pool steps — a q8_0-V - # proposal was written and overruled. The flag pair is what makes this + # Frank, 2026-10-05 evening: "k/v at FP8" for the Swift-1.5 swap — + # llama.cpp's FP8 class is q8_0 (1.0625 B/elem => 8.5 effective bits; + # there is no true fp8 KV dtype). This SUPERSEDES the same-day f16/f16 + # rule written for the Ornith 9B fit. The flag pair is what makes this # observable — a build without the FA path silently falls back, so both # must appear in the argv the supervisor runs. - [ "$(argv_value --cache-type-v)" = "f16" ] + [ "$(argv_value --cache-type-k)" = "q8_0" ] + [ "$(argv_value --cache-type-v)" = "q8_0" ] [ "$(argv_value --flash-attn)" = "on" ] } -@test "the KV budget leaves 2-3 GiB of the 3090 free" { - # Rate, from the Ornith GGUF header (read from the Hub file, 2026-10-05): - # NINE full-attention layers (block_count 33, full_attention_interval 4) - # x 4 kv heads x 256 dim x 2 B = 18 KiB/token for K and the same for V - # => 36 KiB/token (budgeted 37 with slack; MiMo measured 24.5 — one - # fewer layer AND one fewer KV head). The non-KV half: Q8_0 weights - # 9,334 MiB (9,786,060,384 B) + vision tower ~1.3 GiB (mmproj 879 MiB at - # the measured ~1.5x cost law) + DeltaNet state + compute buffers + - # CUDA graphs => 11 GiB budgeted. Two asserts, both about the output - # the operator sees: the pool must still fit with 2 GiB spare, and it - # must be large enough that 5 GiB spare is NOT reachable (an undersized - # pool is as wrong as an oversized one). +@test "the KV budget fits the 2-3 GiB floor and forbids a second 128k slot" { + # Rate, from the Swift-1.5 IQ3_S-mtp GGUF header (read from the Hub + # file, 2026-10-05): SIXTEEN full-attention layers (block_count 65 = 64 + # real layers + the blk.64 MTP head, full_attention_interval 4) x 4 kv + # heads x 256 dim x 1.0625 B = 17 KiB/token for K and the same for V + # => 34 KiB/token at q8_0 (budgeted 37 with slack; f16 would be 64). + # The non-KV half: IQ3_S-mtp weights 11,561 MiB (12,120,016,896 B) + + # vision tower ~1.35 GiB (927,607,488 B F16 mmproj at the measured + # ~1.45x cost law) + DeltaNet states + compute buffers at ubatch 1024 + + # CUDA graphs + the MTP draft context => 17 GiB budgeted. Two asserts, + # both about the output the operator sees: the pool must still fit with + # 2 GiB spare, and a SECOND 131,072 slot must not fit — Frank's "as + # many 128k contexts as fit" has the answer 1 on this card, and the + # pool itself is capped by the 128k law, not by the card. # Arithmetic, not a boot log: change CTX_SIZE and this bites before the - # deploy does. At Ornith's rate the two-200704-slot pool (14.1 GiB KV) - # FAILS the first assert — which is exactly why the shipped shape is - # ONE native slot. + # deploy does. ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" kv_gib=$((ctx * 37 / 1048576)) - [ $((kv_gib + 11 + 2)) -le 24 ] - [ $((kv_gib + 11 + 5)) -gt 24 ] - # And the ceiling that sets the per-slot window: two NATIVE windows cannot - # be paid for in full precision on this card. - native=$((2 * 262144)) - [ $((native * 37 / 1048576 + 11)) -gt 24 ] + [ $((kv_gib + 17 + 2)) -le 24 ] + [ $((2 * kv_gib + 17)) -gt 24 ] } @test "the compute-buffer pair follows the env file and stays self-consistent" { @@ -187,16 +183,22 @@ argv_value() { argv_value --ubatch-size | grep -qE '^[0-9]+$' } -@test "no speculative draft ships by default" { - # The qwen35 GGUF has no MTP tensors, and the cross-model DFlash draft - # is a measured decision the harness makes later, not a default. - ! grep -q -- '--model-draft' "$SERVE" - ! grep -q -- '--spec-type' "$SERVE" - ! grep -q '^DRAFT' "$ENV_FILE" +@test "the in-file MTP head ships as draft-mtp at depth 3" { + # The -mtp GGUF carries the head (blk.64.nextn.*, nextn_predict_layers + # =1): draft-mtp loads it against the main model's own weights — no + # --model-draft and no second draft KV pool. Speculative flags move as + # a set; both must appear in the argv the supervisor runs, and the env + # must carry the same contract the argv executes. + serve_argv + [ "$status" -eq 0 ] + [ "$(argv_value --spec-type)" = "draft-mtp" ] + [ "$(argv_value --spec-draft-n-max)" = "3" ] + grep -qE '^SPEC_TYPE=draft-mtp$' "$ENV_FILE" + grep -qE '^SPEC_DRAFT_N_MAX=3$' "$ENV_FILE" } @test "vision ships by default and MMPROJ_FILE empty opts out of the image path" { - # The Q6_K GGUF carries no `v.*` tensors, so the projector argument IS + # The Swift GGUF carries no vision tensors, so the projector argument IS # the vision capability: no flag, no eyes. serve_argv [ "$status" -eq 0 ]