From 50459e4e31484defa94d3482b82bce864dde5730 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 3 Oct 2026 11:20:33 -0700 Subject: [PATCH] docs: frank-small Q6_K cutover runbook + measured post-boot numbers - new active runbook frank-small-llama-mimo-q6k-2x200704.md: 21,276 used / 2,849 true free (inside the 2-3 GiB floor), decode 94.44 t/s vs 104.84 on Q5_K_M - the rung costs a measured -10 % (weight-bandwidth-bound card), 512px red-PNG vision verified, and the new probe-size law (>=256 px: a 10x10 image undersupplies image tokens and the model answers 'video'), nvidia-smi free-field law (driver holds ~450 MiB the subtraction hides) - frank-small-llama-mimo-q5km.md retired; it keeps the boot-pair table and becomes the Q5 rollback reference (3,571 free, 104.84 t/s measured) - AGENTS.md fleet row, endpoint section, --parallel lesson, runbook index (active row + two retired rows); LLM + root index entries - Q6_K GGUF-vs-GGUF diff proven: template and all 46 KV keys byte-identical to Q5_K_M - the vision probe scare was probe size, not payload --- AGENTS.md | 78 +++++++------ .../LLM/frank-small-llama-mimo-q5km.md | 44 ++++---- .../frank-small-llama-mimo-q6k-2x200704.md | 106 ++++++++++++++++++ serverAppConfigs-docs/LLM/index.md | 27 +++-- serverAppConfigs-docs/index.md | 24 ++-- 5 files changed, 206 insertions(+), 73 deletions(-) create mode 100644 serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md diff --git a/AGENTS.md b/AGENTS.md index 9aafdfb..05e9f04 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser … | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`** (23.15 GiB, ftype 114 ≈ 7.44 bpw: 353 F32 + 159 `Q8_0_ROCMFPX` + 339 `GGML_TYPE_Q6_0_ROCMFPX` tensors — only this fork can read them — with 64 protected output-head tensors), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-03: prose decode **27.6–28.1 t/s** at 4,813-token depth (draft acceptance 396/456 = 0.868, mean acceptance length ≈ 4.4), schema-echo 31.4 t/s at 100 %; `ngram-mod` chained onto MTP measured **−15 % on prose** and was rejected (the chain disables the hybrid's recurrent-state rollback). Cutover 2026-10-03 from the aligned Q8 file (17.0 t/s, acceptance 0.462 — rollback payload still on disk) | 7770 | -| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, 20,530 used / **4,046 free** of 24,576 MiB, 104.8 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V: the freed slot paid for exact values); rollback `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` or `models/MiMo-Q6K/` | 7771 | +| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, 21,276 used / **2,849 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -289,7 +289,7 @@ of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`): `frank-large` through the BF16 mmproj that ships inside its GGUF payload, `frank-primary` through a BF16 mmproj loaded explicitly (this build never autoloads projectors), and `frank-small` through -`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the Q5_K_M +`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is **200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the model's native 262144: the harness must compact before llama.cpp answers a @@ -389,10 +389,13 @@ Q4_K_XL payload and DFlash2 drafter were removed from disk 2026-10-03 `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a **root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`), serving -`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B, -sha256 `04cf4f…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`** -(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q5KM/`, on -port 7771. The runtime arrived 2026-10-01 night for one reason: +`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q6_K** (7,793,710,624 B, +sha256 `ef96d05a…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`** +(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q6K/` (the +projector is a hardlink of the Q5KM copy — the dir is self-contained), on +port 7771. The rung went Q5_K_M → Q6_K on 2026-10-03 the same morning as the +reslice, funded by the freed slots. The runtime arrived 2026-10-01 night for +one reason: **asymmetric KV** — `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's only per-tensor KV lever, and SGLang's `--kv-cache-dtype` moves K and V together. On 2026-10-03 Frank spent the headroom that lever bought: the reslice @@ -411,30 +414,38 @@ drives that term. conversations of 200,704 tokens each** (llama.cpp divides the pool: `n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it), **f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, batch -2048/ubatch 1024 (ubatch 2048 failed the scratch boot). Measured VRAM **20,530 -used / 4,046 free** of 24,576 MiB — above the 2–3 GiB floor Frank asked for, -solved from the boot-pair table in the runbook (non-KV cost ~7,986 MiB at this -shape; f16 K+V is 32 KiB/token across the 8 attention layers). The 2026-10-02 -3 × 180,224 + q8_0-V shape stays one env edit away (`CTX_SIZE=540672 -PARALLEL=3 CACHE_TYPE_V=q8_0`). Three native windows need 18.4 GiB of KV alone -and were never on the table. +2048/ubatch 1024 (ubatch 2048 failed the scratch boot). Measured VRAM **21,276 +used / 2,849 free** of 24,576 MiB — inside the 2–3 GiB floor Frank asked for +(non-KV 8,732 MiB; f16 K+V is 32 KiB/token across the 8 attention layers; +quote `nvidia-smi`'s `free` field, never `total − used` — the driver holds +~450 MiB of reserve the subtraction hides). The 2026-10-02 3 × 180,224 + +q8_0-V shape stays one env edit away (`CTX_SIZE=540672 PARALLEL=3 +CACHE_TYPE_V=q8_0`, and it must pair with Q5_K_M — Q6_K breaks the floor +there). Three native windows need 18.4 GiB of KV alone and were never on the +table. Verified at deploy (2026-10-03): boot logs `n_slots = 2, n_ctx_slot = 200704` -and `loaded multimodal model`; decode **104.84 tok/s** at temp 0 — ~2 % under -the 3-slot q8_0-V shape (106.8), the honest price of exact V; health 200 in -~5 s; native `tool_calls`; thinking in `reasoning_content`; a solid-red PNG -answers "Red." through an `image_url` content part (the legacy top-level -`images` field is ignored by `/v1/chat/completions` — the model then invents a -color). A prompt over the slot is a **400 `exceed_context_size_error`**, which -is why `models.yml` sets `contextWindow: 200704`. +and `loaded multimodal model`; decode **94.44 tok/s** at temp 0 on the probe +that measured 104.84 on Q5_K_M — **the Q6_K rung costs a measured −10 %**: +decode on this card is weight-bandwidth-bound and Q6_K streams ~10 % more +bytes/token; health 200, e2e 33/33; native `tool_calls`; thinking in +`reasoning_content`; a **512×512** solid-red PNG answers `Red` through an +`image_url` content part (two traps: the legacy top-level `images` field is +ignored, and a TINY probe image undersupplies image tokens — a 10×10 PNG made +the model claim "it's a video"; probe with ≥256 px). A prompt over the slot +is a **400 `exceed_context_size_error`**, which is why `models.yml` sets +`contextWindow: 200704`. -Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`. -Rollback: the previous shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` -(3 × 180,224, measured 2,441 MiB free); before that -[[frank-small-llama-mimo-q6k]] (Q6_K file on disk at -`/home/dank/models/MiMo-Q6K/`) and [[frank-small-sglang-mimo-v26]] (venv on -disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was staged to -`~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02). +Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` +(the boot-pair table that prices pool changes lives in the retired +[[frank-small-llama-mimo-q5km]]). +Rollback: `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` +buys back 875 MiB and the −10 % decode rate (3,571 free, 104.84 tok/s, both +measured); the older shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` +pairs with Q5 only; SGLang rollback in +[[frank-small-sglang-mimo-v26]] (venv on disk at +`/home/dank/GitPackages/sglang-cu128/`, INT8 payload staged to +`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`). ### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE) @@ -838,9 +849,11 @@ linked runbooks. 100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident. The 2026-10-03 reslice went the other way (3 → 2 slots, 180,224 → 200,704, pool 401,408) to fund unquantized f16 V, and `contextWindow` followed again; - decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. Both - moves measured; both directions cost or buy window, never pool. See - [[frank-small-llama-mimo-q5km]]. + decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. The same + day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung + itself is a further measured −10 %, weight-bandwidth, not slot geometry). + All moves measured; both slot directions cost or buy window, never pool. + See [[frank-small-llama-mimo-q6k-2x200704]]. - **Price a VRAM budget from two boots, never from a formula — and an mmproj costs ~1.5x its file size.** The frank-small 3090 fit was solved from two measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒ @@ -960,7 +973,7 @@ Active: | Service | Runbook | |---------|---------| | frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md` | -| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q5_K_M, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md` | +| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` | | frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -972,7 +985,8 @@ Retired (rollback targets and history): | ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` | | gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` | | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` | -| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (rollback target) | `frank-small-llama-mimo-q6k.md` | +| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-small-llama-mimo-q6k-2x200704.md`) | `frank-small-llama-mimo-q6k.md` | +| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-small`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-small-llama-mimo-q5km.md` | | SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` | |tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`| | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` | diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md index c9c3ecf..2112efb 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md @@ -1,7 +1,7 @@ --- type: reference -title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF" -description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two conversations of 200,704 tokens, fully unquantized f16 KV, mmproj vision, the boot-pair arithmetic that sizes the pool, and what to check when it misbehaves." +title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)" +description: "The retired 2026-10-02/03 frank-small runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree." tags: - llm - frank-small @@ -14,23 +14,23 @@ tags: timestamp: 2026-10-03 --- -# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090) +# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired) -> **ACTIVE since 2026-10-02, resliced 2026-10-03.** One quant step down from -> [[frank-small-llama-mimo-q6k]] on the same runtime, resized twice on -> 2026-10-02 (three contexts with vision, then the pool that kept 2–3 GiB of -> the card free), and on 2026-10-03 Frank ordered the third shape: **two -> conversations of 200,704 tokens each with the KV VALUES unquantized** — +> **RETIRED 2026-10-03** — superseded by +> [[frank-small-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free +> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same +> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower +> decode is the measured price of the higher rung on this bandwidth-bound +> card. This file remains the rollback reference and the owner of the +> measured boot-pair table. + +> Its own history: one quant step down from [[frank-small-llama-mimo-q6k]] +> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then +> the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two +> conversations of 200,704 tokens with the KV VALUES unquantized** — > *"2 streams with 200700 tokens context or more… if we have extra space on -> the 3090, unquantize the kv."* The freed third slot pays for exact V, and -> the boot lands **20,530 MiB used / 4,046 MiB free** — above the 2–3 GiB -> floor, with full-precision KV. - -> **The tradeoffs of the reslice:** one fewer simultaneous conversation, a -> per-slot window that grew from 180,224 to 200,704 (still below the native -> 262,144), and ~2 % slower decode (104.8 vs 106.8 tok/s) from the larger -> per-token KV traffic. The 3 × 180,224 shape is one env edit away — see -> *Rollback*. +> the 3090, unquantize the kv."* That shape measured 20,530 used / 4,046 +> apparent free (the true `nvidia-smi` free was 3,571 — see *Shape*). ## Identity @@ -90,7 +90,7 @@ measured band. |Decode, temp 0, no thinking|**104.84 tok/s** (128-token completion at ~500-token depth; the 3 × 180,224 q8_0-V shape measured 106.8 — ~2 % is the honest price of exact V)| |Prefill|1,650 t/s on the same shallow 506-token probe (rates are depth- and prompt-dependent; not comparable across shapes)| |Batch geometry|`--batch-size 2048 --ubatch-size 1024` — ubatch 2048 failed the scratch boot; 1024 boots clean| -|Vision|projector loads at boot (`loaded multimodal model`); the red-PNG probe stands from the 2026-10-02 verification — the vision path is unchanged by this reslice| +|Vision|projector loads at boot (`loaded multimodal model`); red-PNG probe passed on 2026-10-02. Caveat discovered on 2026-10-03: probes must use a >= 256 px image — a 10x10 px PNG undersupplies image tokens and the model hallucinates ("it's a video")| |Tools / thinking|unchanged by the reslice: native `tool_calls`, thinking in `reasoning_content`| ## What the slot geometry costs @@ -146,7 +146,7 @@ traffic first. The bats seam fails if a draft flag ships unannounced. |Target|How| |---|---| |Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-small.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V| -|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`| +|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — **this is the deployed rung since 2026-10-03** ([[frank-small-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)| |Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window| |Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit| |SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`| @@ -165,7 +165,9 @@ traffic first. The bats seam fails if a draft flag ships unannounced. outgrew ONE slot. This is not pool exhaustion — the pool is 401,408 tokens and the per-slot window is the binding limit. Shorten the prompt, compact, or roll back to the 262,144 shape above. -- Vision answers a color it never saw: the client sent `images` instead of an - `image_url` content part (see *Client contract*). +- Vision answers a color it never saw, or claims the image "is a video": + either the client sent `images` instead of an `image_url` content part, or + the probe image is too small to supply real image tokens (use >= 256 px; + see *Client contract*). - Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out of VRAM into unified memory halves the rate silently. diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md new file mode 100644 index 0000000..96c302a --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md @@ -0,0 +1,106 @@ +--- +type: reference +title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV" +description: "The active frank-small runbook — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change." +tags: + - llm + - frank-small + - dankclaw + - llama-cpp + - gguf + - rtx3090 +timestamp: 2026-10-03 +--- + +# frank-small — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV + +> **ACTIVE since 2026-10-03.** The third shape Frank bought in two days on +> this card: the freed third slot paid for **exact KV values** +> ([[frank-small-llama-mimo-q5km]] documents that step and remains the +> rollback), and the ~3.5 GiB still free then paid for the **Q6_K quant rung** +> — the same file this endpoint served on 2026-10-01, come back as an +> upgrade rather than a compromise. The pool held at 401,408 (2 × 200,704); +> only the weights moved up. + +## Identity + +| | | +|---|---| +| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB | +| Endpoint | `http://192.168.0.143:7771` — alias `frank-small`, `/v1` OpenAI surface | +| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (`GGML_CUDA_FA_ALL_QUANTS=ON`) | +| Unit | root systemd `frank-small.service` ([`scripts/frank-small.service`](../../scripts/frank-small.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` | +| Serve script | [`scripts/serve-frank-small.sh`](../../scripts/serve-frank-small.sh), tunables [`scripts/frank-small.env`](../../scripts/frank-small.env) | +| Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) | +| Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) | +| Fetch | [`scripts/download-frank-small.sh`](../../scripts/download-frank-small.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy | +| Seam | `tests/frank_small_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) | + +## Shape and KV contract + +`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots, +so each conversation gets `n_ctx_slot = 200704`. `CTX_SIZE` MUST stay +divisible by `PARALLEL`; the seam enforces it. Both K and V are unquantized +(f16) — the flags stay explicit because `--flash-attn` must not be lost if V +ever goes back to `q8_0` (the serve script pins it, the seam pins the pin). + +**The Q6_K rung is a weights-side spend only.** File delta +917,585,920 B +(+875 MiB) against Q5_K_M, same KV, same slot geometry, same batch pair. +Priced from the measured Q5 boot at this shape (non-KV 7,986 MiB): the boot +landed **21,276 MiB used / 2,849 MiB true free** — non-KV measured 8,732 MiB, +inside the 2–3 GiB floor, so the pool stayed at 401,408 and the +12,288-token step-down ladder was never climbed. Quote `nvidia-smi`'s `free` +field, never `total − used`: the driver holds ~450 MiB of reserve the +subtraction hides (that error once overstated this card's slack by a +half-GiB). + +**Pool prices on this card come from boots, never formulas** — the measured +boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV +contract*) and still prices every rollback shape. + +## Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV) + +|Check|Result| +|---|---| +|Boot to healthy|model + projector resident in ~2.3 s (load log `0.02.27`), `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model`, health 200; deploy e2e gate 33/33| +|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung| +|Decode, temp 0, no thinking|**94.44 tok/s** on the identical 506-token probe that measured 104.84 on Q5_K_M — **−10 %**. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably| +|Prefill|2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)| +|Vision|512×512 solid-red PNG → **`Red`** (277 prompt tokens — the image really reached the tower). See the probe-size trap in *Client contract*| +|Tools / thinking|native `tool_calls`, thinking in `reasoning_content`, thinking OFF unless `enable_thinking: true` (MiMo template default; `models.yml` ships it in `extraBody`)| + +## Client contract + +llama.cpp surface, live-verified on this model family: model id `frank-small`, +native tools via `--jinja` (`tool_calls` in the response), thinking in +**`reasoning_content`**, vision via `image_url` content parts. Two vision +traps, both measured here: + +1. A top-level `images` field on `/v1/chat/completions` is silently ignored + and the model invents a color. +2. **The probe image must be big enough to supply real image tokens.** A + 10×10 px solid-red PNG came back "The image is a video, so I can't see its + content" — the engine itself warns at boot that Qwen-VL wants ≥ 1024 image + tokens, and a postage-stamp image lands one patch below usable signal. + 512×512 answers `Red` in 19 tokens. Always probe vision with a ≥ 256 px + image; a wrong-color answer from a tiny image is the probe's fault, not + the endpoint's. + +`~/.omp/agent/models.yml` sets `contextWindow: 200704` — the PER-SLOT window, +so the harness compacts before llama.cpp answers a longer prompt with a 400. + +## Rollback + +|Target|How| +|---|---| +|Q5_K_M at the same shape|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk| +|3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)| +|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — payload staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`| + +## When it misbehaves + +Same decision tree as [[frank-small-llama-mimo-q5km]] (*When it misbehaves*): +`No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming +200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after +a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue +behind two long generations ⇒ the slot ceiling is two, by contract. diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index f85bd5c..6e7675e 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -19,7 +19,7 @@ kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT, a ftype-114 file only that fork can read, with the repo's MTP head drafting 4 tokens at 27.6-28.1 t/s decode, ONE session at 262,144, f16 KV, vision via the BF16 projector)** and -**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M +**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**. > 2026-08-22: the former `vllmScripts` repository is merged into the @@ -81,13 +81,18 @@ frank-small, port 7771] ngram-mod chain onto MTP was measured and REJECTED (-15 %: it disables the hybrid's recurrent state rollback). Rollback: the aligned Q8 file, still on disk -- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02, resliced - 2026-10-03** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) - serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090, - TWO slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K - + f16 V) and the mmproj for vision, 104.8 tok/s decode, 4,046 MiB of the - card left free. The boot-pair pool arithmetic, the slot-window contract, - rollback to 3 x 180,224 or Q6_K +- [[frank-small-llama-mimo-q6k-2x200704]] — **ACTIVE since 2026-10-03** + (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving + bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090, TWO + slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K + + f16 V) and the mmproj for vision — 94.4 tok/s decode (the rung costs a + measured -10 % vs Q5_K_M: weight-bandwidth-bound), 2,849 MiB of the card + left free. Probe-size trap (>=256 px images or the model says "video") + documented. Rollback: one MODEL_FILE line back to Q5_K_M +- [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback + target; payload on disk; owns the measured boot-pair table)**: the same + runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower — + 104.8 tok/s, 3,571 MiB true free - [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since 2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both @@ -252,8 +257,10 @@ frank-small, port 7771] .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract, troubleshooting (2026-09-16 → 09-19) -- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02 (rollback - target)**: llama.cpp (master, CUDA sm_86 build) serving +- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same + rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape — + [[frank-small-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86 + build) serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index ba6ebd8..52db191 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -288,16 +288,20 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since - 2026-10-02, resliced 2026-10-03: a CUDA (sm_86) build of llama.cpp master - serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's - RTX 3090, port 7771 — TWO conversations at 200,704 tokens each with fully - unquantized f16 KV, vision through the repo's mmproj, and the boot-pair - arithmetic that sizes the pool; 104.8 tok/s decode, 4,046 MiB free -- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced - by [[frank-small-llama-mimo-q5km]], payload still on disk): the first - llama.cpp + GGUF shape for this endpoint, Q6_K at the same - 2 x 262,144 / f16-K + q8_0-V contract, 20.9 GiB measured +- [[frank-small-llama-mimo-q6k-2x200704]] — the ACTIVE frank-small runbook + since 2026-10-03: llama.cpp master (CUDA sm_86) serving + bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on Dankclaw's RTX 3090, + port 7771 — TWO conversations at 200,704 tokens each with fully + unquantized f16 KV and mmproj vision; 94.4 tok/s decode (the Q6_K rung + costs a measured −10 % vs Q5_K_M — weight-bandwidth), 2,849 MiB free, and + the ≥256 px image-probe trap (tiny images answer "video") +- [[frank-small-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback + target, payload on disk; owns the measured boot-pair table): same runtime + and shape one rung lower — 104.8 tok/s, 3,571 MiB true free +- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same + Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see + [[frank-small-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape + for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured - [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target): SGLang in a self-contained python3.12 venv serving letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5