diff --git a/AGENTS.md b/AGENTS.md index f344a80..edf3cee 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 | | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 | -| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) on the RTX 3090, **2 × 262,144** (native window) with **asymmetric KV f16 K + q8_0 V** under flash attention, 20,164/24,576 MiB VRAM, 100.4–101.8 tok/s decode, text-only. Swapped 2026-10-02 from Q6_K; rollback `models/MiMo-Q6K/` | 7771 | +| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -243,7 +243,7 @@ merely wedged kernel usually still answers ARP. 1. Stop and mask (self-healing would restart it): - Dankclaw `frank-large` (root unit, vLLM podman on the iGPU): `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - - Dankclaw `frank-small` (root unit, SGLang on the 3090): + - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700): `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary` @@ -277,12 +277,15 @@ When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As -of 2026-10-02 `frank-large` (the BF16 mmproj that ships inside its GGUF -payload) and `frank-primary` (loads the BF16 mmproj explicitly — this build -never autoloads projectors) accept images (`input: [text, image]`); -`frank-small` is **text-only** — the GGUF payload it moved to on 2026-10-02 -ships no projector, and the in-checkpoint MiMo vision tower exists only in the -safetensors layouts. Note also that +of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`): +`frank-large` through the BF16 mmproj that ships inside its GGUF payload, +`frank-primary` through a BF16 mmproj loaded explicitly (this build never +autoloads projectors), and `frank-small` through +`mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the Q5_K_M +GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is +**180224**, the per-slot window, not the model's native 262144: the harness +must compact before llama.cpp answers a longer prompt with a 400. Note also +that the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` — models.yml sets `reasoningContentField` per endpoint to match. @@ -370,11 +373,14 @@ unit/env/serve trio and redeploy. **root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`), serving `bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B, -sha256 `04cf46f2…`) on port 7771. The 2026-10-02 swap was a **quant rung down -from Q6_K**, not a runtime change — the llama.cpp path arrived 2026-10-01 -night for one reason: **asymmetric KV**. `--cache-type-k f16` + -`--cache-type-v q8_0` is the fleet's only per-tensor KV lever; SGLang's -`--kv-cache-dtype` moves K and V together. +sha256 `04cf4f…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`** +(918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q5KM/`, on +port 7771. The 2026-10-02 swap was a **quant rung down from Q6_K**, not a +runtime change — the llama.cpp path arrived 2026-10-01 night for one reason: +**asymmetric KV**. `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's +only per-tensor KV lever; SGLang's `--kv-cache-dtype` moves K and V together. +Vision came the same day via `--mmproj`: the GGUF carries no `v.*` tensors, so +the projector file is the entire vision path (and ~1.4 GiB of the card). **Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144 window. KV exists only in the 8 attention layers @@ -382,19 +388,30 @@ DeltaNet, native 262,144 window. KV exists only in the 8 attention layers the DeltaNet state is a fixed per-slot buffer, so slot count — not context — drives that term. -**The shape:** `--ctx-size 524288 --parallel 2` = TWO full native windows -(llama.cpp divides the pool: `n_ctx_slot = CTX_SIZE / PARALLEL`), -`--flash-attn on` (the q8_0-V path is only cheap under flash attention — the -seam pins it), f16 K + q8_0 V, all layers on the GPU. Measured VRAM -**20,164 / 24,576 MiB**; full f16 V would add ~3.96 GiB and bust the card, -which is the whole justification for the asymmetric contract. +**The shape:** `--ctx-size 540672 --parallel 3` = **THREE conversations of +180,224 tokens each** (llama.cpp divides the pool: +`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the q8_0-V path is +only cheap under flash attention — the seam pins it), f16 K + q8_0 V, all +layers on the GPU, `--mmproj` for vision. Measured VRAM **21,684 used / 2,441 +free** of 24,576 MiB — the 2–3 GiB headroom Frank asked for, landed by +solving the pool against two measured boots: non-KV cost is 7,390 MiB without +a projector (2 × 262,144 boot) and 8,859 MiB with one (3 × 196,608 boot), so +vision costs ~1.4 GiB, not the 0.9 GiB the file size suggests. Three native +windows (786,432 tokens) need 18.4 GiB of KV alone and were never on the +table. Full f16 V would add ~4 GiB more and bust the card — that is the +justification for the asymmetric contract. -Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; decode -**100.4–101.8 tok/s** at temp 0; two concurrent requests answered in -parallel; native `tool_calls`; thinking in `reasoning_content`. **Text-only** -— this GGUF ships no projector, so `models.yml` declares `input: [text]` -(the in-checkpoint vision tower exists only in the safetensors layouts the -SGLang run used). +Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; boot logs +`n_slots = 3, n_ctx_slot = 180224` and `loaded multimodal model`; decode +**106.8 tok/s** at temp 0 (prefill 3,814 t/s) — ~5 % faster than the +2 × 262,144 shape, because a shorter slot means a shorter attention scan; +three concurrent requests answered in parallel and a fourth queued behind a +released slot; native `tool_calls`; thinking in `reasoning_content`; a +solid-red PNG answers "Red." through an `image_url` content part (the legacy +top-level `images` field is ignored by `/v1/chat/completions` — the model then +invents a color). A prompt over the slot is a **400 +`exceed_context_size_error`**, which is why `models.yml` sets +`contextWindow: 180224`. Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`. Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at @@ -701,6 +718,35 @@ linked runbooks. host-RAM hash table) and its window decides which traffic it can see: too short and it proposes on ordinary prose fragments, which costs verification without acceptance. +- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the + per-slot number is the contract — not `ctx-size`.** `n_ctx_slot = + CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT + (`400 exceed_context_size_error: request (184001 tokens) exceeds the + available context size (180224 tokens)`) even when the pool has room. So + frank-small's 2 → 3 reslice moved each conversation from the model's native + 262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or + the harness would discover the ceiling as a 400 on a long conversation. Side + effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs + 100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident. + Measured 2026-10-02. See [[frank-small-llama-mimo-q5km]]. +- **Price a VRAM budget from two boots, never from a formula — and an mmproj + costs ~1.5x its file size.** The frank-small 3090 fit was solved from two + measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒ + **non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV + 14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file + that bills ~1.4 GiB, because the encoder's activation and compute buffers + ride with it. Subtracting the two rows is exact; extrapolating weights+KV+ + "a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then + solve `KV ≤ card − non-KV − headroom` and round to a multiple of + `PARALLEL × 4096`. Recorded in `scripts/frank-small.env` so the next resize + starts from measurement. +- **`images` and `image_url` are NOT interchangeable: only the content part + reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the + model still invents an answer if the client used the legacy top-level + `images` field on `/v1/chat/completions` — frank-small answered "Brown." for + a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The + failure is silent because a hallucinated color is a plausible answer. Probe + vision with a solid color and expect an exact word. - **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that then served frank-large: every tool-using agent on that endpoint was broken diff --git a/scripts/frank-small.env b/scripts/frank-small.env index c30377d..f5c2537 100644 --- a/scripts/frank-small.env +++ b/scripts/frank-small.env @@ -13,8 +13,9 @@ # KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB # The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608 # per slot left only 1.1 GiB free — outside the band. The pool therefore is -# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB used, -# ~2.7 GiB free, the middle of what was asked for. +# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted. +# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and +# the number the bats budget gate reproduces (it prices non-KV at 9 GiB). # # KV arithmetic: attention KV exists only in the 8 full-attention layers # (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md index 71268c8..46714e9 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md @@ -1,7 +1,7 @@ --- type: reference title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF" -description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the VRAM fit, measured rates, and what to check when it misbehaves." +description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves." tags: - llm - frank-small @@ -17,11 +17,17 @@ timestamp: 2026-10-02 # frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090) > **ACTIVE since 2026-10-02.** One quant step down from -> [[frank-small-llama-mimo-q6k]] on the same runtime, the same shape and the -> same KV contract. Frank's order: `bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M` -> on llama.cpp, two contexts, whatever context size fits the 3090. The answer -> to "whatever fits" is unchanged — **2 x 262,144**, because 262,144 is the -> model's native window and two of them already fit. +> [[frank-small-llama-mimo-q6k]] on the same runtime and the same KV +> contract, then **resized twice the same day** on Frank's orders: first to +> three contexts with vision, then down to the pool that keeps 2–3 GiB of the +> card free. Frank's arithmetic request — "as much as can fit while keeping +> overhead open space to 2-3gb on the 3090" — is answered by the two +> measurements in *Shape and the KV contract* below; the answer is +> **3 × 180,224**, not 3 × 262,144. + +> **The tradeoff is the per-slot window.** Three sessions cost 81,920 tokens +> of per-conversation memory versus the two-slot shape: 180,224 tokens each +> instead of the model's native 262,144. See *What three slots cost*. ## Identity @@ -29,15 +35,16 @@ timestamp: 2026-10-02 |---|---| |Endpoint|`192.168.0.143:7771`, served alias `frank-small`| |Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap| -|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f`, at `/home/dank/models/MiMo-Q5KM/` (fetched by `scripts/download-frank-small.sh`)| +|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-small.sh`| |Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`| |Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`| ## Shape and the KV contract -`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots, so -each session gets `n_ctx_slot = 262144`, the model's full native window (boot -log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`). +`--ctx-size 540672 --parallel 3` — llama.cpp divides the pool across slots, so +each conversation gets `n_ctx_slot = 180224` (boot log: `n_slots = 3, +n_ctx_slot = 180224, kv_unified = 'false'`). `CTX_SIZE` MUST stay divisible by +`PARALLEL`; the bats seam enforces it. The KV contract is Frank's: **K unquantized (f16), V `q8_0`** — `--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it: @@ -47,29 +54,64 @@ The KV contract is Frank's: **K unquantized (f16), V `q8_0`** — lie. The serve script pins the trio; the bats seam pins the script. - The VRAM budget: attention KV lives in the 8 full-attention layers only (4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token - (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool ≈ - **12.5 GiB**. Weights 6.40 + KV 12.5 + Gated DeltaNet state (~25 MiB/slot) + - compute buffers/CUDA graphs ⇒ **20,164 MiB measured** against the card's - 24,576 MiB. + (34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool. -**Why the quant step did not buy a bigger window.** The model cannot be run -past its trained 262,144 without scaling, and two of those windows are already -the pool's ceiling. The 0.86 GiB the smaller quant frees is margin, not -geometry. It is also not enough to un-quantize V: f16 V costs 16 KiB/token -instead of 8.5, i.e. +3.96 GiB on this pool, which lands past the card. That -arithmetic is a test (`the KV budget fits the 3090…`), so a future CTX_SIZE or -quant change that breaks the fit fails the suite instead of the boot. +**The pool size came from two boots, not from a formula.** `free`-style +arithmetic on this stack is unreliable because the compute buffers, CUDA +graphs and now the vision tower all bill separately, so the fit was measured +at two shapes and solved between them: -## Measured on this file (2026-10-02) +|Shape|Projector|`nvidia-smi` used|KV at 24.5 KiB/token|Non-KV (weights + buffers + projector)| +|---|---|---|---|---| +|2 × 262,144 (pool 524,288)|none|20,190 MiB|12,800 MiB|**7,390 MiB**| +|3 × 196,608 (pool 589,824)|f16|22,972 MiB|14,113 MiB|**8,859 MiB**| + +The two rows price the projector and its image buffers at **~1.4 GiB** — not +the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own +activation and compute buffers ride along. Solving the second row's non-KV cost +against Frank's 2–3 GiB band: the card is 24,576 MiB, so +`KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB` ⇒ pool ≤ 556,630 tokens, rounded +down to a multiple of 3 × 4,096 ⇒ **540,672**, i.e. 13,265 MiB of KV and +**2,441 MiB free** (measured, 21,684 MiB used). + +**Why 3 × 262,144 was never possible.** Three native windows are 786,432 +tokens = 18.4 GiB of KV *alone*; plus the 8.86 GiB of non-KV cost that is +27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed +6 GiB. + +## Measured on this shape (2026-10-02, 3 × 180,224 + vision) |Check|Result| |---|---| -|Boot to healthy|~20 s, `n_slots = 2`, `n_ctx_slot = 262144`| -|VRAM|20,164 MiB / 24,576 MiB (`nvidia-smi`)| -|Decode, temp 0, no thinking|**100.4–101.8 tok/s** (256–512 token completions at ~900-token depth)| -|Two concurrent requests|both answered, 256 tokens each in 3.2 s wall (slots are real)| -|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"c"}`)| -|Thinking|`reasoning_content` populated on a reasoning-requiring prompt| +|Boot to healthy|~20 s, `n_slots = 3`, `n_ctx_slot = 180224`, `loaded multimodal model` logged| +|VRAM|**21,684 MiB used / 2,441 MiB free** of 24,576 (`nvidia-smi`, steady state; the KV pool is allocated whole at boot, so this is the ceiling)| +|Decode, temp 0, no thinking|**106.8 tok/s** (256-token completion at ~1,300-token depth)| +|Prefill|**3,814 tok/s** (same request)| +|Three concurrent requests|all three answered in 1.7 s wall — real parallel slots| +|Fourth concurrent request|answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires| +|Vision|`image_url` part with a 64×64 solid-red PNG → **"Red."**| +|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"f"}`)| +|Thinking|`reasoning_content` populated (145 chars on a "think first" prompt); `reasoning_effort: "low"` accepted| +|Over-slot prompt|`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)` — refused, never silently truncated| + +Decode is ~5 % **faster** than the 2 × 262,144 shape measured earlier the same +day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention +scan at equal prompt depth, and one less slot's DeltaNet state resident. + +## What three slots cost + +The per-slot window is the whole price. Two consequences the fleet depends on: + +- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml` + sets `contextWindow: 180224` for this endpoint — the harness compacts before + it reaches the ceiling instead of discovering it. +- **frank-primary's compaction prompt no longer fits with room to answer.** + Its trigger is 183,501 tokens — already above this slot — and its summary + budget is 32,768 output tokens. On the two-slot shape (262,144) that left + 32.8k of output room, which is why that number was chosen; on this shape a + single compaction request must be truncated or split. Nothing in the fleet + routes compaction here automatically, but any runbook that claims frank-small + "fits a compaction request in ONE request" is describing the 262,144 shape. ## Client contract @@ -81,10 +123,17 @@ quant change that breaks the fit fails the suite instead of the boot. `extraBody`. A raw client that omits it gets a direct answer — measured, and not a fault. Trivial questions produce no thinking even with the flag on. - Tools: `--jinja` + server-side parsing, verified 2026-10-02. -- **Text-only.** This GGUF carries no vision tensors. The HF repo does ship a - paired `mmproj-MiMo-V2.6-Distill-Qwen-9B-bf16.gguf`, so vision is one - `--mmproj` away if it is ever wanted here (frank-primary is the fleet's - vision endpoint today). +- **Vision is on**, through the projector file (`--mmproj`). Use the + OpenAI-standard `image_url` content part — that path answers correctly + ("Red." on a solid-red PNG). The legacy top-level `images` field is **not** + read by `/v1/chat/completions`: the same request sent that way answered + "Brown.", i.e. it generated from text alone. `/completion` takes `images` + but refuses a request without `messages` in this build, so chat is the only + vision path. +- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-small.env` drops + `--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only + opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot + refusal, never a silent fallback to blindness. ## No spec decode @@ -98,7 +147,9 @@ traffic first. The bats seam fails if a draft flag ships unannounced. |Target|How| |---|---| -|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default| +|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`| +|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2` (measured 20,190 MiB used, 4,386 free) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here| +|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit| |SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`| ## When it misbehaves @@ -107,7 +158,14 @@ traffic first. The bats seam fails if a draft flag ships unannounced. naming the missing file if the binary or GGUF moved. - `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it). -- Requests queue behind each other: two concurrent long generations are the - slot ceiling by contract. +- Requests queue behind each other: three concurrent long generations are the + slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s + for the three, 3.0 s for the fourth). +- `400 … exceeds the available context size (180224 tokens)`: the conversation + outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens + and the per-slot window is the binding limit. Shorten the prompt, compact, or + roll back to the 262,144 shape above. +- Vision answers a color it never saw: the client sent `images` instead of an + `image_url` content part (see *Client contract*). - Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out of VRAM into unified memory halves the rate silently. diff --git a/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md b/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md index 3bb9edc..5854678 100644 --- a/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md +++ b/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md @@ -52,7 +52,10 @@ timestamp: 2026-10-01 trigger 183,501 + a 32,768-token summary budget = 216,269) now fits one request. A single request may grow to the whole pool — a 170,019-token prompt was verified at the 4-slot shape — the grid is - the two-way concurrency contract. + the two-way concurrency contract. **That compaction arithmetic is + specific to a 262,144-token slot**: the ACTIVE shape runs three slots of + 180,224, which 183,501 tokens already exceeds — see *What three slots + cost* in [[frank-small-llama-mimo-q5km]]. - **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256 dim x 1 B), pool 9.31 GiB. - **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 8821399..a2a51a5 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -76,8 +76,9 @@ frank-small, port 7771] - [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090, - 2 x 262,144 with K f16 + V q8_0, 100.4-101.8 tok/s decode, 20,164 MiB - measured. The KV arithmetic, the flash-attn dependency, rollback to Q6_K + THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the + mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free. + The KV arithmetic, the flash-attn dependency, rollback to Q6_K - [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the - failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro): (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 0ab1c04..961451a 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -285,8 +285,9 @@ The LLM inference fleet on the two hosts. Full index: - [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since 2026-10-02: a CUDA (sm_86) build of llama.cpp master serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX - 3090, port 7771 — 2 x 262,144 native windows, the asymmetric KV contract - (f16 K + q8_0 V) and the VRAM arithmetic behind it, ~100 tok/s decode + 3090, port 7771 — THREE conversations at 180,224 tokens each, the + asymmetric KV contract (f16 K + q8_0 V), vision through the repo's mmproj, + and the two VRAM measurements that set the pool size; ~100 tok/s decode - [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced by [[frank-small-llama-mimo-q5km]], payload still on disk): the first llama.cpp + GGUF shape for this endpoint, Q6_K at the same diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 15e3f07..012f75d 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -925,3 +925,71 @@ any repo service. `~/.AGENTTRASH/20261001-frank-large-reclaim/` — 52 G freed. KEPT the 1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's drafter, not a remnant). + +## 2026-10-02 — frank-large cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP + +Frank's order was the highest quality of +`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` on one full context, "whatever +runtime it requires". The answer is forced by the file: `Q8_0_ROCMFPX_AGENT.gguf` +carries tensor type 103 (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115), which stock +llama.cpp, gufo and every other loader on the fleet cannot read — only the +kingjones30/ROCmFPX fork can, so the runtime is pinned to commit +`36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by +`scripts/build-rocmfpx.sh`. Shape: 1 session x 262,144 (native), f16 KV, the +bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` +(n-max 4, `--spec-draft-device ROCm0`). + +- **MTP is the whole performance story: 17.0-17.1 t/s with, 7.60-7.66 without** + (+123 %), one session, temp 0, production stopped per arm, acceptance 0.462 / + mean length ~1.86, at an 8-15 % prefill tax (325-351 vs 381 t/s). The AGENT + routing variant was chosen over the card's plain Q8_0 because it keeps 187 + output-head tensors at plain Q8_0 instead of 2, which measured MTP acceptance + 0.953 vs 0.911. +- Steady GTT occupancy 63.1 GiB of the ~114 GiB aperture; boot to healthy 16 s. +- Rollback: git tag `frank-large-gufo-q4kxl-20261001`. The retired vLLM-era + fetchers moved to `old/`. Runbook + [[frank-large-rocmfpx-q8agent-mtp]]. + +## 2026-10-02 — frank-small: Q5_K_M, then three contexts + vision, sized by two boots + +Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp +with two contexts ("whatever context size fits the 3090"), then **three +contexts** "as much as can fit while keeping overhead open space to 2-3gb on +the 3090", then **"frank-small should have vision"**. + +- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V + under flash attention, `--mmproj` vision, all layers on the 3090. Measured + **21,684 MiB used / 2,441 MiB free** of 24,576 — inside the 2-3 GiB band. +- **The pool came from two boots, not a formula.** 2 x 262,144 with no + projector: 20,190 MiB used, KV 12,800 => non-KV **7,390 MiB**. 3 x 196,608 + with the projector: 22,972 used, KV 14,113 => non-KV **8,859 MiB**. The + difference prices the vision tower at **~1.4 GiB for a 918 MB file** — the + encoder's own buffers ride with it. The first 3-slot deploy (3 x 196,608) + booted healthy but left only 1,153 MiB free, which is how the band got + violated and why the seam now prices non-KV at the measured 9 GiB and + REJECTS that shape. +- **Three contexts cost window, and that is the tradeoff Frank was told:** + 180,224 tokens per conversation, not the native 262,144. A longer prompt is + a hard `400 exceed_context_size_error`, so `models.yml` sets + `contextWindow: 180224`. Consequence: frank-primary's 183,501-token + compaction request no longer fits here with summary room; the two-slot + rollback is the way back. +- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814 + t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention + scan), three concurrent requests answered in parallel in 1.7 s with a fourth + queued at 3.0 s, native `tool_calls`, `reasoning_content` populated, + `reasoning_effort: low` accepted, solid-red PNG answered **"Red."** +- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy + top-level `images` field — that request answered "Brown." for the same red + image, i.e. it generated from text alone. Only the `image_url` content part + reaches the model. `/completion` accepts `images` but this build still + requires `messages`. +- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing + mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=` (explicitly + empty) in the env file is the documented text-only opt-out and returns ~1.4 + GiB to KV. +- Reclaim staged, **not purged, awaiting approval**: + `~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded + payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`, + `Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the + gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].