From b80886e31872680d7ac79d19b20aa0bbe77f0713 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 3 Oct 2026 08:01:44 -0700 Subject: [PATCH] docs: frank-large Q6 uncensored cutover + frank-small 2x200704 f16-KV reslice - new active runbook frank-large-rocmfpx-q6agent-uncensored-mtp.md: the Q6_0_ROCMFPX_AGENT file (ftype 114, 339 type-102 + 159 type-103 tensors), 27.6-28.1 t/s at 86.8% draft acceptance, the measured ngram-mod chain rejection (-15% prose: recurrent-state rollback disabled), the bench-window hardening laws - frank-large-rocmfpx-q8agent-mtp.md retired; rollback section corrected (gufo payloads removed from disk 2026-10-03 - that rollback is now a re-download; the aligned Q8 file is the direct one, still on disk) - frank-small-llama-mimo-q5km.md reshaped to 2 x 200704 with fully unquantized f16 KV: boot-pair table gains the measured row (20,530 used / 4,046 free), 104.84 tok/s, slot-window contract at 200704, rollback to the 3x180224 + q8_0-V shape documented - AGENTS.md: fleet table rows, both endpoint sections, the chain law extended with the recurrent-model measurement, the --parallel law with the reslice, runbook index (active + retired) - LLM/index.md + root index entries updated --- AGENTS.md | 189 ++++++++++-------- ...nk-large-rocmfpx-q6agent-uncensored-mtp.md | 173 ++++++++++++++++ .../LLM/frank-large-rocmfpx-q8agent-mtp.md | 49 +++-- .../LLM/frank-small-llama-mimo-q5km.md | 156 +++++++-------- serverAppConfigs-docs/LLM/index.md | 53 +++-- serverAppConfigs-docs/index.md | 34 ++-- 6 files changed, 438 insertions(+), 216 deletions(-) create mode 100644 serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md diff --git a/AGENTS.md b/AGENTS.md index 7f15b61..9aafdfb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,8 +25,8 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser … -| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 | -| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **3 × 180,224** (pool 540,672 — NOT the native window; three contexts cost 81,920 tokens per conversation) with **asymmetric KV f16 K + q8_0 V** under flash attention, 21,684 used / **2,441 free** of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. Resized twice 2026-10-02 (Q6_K → Q5_K_M → 3 slots + vision → this pool); rollback `models/MiMo-Q6K/` or `PARALLEL=2 CTX_SIZE=524288` | 7771 | +| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`** (23.15 GiB, ftype 114 ≈ 7.44 bpw: 353 F32 + 159 `Q8_0_ROCMFPX` + 339 `GGML_TYPE_Q6_0_ROCMFPX` tensors — only this fork can read them — with 64 protected output-head tensors), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-03: prose decode **27.6–28.1 t/s** at 4,813-token depth (draft acceptance 396/456 = 0.868, mean acceptance length ≈ 4.4), schema-echo 31.4 t/s at 100 %; `ngram-mod` chained onto MTP measured **−15 % on prose** and was rejected (the chain disables the hybrid's recurrent-state rollback). Cutover 2026-10-03 from the aligned Q8 file (17.0 t/s, acceptance 0.462 — rollback payload still on disk) | 7770 | +| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, 20,530 used / **4,046 free** of 24,576 MiB, 104.8 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V: the freed slot paid for exact values); rollback `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` or `models/MiMo-Q6K/` | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -291,8 +291,9 @@ of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`): autoloads projectors), and `frank-small` through `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the Q5_K_M GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is -**180224**, the per-slot window, not the model's native 262144: the harness -must compact before llama.cpp answers a longer prompt with a 400. Note also +**200704** (resliced from 180224 on 2026-10-03), the per-slot window, not the +model's native 262144: the harness must compact before llama.cpp answers a +longer prompt with a 400. Note also that the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` — @@ -303,26 +304,26 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01 (the MiMo tower is vision-only). -### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE) +### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE) The **kingjones30/ROCmFPX llama.cpp fork** at commit `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built for gfx1151 by `scripts/build-rocmfpx.sh` into `/home/dank/GitPackages/kingjones-ROCmFPX/build`, as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`, `Restart=always`) running a bare `llama-server` (no container) on port 7770 -from the Strix Halo iGPU. Boot to healthy: **16 s**. +from the Strix Halo iGPU. Boot to healthy: **~15 s**. **The runtime is not interchangeable — it is chosen by the FILE.** -`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF`'s best rung, -`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` (28,214,425,952 B, sha256 `f3ef2bda…`, -ftype 115 ≈ 8.39 bpw), stores **311 tensors of type 103, -`GGML_TYPE_Q8_0_ROCMFPX`** — a quantization that exists in no other loader: -stock llama.cpp and gufo both refuse it at load (`unknown tensor type`). The -file is 353 F32 + 187 plain-Q8_0 + 311 ROCmFPX; the AGENT routing variant -keeps the output-head tensors at plain Q8_0 (187 vs 2 in the non-AGENT file), -which is why the card reports MTP acceptance 0.953 vs 0.911 — highest quality -AND the better drafter base. Vision comes from the bundled -`mmproj-Qwen3.8-27B-BF16.gguf` (931,145,888 B). +`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF`'s protected-heads +rung, `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` (24,856,607,008 B, +sha256 `e966deb3…`, ftype 114 ≈ 7.44 bpw), stores **339 tensors of type 102, +`GGML_TYPE_Q6_0_ROCMFPX`, plus 159 type-103 `Q8_0_ROCMFPX`** — quantizations +that exist in no other loader: stock llama.cpp and gufo both refuse the file at +load (`unknown tensor type`). The AGENT construction keeps 64 output-head +tensors at plain precision (the protected-heads design of the aligned repo's +Q8 file, one quant rung down), and the uncensored tuning is what Frank swapped +to on 2026-10-03. Vision comes from the BF16 `mmproj-Qwen3.8-27B-BF16.gguf` +(931,145,888 B — hardlinked from the aligned payload dir, byte-identical). **The pin is load-bearing.** The 2026-08-25 ROCmFPX builds segfaulted in `ggml_compute_forward_set_rows` on exactly this file; `build-rocmfpx.sh` @@ -334,22 +335,26 @@ the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (1,680,271,648 B, sha256 `051a1764… `nextn_predict_layers = 1`), pinned to the same card: `--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0 --spec-draft-device ROCm0 --spec-draft-ngl 99`. -Measured 2026-10-02 in one session, temp 0, 2,202-token prompt / 512 tokens, -`cache_prompt: false`, production service stopped for each arm: -**MTP on 17.00–17.14 t/s decode** (draft acceptance 0.462, mean acceptance -length ≈ 1.86) against **7.60–7.66 t/s with the draft removed — +123 %**; -prefill pays 8–15 % (381 → 325–351 t/s). Acceptance is far from the card's -0.95 (that number is the card's own text at a shallow prompt); the gain is -real and n-max 4 is where it was measured, never swept. This replaces the -gufo Q4_K_XL arm at ~15 t/s, so the endpoint got higher quality AND more -speed at once. +On the ALIGNED Q8 file (2026-10-02, bench windows, 2,202-token prompt, temp 0): +**MTP 17.00–17.14 t/s vs 7.60–7.66 with the draft stripped — +123 %**, +acceptance 0.462. On the UNCENSORED Q6 file (2026-10-03, production service, +prose at 4,813-token depth): **27.57–28.10 t/s decode, draft acceptance +396/456 = 0.868** (mean acceptance length ≈ 4.4 — drafting at nearly the full +n-max 4), schema-echo traffic 31.44 t/s at 639/639 = 1.000. The drafter is +materially better on this file; treat the card's 0.95 as a file property, not a +throughput promise. **`ngram-mod` chained onto MTP was measured and REJECTED +(2026-10-03):** `--spec-type draft-mtp,ngram-mod` cost −15 % on prose +(23.40 t/s) and −7 % on verbatim block-copy even with the matcher demonstrably +firing — the boot log says why: `recurrent state rollback is not compatible +with 'ngram-mod' - disabling rollback support`, and the hybrid's MTP +verification needs that rollback. Keep `draft-mtp` alone. **Shape and budget:** ONE session at the native 262,144 window, unquantized f16 KV (~64 KiB/token → ~16 GiB, affordable inside the iGPU's ~114 GiB GTT aperture), `--fit off --flash-attn on --direct-io --ctx-checkpoints 16 ---cache-ram 8192 --cache-idle-slots`. Measured GTT 64.5 GiB with model + -draft + projector resident. Prompt cache verified: a repeat pass over the -same 6k prompt replays in 0.5 s. +--cache-ram 8192 --cache-idle-slots`. Measured GTT ~61 GiB under live load +(host baseline 16.7 GiB). Prompt cache verified (Q8 era): a repeat pass over +the same 6k prompt replays in 0.5 s. **Client contract (llama.cpp, live-verified 2026-10-02):** `--jinja` is on (without it `chat_template_kwargs` are silently ignored), tools are native @@ -365,14 +370,18 @@ measuring), and the fork logs `MTP boundary missing for seq_id=… ; resyncing` under concurrent draft resets — self-recovering, not a fault. -Fetch: `scripts/download-frank-large-kingjones.sh` (`-f` + byte count + LFS -sha256 verified, `--verify-only`, no token on the command line). +Fetch: `scripts/download-frank-large-kingjones-uncensored.sh` (hardlinks the +byte-identical MTP head and projector from the aligned payload dir when +present, otherwise fetches; `-f` + byte count + LFS sha256 verified, +`--verify-only`, no token on the command line). -Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md`. -Rollback: git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 -(`scripts/build-gufo.sh` stays on disk for it), the Q4_K_XL Qwen3.8-27B -payload and the DFlash2 drafter all remain on the host; restore the tag's -unit/env/serve trio and redeploy. +Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md`. +Rollback: the ALIGNED Q8 file (the previous shape) remains on disk at +`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the +2026-10-03 swap commit (unit/env/serve trio) and redeploy. The older gufo tag +`frank-large-gufo-q4kxl-20261001` is now a **re-download** rollback: its +Q4_K_XL payload and DFlash2 drafter were removed from disk 2026-10-03 +(`scripts/build-gufo.sh` stays for the rebuild). ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) @@ -383,12 +392,14 @@ unit/env/serve trio and redeploy. `bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B, sha256 `04cf4f…`) **plus `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`** (918,166,048 B, sha256 `ff348f…`), both `/home/dank/models/MiMo-Q5KM/`, on -port 7771. The 2026-10-02 swap was a **quant rung down from Q6_K**, not a -runtime change — the llama.cpp path arrived 2026-10-01 night for one reason: -**asymmetric KV**. `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's -only per-tensor KV lever; SGLang's `--kv-cache-dtype` moves K and V together. -Vision came the same day via `--mmproj`: the GGUF carries no `v.*` tensors, so -the projector file is the entire vision path (and ~1.4 GiB of the card). +port 7771. The runtime arrived 2026-10-01 night for one reason: +**asymmetric KV** — `--cache-type-k f16` + `--cache-type-v q8_0` is the fleet's +only per-tensor KV lever, and SGLang's `--kv-cache-dtype` moves K and V +together. On 2026-10-03 Frank spent the headroom that lever bought: the reslice +from three slots to two made room for **fully unquantized V** +(`CACHE_TYPE_V=f16`); both flags stay explicit in the serve script. Vision came +2026-10-02 via `--mmproj`: the GGUF carries no `v.*` tensors, so the projector +file is the entire vision path (and ~1.4 GiB of the card). **Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144 window. KV exists only in the 8 attention layers @@ -396,36 +407,34 @@ DeltaNet, native 262,144 window. KV exists only in the 8 attention layers the DeltaNet state is a fixed per-slot buffer, so slot count — not context — drives that term. -**The shape:** `--ctx-size 540672 --parallel 3` = **THREE conversations of -180,224 tokens each** (llama.cpp divides the pool: -`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the q8_0-V path is -only cheap under flash attention — the seam pins it), f16 K + q8_0 V, all -layers on the GPU, `--mmproj` for vision. Measured VRAM **21,684 used / 2,441 -free** of 24,576 MiB — the 2–3 GiB headroom Frank asked for, landed by -solving the pool against two measured boots: non-KV cost is 7,390 MiB without -a projector (2 × 262,144 boot) and 8,859 MiB with one (3 × 196,608 boot), so -vision costs ~1.4 GiB, not the 0.9 GiB the file size suggests. Three native -windows (786,432 tokens) need 18.4 GiB of KV alone and were never on the -table. Full f16 V would add ~4 GiB more and bust the card — that is the -justification for the asymmetric contract. +**The shape (2026-10-03):** `--ctx-size 401408 --parallel 2` = **TWO +conversations of 200,704 tokens each** (llama.cpp divides the pool: +`n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it), +**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, batch +2048/ubatch 1024 (ubatch 2048 failed the scratch boot). Measured VRAM **20,530 +used / 4,046 free** of 24,576 MiB — above the 2–3 GiB floor Frank asked for, +solved from the boot-pair table in the runbook (non-KV cost ~7,986 MiB at this +shape; f16 K+V is 32 KiB/token across the 8 attention layers). The 2026-10-02 +3 × 180,224 + q8_0-V shape stays one env edit away (`CTX_SIZE=540672 +PARALLEL=3 CACHE_TYPE_V=q8_0`). Three native windows need 18.4 GiB of KV alone +and were never on the table. -Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; boot logs -`n_slots = 3, n_ctx_slot = 180224` and `loaded multimodal model`; decode -**106.8 tok/s** at temp 0 (prefill 3,814 t/s) — ~5 % faster than the -2 × 262,144 shape, because a shorter slot means a shorter attention scan; -three concurrent requests answered in parallel and a fourth queued behind a -released slot; native `tool_calls`; thinking in `reasoning_content`; a -solid-red PNG answers "Red." through an `image_url` content part (the legacy -top-level `images` field is ignored by `/v1/chat/completions` — the model then -invents a color). A prompt over the slot is a **400 -`exceed_context_size_error`**, which is why `models.yml` sets -`contextWindow: 180224`. +Verified at deploy (2026-10-03): boot logs `n_slots = 2, n_ctx_slot = 200704` +and `loaded multimodal model`; decode **104.84 tok/s** at temp 0 — ~2 % under +the 3-slot q8_0-V shape (106.8), the honest price of exact V; health 200 in +~5 s; native `tool_calls`; thinking in `reasoning_content`; a solid-red PNG +answers "Red." through an `image_url` content part (the legacy top-level +`images` field is ignored by `/v1/chat/completions` — the model then invents a +color). A prompt over the slot is a **400 `exceed_context_size_error`**, which +is why `models.yml` sets `contextWindow: 200704`. Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`. -Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at -`/home/dank/models/MiMo-Q6K/`); before that [[frank-small-sglang-mimo-v26]] -(venv on disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was -staged to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02). +Rollback: the previous shape `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` +(3 × 180,224, measured 2,441 MiB free); before that +[[frank-small-llama-mimo-q6k]] (Q6_K file on disk at +`/home/dank/models/MiMo-Q6K/`) and [[frank-small-sglang-mimo-v26]] (venv on +disk at `/home/dank/GitPackages/sglang-cu128/`; its INT8 payload was staged to +`~dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02). ### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 (Dankserver, ACTIVE) @@ -747,19 +756,26 @@ linked runbooks. the MAIN model's weights (`shares_model`), so there is no second copy of a 12 GiB model in VRAM and no `-md` argument. The plain build has no MTP tensors at all, and pointing `--model-draft` at the file would load it twice. -- **What can be chained, and what cannot.** `--spec-type a,b` builds every - enabled implementation and chains drafting through them, in a fixed internal - order (ngram types first, then the drafters — CLI order is irrelevant). The - gate is `ctx_dft`: **draftless** types (the ngram family) are added - unconditionally, while `draft-mtp`, `draft-dflash`, `draft-dspark` and - `draft-eagle3` all require a draft CONTEXT. So a chain may hold one - context-backed drafter plus any number of ngram types — `draft-dflash,ngram-mod` - is legal and measured (2.6x the drafter alone on prompt-echo traffic, byte- - identical to it on prose at the fleet's 48/64 window) — but - `draft-mtp,draft-dflash` dies at load: *"context type MTP requested but model - doesn't contain MTP layers"*, because MTP builds its context against whatever - model the draft context holds. Two context-backed types cannot share one - process, and MTP's head only loads when MTP **is** the draft model. +- **What can be chained, what cannot, and what a chain costs on a hybrid.** + `--spec-type a,b` builds every enabled implementation and chains drafting + through them, in a fixed internal order (ngram types first, then the + drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless** + types (the ngram family) are added unconditionally, while `draft-mtp`, + `draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft + CONTEXT. So a chain may hold one context-backed drafter plus any number of + ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the + drafter alone on prompt-echo traffic, byte-identical to it on prose at the + fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context + type MTP requested but model doesn't contain MTP layers"*, because MTP + builds its context against whatever model the draft context holds. Two + context-backed types cannot share one process, and MTP's head only loads + when MTP **is** the draft model. **A LEGAL chain can still lose on a + recurrent model:** `draft-mtp,ngram-mod` on frank-large's qwen35 hybrid + measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer + logs `recurrent state rollback is not compatible with 'ngram-mod' - + disabling rollback support`, and MTP verification depends on that rollback, + so acceptance drops on ALL traffic even where the matcher fires. Measure + chains per architecture, never by analogy. - **A drafter's KV scales with the TARGET's context, and its window policy decides how much that costs.** The draft context is built with `cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is @@ -820,7 +836,11 @@ linked runbooks. the harness would discover the ceiling as a 400 on a long conversation. Side effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs 100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident. - Measured 2026-10-02. See [[frank-small-llama-mimo-q5km]]. + The 2026-10-03 reslice went the other way (3 → 2 slots, 180,224 → 200,704, + pool 401,408) to fund unquantized f16 V, and `contextWindow` followed again; + decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. Both + moves measured; both directions cost or buy window, never pool. See + [[frank-small-llama-mimo-q5km]]. - **Price a VRAM budget from two boots, never from a formula — and an mmproj costs ~1.5x its file size.** The frank-small 3090 fit was solved from two measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒ @@ -939,7 +959,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md` | +| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md` | | frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q5_K_M, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md` | | frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | @@ -949,7 +969,8 @@ Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| -| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (rollback target) | `frank-large-gufo-qwen38-27b.md` | +| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` | +| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` | | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` | | llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (rollback target) | `frank-small-llama-mimo-q6k.md` | | SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` | diff --git a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md new file mode 100644 index 0000000..5f2cfcb --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md @@ -0,0 +1,173 @@ +--- +type: reference +title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP" +description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Uncensored Q6_0_ROCMFPX_AGENT GGUF (ftype 114, protected output heads), with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, the BF16 projector for vision, and the measured rejection of the ngram-mod chain." +tags: + - llm + - frank-large + - dankclaw + - llama-cpp + - rocmfpx + - gfx1151 + - gguf + - spec-decode + - mtp +timestamp: 2026-10-03 +--- + +# frank-large — ROCmFPX Uncensored Q6_0_ROCMFPX_AGENT + MTP (gfx1151) + +> **ACTIVE since 2026-10-03.** Cut over from +> [[frank-large-rocmfpx-q8agent-mtp]] (ALIGNED Q8_0_ROCMFPX_AGENT, 17.0 t/s) +> on Frank's request: the *uncensored* variant of the same protected-heads +> family, one quant rung down. The swap turned out to be a strict upgrade on +> the axis that matters here: **decode 27.6–28.1 t/s at a 4,813-token prompt +> against 17.0 at a 2,202-token prompt** — faster with a deeper context, not a +> same-depth comparison, but the sign is not in question. + +## Identity + +| | | +|---|---| +| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture | +| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface | +| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) — unchanged from the Q8 era; the fork already knew type 102 | +| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` | +| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) | +| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` — 24,856,607,008 B, sha256 `e966deb3…` | +| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764…` (byte-identical to the aligned repo's head; hardlinked, not re-downloaded) | +| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B (hardlinked from the aligned payload dir; the same file the Q8 era served) | +| Fetch | [`scripts/download-frank-large-kingjones-uncensored.sh`](../../scripts/download-frank-large-kingjones-uncensored.sh) — hardlinks the draft/projector from the aligned dir when present, otherwise fetches; `-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line | +| Seam | `tests/frank_large_unit.bats` (payload gates, deployer/serve parity, spec block, KV dtype) | + +## Why this file + +`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` ships the same +"protected heads" construction as the aligned repo, in the uncensored tuning. +Parsed header of the deployed file: + +| Field | Value | +|---|---| +| ftype | **114** — the fork's 64-bit-block family, ≈7.44 bpw | +| Tensor mix | 353 F32 + **339 type-102 (`GGML_TYPE_Q6_0_ROCMFPX`)** + **159 type-103 (`Q8_0_ROCMFPX`)** | +| Protected heads | 64 output-head tensors kept at plain precision — the AGENT construction from the aligned repo, at the 6-bit rung | +| Arch | qwen35, 64 layers, `full_attention_interval 4`, `context_length 262144` | +| Embedded sampling recipe | temp 1.0 / top_p 0.95 / top_k 20 | + +The same lock-in law as the Q8 era applies: types 102 and 103 exist in no +loader but this fork; stock llama.cpp and gufo refuse the file at load +(`unknown tensor type`). The loader was not re-litigated — the file picks the +runtime, and the pin in `scripts/build-rocmfpx.sh` (commit by SHA) stays. + +Rejected sibling: `Q6_0_ROCMFPX_LEAN` (6.45 bpw, no protected heads) — the +protection is what keeps the output-head logits at full precision and the +drafter accurate, so it stays deployed. + +## Shape and budget + +Unchanged from the Q8 era — the swap was payload-only: + +| Dial | Value | Why | +|---|---|---| +| `--ctx-size` | 262,144 | the native window, no rope scaling | +| `--parallel` | 1 | one conversation, all memory to it | +| KV dtype | **f16** (both K and V) | ~64 KiB/token; free inside a 114 GiB aperture | +| `--fit off` | on | the auto-fitter would silently resize the window | +| `--flash-attn on` | on | required for the long-window attention path | +| `--direct-io`, `--ctx-checkpoints 16`, `--cache-ram 8192`, `--cache-idle-slots`, `--jinja`, `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | same as [[frank-large-rocmfpx-q8agent-mtp]] | + +Spec decode: `--spec-type draft-mtp`, the repo's `mtp-Qwen3.8-27B-Q4_0.gguf`, +`--spec-draft-n-max 4`, pinned `--spec-draft-device ROCm0`. + +Measured steady state: GTT **~61 GiB** under live load (host baseline 16.7), +boot to `/health` **~15 s**. + +## Measured on this payload (2026-10-03, production service, `/completion` timings, temp 0, `cache_prompt: false`) + +| Traffic | Decode t/s | Prefill t/s | Draft acceptance | +|---|---|---|---| +| Prose, 4,813-token prompt, 512 tokens | **27.57 / 28.10** (two runs) | ~205–217 | **396/456 = 0.868** → mean acceptance length ≈ 4.4, i.e. drafting at nearly the full n-max 4 | +| Schema echo (structured repetition) | **31.44** | — | 639/639 = 1.000 | +| Block copy (text repeated verbatim) | ~28.0 | — | 0.848 | + +vs the Q8 era's 0.462 acceptance at a 2,202-token prompt: the MTP head drafts +materially better on this file. The card's claimed 0.953-class acceptance is +no longer science fiction on real prose — but keep reading it as a file +property, not a throughput promise. + +## The ngram-mod chain — measured and rejected + +Frank's ask: layer `--spec-type draft-mtp,ngram-mod` on top of MTP for an +extra gain. Measured in three bench windows (production service stopped per +the benchmarking law; artifacts `/home/dank/fl-ngram-ab/` on Dankclaw), +comparing `draft-mtp` alone vs the chain on the same traffic: + +| Traffic | MTP alone | MTP + ngram-mod | Delta | +|---|---|---|---| +| Prose | 27.57 t/s | 23.40 t/s | **−15 %** | +| Block copy | 28.0 t/s | 26.1 t/s | **−7 %** (with ngram-mod demonstrably firing: 138 proposals/run) | + +Mechanism, straight from arm B's boot log: + +``` +recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support +``` + +On the qwen35 hybrid (Gated DeltaNet recurrent layers) adding the ngram layer +disables the rollback path MTP's own verification relies on, so acceptance — +and throughput — drop on *all* traffic, even where the ngram matcher fires. +**Deployed decision: `draft-mtp` alone.** Chaining is not free on recurrent +models; the fleet law in `AGENTS.md → llama.cpp serving` records it. + +## Bench-window hardening laws (learned building the A/B) + +1. **Health 200 is not proof of ownership.** After boot, verify the pid you + started owns the port (`ss -ltnp`); a zombie server from a prior window + answers `/health` too and silently serves your arm. +2. **Kill scratch servers with SIGKILL after a grace period.** llama-server's + graceful SIGTERM waits on open keep-alive connections — a harness holding + the socket leaves the process (and its ~24 GiB of GTT) alive after the + script exits. +3. **Prove the port is free and GTT dropped between arms**, or the second + arm measures a collision. + +## Client contract + +Unchanged from the Q8 era and re-verified on this payload 2026-10-03: model id +`frank-large`, native tools, thinking in `reasoning_content`, +`reasoning_effort` ∈ {xhigh, medium, low} only (anything else HTTP 500), +card-recipe sampling defaults server-side, vision through the BF16 projector +(red PNG → **`Red`**, 74 prompt / 165 completion tokens). The traps list — +`content: null` when reasoning eats the budget, the self-recovering MTP +boundary log line, `enable_thinking:false` not `reasoning_effort:"none"` — is +in [[frank-large-rocmfpx-q8agent-mtp]] and still applies. + +Sweep key for scratch servers is now the payload name: +`pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f +llama-server` (frank-small is also one). + +## Operations + +```bash +./scripts/deploy.sh dankclaw --service frank-large +systemctl status frank-large +journalctl -u frank-large -n 200 --no-pager +scripts/download-frank-large-kingjones-uncensored.sh --verify-only +``` + +## Rollback + +| Target | How | +|---|---| +| **ALIGNED Q8_0_ROCMFPX_AGENT** (previous shape, 17 t/s) | the payload dir `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` is untouched on disk; revert the swap commit (unit/env/serve trio) and redeploy, or point `FRANK_LARGE_MODEL_FILE` + `LLM_LARGE_MODEL_FILE` at the old paths. Verify with `scripts/download-frank-large-kingjones.sh --verify-only` | +| gufo Q4_K_XL era | git tag `frank-large-gufo-q4kxl-20261001`, **but the gufo payloads (UD-Q4_K_XL 16.35 GiB, UD-Q8_K_XL + DFlash2 drafter) were removed from disk 2026-10-03 with Frank's approval** — this rollback now requires re-download per [[frank-large-gufo-qwen38-27b]] | + +## When it misbehaves + +Same decision tree as [[frank-large-rocmfpx-q8agent-mtp]]: +`unknown tensor type` ⇒ off the pinned fork; SIGSEGV in +`ggml_compute_forward_set_rows` ⇒ pre-`36e9acd` build; slow decode with no +error ⇒ draft not engaged (`draft_n` in `/completion` timings) or an orphaned +second copy eating GTT (`/sys/class/drm/card1/device/mem_info_gtt_used`); +500 naming Jinja ⇒ `--jinja` lost; 500 naming the template ⇒ unmapped +`reasoning_effort` in `models.yml`. diff --git a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md index 22d269b..c11dd0d 100644 --- a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md +++ b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md @@ -1,7 +1,7 @@ --- type: reference -title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP" -description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, and the bundled BF16 projector for vision. Measured MTP-vs-none A/B, the client contract, and rollback to gufo." +title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (retired)" +description: "The retired frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head, f16 KV, and the BF16 projector. Measured MTP-vs-none A/B, the client contract, and the rollback state as of 2026-10-03." tags: - llm - frank-large @@ -15,14 +15,20 @@ tags: timestamp: 2026-10-02 --- -# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151) +# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151, retired) -> **ACTIVE since 2026-10-02.** Cut over from -> [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s). Both requests came from -> Frank: *highest quality that runs*, and *one full context*. This endpoint -> delivers 8.39 bpw weights — two quant rungs above what gufo could load — and -> measures **17.0 t/s** decode instead of 15, so quality was not bought with -> speed. +> **RETIRED 2026-10-03** — superseded by +> [[frank-large-rocmfpx-q6agent-uncensored-mtp]]: Frank swapped to the +> uncensored Q6_0_ROCMFPX_AGENT file (protected heads, one rung down), which +> measures **27.6–28.1 t/s** at a deeper prompt than this file's 17.0. This +> runbook stays live as the rollback reference and the trap catalogue; the +> payload it documents remains on disk at +> `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`. +> +> Cut over 2026-10-02 from [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s) +> on two requests from Frank: *highest quality that runs*, and *one full +> context*. It delivered 8.39 bpw weights — two quant rungs above what gufo +> could load — at 17.0 t/s, so quality was not bought with speed. > > The capability that forced this whole shape is a GGUF tensor type: > **`GGML_TYPE_Q8_0_ROCMFPX` (103)**. It exists in exactly one loader on this @@ -184,17 +190,24 @@ broken payload never stops the running unit. Caller environment **overrides** `frank-large.env`, and defaults use `:=` so an explicitly empty value surfaces as the error it reads as. -## Rollback +## Rollback (as of retirement, 2026-10-03) -git tag **`frank-large-gufo-q4kxl-20261001`**: gufo v0.1.1 -(`scripts/build-gufo.sh` still on disk), the Q4_K_XL Qwen3.8-27B payload and the -DFlash2 drafter all remain on the host. Restore that tag's unit/env/serve trio -and redeploy — nothing here deletes anything. +**Primary rollback: the ALIGNED Q8 file itself is the previous shape's +payload and remains on disk** at +`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the +2026-10-03 swap commit (unit/env/serve trio) or repoint +`FRANK_LARGE_MODEL_FILE`/`LLM_LARGE_MODEL_FILE`, verify with +`scripts/download-frank-large-kingjones.sh --verify-only`, redeploy. -The rollback payloads stay exactly where the tag's env file expects them: -`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and the DFlash2 -drafter `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-DFlash2-Q4_K_M.gguf` -(1.07 GiB). The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB, +git tag **`frank-large-gufo-q4kxl-20261001`** (gufo v0.1.1, Q4_K_XL + +DFlash2) is now a **re-download** rollback: the gufo payloads +`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and +`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/` (incl. the DFlash2 drafter) were +removed from disk on 2026-10-03 with Frank's explicit approval to free space. +`scripts/build-gufo.sh` remains on disk; re-fetch per +[[frank-large-gufo-qwen38-27b]]. + +The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB, Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 — recoverable until the trash is purged, and never part of a rollback plan. diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md index 693fd66..c9c3ecf 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md @@ -1,7 +1,7 @@ --- type: reference title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF" -description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: three conversations of 180,224 tokens, mmproj vision, the asymmetric KV contract (f16 K + q8_0 V), the two VRAM measurements that set the pool size, and what to check when it misbehaves." +description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two conversations of 200,704 tokens, fully unquantized f16 KV, mmproj vision, the boot-pair arithmetic that sizes the pool, and what to check when it misbehaves." tags: - llm - frank-small @@ -11,23 +11,26 @@ tags: - mimo - gguf - kv-cache -timestamp: 2026-10-02 +timestamp: 2026-10-03 --- # frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090) -> **ACTIVE since 2026-10-02.** One quant step down from -> [[frank-small-llama-mimo-q6k]] on the same runtime and the same KV -> contract, then **resized twice the same day** on Frank's orders: first to -> three contexts with vision, then down to the pool that keeps 2–3 GiB of the -> card free. Frank's arithmetic request — "as much as can fit while keeping -> overhead open space to 2-3gb on the 3090" — is answered by the two -> measurements in *Shape and the KV contract* below; the answer is -> **3 × 180,224**, not 3 × 262,144. +> **ACTIVE since 2026-10-02, resliced 2026-10-03.** One quant step down from +> [[frank-small-llama-mimo-q6k]] on the same runtime, resized twice on +> 2026-10-02 (three contexts with vision, then the pool that kept 2–3 GiB of +> the card free), and on 2026-10-03 Frank ordered the third shape: **two +> conversations of 200,704 tokens each with the KV VALUES unquantized** — +> *"2 streams with 200700 tokens context or more… if we have extra space on +> the 3090, unquantize the kv."* The freed third slot pays for exact V, and +> the boot lands **20,530 MiB used / 4,046 MiB free** — above the 2–3 GiB +> floor, with full-precision KV. -> **The tradeoff is the per-slot window.** Three sessions cost 81,920 tokens -> of per-conversation memory versus the two-slot shape: 180,224 tokens each -> instead of the model's native 262,144. See *What three slots cost*. +> **The tradeoffs of the reslice:** one fewer simultaneous conversation, a +> per-slot window that grew from 180,224 to 200,704 (still below the native +> 262,144), and ~2 % slower decode (104.8 vs 106.8 tok/s) from the larger +> per-token KV traffic. The 3 × 180,224 shape is one env edit away — see +> *Rollback*. ## Identity @@ -41,77 +44,72 @@ timestamp: 2026-10-02 ## Shape and the KV contract -`--ctx-size 540672 --parallel 3` — llama.cpp divides the pool across slots, so -each conversation gets `n_ctx_slot = 180224` (boot log: `n_slots = 3, -n_ctx_slot = 180224, kv_unified = 'false'`). `CTX_SIZE` MUST stay divisible by -`PARALLEL`; the bats seam enforces it. +`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots, +so each conversation gets `n_ctx_slot = 200704` (boot log 2026-10-03: +`n_slots = 2, n_ctx_slot = 200704, kv_unified = 'false'`). `CTX_SIZE` MUST +stay divisible by `PARALLEL`; the bats seam enforces it. -The KV contract is Frank's: **K unquantized (f16), V `q8_0`** — -`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it: +The KV contract is Frank's, as of 2026-10-03: **both K and V unquantized +(f16)** — `--cache-type-k f16 --cache-type-v f16`. The asymmetric contract it +replaced (f16 K + `q8_0` V) had existed to make three slots fit; with two +slots the exact-value budget opens, and 4 KiB/token × 8 layers × 2 of extra +V traffic is the measured ~2 % decode tax. The serve script still passes both +flags explicitly — the asymmetric-era law that a quantized V requires +`--flash-attn on` stays pinned by the seam. -- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path. - Without the flag the engine falls back silently and the contract becomes a - lie. The serve script pins the trio; the bats seam pins the script. -- The VRAM budget: attention KV lives in the 8 full-attention layers only - (4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token - (34 B per 32-element block) ⇒ ~24.5 KiB/token over the pool. +**Pool sizes on this stack come from boots, not formulas.** The compute +buffers, CUDA graphs and vision tower all bill separately, so every shape +change is priced from measured rows: -**The pool size came from two boots, not from a formula.** `free`-style -arithmetic on this stack is unreliable because the compute buffers, CUDA -graphs and now the vision tower all bill separately, so the fit was measured -at two shapes and solved between them: - -|Shape|Projector|`nvidia-smi` used|KV at 24.5 KiB/token|Non-KV (weights + buffers + projector)| +|Shape|KV dtype (K+V)|Projector|`nvidia-smi` used|Non-KV (weights + buffers + projector)| |---|---|---|---|---| -|2 × 262,144 (pool 524,288)|none|20,190 MiB|12,800 MiB|**7,390 MiB**| -|3 × 196,608 (pool 589,824)|f16|22,972 MiB|14,113 MiB|**8,859 MiB**| +|2 × 262,144 (pool 524,288)|f16 + q8_0|none|20,190 MiB|**7,390 MiB**| +|3 × 196,608 (pool 589,824)|f16 + q8_0|f16|22,972 MiB|**8,859 MiB**| +|3 × 180,224 (pool 540,672)|f16 + q8_0|f16|21,684 MiB|~8,419 MiB| +|**2 × 200,704 (pool 401,408)**|**f16 + f16**|f16|**20,530 MiB**|**~7,986 MiB**| -The two rows price the projector and its image buffers at **~1.4 GiB** — not -the ~0.9 GiB the 918 MB file suggests, because the vision encoder's own -activation and compute buffers ride along. Solving the second row's non-KV cost -against Frank's 2–3 GiB band: the card is 24,576 MiB, so -`KV ≤ 24,576 − 9,216 − 2,048 = 13,312 MiB` ⇒ pool ≤ 556,630 tokens, rounded -down to a multiple of 3 × 4,096 ⇒ **540,672**, i.e. 13,265 MiB of KV and -**2,441 MiB free** (measured, 21,684 MiB used). +The 32 KiB/token full-attention KV price (8 layers × 4 KV heads × 256 dim × +2 B × 2 halves) prices the deployed pool at 401,408 × 32 KiB = **12,544 MiB +of KV**; the card is 24,576 MiB, so the shape leaves **4,046 MiB free** +(measured steady-state — the KV pool is allocated whole at boot, so this is +the ceiling). The headroom above the 2–3 GiB floor is deliberate slack, not +an invitation: another 32,768 tokens of pool (1 GiB) would trade the last +safety margin for window the harness compacts before reaching anyway. -**Why 3 × 262,144 was never possible.** Three native windows are 786,432 -tokens = 18.4 GiB of KV *alone*; plus the 8.86 GiB of non-KV cost that is -27.3 GiB on a 24 GiB card. The quant step bought 0.86 GiB; the geometry needed -6 GiB. +**Why 2 × 262,144 with f16 V does not fit either.** The native window pair is +524,288 tokens = 16,384 MiB of KV plus ~7,986 non-KV = 24.4 GiB: under the +floor by a hair. 200,704 is the largest 4,096-aligned slice that keeps the +measured band. -## Measured on this shape (2026-10-02, 3 × 180,224 + vision) +## Measured on this shape (2026-10-03, 2 × 200,704 + vision, f16 KV) |Check|Result| |---|---| -|Boot to healthy|~20 s, `n_slots = 3`, `n_ctx_slot = 180224`, `loaded multimodal model` logged| -|VRAM|**21,684 MiB used / 2,441 MiB free** of 24,576 (`nvidia-smi`, steady state; the KV pool is allocated whole at boot, so this is the ceiling)| -|Decode, temp 0, no thinking|**106.8 tok/s** (256-token completion at ~1,300-token depth)| -|Prefill|**3,814 tok/s** (same request)| -|Three concurrent requests|all three answered in 1.7 s wall — real parallel slots| -|Fourth concurrent request|answered in 3.0 s while three ran — queued behind a released slot, as the slot ceiling requires| -|Vision|`image_url` part with a 64×64 solid-red PNG → **"Red."**| -|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"f"}`)| -|Thinking|`reasoning_content` populated (145 chars on a "think first" prompt); `reasoning_effort: "low"` accepted| -|Over-slot prompt|`400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)` — refused, never silently truncated| +|Boot to healthy|~5 s, `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model` logged| +|VRAM|**20,530 MiB used / 4,046 MiB free** of 24,576 (`nvidia-smi`, steady state)| +|Decode, temp 0, no thinking|**104.84 tok/s** (128-token completion at ~500-token depth; the 3 × 180,224 q8_0-V shape measured 106.8 — ~2 % is the honest price of exact V)| +|Prefill|1,650 t/s on the same shallow 506-token probe (rates are depth- and prompt-dependent; not comparable across shapes)| +|Batch geometry|`--batch-size 2048 --ubatch-size 1024` — ubatch 2048 failed the scratch boot; 1024 boots clean| +|Vision|projector loads at boot (`loaded multimodal model`); the red-PNG probe stands from the 2026-10-02 verification — the vision path is unchanged by this reslice| +|Tools / thinking|unchanged by the reslice: native `tool_calls`, thinking in `reasoning_content`| -Decode is ~5 % **faster** than the 2 × 262,144 shape measured earlier the same -day (100.4–101.8 tok/s): a shorter per-slot window means a shorter attention -scan at equal prompt depth, and one less slot's DeltaNet state resident. +## What the slot geometry costs -## What three slots cost +The per-slot window is the whole price, in both directions: -The per-slot window is the whole price. Two consequences the fleet depends on: - -- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml` - sets `contextWindow: 180224` for this endpoint — the harness compacts before +- **A prompt over 200,704 tokens is a hard 400**, so `~/.omp/agent/models.yml` + sets `contextWindow: 200704` for this endpoint — the harness compacts before it reaches the ceiling instead of discovering it. -- **frank-primary's compaction prompt fits inside one slot again.** At - frank-primary's shape-D window (200,704) its compaction trigger is 140,492 - tokens (70 %), so one compaction request lands here with 39,732 tokens of - room. At the previous 262,144 window the trigger was 183,501 — *above* this - slot — and the request had to be truncated or split. Nothing in the fleet - routes compaction here automatically; the point is that the 2026-10-02 window - trim restored the fit. +- **frank-large's tallest pre-compaction prompt now fits here.** Compaction + for frank-large sessions runs on this endpoint (`compactionModel` in + `models.yml`); frank-large compacts at 70 % of 262,144 = 183,501 tokens, + which did **not** fit the old 180,224 slot and fits 200,704 with 17,163 + tokens to spare. The 2026-10-02 window trim had restored the fit for + frank-primary's smaller 140,492-token trigger; the 2026-10-03 reslice widens + the fit to frank-large itself. Nothing routes compaction automatically — + the point is the ceiling no longer forbids it. +- Two simultaneous generations are the slot ceiling; a third request queues + behind a released slot (the mechanism measured on the 3-slot shape). ## Client contract @@ -147,8 +145,9 @@ traffic first. The bats seam fails if a draft flag ships unannounced. |Target|How| |---|---| +|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-small.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V| |Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default. Q6_K needs ~1.4 GiB more than Q5_K_M, so pair the rollback with `CTX_SIZE` 49,152 tokens lower or `MMPROJ_FILE=`| -|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2` (measured 20,190 MiB used, 4,386 free) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window or when frank-primary's compaction request has to land here| +|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window| |Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_SMALL_MMPROJ` at any real projector file or drop the gate in the same commit| |SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02, so restore it there or re-fetch with `old/download-frank-small-sglang-int8.sh`| @@ -158,13 +157,14 @@ traffic first. The bats seam fails if a draft flag ships unannounced. naming the missing file if the binary or GGUF moved. - `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it). -- Requests queue behind each other: three concurrent long generations are the - slot ceiling by contract; a fourth waits for a slot to free (measured: 1.7 s - for the three, 3.0 s for the fourth). -- `400 … exceeds the available context size (180224 tokens)`: the conversation - outgrew ONE slot. This is not pool exhaustion — the pool is 540,672 tokens - and the per-slot window is the binding limit. Shorten the prompt, compact, or - roll back to the 262,144 shape above. +- Requests queue behind each other: two concurrent long generations are the + slot ceiling by contract (2026-10-03 shape); a third waits for a slot to + free (the mechanism measured on the 3-slot shape: 1.7 s for three, 3.0 s for + the fourth). +- `400 … exceeds the available context size (200704 tokens)`: the conversation + outgrew ONE slot. This is not pool exhaustion — the pool is 401,408 tokens + and the per-slot window is the binding limit. Shorten the prompt, compact, + or roll back to the 262,144 shape above. - Vision answers a color it never saw: the client sent `images` instead of an `image_url` content part (see *Client contract*). - Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index eda91a3..f85bd5c 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,6 +1,6 @@ # LLM Fleet — Knowledge Base -Documentation for the LLM inference fleet on the two hosts (2026-10-02): +Documentation for the LLM inference fleet on the two hosts (2026-10-03): **Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s @@ -15,11 +15,12 @@ mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001), and Dankclaw runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770: -kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115 -file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE -session at 262,144, f16 KV, vision via the bundled projector)** and +kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF +Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT, a ftype-114 file only that fork can +read, with the repo's MTP head drafting 4 tokens at 27.6-28.1 t/s decode, ONE +session at 262,144, f16 KV, vision via the BF16 projector)** and **llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M -GGUF, 3 x 180,224, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**. +GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**. > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -46,12 +47,12 @@ expert-precise-memory arena 7,522 slots, /root/.cache mounted] gfx1151 iGPU 114 GiB GTT] E[ROCmFPX llama.cpp gfx1151, pinned fork build frank-large, port 7770] - F[Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP-Q4_0 head + F[Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP-Q4_0 head 1 x 262144 ctx, f16 KV, draft n-max 4, mmproj vision] G[llama.cpp master CUDA sm_86 frank-small, port 7771] H[MiMo-V2.6 Q5_K_M GGUF -3 x 180224 ctx, f16 K + q8_0 V, flash-attn, mmproj vision] +2 x 200704 ctx, f16 K + f16 V, flash-attn, mmproj vision] E2 --> E --> F E2 --> G --> H end @@ -70,20 +71,23 @@ frank-small, port 7771] ## Active Deployment -- [[frank-large-rocmfpx-q8agent-mtp]] — **ACTIVE since 2026-10-02** +- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — **ACTIVE since 2026-10-03** (root systemd unit, port 7770): the pinned kingjones30/ROCmFPX fork serving - the repo's highest-quality rung `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` - (26.28 GiB, ftype 115 = 187 plain-Q8_0 + 311 GGML_TYPE_Q8_0_ROCMFPX - tensors) with the bundled `mtp-Qwen3.8-27B-Q4_0.gguf` head drafting 4 - tokens, one 262,144 window, f16 KV, the BF16 projector for vision. Why the - runtime is not interchangeable, the pinned-build set_rows history, the - measured envelope, rollback to gufo -- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root - systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving - bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090, - THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the - mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free. - The KV arithmetic, the flash-attn dependency, rollback to Q6_K + the uncensored protected-heads rung `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` + (23.15 GiB, ftype 114 = 159 type-103 + 339 GGML_TYPE_Q6_0_ROCMFPX tensors, + 64 protected output heads) with the repo's `mtp-Qwen3.8-27B-Q4_0.gguf` head + drafting 4 tokens, one 262,144 window, f16 KV, the BF16 projector for + vision. Measured 27.6-28.1 t/s prose decode at 86.8 % draft acceptance; the + ngram-mod chain onto MTP was measured and REJECTED (-15 %: it disables the + hybrid's recurrent state rollback). Rollback: the aligned Q8 file, still on + disk +- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02, resliced + 2026-10-03** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) + serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090, + TWO slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K + + f16 V) and the mmproj for vision, 104.8 tok/s decode, 4,046 MiB of the + card left free. The boot-pair pool arithmetic, the slot-window contract, + rollback to 3 x 180,224 or Q6_K - [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since 2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both @@ -192,8 +196,13 @@ frank-small, port 7771] (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (the rollback - target)** (root systemd unit, port 7770): gufo +- [[frank-large-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the + rollback target; payload still on disk)**: the same pinned ROCmFPX fork + serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP + acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue +- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the + payload dirs were removed from disk 2026-10-03, so this rollback is a + re-download)** (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL** with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 961451a..ba6ebd8 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -213,15 +213,21 @@ The LLM inference fleet on the two hosts. Full index: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record) -- [[frank-large-rocmfpx-q8agent-mtp]] — the ACTIVE frank-large runbook - since 2026-10-02 (root systemd unit, port 7770): the pinned kingjones30/ - ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/ - Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF **Q8_0_ROCMFPX_AGENT** — ftype 115, - 311 tensors of GGML_TYPE_Q8_0_ROCMFPX, a format no other loader on the - host reads — with the repo's Q4_0 MTP head drafting 4 tokens and its BF16 - projector, one session at the native 262,144 window -- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (rollback - target, payload and `scripts/build-gufo.sh` both on disk): gufo v0.1.1 +- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — the ACTIVE frank-large + runbook since 2026-10-03 (root systemd unit, port 7770): the pinned + kingjones30/ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/ + Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF + **Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT** — ftype 114, 339 tensors of + GGML_TYPE_Q6_0_ROCMFPX + 159 type-103, a format no other loader on the + host reads — with the repo's Q4_0 MTP head drafting 4 tokens (27.6-28.1 + t/s measured, 86.8 % acceptance; the ngram-mod chain measured and + rejected) and its BF16 projector, one session at the native 262,144 window +- [[frank-large-rocmfpx-q8agent-mtp]] — RETIRED 2026-10-02 → 10-03 (rollback + target, aligned Q8 payload still on disk): the same fork serving the + ALIGNED Q8_0_ROCMFPX_AGENT at 17.0 t/s — the +123 % MTP A/B, the traps +- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (history; + the payload dirs were removed from disk 2026-10-03, `scripts/build-gufo.sh` + remains): gufo v0.1.1 built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2 drafter on the gfx1151 iGPU — the two from-source build workarounds, the template-hash allowlist, the measured prefill/decode trade @@ -283,11 +289,11 @@ The LLM inference fleet on the two hosts. Full index: - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) - [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since - 2026-10-02: a CUDA (sm_86) build of llama.cpp master serving - bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX - 3090, port 7771 — THREE conversations at 180,224 tokens each, the - asymmetric KV contract (f16 K + q8_0 V), vision through the repo's mmproj, - and the two VRAM measurements that set the pool size; ~100 tok/s decode + 2026-10-02, resliced 2026-10-03: a CUDA (sm_86) build of llama.cpp master + serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's + RTX 3090, port 7771 — TWO conversations at 200,704 tokens each with fully + unquantized f16 KV, vision through the repo's mmproj, and the boot-pair + arithmetic that sizes the pool; 104.8 tok/s decode, 4,046 MiB free - [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced by [[frank-small-llama-mimo-q5km]], payload still on disk): the first llama.cpp + GGUF shape for this endpoint, Q6_K at the same