diff --git a/scripts/download-frank-small.sh b/old/download-frank-small-sglang-int8.sh similarity index 100% rename from scripts/download-frank-small.sh rename to old/download-frank-small-sglang-int8.sh diff --git a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md index 8d3b8b8..e0e8557 100644 --- a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md +++ b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md @@ -1,7 +1,7 @@ --- type: reference -title: "gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)" -description: "The RETIRED frank-large runbook (2026-09-28 → 2026-10-01, rollback target via tag frank-large-gufo-q4kxl-20261001) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing." +title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-large)" +description: "The ACTIVE frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing." tags: - llm - gufo @@ -16,13 +16,18 @@ tags: timestamp: 2026-09-28 --- -# frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) +# frank-large — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2) -> **RETIRED 2026-10-01** — replaced by [[frank-large-vllm-mimo-v26-bf16]] -> (vLLM TheRock + MiMo-V2.6 BF16). Rollback: git tag -> `frank-large-gufo-q4kxl-20261001`; the gufo build (`build-gufo.sh`), the -> Q4_K_XL payload and the DFlash2 drafter all remain on the host. -> **Originally ACTIVE 2026-09-28 → 2026-10-01.** Replaces +> **ACTIVE again since 2026-10-01 night** — Frank rolled frank-large back +> after one day of [[frank-large-vllm-mimo-v26-bf16]] (battery decode +> 6.19-9.01 t/s; every ≥6-bit vLLM quant path dead on gfx1151). The shape +> moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF +> UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was +> reclaimed), and the session grid is **one session at the full native +> 262,144 window** (`SESSIONS=1 CONTEXT_SIZE=262144`, f16 KV). The +> Swift-1.5 Q8 history below is retained for the engine/template law, +> which still governs. +> **Originally ACTIVE 2026-09-28.** Replaces > [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth > b11030 prebuilt), which had replaced > [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same diff --git a/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md b/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md index fa1b71b..6000864 100644 --- a/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md +++ b/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md @@ -1,7 +1,7 @@ --- type: reference -title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP" -description: "The active frank-large runbook — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU: identity, the measured roof math, the cross-model Qwen3.5-9B MTP drafter and its acceptance curve, the greedy-text numerics trap, the client-contract field change, deployed flags, and what to check when it misbehaves." +title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)" +description: "The RETIRED frank-large runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-large-gufo-qwen38-27b]] by Frank the same night." tags: - llm - frank-large @@ -17,12 +17,12 @@ timestamp: 2026-10-01 # frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP) -> **ACTIVE since 2026-10-01.** Replaces [[frank-large-gufo-qwen38-27b]] -> (gufo + Qwen3.8-27B GGUF, now the rollback target, git tag -> `frank-large-gufo-q4kxl-20261001`). Same port (7770), same served alias -> (`frank-large`). This is the fleet's second vLLM endpoint and the first -> CROSS-MODEL speculative-decode deployment: the target is Xiaomi's distill, -> the draft head is Qwen's. +> **RETIRED 2026-10-01 night, one day active.** Frank's call after the +> autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft) +> against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM +> 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only; +> torchao config unwired). Back to [[frank-large-gufo-qwen38-27b]] +> (Q4_K_XL, 1 x 262144). The measured content below stands as the record. ## Identity diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md new file mode 100644 index 0000000..c2d0237 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md @@ -0,0 +1,107 @@ +--- +type: reference +title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF" +description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the flash-attention dependency that comes with quantized V, the VRAM fit, and what to check when it misbehaves." +tags: + - llm + - frank-small + - dankclaw + - llama-cpp + - rtx3090 + - mimo + - gguf + - kv-cache +timestamp: 2026-10-01 +--- + +# frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090) + +> **ACTIVE since 2026-10-01 night.** Replaces +> [[frank-small-sglang-mimo-v26]] (SGLang + INT8-W8A16-AutoRound safetensors, +> same architecture). Rolled in by Frank's night order: move this endpoint to +> the GGUF path for **asymmetric KV** — unquantized K + q8_0 V — which only +> llama.cpp exposes per-tensor. + +## Identity + +|Field|Value| +|---|---| +|Endpoint|`192.168.0.143:7771`, served alias `frank-small`| +|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`| +|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`| +|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`| +|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam), `deploy_llm_small` gates in `scripts/deploy-remote.sh`| + +## Shape and the KV contract + +`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots, +so each session gets `n_ctx_slot = 262144`, the model's full native window +(boot log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`). + +Frank's KV contract: **K unquantized (f16), V `q8_0`** — +`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it: + +- **`--flash-attn on` is mandatory**: quantized V exists only on the FA + path. Without the flag the engine falls back silently and the contract + becomes a lie. The serve script pins the trio; the bats seam pins the + script. +- The VRAM budget: attention KV lives in the 8 full-attention layers only + (4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token + (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool + ≈ **12.6 GiB**. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈ + **20.9 GiB measured** against 24.5 GiB — the "room for checkpoints" the + order asked for. If graph capture OOMs, halve `CTX_SIZE`; never quantize + K to fix it. + +## Client contract + +- Thinking arrives in **`reasoning_content`** (`--reasoning-format + deepseek`) — the same field SGLang's qwen3 parser used, so `models.yml` + `reasoningContentField` did NOT move. Live-verified 2026-10-01 22:57. +- **Thinking is off unless the request asks**: the MiMo GGUF template + defaults `enable_thinking` false; `models.yml` now ships + `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` + in `extraBody`. A raw client that omits it gets a no-thinking answer + (measured — this is not a fault). +- Tools: `--jinja` + server-side tool parsing, verified + `tool_calls` round-trip 2026-10-01 23:0x. +- **Text-only now.** The GGUF carries no vision projector; the + in-checkpoint tower existed only in the SGLang safetensors. `models.yml` + `input: [text]`. + +## No spec decode (yet) + +The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the +cross-model DFlash draft (`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`, +Q8_0) is an optimization-loop candidate, not a default — `--spec-type +draft-dflash` needs acceptance measured on this traffic first. The bats +seam fails if a draft flag ships unannounced. + +## Build + +No ccache, CUDA 12.4 system nvcc with `--gcc-install-dir`-free gcc-13 host +compiler (nvcc 12.4 rejects gcc ≥ 14): + +``` +cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \ + -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \ + -DLLAMA_CURL=OFF -DGGML_CCACHE=OFF +cmake --build build-cuda -j 16 --target llama-server +``` + +The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) and the INT8 +AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`) +stay on disk as the rollback: repoint the unit's serve script and env from +git history (`git show frank-large-gufo-q4kxl-20261001` era is BEFORE this; +use the parent of the 2026-10-01 night swap commit). + +## When it misbehaves + +- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script + exits 1 naming the missing file if the binary or GGUF moved. +- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM + node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it). +- Requests hang at first token while `/health` is 200: both slots busy — + two concurrent long generations are the ceiling by contract. +- Slow decode after a CTX change: check `nvidia-smi` — swapping KV out of + VRAM into unified memory halves the rate silently. diff --git a/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md b/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md index 648daf3..e16a22b 100644 --- a/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md +++ b/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md @@ -1,7 +1,7 @@ --- type: reference -title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16" -description: "The active frank-small runbook — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode shape, the CUDA 13 nvcc toolchain the JIT needs, deployed flags, the client contract, and what to check when it misbehaves." +title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)" +description: "The RETIRED frank-small runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-small-llama-mimo-q6k]] the same night." tags: - llm - frank-small @@ -17,9 +17,10 @@ timestamp: 2026-10-01 # frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP) -> **ACTIVE since 2026-10-01.** Replaces [[frank-small-gemma4-12b]] -> (llama.cpp Gemma4-12B QAT Q4_K_M, now the rollback target). Same port -> (7771), same alias (`frank-small`), same client contract surface. +> **RETIRED 2026-10-01, hours after cutover.** Replaced by +> [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF, +> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express. +> The venv and the INT8 payload stay on disk as the rollback. ## Identity @@ -118,8 +119,11 @@ change lands. Eight safetensors shards (12,891,718,440 B total) + tokenizer and config files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`, -all LFS sha256-verified by `scripts/download-frank-small.sh` (`--verify-only` -re-checks). The serve script refuses to start with a missing shard. +all LFS sha256-verified by `old/download-frank-small-sglang-int8.sh` +(`--verify-only` re-checks) — the script moved to `old/` when the endpoint +cut over to llama.cpp + GGUF on 2026-10-01 night, and +`scripts/download-frank-small.sh` now fetches the GGUF quant instead. The +serve script refuses to start with a missing shard. ## Client contract diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index a3fc1be..a07f003 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -10,12 +10,11 @@ headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, uti reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on, measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw -runs two endpoints on the same arch — -vLLM (digest-pinned kyuz0 TheRock gfx1151 image; frank-large, -XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 with cross-model MTP -spec decode from the Qwen3.5-9B head on the gfx1151 iGPU, 4 x 262144 ctx, -port 7770) and SGLang (frank-small, the same model as an INT8 W8A16 -AutoRound conversion on the RTX 3090, 2 x 262144 ctx, port 7771). +runs two endpoints — **gufo v0.1.1 (frank-large, port 7770: Qwen3.8-27B +UD-Q4_K_XL + DFlash2, ONE session at 262,144, f16 KV on the gfx1151 iGPU, +rolled back from the one-day vLLM MiMo experiment 2026-10-01 night)** and +**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K +GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**. > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -39,14 +38,14 @@ fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] E2[RTX 3090 (PCIe x4 dock) + gfx1151 iGPU 114 GiB GTT] - E[vLLM TheRock gfx1151, digest-pinned + E[gufo v0.1.1 built from source frank-large, port 7770] - F[MiMo-V2.6-Distill-Qwen-9B BF16 + Qwen3.5-9B MTP head -4 x 262144 ctx, spec 2 tok, qwen3/qwen3_coder parsers] - G[SGLang venv cu130 + F[Qwen3.8-27B UD-Q4_K_XL + DFlash2 drafter +1 x 262144 ctx, f16 KV, greedy, adaptive draft 7] + G[llama.cpp master CUDA sm_86 frank-small, port 7771] - H[MiMo-V2.6 INT8 W8A16 AutoRound -2 x 262144 ctx, pinned pool 606208, in-checkpoint vision] + H[MiMo-V2.6 Q6_K GGUF +2 x 262144 ctx, f16 K + q8_0 V, flash-attn] E2 --> E --> F E2 --> G --> H end @@ -65,7 +64,9 @@ frank-small, port 7771] ## Active Deployment -- [[frank-primary-glm53-flash-w4a16]] — the ACTIVE frank-primary runbook +- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the +- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro): + (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork (digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`, no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric @@ -162,23 +163,24 @@ frank-small, port 7771] (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[frank-large-vllm-mimo-v26-bf16]] — ACTIVE frank-large runbook since - 2026-10-01 (root systemd unit, port 7770): the digest-pinned +- [[frank-large-gufo-qwen38-27b]] — **the ACTIVE frank-large runbook + again since 2026-10-01 night** (root systemd unit, port 7770): gufo + (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from + source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL** + with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native + 262,144 window, unquantized f16 KV. Rolled back by Frank after one day + of [[frank-large-vllm-mimo-v26-bf16]]. The two from-source build + workarounds, the client-contract delta, the measured prefill/decode + trade against llama.cpp +- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day; + rollback by Frank after the autoresearch loop): the digest-pinned kyuz0/vllm-therock-gfx1151 image serving - XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU, - 4 x 262144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B - (the fleet's first; 71%/44% per-position acceptance measured) — - the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured - MTP-vs-DFlash table (19.8 vs 17.4 t/s decode), the greedy-text numerics - trap, the `reasoning` field change, troubleshooting -- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-09-28 → 2026-10-01 - (replaced by [[frank-large-vllm-mimo-v26-bf16]]; rollback target via git - tag frank-large-gufo-q4kxl-20261001, engine + payload + drafter still on - the host): gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, - built from source on Dankclaw, serving Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL - with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — the two - from-source build workarounds, the client-contract delta, the measured - prefill/decode trade against llama.cpp + XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx, + CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record: + battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET- + NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text + numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151 + (quark int8 rejected, FP8 kernels CDNA-only, torchao unwired) - [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp (unsloth b11030, ROCm gfx1151) serving @@ -212,14 +214,22 @@ frank-small, port 7771] .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract, troubleshooting (2026-09-16 → 09-19) -- [[frank-small-sglang-mimo-v26]] — ACTIVE frank-small runbook since - 2026-10-01 (root systemd unit, port 7771): SGLang in a self-contained - python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16- - AutoRound on the RTX 3090 — the same architecture frank-large now serves - in BF16 on the iGPU; 2 x 262144 ctx on a pinned 606,208-token fp8 pool, - NO spec decode (zero mtp.* tensors shipped), in-checkpoint vision. +- [[frank-small-llama-mimo-q6k]] — **the ACTIVE frank-small runbook + since 2026-10-01 night** (root systemd unit, port 7771): llama.cpp + (master, CUDA sm_86 build) serving bartowski/MiMo-V2.6-Distill- + Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO native 262,144-token + sessions with ASYMMETRIC KV (K f16 unquantized + V q8_0, which forces + flash-attn), ~20.9/24.6 GiB measured, text-only (the GGUF has no + vision), reasoning_content, no spec decode by default +- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced + by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot + express; venv + INT8 payload on disk as rollback): SGLang in a + self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8 + W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision, + 83.9 tok/s decode - [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by - [[frank-small-sglang-mimo-v26]]; rollback target, payload and binary + [[frank-small-sglang-mimo-v26]] → [[frank-small-llama-mimo-q6k]]; + rollback target, payload and binary intact on disk): mainline llama.cpp serving intact on disk): mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX 3090, with the separate MTP assistant sidecar chained behind ngram-mod,