docs: record the frank-small llama.cpp Q6_K cutover and the frank-large gufo rollback
This commit is contained in:
Frank
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The RETIRED frank-large runbook (2026-09-28 → 2026-10-01, rollback target via tag frank-large-gufo-q4kxl-20261001) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The ACTIVE frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- gufo
|
||||
@@ -16,13 +16,18 @@ tags:
|
||||
timestamp: 2026-09-28
|
||||
---
|
||||
|
||||
# frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2)
|
||||
# frank-large — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)
|
||||
|
||||
> **RETIRED 2026-10-01** — replaced by [[frank-large-vllm-mimo-v26-bf16]]
|
||||
> (vLLM TheRock + MiMo-V2.6 BF16). Rollback: git tag
|
||||
> `frank-large-gufo-q4kxl-20261001`; the gufo build (`build-gufo.sh`), the
|
||||
> Q4_K_XL payload and the DFlash2 drafter all remain on the host.
|
||||
> **Originally ACTIVE 2026-09-28 → 2026-10-01.** Replaces
|
||||
> **ACTIVE again since 2026-10-01 night** — Frank rolled frank-large back
|
||||
> after one day of [[frank-large-vllm-mimo-v26-bf16]] (battery decode
|
||||
> 6.19-9.01 t/s; every ≥6-bit vLLM quant path dead on gfx1151). The shape
|
||||
> moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF
|
||||
> UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was
|
||||
> reclaimed), and the session grid is **one session at the full native
|
||||
> 262,144 window** (`SESSIONS=1 CONTEXT_SIZE=262144`, f16 KV). The
|
||||
> Swift-1.5 Q8 history below is retained for the engine/template law,
|
||||
> which still governs.
|
||||
> **Originally ACTIVE 2026-09-28.** Replaces
|
||||
> [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
|
||||
> b11030 prebuilt), which had replaced
|
||||
> [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP"
|
||||
description: "The active frank-large runbook — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU: identity, the measured roof math, the cross-model Qwen3.5-9B MTP drafter and its acceptance curve, the greedy-text numerics trap, the client-contract field change, deployed flags, and what to check when it misbehaves."
|
||||
title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
|
||||
description: "The RETIRED frank-large runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-large-gufo-qwen38-27b]] by Frank the same night."
|
||||
tags:
|
||||
- llm
|
||||
- frank-large
|
||||
@@ -17,12 +17,12 @@ timestamp: 2026-10-01
|
||||
|
||||
# frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
|
||||
|
||||
> **ACTIVE since 2026-10-01.** Replaces [[frank-large-gufo-qwen38-27b]]
|
||||
> (gufo + Qwen3.8-27B GGUF, now the rollback target, git tag
|
||||
> `frank-large-gufo-q4kxl-20261001`). Same port (7770), same served alias
|
||||
> (`frank-large`). This is the fleet's second vLLM endpoint and the first
|
||||
> CROSS-MODEL speculative-decode deployment: the target is Xiaomi's distill,
|
||||
> the draft head is Qwen's.
|
||||
> **RETIRED 2026-10-01 night, one day active.** Frank's call after the
|
||||
> autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft)
|
||||
> against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM
|
||||
> 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only;
|
||||
> torchao config unwired). Back to [[frank-large-gufo-qwen38-27b]]
|
||||
> (Q4_K_XL, 1 x 262144). The measured content below stands as the record.
|
||||
|
||||
## Identity
|
||||
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF"
|
||||
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the flash-attention dependency that comes with quantized V, the VRAM fit, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rtx3090
|
||||
- mimo
|
||||
- gguf
|
||||
- kv-cache
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
|
||||
|
||||
> **ACTIVE since 2026-10-01 night.** Replaces
|
||||
> [[frank-small-sglang-mimo-v26]] (SGLang + INT8-W8A16-AutoRound safetensors,
|
||||
> same architecture). Rolled in by Frank's night order: move this endpoint to
|
||||
> the GGUF path for **asymmetric KV** — unquantized K + q8_0 V — which only
|
||||
> llama.cpp exposes per-tensor.
|
||||
|
||||
## Identity
|
||||
|
||||
|Field|Value|
|
||||
|---|---|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|
||||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
|
||||
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|
||||
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
|
||||
|
||||
## Shape and the KV contract
|
||||
|
||||
`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots,
|
||||
so each session gets `n_ctx_slot = 262144`, the model's full native window
|
||||
(boot log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
|
||||
|
||||
Frank's KV contract: **K unquantized (f16), V `q8_0`** —
|
||||
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
|
||||
|
||||
- **`--flash-attn on` is mandatory**: quantized V exists only on the FA
|
||||
path. Without the flag the engine falls back silently and the contract
|
||||
becomes a lie. The serve script pins the trio; the bats seam pins the
|
||||
script.
|
||||
- The VRAM budget: attention KV lives in the 8 full-attention layers only
|
||||
(4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
|
||||
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool
|
||||
≈ **12.6 GiB**. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈
|
||||
**20.9 GiB measured** against 24.5 GiB — the "room for checkpoints" the
|
||||
order asked for. If graph capture OOMs, halve `CTX_SIZE`; never quantize
|
||||
K to fix it.
|
||||
|
||||
## Client contract
|
||||
|
||||
- Thinking arrives in **`reasoning_content`** (`--reasoning-format
|
||||
deepseek`) — the same field SGLang's qwen3 parser used, so `models.yml`
|
||||
`reasoningContentField` did NOT move. Live-verified 2026-10-01 22:57.
|
||||
- **Thinking is off unless the request asks**: the MiMo GGUF template
|
||||
defaults `enable_thinking` false; `models.yml` now ships
|
||||
`chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}`
|
||||
in `extraBody`. A raw client that omits it gets a no-thinking answer
|
||||
(measured — this is not a fault).
|
||||
- Tools: `--jinja` + server-side tool parsing, verified
|
||||
`tool_calls` round-trip 2026-10-01 23:0x.
|
||||
- **Text-only now.** The GGUF carries no vision projector; the
|
||||
in-checkpoint tower existed only in the SGLang safetensors. `models.yml`
|
||||
`input: [text]`.
|
||||
|
||||
## No spec decode (yet)
|
||||
|
||||
The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the
|
||||
cross-model DFlash draft (`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`,
|
||||
Q8_0) is an optimization-loop candidate, not a default — `--spec-type
|
||||
draft-dflash` needs acceptance measured on this traffic first. The bats
|
||||
seam fails if a draft flag ships unannounced.
|
||||
|
||||
## Build
|
||||
|
||||
No ccache, CUDA 12.4 system nvcc with `--gcc-install-dir`-free gcc-13 host
|
||||
compiler (nvcc 12.4 rejects gcc ≥ 14):
|
||||
|
||||
```
|
||||
cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
|
||||
-DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
|
||||
-DLLAMA_CURL=OFF -DGGML_CCACHE=OFF
|
||||
cmake --build build-cuda -j 16 --target llama-server
|
||||
```
|
||||
|
||||
The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) and the INT8
|
||||
AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
|
||||
stay on disk as the rollback: repoint the unit's serve script and env from
|
||||
git history (`git show frank-large-gufo-q4kxl-20261001` era is BEFORE this;
|
||||
use the parent of the 2026-10-01 night swap commit).
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script
|
||||
exits 1 naming the missing file if the binary or GGUF moved.
|
||||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
|
||||
node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||||
- Requests hang at first token while `/health` is 200: both slots busy —
|
||||
two concurrent long generations are the ceiling by contract.
|
||||
- Slow decode after a CTX change: check `nvidia-smi` — swapping KV out of
|
||||
VRAM into unified memory halves the rate silently.
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16"
|
||||
description: "The active frank-small runbook — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode shape, the CUDA 13 nvcc toolchain the JIT needs, deployed flags, the client contract, and what to check when it misbehaves."
|
||||
title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
|
||||
description: "The RETIRED frank-small runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-small-llama-mimo-q6k]] the same night."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
@@ -17,9 +17,10 @@ timestamp: 2026-10-01
|
||||
|
||||
# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
|
||||
|
||||
> **ACTIVE since 2026-10-01.** Replaces [[frank-small-gemma4-12b]]
|
||||
> (llama.cpp Gemma4-12B QAT Q4_K_M, now the rollback target). Same port
|
||||
> (7771), same alias (`frank-small`), same client contract surface.
|
||||
> **RETIRED 2026-10-01, hours after cutover.** Replaced by
|
||||
> [[frank-small-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
|
||||
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
|
||||
> The venv and the INT8 payload stay on disk as the rollback.
|
||||
|
||||
## Identity
|
||||
|
||||
@@ -118,8 +119,11 @@ change lands.
|
||||
|
||||
Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
|
||||
files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
|
||||
all LFS sha256-verified by `scripts/download-frank-small.sh` (`--verify-only`
|
||||
re-checks). The serve script refuses to start with a missing shard.
|
||||
all LFS sha256-verified by `old/download-frank-small-sglang-int8.sh`
|
||||
(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
|
||||
cut over to llama.cpp + GGUF on 2026-10-01 night, and
|
||||
`scripts/download-frank-small.sh` now fetches the GGUF quant instead. The
|
||||
serve script refuses to start with a missing shard.
|
||||
|
||||
## Client contract
|
||||
|
||||
|
||||
@@ -10,12 +10,11 @@ headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, uti
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
runs two endpoints on the same arch —
|
||||
vLLM (digest-pinned kyuz0 TheRock gfx1151 image; frank-large,
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 with cross-model MTP
|
||||
spec decode from the Qwen3.5-9B head on the gfx1151 iGPU, 4 x 262144 ctx,
|
||||
port 7770) and SGLang (frank-small, the same model as an INT8 W8A16
|
||||
AutoRound conversion on the RTX 3090, 2 x 262144 ctx, port 7771).
|
||||
runs two endpoints — **gufo v0.1.1 (frank-large, port 7770: Qwen3.8-27B
|
||||
UD-Q4_K_XL + DFlash2, ONE session at 262,144, f16 KV on the gfx1151 iGPU,
|
||||
rolled back from the one-day vLLM MiMo experiment 2026-10-01 night)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
|
||||
GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -39,14 +38,14 @@ fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[vLLM TheRock gfx1151, digest-pinned
|
||||
E[gufo v0.1.1 built from source
|
||||
frank-large, port 7770]
|
||||
F[MiMo-V2.6-Distill-Qwen-9B BF16 + Qwen3.5-9B MTP head
|
||||
4 x 262144 ctx, spec 2 tok, qwen3/qwen3_coder parsers]
|
||||
G[SGLang venv cu130
|
||||
F[Qwen3.8-27B UD-Q4_K_XL + DFlash2 drafter
|
||||
1 x 262144 ctx, f16 KV, greedy, adaptive draft 7]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[MiMo-V2.6 INT8 W8A16 AutoRound
|
||||
2 x 262144 ctx, pinned pool 606208, in-checkpoint vision]
|
||||
H[MiMo-V2.6 Q6_K GGUF
|
||||
2 x 262144 ctx, f16 K + q8_0 V, flash-attn]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -65,7 +64,9 @@ frank-small, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-primary-glm53-flash-w4a16]] — the ACTIVE frank-primary runbook
|
||||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
|
||||
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
|
||||
@@ -162,23 +163,24 @@ frank-small, port 7771]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — ACTIVE frank-large runbook since
|
||||
2026-10-01 (root systemd unit, port 7770): the digest-pinned
|
||||
- [[frank-large-gufo-qwen38-27b]] — **the ACTIVE frank-large runbook
|
||||
again since 2026-10-01 night** (root systemd unit, port 7770): gufo
|
||||
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
|
||||
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
|
||||
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
|
||||
262,144 window, unquantized f16 KV. Rolled back by Frank after one day
|
||||
of [[frank-large-vllm-mimo-v26-bf16]]. The two from-source build
|
||||
workarounds, the client-contract delta, the measured prefill/decode
|
||||
trade against llama.cpp
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
|
||||
rollback by Frank after the autoresearch loop): the digest-pinned
|
||||
kyuz0/vllm-therock-gfx1151 image serving
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
|
||||
4 x 262144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B
|
||||
(the fleet's first; 71%/44% per-position acceptance measured) —
|
||||
the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
|
||||
MTP-vs-DFlash table (19.8 vs 17.4 t/s decode), the greedy-text numerics
|
||||
trap, the `reasoning` field change, troubleshooting
|
||||
- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-09-28 → 2026-10-01
|
||||
(replaced by [[frank-large-vllm-mimo-v26-bf16]]; rollback target via git
|
||||
tag frank-large-gufo-q4kxl-20261001, engine + payload + drafter still on
|
||||
the host): gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine,
|
||||
built from source on Dankclaw, serving Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL
|
||||
with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — the two
|
||||
from-source build workarounds, the client-contract delta, the measured
|
||||
prefill/decode trade against llama.cpp
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
|
||||
CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record:
|
||||
battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET-
|
||||
NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
|
||||
numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
|
||||
(quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
@@ -212,14 +214,22 @@ frank-small, port 7771]
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-small-sglang-mimo-v26]] — ACTIVE frank-small runbook since
|
||||
2026-10-01 (root systemd unit, port 7771): SGLang in a self-contained
|
||||
python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-
|
||||
AutoRound on the RTX 3090 — the same architecture frank-large now serves
|
||||
in BF16 on the iGPU; 2 x 262144 ctx on a pinned 606,208-token fp8 pool,
|
||||
NO spec decode (zero mtp.* tensors shipped), in-checkpoint vision.
|
||||
- [[frank-small-llama-mimo-q6k]] — **the ACTIVE frank-small runbook
|
||||
since 2026-10-01 night** (root systemd unit, port 7771): llama.cpp
|
||||
(master, CUDA sm_86 build) serving bartowski/MiMo-V2.6-Distill-
|
||||
Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO native 262,144-token
|
||||
sessions with ASYMMETRIC KV (K f16 unquantized + V q8_0, which forces
|
||||
flash-attn), ~20.9/24.6 GiB measured, text-only (the GGUF has no
|
||||
vision), reasoning_content, no spec decode by default
|
||||
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
|
||||
by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
|
||||
express; venv + INT8 payload on disk as rollback): SGLang in a
|
||||
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
|
||||
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
|
||||
83.9 tok/s decode
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||||
[[frank-small-sglang-mimo-v26]]; rollback target, payload and binary
|
||||
[[frank-small-sglang-mimo-v26]] → [[frank-small-llama-mimo-q6k]];
|
||||
rollback target, payload and binary intact on disk): mainline llama.cpp serving
|
||||
intact on disk): mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
|
||||
Reference in New Issue
Block a user