dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-small: SGLang MiMo-V2.6-Distill-Qwen-9B INT8 (qwen3_5 hybrid, RTX 3090)

Browse Source 
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback).
SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0
nvcc toolchain for the sgl-kernel/flashinfer JIT.

Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba
states pinned at 32 slots (the engine autofills the ssm pool to a fixed
~5.1 GB regardless of dtype), eager prefill (no headroom for graph
capture), no spec decode (the checkpoint ships no mtp.* tensors despite
mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB
ghost at 0% acceptance).

Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k
requests, vision (in-checkpoint tower), reasoning_content field.
This commit is contained in:
Frank
2026-10-01 06:55:48 -07:00
parent 7da504a651
commit 0c5e054fb8
12 changed files with 1005 additions and 328 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/index.md
+9 -3
View File
@@ -269,10 +269,16 @@ The LLM inference fleet on the two hosts. Full index:
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
2026-09-17: mainline llama.cpp serving
- [[frank-small-sglang-mimo-v26]] — the ACTIVE frank-small runbook since
2026-10-01: SGLang in a self-contained python3.12 venv serving
letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
4 x 151,552 slots, bf16 mamba states, eager prefill
- [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
(MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
(MTP assistant sidecar + ngram-mod, q8_0 KV) on
Dankclaw's RTX 3090, port 7771
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16