frank-small: SGLang MiMo-V2.6-Distill-Qwen-9B INT8 (qwen3_5 hybrid, RTX 3090)
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback). SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0 nvcc toolchain for the sgl-kernel/flashinfer JIT. Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba states pinned at 32 slots (the engine autofills the ssm pool to a fixed ~5.1 GB regardless of dtype), eager prefill (no headroom for graph capture), no spec decode (the checkpoint ships no mtp.* tensors despite mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB ghost at 0% acceptance). Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k requests, vision (in-checkpoint tower), reasoning_content field.
This commit is contained in:
Frank committed
1 parent
7da504a651
commit
0c5e054fb8
12 files changed
+1005
-328
No files matched your search
@@ -13,7 +13,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
|
||||
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
|
||||
the gfx1151 iGPU) and `frank-small` (SGLang on the RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
(Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
|
||||
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **4 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 8,311,013,376/rank, util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Five full-length contexts BOOT but leave ~0 bytes free at runtime — a real client request (995-token prompt, `max_tokens=261149`) killed the worker with `torch.OutOfMemoryError ... 0 bytes free` — so four slots carry the deployed shape and its VRAM headroom. Six do not fit at all (6×241664 asked 10.70 GiB vs 9.18 GiB free). fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
|
||||
| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
|
||||
| Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 4 slots of 151,552, bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -243,7 +243,7 @@ merely wedged kernel usually still answers ARP.
|
||||
1. Stop and mask (self-healing would restart it):
|
||||
- Dankclaw `frank-large` (root unit, gufo on the iGPU):
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
|
||||
- Dankclaw `frank-small` (root unit, SGLang on the 3090):
|
||||
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
|
||||
- Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
|
||||
`sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
|
||||
@@ -277,19 +277,20 @@ When renaming a service, retiring an endpoint, or moving a model: update the
|
||||
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-09-26 both `frank-small` and `frank-primary` accept images
|
||||
of 2026-10-01 both `frank-small` and `frank-primary` accept images
|
||||
(`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly
|
||||
(this build never autoloads projectors); `frank-large` remains text-only
|
||||
because its stock llama.cpp deployment loads no
|
||||
vision projector (the mmproj is not loaded). Note also that
|
||||
the reasoning channel differs by runtime: llama.cpp endpoints emit
|
||||
`reasoning_content`, vLLM emits `reasoning` — academic since the
|
||||
2026-09-26 swap, as every endpoint in the fleet is llama.cpp now.
|
||||
(this build never autoloads projectors), and frank-small's MiMo payload
|
||||
bakes its vision tower INTO the checkpoint (no projector file at all);
|
||||
`frank-large` remains text-only because its gufo deployment loads no
|
||||
vision projector. Note also that
|
||||
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
|
||||
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
|
||||
models.yml sets `reasoningContentField` per endpoint to match.
|
||||
There is **no audio**
|
||||
input anywhere in the fleet except frank-small: that capability left with the
|
||||
FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
|
||||
projector's audio encoder (frank-small transcribes WAV verbatim even though
|
||||
`audio` is omitted from its `input` list — OMP's schema rejects it).
|
||||
input anywhere in the fleet: that capability left with the FLM NPU
|
||||
endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
|
||||
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
|
||||
(the MiMo tower is vision-only).
|
||||
|
||||
### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE)
|
||||
|
||||
@@ -397,31 +398,46 @@ no longer a one-minute flip. **Engine** —
|
||||
Qwen-Sharp back. The DFlash2 drafter was never reclaimed and still lives in
|
||||
`Qwen3.8-27B-UD-Q8_K_XL/`.
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
### frank-small — SGLang on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
Mainline llama.cpp master `37b53fd` (CUDA build with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
|
||||
`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
|
||||
serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
|
||||
The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
|
||||
nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
|
||||
matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
|
||||
port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
|
||||
(four FULL 262,144-token slots — the measured card ceiling), batch
|
||||
**8192/1024**, greedy default. Deployed flags are in
|
||||
`scripts/frank-small.env`; the serve script refuses to start if the
|
||||
weights, the projector, the sidecar or the binary are missing.
|
||||
SGLang (latest) in a **self-contained python3.12 venv**
|
||||
(`/home/dank/GitPackages/sglang-cu128/`, torch 2.13.0+cu130) as a **root
|
||||
systemd unit** (repo `scripts/frank-small.service`, `relink_unit`ed into
|
||||
`/etc/systemd/system/`), serving
|
||||
`letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound` (9B dense
|
||||
qwen3_5 hybrid — 8 full-attention + 24 Gated DeltaNet layers — INT8 W8A16
|
||||
AutoRound, 12.0 GiB, vision tower baked into the checkpoint). Port 7771.
|
||||
|
||||
Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
|
||||
chat completion returns content plus `reasoning_content`; a tool call
|
||||
returns `tool_calls`; an image request is answered from the projector;
|
||||
an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
|
||||
max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
|
||||
acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
|
||||
ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
|
||||
100 is WORSE at long ctx) is tabled in the runbook.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
|
||||
Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
**The shape is memory-pinned, and the sharp edge is the mamba pool.** The
|
||||
DeltaNet ssm state pool autofills to a fixed ~5.1 GB budget *regardless of
|
||||
dtype* (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
|
||||
pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` (795 MiB).
|
||||
The KV pool is then **pinned at 606,208 tokens** (`--max-total-tokens`) —
|
||||
the engine's autofit picks 687,388 and leaves 0.53 GB, which Triton's
|
||||
deferred kernel loads eat and the first long request OOMs on. Four
|
||||
151,552-token slots; a single request may grow to the whole pool (170k
|
||||
verified). Prefill runs **eager** (`--disable-prefill-cuda-graph`) — there
|
||||
is no headroom for capture; decode graphs are on.
|
||||
|
||||
**No speculative decode — a checkpoint fact.** `text_config` declares
|
||||
`mtp_num_hidden_layers: 1`, but the shipped safetensors (base BF16 and this
|
||||
INT8 conversion alike) carry zero `mtp.*` tensors; NEXTN would draft with
|
||||
an uninitialised 4.24 GiB ghost model at 0% acceptance. The seam gates
|
||||
that no `--speculative-*` flag reaches the engine.
|
||||
|
||||
**The venv carries a CUDA 13.0 nvcc extracted from debs** (system nvcc is
|
||||
12.4) so the sgl-kernel/flashinfer JIT compiles for sm86; three local
|
||||
patches ride in the tree (GCC-16 shim, CCCL symlink, rsqrt noexcept).
|
||||
Details in the runbook.
|
||||
|
||||
Verified at deploy (2026-10-01): `/v1/models` reports `frank-small`;
|
||||
chat returns content plus `reasoning_content` (SGLang's qwen3 parser uses
|
||||
llama.cpp's field name); a red-square image answers "Red"
|
||||
(`image_tokens: 64`); 4 concurrent 54k-token requests all answer; a
|
||||
170,019-token prompt prefills in 50.5 s (~3,370 t/s eager). Decode
|
||||
**83.9 tok/s**.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md`.
|
||||
Rollback: [[frank-small-gemma4-12b]] (payload and binary intact on disk).
|
||||
|
||||
### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
|
||||
|
||||
@@ -803,7 +819,7 @@ Active:
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-small (SGLang MiMo-V2.6 INT8, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md` |
|
||||
| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
@@ -826,6 +842,7 @@ Retired (rollback targets and history):
|
||||
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
||||
| vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
|
||||
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 10-01 | `frank-small-gemma4-12b.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
|
||||
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
|
||||
|
||||
Reference in new issue
Block a user