rename frank-small → frank-secondary (Dankclaw llama.cpp :7771)
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
This commit is contained in:
Frank committed
1 parent
c71a3066c0
commit
87721ab0fe
87 files changed
+460
-460
No files matched your search
@@ -0,0 +1,180 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-secondary on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
|
||||
description: "The RETIRED frank-secondary runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-secondary-llama-mimo-q6k]] the same night."
|
||||
tags:
|
||||
- llm
|
||||
- frank-secondary
|
||||
- dankclaw
|
||||
- sglang
|
||||
- rtx3090
|
||||
- qwen
|
||||
- mtp
|
||||
- vision
|
||||
- int8
|
||||
timestamp: 2026-10-01
|
||||
---
|
||||
|
||||
# frank-secondary — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
|
||||
|
||||
> **RETIRED 2026-10-01, hours after cutover.** Replaced by
|
||||
> [[frank-secondary-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
|
||||
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express.
|
||||
> The venv (`/home/dank/GitPackages/sglang-cu128/`) stays in place; the INT8
|
||||
> payload was staged to
|
||||
> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 when
|
||||
> the llama.cpp endpoint was two runtimes deep — recoverable until the trash
|
||||
> is purged.
|
||||
|
||||
## Identity
|
||||
|
||||
- **Endpoint:** `frank-secondary` at `192.168.0.143:7771` (Dankclaw).
|
||||
- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
|
||||
`/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
|
||||
transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
|
||||
systemd unit `scripts/frank-secondary.service` (`User=dank`,
|
||||
`LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-secondary.sh` with
|
||||
tunables from `scripts/frank-secondary.env`.
|
||||
- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
|
||||
(9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
|
||||
|
||||
## Shape
|
||||
|
||||
- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
|
||||
(4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
|
||||
DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
|
||||
tower (27 layers) in the safetensors — no separate mmproj file.
|
||||
- **Context:** two 262,144-token slots at the model's native window
|
||||
(`--context-length 262144`, `--max-running-requests 2`) against a
|
||||
**pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
|
||||
reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
|
||||
~82k tokens of pool slack, and a compaction request (frank-primary's
|
||||
trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
|
||||
one request. A single request may grow to the whole pool — a
|
||||
170,019-token prompt was verified at the 4-slot shape — the grid is
|
||||
the two-way concurrency contract. **That compaction arithmetic is
|
||||
specific to a 262,144-token slot**: the ACTIVE shape runs three slots of
|
||||
180,224, which 183,501 tokens already exceeds — see *What three slots
|
||||
cost* in [[frank-secondary-llama-mimo-q5km]].
|
||||
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
|
||||
dim x 1 B), pool 9.31 GiB.
|
||||
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
|
||||
ssm state pool is a first-class consumer. The engine autofills the
|
||||
mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
|
||||
(fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
|
||||
pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
|
||||
MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
|
||||
leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
|
||||
CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
|
||||
dies in capture, or later in the first long request once Triton's
|
||||
deferred kernel loads eat the remainder). Hence
|
||||
`--disable-prefill-cuda-graph`; decode graphs are untouched.
|
||||
- **Quantization:** `--quantization auto-round` — the checkpoint's
|
||||
`quantization_config.json` declares `quant_method: "auto-round"` and
|
||||
SGLang loads it natively (vision tower, `embed_tokens` and
|
||||
`linear_attn.in_proj` stay fp16).
|
||||
|
||||
## Spec method
|
||||
|
||||
**None — and this is a checkpoint fact, not a tuning choice.** The
|
||||
`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
|
||||
safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
|
||||
`model.safetensors.index.json` for both this INT8 conversion and the base
|
||||
`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
|
||||
`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
|
||||
boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
|
||||
decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
|
||||
paying the 4-token verify tax for no token. The serve script passes no
|
||||
`--speculative-*` flag, and the seam gates that.
|
||||
|
||||
## CUDA 13 toolchain (the sharp edge)
|
||||
|
||||
The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
|
||||
flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
|
||||
venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
|
||||
`--compress-mode`) and its headers clash with the 13.0 device passes.
|
||||
|
||||
Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):
|
||||
|
||||
| Package | Supplies |
|
||||
|---|---|
|
||||
| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
|
||||
| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
|
||||
| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
|
||||
| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
|
||||
| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
|
||||
| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |
|
||||
|
||||
Three local modifications the tree carries:
|
||||
|
||||
1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
|
||||
`-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
|
||||
2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
|
||||
`../../../include` / `../../../lib64` (the deb ships them as real
|
||||
near-empty dirs; the stub pass must see the 13.0 headers, not
|
||||
`/usr/include` 12.4).
|
||||
3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
|
||||
includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
|
||||
`noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
|
||||
the unpatched 13.0 declarations are incompatible in the C++ host stub).
|
||||
|
||||
JIT caches: `~/.cache/sglang/jit/sm86/` and
|
||||
`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
|
||||
change lands.
|
||||
|
||||
## Payload
|
||||
|
||||
Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
|
||||
files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
|
||||
all LFS sha256-verified by `old/download-frank-secondary-sglang-int8.sh`
|
||||
(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
|
||||
cut over to llama.cpp + GGUF on 2026-10-01 night, and
|
||||
`scripts/download-frank-secondary.sh` now fetches the GGUF quant instead. The
|
||||
serve script refuses to start with a missing shard.
|
||||
|
||||
## Client contract
|
||||
|
||||
- **Alias:** `frank-secondary` (the `id` at `/v1/models`).
|
||||
- **Reasoning:** extracted to **`reasoning_content`** (live-verified
|
||||
2026-10-01 — this SGLang build's qwen3 parser uses the same field name
|
||||
as llama.cpp; models.yml sets `reasoningContentField:
|
||||
"reasoning_content"`). `--reasoning-parser qwen3`.
|
||||
- **Tools:** `--tool-call-parser qwen3_coder`.
|
||||
- **Vision:** supported (in-checkpoint tower).
|
||||
- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
|
||||
`--top-k` — sampling rides on the request; the harness sends the Qwen
|
||||
recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
|
||||
- **Reasoning effort:** the full OpenAI set is accepted (the chat template
|
||||
has no effort gate).
|
||||
|
||||
## Measured
|
||||
|
||||
- Boot ~60 s with warm JIT caches (several minutes cold — the first boot
|
||||
compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
|
||||
above is what makes that compile succeed).
|
||||
- Decode **83.9 tok/s** (no spec decode — nothing to accept).
|
||||
- Prefill ~2,050–3,370 t/s eager (4-slot shape: a 170,019-token prompt in
|
||||
50.5 s, ~3,370 t/s; 2-slot shape: a 200,000-token prompt in 97.3 s wall
|
||||
incl. 8 decode tokens, ~2,056 t/s).
|
||||
- 4-slot shape: 4 concurrent 54,027-token requests all answered in ~17 s
|
||||
(216k tokens of live KV at once). 2-slot shape: two concurrent
|
||||
50,000-token requests both answered (28.8 s wall).
|
||||
- **Compaction fit (the reslice's purpose):** a 200,000-token prompt —
|
||||
well past frank-primary's 183,501-token compaction trigger (70% of its
|
||||
262,144 window) — prefills and completes in ONE request.
|
||||
- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
|
||||
|
||||
## Rollback
|
||||
|
||||
[[frank-secondary-llama-mimo-q5km]] is the live shape two steps forward; the
|
||||
practical rollback today is [[frank-secondary-llama-mimo-q6k]] (payload at
|
||||
`/home/dank/models/MiMo-Q6K/`). This era's own rollback, the llama.cpp
|
||||
Gemma4-12B QAT payload, was staged to
|
||||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
|
||||
2026-10-02; its unit/env/serve trio remains in `old/`.
|
||||
|
||||
# Citations
|
||||
|
||||
- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
|
||||
- SGLang: https://github.com/sgl-project/sglang
|
||||
- Serve script: `scripts/serve-frank-secondary.sh`; tunables: `scripts/frank-secondary.env`
|
||||
Reference in new issue
Block a user