frank-small: SGLang MiMo-V2.6-Distill-Qwen-9B INT8 (qwen3_5 hybrid, RTX 3090)
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback). SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0 nvcc toolchain for the sgl-kernel/flashinfer JIT. Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba states pinned at 32 slots (the engine autofills the ssm pool to a fixed ~5.1 GB regardless of dtype), eager prefill (no headroom for graph capture), no spec decode (the checkpoint ships no mtp.* tensors despite mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB ghost at 0% acceptance). Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k requests, vision (in-checkpoint tower), reasoning_content field.
This commit is contained in:
1 parent
7da504a651
commit
0c5e054fb8
12 files changed
+1005
-328
No files matched your search
@@ -167,11 +167,12 @@ argv_value() {
|
||||
|
||||
@test "frank-large stays text-only" {
|
||||
# The payload has no mmproj, so an --mmproj would point at nothing. The
|
||||
# fleet's image path is frank-small; models.yml declares frank-large
|
||||
# text-only.
|
||||
# fleet's image path is frank-small: its SGLang MiMo payload carries the
|
||||
# vision tower inside the checkpoint (no projector file at all), and
|
||||
# models.yml declares frank-large text-only.
|
||||
serve_argv
|
||||
! argv_has --mmproj
|
||||
grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
grep -q 'MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound' "$REPO_ROOT/scripts/frank-small.env"
|
||||
}
|
||||
|
||||
@test "the engine is asked for greedy decoding and for preserved prior reasoning" {
|
||||
|
||||
Reference in new issue
Block a user