0 Fork 0

Files

T

Frank 71bc1735d3 docs(frank-zealot): re-key the exact-KV shape across the vault

AGENTS.md: topology row, the frank-zealot runbook section (shape, KV
accounting, MTP verdict, compaction-hub concurrency), and two Key
Lessons bullets -- the draft-context bill restated with the real knob
names, plus a new lesson that a quant's scale overhead belongs in its
byte rate (the 6.25 % invented by doubling 13,056.00 MiB).

LLM/frank-zealot-llama-prism-bonsai9-mtp.md: retitled to the live shape;
the 3-slot q8_0 boot ledger and its measurements are relabelled as the
arm this replaced (byte-identical numbers, honest attribution), the
corrected f16 rate derivation is added, the MTP census is labelled
q8_0-era, the fallback rungs are re-derived for the 524,288 pool, and the
boot-ledger section carries an explicit PROJECTION that must be replaced
with the measured ledger from the first f16 boot.

Pointers re-pointed (live-shape sentences only; every historical ledger
left byte-identical): LLM/index.md, vault index.md, Dankclaw/OpenWebUI.md,
and the retired-runbook "today" pointers in mimo-q5km, mimo-q6k,
qwen35-9b-q6k, swift15, sglang-mimo-v26. log.md gains the dated entry.

2026-10-18 21:02:42 -07:00

9.7 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

frank-zealot — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)

RETIRED 2026-10-01, hours after cutover (AGENTS.md Runbook Index). Replaced by frank-zealot-llama-mimo-q6k — llama.cpp CUDA on the MiMo Q6_K GGUF, chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express. :7771 today runs frank-zealot-llama-prism-bonsai9-mtp (llama.cpp PrismML fork build 10777, Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf, 2 × 262,144, exact f16 K + f16 V, MTP off by measurement). The venv (/home/dank/GitPackages/sglang-cu128/) is still on disk; the INT8 payload was staged to /home/dank/.AGENTTRASH/20261002-retired-llm-payloads/ on 2026-10-02 and that stage batch is gone (ls /home/dank/.AGENTTRASH/, 2026-10-18 — only the 20261005/06/07/10/17 batches remain), so this runtime needs old/download-frank-zealot-sglang-int8.sh to re-stage.

Identity

Endpoint: frank-small at 192.168.0.143:7771 (Dankclaw) — the id became frank-secondary on 2026-10-06 and frank-zealot on 2026-10-09.
Runtime: SGLang (latest) in a self-contained python3.12 venv at /home/dank/GitPackages/sglang-cu128/ (torch 2.13.0+cu130, transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root systemd unit scripts/frank-zealot.service (User=dank, LimitMEMLOCK=infinity) launched by scripts/serve-frank-zealot.sh with tunables from scripts/frank-zealot.env.
Model: letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).

Shape

Architecture qwen3_5 hybrid: 32 layers = 8 full-attention (4 KV heads x 256 head_dim, partial_rotary_factor: 0.25) + 24 Gated DeltaNet. max_position_embeddings: 262144, and a qwen3_5_vision tower (27 layers) in the safetensors — no separate mmproj file.
Context: two 262,144-token slots at the model's native window (--context-length 262144, --max-running-requests 2) against a pinned 606,208-token pool (--max-total-tokens) — 2026-10-01 reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack, and a compaction request (frank-archon's trigger 183,501 + a 32,768-token summary budget = 216,269) now fits one request. A single request may grow to the whole pool — a 170,019-token prompt was verified at the 4-slot shape — the grid is the two-way concurrency contract. That compaction arithmetic is specific to a 262,144-token slot: the shape that followed on 2026-10-02 ran three slots of 180,224, which 183,501 tokens already exceeds — see What three slots cost in frank-zealot-llama-mimo-q5km; :7771 today runs two 262,144-token conversations over one 524,288-position pool with exact f16 KV.
KV: fp8_e4m3 — 16 KiB/token (8 attn layers x 2 x 4 heads x 256 dim x 1 B), pool 9.31 GiB.
The memory shape (the 4.5 GiB the naive math misses): the DeltaNet ssm state pool is a first-class consumer. The engine autofills the mamba slot count to spend a fixed ~5.1 GB budget regardless of dtype (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are pinned: --mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32 = 795 MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and dies in capture, or later in the first long request once Triton's deferred kernel loads eat the remainder). Hence --disable-prefill-cuda-graph; decode graphs are untouched.
Quantization: --quantization auto-round — the checkpoint's quantization_config.json declares quant_method: "auto-round" and SGLang loads it natively (vision tower, embed_tokens and linear_attn.in_proj stay fp16).

Spec method

None — and this is a checkpoint fact, not a tuning choice. The text_config declares mtp_num_hidden_layers: 1, but the shipped safetensors carry zero mtp.* tensors: verified 2026-10-01 against model.safetensors.index.json for both this INT8 conversion and the base XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (32 layers, indices 0–31, no mtp./nextn./layers.32 keys in either). With NEXTN enabled, SGLang boots a 4.24 GiB draft model (Qwen3_5ForCausalLMMTP) from nothing and decodes at accept len: 1.00, accept rate: 0.00 — every proposal rejected, paying the 4-token verify tax for no token. The serve script passes no --speculative-* flag, and the seam gates that.

CUDA 13 toolchain (the sharp edge)

The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack, flashinfer) compiles with a CUDA 13.0 nvcc extracted from debs into the venv at cuda-nvcc/usr/local/cuda-13.0/ — the system nvcc is 12.4 (no --compress-mode) and its headers clash with the 13.0 device passes.

Components (all from the ubuntu2404 CUDA repo, dpkg -x'd, not installed):

Package	Supplies
`cuda-nvcc-13-0` (13.0.88)	`bin/nvcc`, `ptxas`, `cudafe++`, …
`libnvvm-13-0` (13.0.48)	`nvvm/bin/cicc` (the device compiler)
`cuda-cudart-dev-13-0` (13.0.88)	`include/cuda_runtime.h`, `lib64/stubs`
`cuda-crt-13-0` (13.0.88)	`include/crt/host_runtime.h` (2-arg `__cudaLaunch`)
`libnvptxcompiler-13-0` (13.0.88)	static lib + `nvPTXCompiler.h`
`cuda-cccl-13-0` (13.0.50)	`cccl/cuda/...` headers (libcudacxx)

Three local modifications the tree carries:

bin/nvcc is a shim over bin/nvcc.real adding -allow-unsupported-compiler (host GCC 16 > nvcc's max 13).
targets/x86_64-linux/include and lib are symlinks to ../../../include / ../../../lib64 (the deb ships them as real near-empty dirs; the stub pass must see the 13.0 headers, not /usr/include 12.4).
include/cuda → cccl/cuda symlink (CCCL 3.x layout; flashinfer includes <cuda/functional>), and crt/math_functions.h patched with noexcept on rsqrt/rsqrtf (glibc 2.42 declares them noexcept(true); the unpatched 13.0 declarations are incompatible in the C++ host stub).

JIT caches: ~/.cache/sglang/jit/sm86/ and ~/.cache/sglang/.cache/flashinfer/0.6.18/86/ — clear both when a toolchain change lands.

Payload

Eight safetensors shards (12,891,718,440 B total) + tokenizer and config files lived under /home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/ (absent from the 2026-10-18 census — see the banner), all LFS sha256-verified by old/download-frank-zealot-sglang-int8.sh (--verify-only re-checks) — the script moved to old/ when the endpoint cut over to llama.cpp + GGUF on 2026-10-01 night, and scripts/download-frank-zealot.sh now fetches the GGUF quant instead. The serve script refuses to start with a missing shard.

Client contract

Alias: frank-small (the id at /v1/models).
Reasoning: extracted to reasoning_content (live-verified 2026-10-01 — this SGLang build's qwen3 parser uses the same field name as llama.cpp; models.yml sets reasoningContentField: "reasoning_content"). --reasoning-parser qwen3.
Tools: --tool-call-parser qwen3_coder.
Vision: supported (in-checkpoint tower).
Sampling: SGLang has NO server-wide --temperature/--top-p/ --top-k — sampling rides on the request; the harness sends the Qwen recipe (temp 0.7, top_p 0.8) via models.yml compat.extraBody.
Reasoning effort: the full OpenAI set is accepted (the chat template has no effort gate).

Measured

Boot ~60 s with warm JIT caches (several minutes cold — the first boot compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain above is what makes that compile succeed).
Decode 83.9 tok/s (no spec decode — nothing to accept).
Prefill ~2,050–3,370 t/s eager (4-slot shape: a 170,019-token prompt in 50.5 s, ~3,370 t/s; 2-slot shape: a 200,000-token prompt in 97.3 s wall incl. 8 decode tokens, ~2,056 t/s).
4-slot shape: 4 concurrent 54,027-token requests all answered in ~17 s (216k tokens of live KV at once). 2-slot shape: two concurrent 50,000-token requests both answered (28.8 s wall).
Compaction fit (the reslice's purpose): a 200,000-token prompt — well past frank-archon's 183,501-token compaction trigger (70% of its 262,144 window) — prefills and completes in ONE request.
Vision: a 64x64 red square answers "Red" with image_tokens: 64.

Rollback

frank-zealot-llama-mimo-q5km was the live shape two steps forward; the practical rollback at the time was frank-zealot-llama-mimo-q6k (payload at /home/dank/models/MiMo-Q6K/, still on disk 2026-10-18). This era's own rollback, the llama.cpp Gemma4-12B QAT payload, was staged to /home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/ on 2026-10-02; its unit/env/serve trio remains in old/.

Citations

Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
SGLang: https://github.com/sgl-project/sglang
Serve script: scripts/serve-frank-zealot.sh; tunables: scripts/frank-zealot.env

9.7 KiB Raw Blame History Unescape Escape