frank-small: SGLang MiMo-V2.6-Distill-Qwen-9B INT8 (qwen3_5 hybrid, RTX 3090)

Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback). SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0 nvcc toolchain for the sgl-kernel/flashinfer JIT. Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba states pinned at 32 slots (the engine autofills the ssm pool to a fixed ~5.1 GB regardless of dtype), eager prefill (no headroom for graph capture), no spec decode (the checkpoint ships no mtp.* tensors despite mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB ghost at 0% acceptance). Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k requests, vision (in-checkpoint tower), reasoning_content field.
2026-10-01 06:55:48 -07:00
1 parent 7da504a651
commit 0c5e054fb8
@@ -13,7 +13,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
  via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on
-  the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
+  the gfx1151 iGPU) and `frank-small` (SGLang on the RTX 3090). Caddy on
  Dankclaw reverse-proxies both hosts.
 - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
  (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 |--------|---------|---------|-------|------|
 | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **4 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 8,311,013,376/rank, util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Five full-length contexts BOOT but leave ~0 bytes free at runtime — a real client request (995-token prompt, `max_tokens=261149`) killed the worker with `torch.OutOfMemoryError ... 0 bytes free` — so four slots carry the deployed shape and its VRAM headroom. Six do not fit at all (6×241664 asked 10.70 GiB vs 9.18 GiB free). fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
 | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
-| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
+| Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 4 slots of 151,552, bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 |
 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -243,7 +243,7 @@ merely wedged kernel usually still answers ARP.
 1. Stop and mask (self-healing would restart it):
   - Dankclaw `frank-large` (root unit, gufo on the iGPU):
     `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
-   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
+   - Dankclaw `frank-small` (root unit, SGLang on the 3090):
     `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
   - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700):
     `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
@@ -277,19 +277,20 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-09-26 both `frank-small` and `frank-primary` accept images
+of 2026-10-01 both `frank-small` and `frank-primary` accept images
 (`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly
-(this build never autoloads projectors); `frank-large` remains text-only
+(this build never autoloads projectors), and frank-small's MiMo payload
-because its stock llama.cpp deployment loads no
+bakes its vision tower INTO the checkpoint (no projector file at all);
-vision projector (the mmproj is not loaded). Note also that
+`frank-large` remains text-only because its gufo deployment loads no
-the reasoning channel differs by runtime: llama.cpp endpoints emit
+vision projector. Note also that
-`reasoning_content`, vLLM emits `reasoning` — academic since the
+the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
-2026-09-26 swap, as every endpoint in the fleet is llama.cpp now.
+parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
 models.yml sets `reasoningContentField` per endpoint to match.
 There is **no audio**
-input anywhere in the fleet except frank-small: that capability left with the
+input anywhere in the fleet: that capability left with the FLM NPU
-FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4
+endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
-projector's audio encoder (frank-small transcribes WAV verbatim even though
+audio encoder, and left again with the Gemma4 retirement on 2026-10-01
-`audio` is omitted from its `input` list — OMP's schema rejects it).
+(the MiMo tower is vision-only).
 ### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE)
@@ -397,31 +398,46 @@ no longer a one-minute flip. **Engine** —
 Qwen-Sharp back. The DFlash2 drafter was never reclaimed and still lives in
 `Qwen3.8-27B-UD-Q8_K_XL/`.
-### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
+### frank-small — SGLang on the RTX 3090 (Dankclaw, ACTIVE)
-Mainline llama.cpp master `37b53fd` (CUDA build with
+SGLang (latest) in a **self-contained python3.12 venv**
-`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
+(`/home/dank/GitPackages/sglang-cu128/`, torch 2.13.0+cu130) as a **root
-`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
+systemd unit** (repo `scripts/frank-small.service`, `relink_unit`ed into
-serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
+`/etc/systemd/system/`), serving
-The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
+`letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound` (9B dense
-nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
+qwen3_5 hybrid — 8 full-attention + 24 Gated DeltaNet layers — INT8 W8A16
-matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
+AutoRound, 12.0 GiB, vision tower baked into the checkpoint). Port 7771.
 port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
 (four FULL 262,144-token slots — the measured card ceiling), batch
 **8192/1024**, greedy default. Deployed flags are in
 `scripts/frank-small.env`; the serve script refuses to start if the
 weights, the projector, the sidecar or the binary are missing.
-Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
+**The shape is memory-pinned, and the sharp edge is the mamba pool.** The
-chat completion returns content plus `reasoning_content`; a tool call
+DeltaNet ssm state pool autofills to a fixed ~5.1 GB budget *regardless of
-returns `tool_calls`; an image request is answered from the projector;
+dtype* (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
-an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
+pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` (795 MiB).
-max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
+The KV pool is then **pinned at 606,208 tokens** (`--max-total-tokens`) —
-acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
+the engine's autofit picks 687,388 and leaves 0.53 GB, which Triton's
-ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
+deferred kernel loads eat and the first long request OOMs on. Four
-100 is WORSE at long ctx) is tabled in the runbook.
+151,552-token slots; a single request may grow to the whole pool (170k
-Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
+verified). Prefill runs **eager** (`--disable-prefill-cuda-graph`) — there
-Rollback: [[frank-small-qwen38-gsq-rco]].
+is no headroom for capture; decode graphs are on.
 **No speculative decode — a checkpoint fact.** `text_config` declares
 `mtp_num_hidden_layers: 1`, but the shipped safetensors (base BF16 and this
 INT8 conversion alike) carry zero `mtp.*` tensors; NEXTN would draft with
 an uninitialised 4.24 GiB ghost model at 0% acceptance. The seam gates
 that no `--speculative-*` flag reaches the engine.
 **The venv carries a CUDA 13.0 nvcc extracted from debs** (system nvcc is
 12.4) so the sgl-kernel/flashinfer JIT compiles for sm86; three local
 patches ride in the tree (GCC-16 shim, CCCL symlink, rsqrt noexcept).
 Details in the runbook.
 Verified at deploy (2026-10-01): `/v1/models` reports `frank-small`;
 chat returns content plus `reasoning_content` (SGLang's qwen3 parser uses
 llama.cpp's field name); a red-square image answers "Red"
 (`image_tokens: 64`); 4 concurrent 54k-token requests all answer; a
 170,019-token prompt prefills in 50.5 s (~3,370 t/s eager). Decode
 **83.9 tok/s**.
 Runbook: `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md`.
 Rollback: [[frank-small-gemma4-12b]] (payload and binary intact on disk).
 ### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
@@ -803,7 +819,7 @@ Active:
 | Service | Runbook |
 |---------|---------|
 | frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` |
-| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
+| frank-small (SGLang MiMo-V2.6 INT8, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md` |
 | frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -826,6 +842,7 @@ Retired (rollback targets and history):
 | Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
 | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
 | vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` |
 | llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 10-01 | `frank-small-gemma4-12b.md` |
 | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
 | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
 | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
@@ -1177,23 +1177,39 @@ deploy_llm_large() {
    wait_for_health frank-large http://127.0.0.1:7770/health 1200
 }
-# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a
+# deploy_llm_small — frank-small is SGLang (python3.12 venv) on the RTX 3090,
-# ROOT unit: the runtime is a native CUDA binary, so there is no container
+# as a ROOT unit: the runtime is a self-contained venv, so there is no
-# image to install and no generated unit to avoid. relink_unit points
+# container image to install and no generated unit to avoid. relink_unit
-# /etc/systemd/system/frank-small.service at the repo file and boot persistence
+# points /etc/systemd/system/frank-small.service at the repo file and boot
-# is that unit's [Install] section.
+# persistence is that unit's [Install] section.
 #
 # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
 # not shared: frank-large's engine runs on the gfx1151 iGPU.
-LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
+LLM_SMALL_MODEL_DIR="${LLM_SMALL_MODEL_DIR:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
 LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/sglang-cu128/bin/sglang}"
 deploy_llm_small() {
-    if [[ ! -f "$LLM_SMALL_MODEL" ]]; then
+    # Refuse rather than restart-loop: a unit whose runtime or payload is
-        echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2
+    # missing never becomes healthy, and a green deploy would leave the
-        echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2
+    # endpoint dark with nothing in the log to say why.
    if [[ ! -x "$LLM_SMALL_RUNTIME" ]]; then
        echo "ERROR: SGLang runtime absent at $LLM_SMALL_RUNTIME" >&2
        echo "frank-small: build the venv (python3.12 -m venv /home/dank/GitPackages/sglang-cu128" >&2
        echo "frank-small: and pip install 'sglang[all]' into it) first." >&2
        return 1
    fi
    if [[ ! -f "$LLM_SMALL_MODEL_DIR/model.safetensors.index.json" ]]; then
        echo "ERROR: payload index absent at $LLM_SMALL_MODEL_DIR" >&2
        echo "frank-small: run scripts/download-frank-small.sh first;" >&2
        echo "frank-small: refusing to start a unit that cannot load." >&2
        return 1
    fi
    local count
    count="$(find "$LLM_SMALL_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)"
    if [[ "$count" -ne 8 ]]; then
        echo "ERROR: payload shard count wrong: $LLM_SMALL_MODEL_DIR (found $count, want 8)" >&2
        return 1
    fi
    relink_unit frank-small
    run_privileged systemctl daemon-reload
    run_privileged systemctl enable frank-small.service
@@ -0,0 +1,93 @@
 #!/usr/bin/env bash
 # Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-small
 # (SGLang, RTX 3090). Public repo — no auth header on the command line.
 #
 # Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
 # oversized file on a 31 GiB payload; a fresh -f fetch plus size+sha256 is
 # the house rule.
 #
 # Usage:
 #   scripts/download-frank-small.sh [DEST_DIR] [--verify-only]
 #
 # LFS shards are verified against the Hub LFS sha256; non-LFS text files
 # against their declared byte size. Idempotent: complete files are skipped.
 set -euo pipefail
 REPO="letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound"
 BASE="https://huggingface.co/${REPO}/resolve/main"
 DEST=""
 VERIFY_ONLY=0
 for arg in "$@"; do
    case "$arg" in
        --verify-only) VERIFY_ONLY=1 ;;
        *) DEST="$arg" ;;
    esac
 done
 DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
 # name bytes sha256(- = non-LFS, size-only check)
 FILES=(
  "model-00001-of-00008.safetensors 2946296312 ddb6117c04cb6feaf7c265e5d75efadf21949037d73253a8e4521d40e164366d"
  "model-00002-of-00008.safetensors 1402134792 2dc78d55d1d33d7b3b21dd6f8b4b817b62c2fb1376e6448b322428798c10e257"
  "model-00003-of-00008.safetensors 1427365904 5c797d14e345309d67c78306b9de4aa01a21dbd92dd98e40b0ad69b2900e9203"
  "model-00004-of-00008.safetensors 1420041856 7402f852e2c7a24ebf4694b98a220e71dfb2c93ba3776272cf2acfc65acd8126"
  "model-00005-of-00008.safetensors 1410655648 991cbf2eda2ff91fc3ffbb15591d7fcac340fb10d9c33bd3da20010e2f3f4090"
  "model-00006-of-00008.safetensors 1427627856 efc42448054cfb5e3061e154ba4539204088ce95176d689a95b58de885ef7eeb"
  "model-00007-of-00008.safetensors 2035442408 75a101fc71ae6f2e1f5c4816f711d62cf1d7a0e02ef6e5182361b8c11e72b626"
  "model-00008-of-00008.safetensors 822153664 0fb90cb50e2d336fbad58bd4c01880eeddbf537c1dca1cf0ae291195d5efbb4b"
  "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
  "model.safetensors.index.json 106278 -"
  "config.json 4608 -"
  "generation_config.json 161 -"
  "quantization_config.json 1654 -"
  "chat_template.jinja 3916 -"
  "tokenizer_config.json 1168 -"
  "preprocessor_config.json 443 -"
  "processor_config.json 1191 -"
  "README.md 8448 -"
 )
 mkdir -p "$DEST"
 status=0
 for entry in "${FILES[@]}"; do
  read -r name size sha <<<"$entry"
  f="$DEST/$name"
  have=0
  if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
    if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
      have=1
    fi
  fi
  if (( have )); then
    echo "OK        $name"
    continue
  fi
  if (( VERIFY_ONLY )); then
    echo "MISSING   $name"
    status=1
    continue
  fi
  echo "FETCHING  $name"
  tmp="$f.part"
  # no -C resume; -f + --fail-with-body so an HTTP error is not a success
  curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name"
  mv "$tmp" "$f"
  if [[ "$(stat -c%s "$f")" != "$size" ]]; then
    echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)"
    status=1
    continue
  fi
  if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s  %s\n' "$sha" "$f"); then
    echo "SHA FAIL  $name"
    status=1
    continue
  fi
  echo "VERIFIED  $name"
 done
 if (( status == 0 )); then
  echo "DOWNLOAD_DONE $DEST"
 else
  echo "DOWNLOAD_FAILED $DEST" >&2
 fi
 exit "$status"
@@ -1,46 +1,36 @@
-# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
+# Tunables for frank-small (letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-
-# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
+# W8A16-AutoRound, qwen3_5 hybrid, in-checkpoint vision) on SGLang,
-# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
+# Dankclaw's RTX 3090, since 2026-10-01.
 #
-# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
+# Runtime: SGLang in a python3.12 venv (pip "sglang[all]"):
-#   cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
+#   python3.12 -m venv /home/dank/GitPackages/sglang-cu128
-#     -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
+#   /home/dank/GitPackages/sglang-cu128/bin/pip install "sglang[all]"
-#   cmake --build build-cuda --target llama-server -j
+# The venv carries its own CUDA torch + sgl-kernel + flashinfer; it is
-# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
+# self-contained and does not touch the host Python (3.14) or the ROCm
-# attention, and without the flag quantized-KV prefill collapses ~12x on
+# stack that serves frank-large.
 # CUDA (see the CUDA lesson in AGENTS.md).
 #
-# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
+# Model: MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound — a Qwen3.5-9B
-# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
+# distillation at INT8 W8A16 (AutoRound, group 128, symmetric). 8
-# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
+# safetensors shards, 12.0 GiB total. The vision tower
-# per-layer token embeddings (PLE) inside the file, and a sliding-window
+# (qwen3_5_vision, 27 layers) and embed_tokens stay fp16 inside the
-# attention pattern (attention.sliding_window = 1024) -- most layers cache
+# checkpoint, so the image path needs no separate projector file.
-# only a 1024-position window, which is why the KV pool is a fraction of
+# Architecture: 32 layers, 8 full-attention (4 KV heads x 256 head_dim)
-# the full-ctx x all-layers number.
+# + 24 Gated DeltaNet, native 262144 window. text_config declares
 # mtp_num_hidden_layers = 1, but the shipped safetensors carry NO mtp.*
 # tensors (verified against model.safetensors.index.json 2026-10-01 —
 # base BF16 and this INT8 conversion both omit the head), so there is no
 # spec decode: SGLang would draft with an uninitialised 4.24 GiB ghost
 # model at 0% acceptance and pay the verify tax for nothing.
 #
-# MTP: the Multi-Token Prediction head ships as a SEPARATE file
+# SIZING (RTX 3090), measured 2026-10-01: the 0.98 budget holds the
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
+# checkpoint (11,354 MiB) + the DeltaNet ssm state pool (795 MiB at 32
-# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
+# bf16 slots) + the KV pool. The engine's autofit picks a 687,388-token
-# is passed via --model-draft with --spec-type draft-mtp, so the draft
+# pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads
-# context loads the assistant model. The same --spec-type value chains the
+# take most of it after boot, and eager-prefill activations then OOM on
-# draftless ngram-mod matcher, which attaches unconditionally; two
+# the first long request. The pool is therefore PINNED at 606,208 tokens
-# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
+# (4 x 151,552), which leaves ~0.8 GB for the prefill workspace; a 170k
-# is the only legal shape that carries both.
+# single request and 4 concurrent 54k requests are verified against it.
-#
+# The ssm pool is the 4.5 GiB the weights-only arithmetic misses.
 # SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
 # FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
 # MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
 # 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
 # card, and the KV pool allocates upfront -- so 4 is the max. Explicit
 # CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
 # which would shrink each slot below the full 262144 window once
 # parallel > 1.
 #
 # Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
 # params; it carries an audio encoder too). The fleet's image path lives
 # here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
 # WORKS: input_audio content parts transcribe verbatim (verified
 # 2026-09-17 with a 16k mono WAV).
 #
 # Sampling: greedy by default (the fleet contract); a client that sends a
 # temperature gets its own, and the harness applies per-model overrides
@@ -50,69 +40,41 @@
 # reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
 # AGENTS.md). This box is CUDA, but the rule keeps env files portable.
-# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
+# SGLang runtime: python3.12 venv, self-contained CUDA stack.
-# FA_ALL_QUANTS=ON.
+SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang
 LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
-# Model package. The MTP sidecar file is required for speculative decoding.
+# Model package (8 shards + configs, sha256-verified by
-MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
+# scripts/download-frank-small.sh).
-MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
+MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
 MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
 DRAFT_FILE=mtp-gemma-4-12B-it.gguf
 # Fleet identity and network. 7771 is this service's own endpoint on
-# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
+# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models
-# what clients ask for -- the harness entry in models.yml names it.
+# and what clients ask for -- the harness entry in models.yml names it.
 ALIAS=frank-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7771
-# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
+# Four slots on fp8 KV. The pool is pinned (see SIZING above): 606,208
-# pool; --parallel divides it). See the SIZING note above for the fit
+# tokens = 4 x 151,552, so the slot window is 151,552. A single request
-# arithmetic; a fifth slot does not fit.
+# may still grow to the whole pool (a 170k prompt is verified); the grid
-CTX_SIZE=1048576
+# is the four-way concurrency contract.
-PARALLEL=4
+MEM_FRACTION=0.98
-CACHE_TYPE_K=q8_0
+MAX_TOTAL_TOKENS=606208
-CACHE_TYPE_V=q8_0
+CONTEXT_LENGTH=151552
-# CPU RAM cache cap in MiB (build default 8192). This host is memory-tight
+MAX_RUNNING=4
-# (frank-large's 70 GiB GTT plus the rest of the stack), and Gemma4 runs
+KV_DTYPE=fp8_e4m3
-# fully on the 3090 (-ngl 999), so the CPU-side cache is not the hot path;
+QUANTIZATION=auto-round
-# cap it at 2048 to give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet
+# DeltaNet (mamba) recurrent states: the ssm state pool is the single
-# model, so the >12.8 GiB CACHE_RAM floor does not apply.
+# biggest hidden consumer on this card. The engine autofills the slot
-CACHE_RAM=2048
+# COUNT to spend a fixed ~5.1 GB budget regardless of dtype (fp32 at 110
 # slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below:
 # bf16 dtype + 32 slots = 795 MiB.
 MAMBA_SSM_DTYPE=bfloat16
 # 32 slots is 8x the 4-way concurrency: enough for the radix mamba cache
 # to keep shared-prefix states, small enough to leave the KV pool room.
 MAX_MAMBA_CACHE_SIZE=32
-# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
+# Reasoning is extracted into reasoning_content; tools use the qwen3_coder
-# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
+# grammar (both parsers per the SGLang Qwen3.5 cookbook).
-# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
+REASONING_PARSER=qwen3
-# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
+TOOLCALL_PARSER=qwen3_coder
 # compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
 # tried. Idle VRAM is 23,300 MiB at this geometry.
 BATCH_SIZE=8192
 UBATCH_SIZE=1024
 # CPU threads: this service runs entirely on the GPU (NGL=999), so threads
 # cover tokenization, sampling and the HTTP layer only. Deliberately modest
 # -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
 NUM_THREADS=8
 THREAD_BATCH=8
 # Every layer on the 3090, with flash attention (required for q8 KV).
 NGL=999
 FLASH_ATTN=on
 # Speculative decoding: the MTP sidecar (draft context) chained with the
 # draftless ngram-mod matcher. Draft depth and busy-wait level are made
 # explicit here (both were the implicit build defaults: 3 and 50).
 # Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
 # than depth 3 + poll 50 (257-302 t/s) — the longer verification round
 # costs more attention re-read at long context than the extra accepted
 # token earns. Keep 3/50.
 SPEC_TYPE=draft-mtp,ngram-mod
 SPEC_DRAFT_N_MAX=3
 POLL_LEVEL=50
 # Greedy default (fleet contract); reasoning is extracted into
 # reasoning_content with an unlimited budget (the model decides).
 TEMPERATURE=0.0
 REASONING_FORMAT=deepseek
 REASONING_BUDGET=-1
@@ -1,5 +1,5 @@
 [Unit]
-Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small)
+Description=SGLang MiMo-V2.6-Distill-Qwen-9B-INT8 (qwen3_5 hybrid, MTP+vision) 4x172k fp8 KV (dankclaw, frank-small)
 After=network-online.target
 Wants=network-online.target
@@ -8,8 +8,7 @@ Type=simple
 User=dank
 Group=dank
 # CUDA pins its staging buffers, so the process needs the memlock budget even
-# though the weights themselves are copied into VRAM. Parity with the halogen
+# though the weights themselves are copied into VRAM.
 # quadlet's Ulimit=memlock=-1:-1 and the retired ik unit.
 LimitMEMLOCK=infinity
 Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
@@ -1,37 +1,28 @@
 #!/usr/bin/env bash
-# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as
+# Serve letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound as
-# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp.
+# frank-small on Dankclaw's RTX 3090 via SGLang.
 #
-# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M
+# Topology: the whole checkpoint (12.0 GiB, INT8 W8A16 AutoRound) lives in
-# weights plus the BF16 vision projector and q8_0 KV inside the card's
+# VRAM; the 3090's 24 GiB holds weights + a ~11.8 GiB fp8 KV pool at
-# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU
+# --mem-fraction-static 0.98. The host CPU only tokenizes.
 # belongs to the halogen engine that serves frank-large.
 #
-# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file
+# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated
-# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is
+# DeltaNet, native 262144 window, and a vision tower (qwen3_5_vision)
-# passed via --model-draft with --spec-type draft-mtp: the draft context
+# baked into the safetensors: no separate projector file, the image path
-# loads the assistant model. The same --spec-type value chains the
+# is in the checkpoint. The config declares an MTP layer, but the shipped
-# draftless ngram-mod matcher, which attaches unconditionally; two
+# weights carry none (no mtp.* tensors in the index), so this service runs
-# context-backed drafters cannot share one process, so this chain is the
+# WITHOUT speculative decoding — a NEXTN draft would be an uninitialised
-# only shape that carries both.
+# 4.24 GiB ghost model drafting at 0% acceptance.
 #
-# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
+# Context: CONTEXT_LENGTH x MAX_RUNNING=4 slots must fit the fp8 KV pool,
-# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
+# which the engine autofits around the weights AND the DeltaNet ssm state
-# maximises the TOTAL pool instead, which would divide a smaller window
+# pool (see frank-small.env). The mamba pool defaults to ~5.1 GB, so it is
-# across slots. The 4-slot layout is measured to fit the card (see the
+# pinned (bf16, 32 slots). With the grid at its measured fit the card has
-# SIZING note in frank-small.env); more slots would fail the KV allocation.
+# no headroom for prefill CUDA-graph capture, so prefill runs eager
 # (--disable-prefill-cuda-graph); decode graphs are untouched.
 #
-# Vision + audio: --mmproj loads the BF16 projector, which also carries
+# Sampling: greedy by default (the fleet contract); clients that send their
-# an audio encoder — input_audio content parts transcribe verbatim
+# own temperature get their own values.
 # (verified 2026-09-17). The fleet's image path lives here, which is why
 # halogen no longer sets HALOGEN_VISION_TOWER.
 #
 # Sampling: greedy by default (the fleet contract). Clients that send their own
 # temperature get their own values, and the harness overrides per model from
 # models.yml. --jinja is pinned even though this build defaults it on: the
 # model's own template supplies reasoning_effort, preserve_thinking and the
 # tool-call grammar, and a future build that flips that default must not
 # silently take tools away.
 #
 # Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it.
 set -euo pipefail
@@ -46,101 +37,61 @@ if [[ -f "$ENV_FILE" ]]; then
 fi
 # --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}"
+: "${SGLANG_BIN:=/home/dank/GitPackages/sglang-cu128/bin/sglang}"
-: "${MODEL_DIR:=/home/dank/models/Gemma4-12B-QAT}"
+: "${MODEL_DIR:=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}"
 : "${MODEL_FILE:=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}"
 : "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}"
 # NOTE: the fallback above was the retired ISTA-DASLab Qwen3.8-27B GSQ-RCO
 # payload, which frank-small served on 2026-09-16/17 and which no longer
 # exists on disk. frank-small.env has always overridden it, so the drift was
 # invisible — but a missing or unreadable env file would have left this
 # script pointing at a payload that is not there, turning a config problem
 # into a "missing asset" that names the wrong model.
 : "${DRAFT_FILE:=}"
 : "${ALIAS:=frank-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7771}"
-: "${CTX_SIZE:=}"
+: "${MEM_FRACTION:=0.98}"
-: "${PARALLEL:=1}"
+: "${CONTEXT_LENGTH:=151552}"
-: "${CACHE_TYPE_K:=f16}"
+: "${MAX_TOTAL_TOKENS:=606208}"
-: "${CACHE_TYPE_V:=f16}"
+: "${MAX_RUNNING:=4}"
-: "${CACHE_RAM:=8192}"
+: "${KV_DTYPE:=fp8_e4m3}"
-: "${BATCH_SIZE:=2048}"
+: "${QUANTIZATION:=auto-round}"
-: "${UBATCH_SIZE:=512}"
+: "${REASONING_PARSER:=qwen3}"
-# Decode-shape knobs: the MTP drafter's per-round proposal length (build
+: "${TOOLCALL_PARSER:=qwen3_coder}"
-# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's
+: "${MAMBA_SSM_DTYPE:=bfloat16}"
-# busy-wait level (build default 50).
+: "${MAX_MAMBA_CACHE_SIZE:=32}"
 : "${SPEC_DRAFT_N_MAX:=3}"
 : "${POLL_LEVEL:=50}"
 : "${NUM_THREADS:=8}"
 : "${THREAD_BATCH:=8}"
 : "${NGL:=999}"
 : "${FLASH_ATTN:=on}"
 : "${SPEC_TYPE:=draft-mtp}"
 : "${TEMPERATURE:=0.0}"
 : "${REASONING_FORMAT:=deepseek}"
 : "${REASONING_BUDGET:=-1}"
 # Fail loudly rather than half-loading a model into VRAM: a missing file here
 # is a download that has not finished, and the unit's restart policy would
 # otherwise turn that into a crash loop that reads like a runtime fault.
-for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do
+for path in "$SGLANG_BIN" "$MODEL_DIR/model.safetensors.index.json" \
    "$(cd -- "$(dirname -- "$SGLANG_BIN")/.." && pwd)/cuda-nvcc/usr/local/cuda-13.0/bin/nvcc"; do
    [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; }
 done
-if [[ -n "${DRAFT_FILE:-}" ]]; then
+shard_count=$(find "$MODEL_DIR" -maxdepth 1 -name 'model-*.safetensors' | wc -l)
-    [[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; }
+if [[ "$shard_count" -ne 8 ]]; then
    echo "frank-small: expected 8 safetensors shards in $MODEL_DIR, found $shard_count" >&2
    exit 1
 fi
 # The UVM device node is created lazily by nvidia-modprobe on the first CUDA
 # open after boot; make sure it exists before the server needs it.
 nvidia-modprobe -u -c=0 2>/dev/null || true
-MMPROJ_ARGS=()
+# FlashInfer JITs its attention kernels with nvcc at boot. The host's
-if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then
+# /usr/bin/nvcc is CUDA 12.4 and dies on `--compress-mode` (a 12.8+
-    MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
+# option), so the venv carries a matching CUDA 13.0 nvcc (extracted from
-fi
+# the cuda-nvcc-13-0 deb); it must shadow the system one on PATH.
 export PATH="$(cd -- "$(dirname -- "$SGLANG_BIN")/.." && pwd)/cuda-nvcc/usr/local/cuda-13.0/bin:$PATH"
-SPEC_ARGS=()
+# Sampling: SGLang has no server-wide sampling defaults; the harness sends
-if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then
+# its per-model recipe from models.yml (compat.extraBody), and other clients
-    SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
+# get the engine's own defaults.
-fi
+exec "$SGLANG_BIN" serve \
-
+    --model-path "$MODEL_DIR" \
-DRAFT_ARGS=()
+    --served-model-name "$ALIAS" \
 if [[ -n "${DRAFT_FILE:-}" ]]; then
    DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE")
 fi
 # CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against
 # whatever the weights, projector, draft context and compute buffers leave
 # on the card; the engine prints the chosen value at startup.
 if [[ -n "${CTX_SIZE:-}" ]]; then
    CTX_ARGS=(--ctx-size "$CTX_SIZE")
 else
    CTX_ARGS=(--fit on)
 fi
 exec "$LLAMA_BIN" \
    --model "$MODEL_DIR/$MODEL_FILE" \
    "${MMPROJ_ARGS[@]}" \
    "${DRAFT_ARGS[@]}" \
    --alias "$ALIAS" \
    --host "$BIND_ADDR" \
    --port "$LISTEN_PORT" \
-    --gpu-layers "$NGL" \
+    --mem-fraction-static "$MEM_FRACTION" \
-    --flash-attn "$FLASH_ATTN" \
+    --context-length "$CONTEXT_LENGTH" \
-    "${CTX_ARGS[@]}" \
+    --max-running-requests "$MAX_RUNNING" \
-    --parallel "$PARALLEL" \
+    --kv-cache-dtype "$KV_DTYPE" \
-    --batch-size "$BATCH_SIZE" \
+    --quantization "$QUANTIZATION" \
-    --ubatch-size "$UBATCH_SIZE" \
+    --reasoning-parser "$REASONING_PARSER" \
-    --threads "$NUM_THREADS" \
+    --tool-call-parser "$TOOLCALL_PARSER" \
-    --threads-batch "$THREAD_BATCH" \
+    --mamba-ssm-dtype "$MAMBA_SSM_DTYPE" \
-    --poll "$POLL_LEVEL" \
+    --max-mamba-cache-size "$MAX_MAMBA_CACHE_SIZE" \
-    --cache-type-k "$CACHE_TYPE_K" \
+    --disable-prefill-cuda-graph \
-    --cache-type-v "$CACHE_TYPE_V" \
+    --max-total-tokens "$MAX_TOTAL_TOKENS"
    --cache-ram "$CACHE_RAM" \
    "${SPEC_ARGS[@]}" \
    --jinja \
    --metrics \
    --temp "$TEMPERATURE" \
    --reasoning-format "$REASONING_FORMAT" \
    --reasoning-budget "$REASONING_BUDGET"
@@ -0,0 +1,156 @@
 ---
 type: reference
 title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16"
 description: "The active frank-small runbook — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode shape, the CUDA 13 nvcc toolchain the JIT needs, deployed flags, the client contract, and what to check when it misbehaves."
 tags:
  - llm
  - frank-small
  - dankclaw
  - sglang
  - rtx3090
  - qwen
  - mtp
  - vision
  - int8
 timestamp: 2026-10-01
 ---
 # frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
 > **ACTIVE since 2026-10-01.** Replaces [[frank-small-gemma4-12b]]
 > (llama.cpp Gemma4-12B QAT Q4_K_M, now the rollback target). Same port
 > (7771), same alias (`frank-small`), same client contract surface.
 ## Identity
 - **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
 - **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
  `/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
  transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
  systemd unit `scripts/frank-small.service` (`User=dank`,
  `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with
  tunables from `scripts/frank-small.env`.
 - **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
  (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
 ## Shape
 - **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
  (4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
  DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
  tower (27 layers) in the safetensors — no separate mmproj file.
 - **Context:** four 151,552-token slots (`--context-length 151552`,
  `--max-running-requests 4`) against a **pinned** 606,208-token pool
  (`--max-total-tokens`). A single request may grow to the whole pool —
  a 170,019-token prompt is verified — the grid is the four-way
  concurrency contract.
 - **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
  dim x 1 B), pool 9.31 GiB.
 - **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
  ssm state pool is a first-class consumer. The engine autofills the
  mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
  (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
  pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
  MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
  leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
  CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
  dies in capture, or later in the first long request once Triton's
  deferred kernel loads eat the remainder). Hence
  `--disable-prefill-cuda-graph`; decode graphs are untouched.
 - **Quantization:** `--quantization auto-round` — the checkpoint's
  `quantization_config.json` declares `quant_method: "auto-round"` and
  SGLang loads it natively (vision tower, `embed_tokens` and
  `linear_attn.in_proj` stay fp16).
 ## Spec method
 **None — and this is a checkpoint fact, not a tuning choice.** The
 `text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
 safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
 `model.safetensors.index.json` for both this INT8 conversion and the base
 `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
 `mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
 boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
 decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
 paying the 4-token verify tax for no token. The serve script passes no
 `--speculative-*` flag, and the seam gates that.
 ## CUDA 13 toolchain (the sharp edge)
 The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
 flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
 venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
 `--compress-mode`) and its headers clash with the 13.0 device passes.
 Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):
 | Package | Supplies |
 |---|---|
 | `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
 | `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
 | `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
 | `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
 | `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
 | `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |
 Three local modifications the tree carries:
 1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
   `-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
 2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
   `../../../include` / `../../../lib64` (the deb ships them as real
   near-empty dirs; the stub pass must see the 13.0 headers, not
   `/usr/include` 12.4).
 3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
   includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
   `noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
   the unpatched 13.0 declarations are incompatible in the C++ host stub).
 JIT caches: `~/.cache/sglang/jit/sm86/` and
 `~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
 change lands.
 ## Payload
 Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
 files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
 all LFS sha256-verified by `scripts/download-frank-small.sh` (`--verify-only`
 re-checks). The serve script refuses to start with a missing shard.
 ## Client contract
 - **Alias:** `frank-small` (the `id` at `/v1/models`).
 - **Reasoning:** extracted to **`reasoning_content`** (live-verified
  2026-10-01 — this SGLang build's qwen3 parser uses the same field name
  as llama.cpp; models.yml sets `reasoningContentField:
  "reasoning_content"`). `--reasoning-parser qwen3`.
 - **Tools:** `--tool-call-parser qwen3_coder`.
 - **Vision:** supported (in-checkpoint tower).
 - **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
  `--top-k` — sampling rides on the request; the harness sends the Qwen
  recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
 - **Reasoning effort:** the full OpenAI set is accepted (the chat template
  has no effort gate).
 ## Measured
 - Boot ~60 s with warm JIT caches (several minutes cold — the first boot
  compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
  above is what makes that compile succeed).
 - Decode **83.9 tok/s** (no spec decode — nothing to accept).
 - Prefill ~2,950–3,370 t/s eager (a 170,019-token prompt in 50.5 s).
 - 4 concurrent 54,027-token requests all answer in ~17 s (216k tokens of
  live KV at once).
 - Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
 ## Rollback
 [[frank-small-gemma4-12b]] — the llama.cpp Gemma4-12B QAT payload and binary
 remain intact on disk; its unit/env/serve trio is in `old/` after the
 cutover.
 # Citations
 - Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
 - SGLang: https://github.com/sgl-project/sglang
 - Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: "halogen-flash-server on Dankclaw (frank-large)"
+title: halogen-flash-server on Dankclaw (frank-large)
-description: "The RETIRED frank-large runbook (2026-09-16 → 09-19, rollback target) — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. Replaced by [[frank-large-ciru-orca]] on 2026-09-19."
+description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
 tags:
  - llm
  - halogen
@@ -13,3 +13,466 @@ tags:
  - rocm
 timestamp: 2026-09-13
 ---
 # halogen-flash-server on Dankclaw (frank-large)
 > **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp /
 > kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
 > [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
 > [[frank-large-ik-uncensored]], the current rollback), and came back as the
 > shape the fleet asked for: two 200k-token conversations rather than the
 > native-context pair. Deployed as a rootless Quadlet container,
 > `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090
 > it does not touch now belongs to [[frank-small-qwen38-gsq-rco]].
 The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
 architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
 Upstream's own comparison table measures it at **roughly 4x the prefill
 throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
 ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
 prompt with real context. That, not decode, is the reason for the swap.
 ## Shape
 One container runs both halves of the stack:
 | half | port | reachable |
 |---|---|---|
 | engine | 8730 | container loopback only — **unauthenticated, never publish it** |
 | OpenAI front-end | 8731 | published as host **7770** |
 The `all` mode is upstream's default and the shape a single-container host
 wants: the front-end talks to the engine over loopback inside the container, so
 there is one process tree, one log, and one published port.
 `HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
 `/health`, and in every response body. Requests are **never rejected for naming
 a different model** — the server serves whatever asks. Keeping the fleet alias
 is therefore free, and it means Hermes, the harness config, the OpenWebUI
 discovery, the `services.json` probe and the docs needed no rename.
 ## Weights
 `peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):
 | file | size | role |
 |---|---|---|
 | `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
 | `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
 | `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
 | `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
 `HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
 only** — the entrypoint acts only when the checkpoint is absent, so a restart
 never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
 mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
 `/home/dank/models/halogen`.
 **Refreshing the weights.** Delete the mount's contents (or the whole
 directory, then recreate it **empty**) and start the unit: the entrypoint sees
 the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
 traps. First, the bind mount's source directory must *exist* — podman refuses
 the container with `Error: statfs /home/dank/models/halogen: no such file or
 directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
 directory and the fetch starts on the next start. Second, "refresh" only means
 "fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
 (`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
 2,383,306,048 → **2,478,095,488**) while the checkpoint returned
 byte-identical in size (124,068,083,904) — so an update can live entirely in
 the sidecars, and equal file sizes are not proof of an unchanged repo.
 `overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
 refresh) comes along with the repo but is unused: it trades the `o_proj`
 calibration for about 2 % of decode, and the quality sidecar is the default for
 a reason.
 ## Deployed flags
 Everything else is configured by environment variable — there is no config
 file, and the flags are read once at startup. These are the departures from
 upstream's defaults:
 | flag | value | why |
 |---|---|---|
 | `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
 | `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
 | `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
 | `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
 | `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
 | `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
 | `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
 | `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
 | `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
 Everything else is upstream's default, deliberately: prompt cache mode `2`
 (two entries per conversation — end-of-system-prompt and end-of-history — which
 already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
 stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
 temperature 0, and it stands down whenever a second conversation is
 generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
 gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
 ## Memory model
 Three numbers explain the whole box:
 - **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
  GPU in place, never copied. The kernel counts the locked pages as reclaimable
  file cache, so `free`, `MemAvailable` and every dashboard built on them
  **overstate free memory by the size of the model**. The server prints the
  corrected figure at startup; believe that line, not Grafana.
 - **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
  ~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger
  reads the two together in the engine's own words: `400384 positions, ~27.2
  GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so
  ~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a
  vision tower. The engine's device-side budget stops at roughly 47 GiB on a
  128 GB machine *with the weights pinned* — and lower on a machine carrying
  more besides this server, which Dankclaw is.
 - **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
  table is read through. Too little and long prompts crawl with the disk busy
  and the process in uninterruptible sleep — that is a cache problem, not a
  memory problem, and the fix is a *smaller* pool, not a bigger one.
 `HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
 the pool itself if the configured one will not fit, printing what it chose.
 Read those lines on every restart; they are the ground truth for sizing.
 **Measured ledger, seven restarts, read live 2026-09-12.** The engine's own
 startup account, including three deliberate experiments:
 | pool | max_tok | reserve | KV | arena | need | budget | verdict |
 |---|---|---|---|---|---|---|---|
 | 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
 | 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
 | 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
 | 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
 | 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
 | 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
 | **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** |
 | 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune |
 ### The sizing rule
 One budget covers the pool, the arena **and** the vision tower together:
 ```
 device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
              = 121.2    - 67.7             - reserve
                                    reserve 24 -> 29.5 GiB
                                    reserve 15 -> 38.5 GiB
 need = KV (28 KiB/position) + arena + vision (1.97 measured)
 arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
 ```
 The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
 121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
 Use the unrounded value. Two of the three terms are fixed hardware and
 checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which
 buys the *fit margin*, not the host's memory. It is a term in this inequality,
 not an allocation, and the section at the end of this page shows the live test
 that proves it.
 Trust the engine's printed requirement over this model wherever they disagree:
 a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5
 of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close
 enough to plan with, and the authoritative number to cut the other way.
 The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
 point to 0.1 GiB — including upstream's three published "on the device" figures
 (524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
 **pool + arena totals at the default `max_tok`**, not pool alone. That is why
 their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
 49.4. Reading those figures as pool-only is the trap that makes a doubling look
 affordable when it is not.
 At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
 13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
 ceiling, not a conservative setting.** The corner check agrees: at 32768 only
 7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
 262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
 = 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
 fits unmodified.
 ### What the reserve actually costs
 The device budget is not the whole story. Memory position by configuration
 (401,408 is the 2026-09-13 deployed row, read from its boot ledger):
 | config | held | host left | contiguous 2 MiB blocks |
 |---|---|---|---|
 | 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
 | 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
 | 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
 | **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** |
 | **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** |
 At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
 engine volunteered the stall note. It did not print it at the 401,408 trim, and
 it prints it again after the 2026-09-16 retune. Three data points, and they do
 not line up with a "free memory below X" reading: 310 blocks printed, 1396 did
 not, 475 did. So treat the note as "the machine is tight" and read the
 `host memory left` line rather than waiting to be told:
 ```
 halogen: NOTE this server now holds most of the machine, and 310 contiguous
 2 MiB blocks remain. It runs best on a host of its own. If other large
 processes share this machine they will compete for what is left, and when it
 runs out BOTH they and this server can stall for minutes at a time at full
 CPU with no disk activity and no output, which is not a crash and needs no
 restart.
 ```
 That is the failure mode to watch, and it is the least diagnosable of the
 three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
 looks like a hang, for the neighbours as much as for frank-large. `free` is
 useless for spotting it — `MemAvailable` still reads ~74 GiB because the
 page cache counts as reclaimable. Watch the `host memory left for everything
 else` line and the contiguous-block count at boot instead.
 28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
 so it is survivable.
 ### The reserve cannot buy air — ceiling 16
 The last row above is the experiment that proves it. Raising the reserve from
 15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1
 GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an
 allocation; the actual footprint is `weights + KV + arena + vision`, none of
 which the reserve touches. So the knob has exactly two regimes:
 | reserve | budget | vs required 36.5 | effect |
 |---|---|---|---|
 | 15 | 38.5 | +2.0 | deployed |
 | 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless |
 | 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term |
 | **18** | **35.5** | **−1.0** | **clamps the pool to 262144** |
 | 24 | 29.5 | −7.0 | clamps the pool to 262144 |
 The 36.5 requirement is the engine's own, read from a clamped boot:
 `524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only
 thing a bigger reserve can do is take the doubling away.
 **To actually free host memory, change the pool or the arena**, and the reserve
 then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
 7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both
 are one-line edits that the fit check will confirm on the next boot. The
 2026-09-13 trim to 401,408 was this remedy applied at half strength —
 `HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB
 with a 4.5x improvement in contiguous blocks (310 → 1396), less than the
 modelled 3.4 because the engine holds the freed KV as working slack.
 ### The `WARNING` line only appears when the fit check had to move
 This one cost me a wrong conclusion, so it is worth stating plainly. When the
 configured pool already fits, the engine reserves it **and prints nothing** —
 the 09-11 baseline boot goes straight from the pre-flight line to
 `checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
 its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
 after a clamp**.
 So the absence of a warning in old logs is not evidence of headroom. Both
 `262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
 baseline has lived in that state since it was deployed without incident,
 because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
 at that point). The engine's budget is a conservative floor, not a hard wall.
 ### Two failure modes, and only one is dangerous
 Raising `max_tok` past the ceiling does not shrink the pool — the arena alone
 outgrows the budget, and the server cannot allocate at all:
 ```
 kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB).
 kv pool: WARNING even 262144 positions (the context, the smallest pool that can
         serve one request) needs ~44.5 GiB and host RAM cannot spare it.
 dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)
 ```
 This is the quiet one: the unit stays `active`, the container reports
 `Up (starting)`, and `deploy.sh` sits in its health wait for the whole
 `LLM_WAIT` rather than failing fast. It confirms upstream's warning that a
 large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be
 slower".
 Under-running the ceiling is far more benign and far easier to miss. At 524288
 with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and
 `deploy.sh` still reported healthy in 12 seconds:
 ```
 kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
         RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).
 ```
 A green health check does not mean the pool you asked for is the pool you got —
 read the `kv pool:` lines after every restart.
 The 47.7 GiB n-gram lookup table is never held in RAM; it is read through
 the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no
 `/proc/meminfo` figure at all.
 **Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
 decode numbers were measured with, and it means this container holds most of
 the host. If a co-tenant then takes the remainder, both it and the engine can
 stall for minutes at a time at full CPU with no disk activity and no output.
 That is documented behaviour, not a crash, and it needs no restart.
 `HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
 decode speed — a last resort, not a tuning option.
 ## Client contract
 The differences from llama.cpp that bite are all in the request, not the
 deployment:
 - **No server-side sampler defaults.** `temperature` absent or `0` is
  **greedy** decode; above 0 the request samples on the same drafter. Supported
  fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
  `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
  with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
  value outside its defined range (values are refused, never clamped).
  The fleet runs greedy for clients that stay silent — deterministic, and
  byte-identical with the drafter.
 - **The token budget covers thinking.** The default is **8192**, and a budget
  that runs out mid-thought returns `finish_reason: "length"` with an **empty
  `content`** and the partial reasoning in `reasoning_content` — which most
  clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
  request is a 400 rather than a silent truncation. `max_tokens`,
  `max_completion_tokens` and `max_output_tokens` all work and must agree if
  more than one is sent.
 - **A budget above (window − compaction-threshold) is a guaranteed 400 zone.**
  The fit check (`prompt + max_tokens <= HALOGEN_CTX`) refuses, and the harness
  only compacts at its own threshold — with OMP at 70 % of a 200,192 window
  (140,134), any prompt in (136,192, 140,134] with a 64,000 budget overflows
  the window while sitting below the compaction trigger. That collided live on
  2026-09-17 (prompt 138,898 + 64,000 = 202,898 > 200,192). The fix is
  harness-side: keep the declared `maxTokens` at or under
  `window − threshold` (57,344 chosen) so no pre-compaction request can miss
  the fit check. Do not fix it by raising `HALOGEN_CTX` past pool/2 — that
  breaks the two-resident-conversations property.
 - **`reasoning_effort`** is accepted at the top level and inside
  `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
  model's default is `xhigh`. `minimal` aliases the template's `low` and still
  reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
  honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
 - `/v1/models`, `/health` and completions all report the served id, and
  `reasoning_content` carries the reasoning trace.
 **Pool admission is the one arithmetic trap.** A request reserves its prompt
 **plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
 the request waits in arrival order, and after the queue timeout (3600 s) it
 gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
 the pool can therefore never be admitted at all. The 2026-09-16 shape is a
 400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two
 conversations at the ceiling fit the pool exactly, so a pair that is both
 near-full and both generating is pool-bound — the second waits for the first to
 release its region. The pool binds **how many** conversations stay resident,
 not how long one may be; the older 524,288 / 401,408 numbers live in the ledger
 tables above.
 ## Vision
 **None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
 the tower is a 1.97 GiB charge against the same device budget as the pool and
 the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]],
 which loads the BF16 projector alongside its own weights. This server refuses
 image content parts and says why, rather than ignoring them:
 ```json
 {"error": {"message": "this server does not accept images: the engine was
 started without a vision tower (HALOGEN_VISION_TOWER)", "type":
 "invalid_request_error"}}
 ```
 The llama.cpp side handles images the same way this engine used to: a `data:`
 URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs
 refused deliberately (fetching one would make the server issue outbound
 requests to wherever a client pointed it). Audio and video have no path
 anywhere in the fleet — that left with [[frank-npu]].
 ## Deploy and operate
 ```bash
 ./scripts/deploy.sh dankclaw --service frank-large
 ```
 The handler retires the ik-era **root** unit first (stop, disable, move
 `/etc/systemd/system/frank-large.service` into
 `~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
 `disable` may delete the file outright, which the handler now reports), stops
 the Quadlet's own unit if it is running, then pulls
 `ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the
 `localhost/` alias the unit references, and **waits for port 7770 to be free**
 before installing and starting anything.
 That last gate is not paranoia. On the first cutover attempt the ik server
 logged "Deactivated successfully", and pasta — rootless podman's userspace
 network stack, which binds the published port at container CREATE — still
 reported `Address already in use` a second later, exiting the unit with code
 126. Only `Restart=on-failure` recovered it, minutes later. A deploy that
 starts a doomed container has not deployed anything, so `wait_port_free` polls
 every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with
 the socket table if it never clears.
 Boot persistence comes from `[Install] WantedBy=default.target` in the unit
 file plus linger on the deploy user — generated units **cannot** be
 `systemctl --user enable`d on this host.
 ```bash
 systemctl --user status frank-large.service     # the generated unit
 podman logs frank-large | head -40              # version banner + memory ledger
 curl -s http://127.0.0.1:7770/health            # 200 only after the engine PONGs
 curl -s http://127.0.0.1:7770/v1/models
 ```
 `/health` asks the **engine** for a PONG before answering, so a 200 means
 "ready to serve", not merely "process alive". During the first-start download
 it stays unhealthy by design.
 ## Troubleshooting
 | symptom | cause |
 |---|---|
 | `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
 | starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
 | long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
 | a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
 | both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
 | `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |
 `HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is
 already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is
 deliberately **not** set: it logs every allocation over 64 MB, which is
 steady-state noise on a shared host — raise it only for a live OOM report.
 The lines worth grepping first:
 ```bash
 podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
 ```
 Retrieval: `podman logs frank-large` covers the **current** instance only —
 the Quadlet runs `--replace --rm`, so everything before the last recreate is
 gone from the container log. `journalctl --user -u frank-large.service` is
 the durable record. Both are rootless: `sudo podman logs frank-large` looks
 in root's storage and answers "no such container".
 ## Rollback
 The predecessor is intact: the ik payload stays at
 `/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the
 ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env,
 serve script and the Qwen-Sharp template are archived in the repo under
 `scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back
 is a `git revert` of this change set plus a deploy — **but the Quadlet must be
 removed from `~/.config/containers/systemd/` first**, or the restarted root
 unit and the still-installed container both claim port 7770. The port gate in
 the handler now covers exactly that race. See [[frank-large-ik-uncensored]].
 The `amd_iommu=off` kernel argument that came with this change is separate and
 reverts on its own — see [[dankclaw-iommu-translation-off]].
 ## Citations
 - [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
 - [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
 - [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model
@@ -269,10 +269,16 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
+- [[frank-small-sglang-mimo-v26]] — the ACTIVE frank-small runbook since
-  2026-09-17: mainline llama.cpp serving
+  2026-10-01: SGLang in a self-contained python3.12 venv serving
  letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
  hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
  Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
  4 x 151,552 slots, bf16 mamba states, eager prefill
 - [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
  mainline llama.cpp serving
  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
-  (MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
+  (MTP assistant sidecar + ngram-mod, q8_0 KV) on
  Dankclaw's RTX 3090, port 7771
 - [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
  rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
@@ -303,8 +303,8 @@
    "section": "AI",
    "details": {
      "role": "Fast utility model",
-      "spec": "Gemma4-12B QAT (Q4_K_M, 7.4 GiB) — 1M-token pool across four conversations; fastest at images.",
+      "spec": "MiMo-V2.6-Distill-Qwen-9B (INT8 W8A16, 12 GiB) on SGLang — 606k-token pool across four 151k-token conversations; 84 tok/s decode; sees images.",
-      "inputs": ["text", "image", "audio"],
+      "inputs": ["text", "image"],
      "use": "Agentic search, RAG, research subagents, translation, roleplay, and summarization — any small task that doesn't need much intelligence."
    },
    "url": "",
@@ -167,11 +167,12 @@ argv_value() {
@test "frank-large stays text-only" {
    # The payload has no mmproj, so an --mmproj would point at nothing. The
-    # fleet's image path is frank-small; models.yml declares frank-large
+    # fleet's image path is frank-small: its SGLang MiMo payload carries the
-    # text-only.
+    # vision tower inside the checkpoint (no projector file at all), and
    # models.yml declares frank-large text-only.
    serve_argv
    ! argv_has --mmproj
-    grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh"
+    grep -q 'MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound' "$REPO_ROOT/scripts/frank-small.env"
 }
@test "the engine is asked for greedy decoding and for preserved prior reasoning" {
@@ -1,25 +1,30 @@
 #!/usr/bin/env bats
-# Seam: the frank-small mainline-llama.cpp root unit on dankclaw.
+# Seam: the frank-small SGLang root unit on dankclaw.
-# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large
+# frank-small is the RTX 3090 endpoint; since 2026-10-01 it serves
-# went back to the halogen engine on the iGPU. This seam pins what the flags
+# letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid,
-# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet
+# INT8 W8A16 AutoRound, in-checkpoint vision) via SGLang in a
-# is involved), the repo-relative paths the unit executes, the separate client
+# python3.12 venv. This seam pins what the flags alone do not say: boot
-# port, the MTP contract (the head is a separate sidecar file loaded as the
+# persistence for a NATIVE runtime (no generated quadlet is involved), the
-# draft model, chained behind ngram-mod), the vision handoff from frank-large,
+# repo-relative paths the unit executes, the separate client port, the
-# and the KV budget: q8_0 quantized KV whose context --fit sizes to the card.
+# no-spec-decode gate (the checkpoint ships NO mtp.* tensors, so a NEXTN
 # draft would be an uninitialised ghost model), the 8-shard payload gate,
 # and the KV budget: fp8 KV at 0.98 utilization whose 4-slot grid must fit
 # the card's 24 GiB.
 load helpers
 UNIT="$REPO_ROOT/scripts/frank-small.service"
 SERVE="$REPO_ROOT/scripts/serve-frank-small.sh"
 ENV_FILE="$REPO_ROOT/scripts/frank-small.env"
 DOWNLOAD="$REPO_ROOT/scripts/download-frank-small.sh"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
-@test "frank-small unit, serve script and env exist in scripts/" {
+@test "frank-small unit, serve script, env and download script exist in scripts/" {
    [ -f "$UNIT" ]
    [ -f "$SERVE" ]
    [ -f "$ENV_FILE" ]
    [ -f "$DOWNLOAD" ]
 }
@test "frank-small is boot-persistent, self-healing and unprivileged" {
@@ -35,64 +40,67 @@ ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
    grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT"
 }
-@test "frank-small offloads every layer to the 3090 with flash attention" {
+@test "frank-small serves the MiMo-V2.6 checkpoint from the self-contained SGLang venv" {
-    grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE"
+    grep -q '^SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang$' "$ENV_FILE"
-    grep -q '^NGL=999$' "$ENV_FILE"
+    grep -q '^MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound$' "$ENV_FILE"
-    grep -q -- '--gpu-layers "$NGL"' "$SERVE"
+    grep -q -- '--model-path "$MODEL_DIR"' "$SERVE"
-    grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
+    # The AutoRound checkpoint declares its quant method; SGLang must load it
-    grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE"
+    # as auto-round, not re-quantize online.
-    # Prometheus scrapes this endpoint; without --metrics there is no /metrics.
+    grep -q '^QUANTIZATION=auto-round$' "$ENV_FILE"
-    grep -q -- '--metrics' "$SERVE"
+    grep -q -- '--quantization "$QUANTIZATION"' "$SERVE"
 }
-@test "frank-small caps its CPU RAM cache below the build default" {
+@test "frank-small runs WITHOUT speculative decoding (no MTP weights shipped)" {
-    # Without the flag the 8192 MiB default returns and the env cap is a
+    # The config declares mtp_num_hidden_layers=1 but the safetensors index
-    # silent no-op; this memory-tight host needs the cap to hold.
+    # carries zero mtp.* tensors (base and INT8 conversion alike, verified
-    grep -q -- '--cache-ram "$CACHE_RAM"' "$SERVE"
+    # 2026-10-01). SGLang would draft with an uninitialised 4.24 GiB model
-}
+    # at 0% acceptance; the serve script must not pass any speculative flag.
-
+    ! grep -q -- '--speculative-' "$SERVE"
-@test "frank-small drafts with the MTP sidecar chained behind ngram-mod" {
+    ! grep -q '^SPEC_' "$ENV_FILE"
    grep -q '^SPEC_TYPE=draft-mtp,ngram-mod$' "$ENV_FILE"
    grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
    # The Gemma4 MTP head is a separate assistant file (gemma4-assistant, 4
    # nextn layers); it loads as --model-draft and shares the target's last
    # layers' KV. ngram-mod is draftless, so the chain is legal.
    grep -q '^DRAFT_FILE=mtp-gemma-4-12B-it\.gguf$' "$ENV_FILE"
    grep -q -- '--model-draft "$MODEL_DIR/$DRAFT_FILE"' "$SERVE"
 }
@test "frank-small carries the fleet's image path, and only it" {
-    grep -q '^MMPROJ_PATH=.*mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16\.gguf$' "$ENV_FILE"
+    # The vision tower is baked into the safetensors (qwen3_5_vision): no
-    grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE"
+    # separate projector file, and the serve script must not look for one.
    ! grep -q '^MMPROJ_PATH=' "$ENV_FILE"
    ! grep -q -- '--mmproj' "$SERVE"
    ! grep -q '^ENABLE_VISION=1$' "$ENV_LARGE"
 }
-@test "frank-small's slot grid is full-window and fits the 24 GiB card" {
+@test "frank-small's slot grid is fp8 KV and fits the measured card budget" {
-    local parallel cache_k cache_v ctx fixed per_slot
+    local ctx running kv ssmdtype mambasize required
-    parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
+    ctx="$(sed -nE 's/^CONTEXT_LENGTH=([0-9]+)$/\1/p' "$ENV_FILE")"
-    cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
+    running="$(sed -nE 's/^MAX_RUNNING=([0-9]+)$/\1/p' "$ENV_FILE")"
-    cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
+    kv="$(sed -nE 's/^KV_DTYPE=(.+)$/\1/p' "$ENV_FILE")"
-    ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
+    ssmdtype="$(sed -nE 's/^MAMBA_SSM_DTYPE=(.+)$/\1/p' "$ENV_FILE")"
-    [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ]
+    mambasize="$(sed -nE 's/^MAX_MAMBA_CACHE_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
-    # Every slot must be the model's FULL trained window (262144), not a
+    [ -n "$ctx" ] && [ -n "$running" ] && [ -n "$kv" ]
-    # divided one -- CTX_SIZE is the total pool and --parallel divides it.
+    # fp8 KV is the only dtype whose 16 KiB/token makes four slots fit;
-    [ $((ctx / parallel)) -eq 262144 ]
+    # 64 tokens per MiB of pool.
-    [ $((ctx % parallel)) -eq 0 ]
+    case "$kv" in fp8*) ;; *) return 1 ;; esac
-    [ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ]
+    # The DeltaNet ssm state pool is the hidden fit cost: the engine
-    # Fit budget from the measured boot arithmetic on the 3090 (24576
+    # autofills mamba slots to spend a fixed ~5.1 GB budget, so dtype
-    # MiB): one full-context slot idles at 11,996 MiB, four at 22,346,
+    # alone does not shrink it — the SLOT COUNT must be pinned (bf16,
-    # so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part
+    # bounded). These two gates are what make the measured pool below
-    # (weights, projector, sidecar draft, compute buffers, CUDA context)
+    # exist; change either and re-measure the pool at boot.
-    # is 11996-3450 = 8546 MiB. If the config asks for more slots than
+    [ "$ssmdtype" = "bfloat16" ]
-    # the card holds, this inequality fails before deploy instead of the
+    [ -n "$mambasize" ] && [ "$mambasize" -ge "$running" ] && [ "$mambasize" -le 64 ]
-    # KV allocation failing at boot.
+    grep -q -- '--mamba-ssm-dtype "$MAMBA_SSM_DTYPE"' "$SERVE"
-    fixed=8546
+    grep -q -- '--max-mamba-cache-size "$MAX_MAMBA_CACHE_SIZE"' "$SERVE"
-    per_slot=3450
+    # The slot window is inside the model's native 262144 window.
-    [ $((fixed + parallel * per_slot)) -le 24576 ]
+    [ "$ctx" -le 262144 ]
-    # Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option,
+    # The grid must equal the pinned pool. The engine's autofit picks a
-    # without which prefill collapses about 12x (see the CUDA lesson in
+    # bigger pool (687,388 tokens) but leaves ~0.53 GB — Triton's
-    # AGENTS.md). The build-cuda tree must carry it.
+    # deferred kernel loads take most of it and eager-prefill activations
-    grep -q '^FLASH_ATTN=on$' "$ENV_FILE"
+    # then OOM on the first long request. 606,208 (4 x 151,552) is the
    # verified ceiling: 170k single request and 4 concurrent 54k requests
    # run against it (2026-10-01). A weights or mamba-flag change moves
    # this number; re-measure before trusting the grid.
    pinned_pool=606208
    required=$((running * ctx))
    [ "$required" -le "$pinned_pool" ]
    grep -q -- '--max-total-tokens "$MAX_TOTAL_TOKENS"' "$SERVE"
    grep -q -- '--disable-prefill-cuda-graph' "$SERVE"
 }
@test "frank-small runs on its own port beside frank-large's" {
@@ -106,16 +114,21 @@ ENV_LARGE="$REPO_ROOT/scripts/frank-large.env"
    ! grep -q '^PORT=7771$' "$ENV_LARGE"
 }
-@test "the deploy handler installs the root unit and refuses a missing model" {
+@test "the deploy handler installs the root unit and refuses a missing payload" {
    grep -q 'deploy_llm_small' "$REMOTE_DEPLOY"
    grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY"
    grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY"
    grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY"
-    grep -q 'Gemma4-12B-QAT' "$REMOTE_DEPLOY"
+    grep -q 'MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound' "$REMOTE_DEPLOY"
    grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY"
    grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY"
 }
@test "frank-small's serve script refuses to start on a half-finished download" {
    # The payload is 8 safetensors shards; the serve script counts them and
    # the download script pins each one's LFS sha256.
    grep -q 'frank-small: missing' "$SERVE"
    grep -q 'expected 8 safetensors shards' "$SERVE"
    grep -q 'model-00001-of-00008.safetensors' "$DOWNLOAD"
    grep -q 'model-00008-of-00008.safetensors' "$DOWNLOAD"
 }