From 0c5e054fb88e9ac6bcfc1d99d769597816e431cf Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 1 Oct 2026 06:55:48 -0700 Subject: [PATCH] frank-small: SGLang MiMo-V2.6-Distill-Qwen-9B INT8 (qwen3_5 hybrid, RTX 3090) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback). SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0 nvcc toolchain for the sgl-kernel/flashinfer JIT. Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba states pinned at 32 slots (the engine autofills the ssm pool to a fixed ~5.1 GB regardless of dtype), eager prefill (no headroom for graph capture), no spec decode (the checkpoint ships no mtp.* tensors despite mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB ghost at 0% acceptance). Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k requests, vision (in-checkpoint tower), reasoning_content field. --- AGENTS.md | 93 ++-- scripts/deploy-remote.sh | 34 +- scripts/download-frank-small.sh | 93 ++++ scripts/frank-small.env | 158 +++--- scripts/frank-small.service | 5 +- scripts/serve-frank-small.sh | 169 +++---- .../LLM/frank-small-sglang-mimo-v26.md | 156 ++++++ .../LLM/halogen-flash-server.md | 467 +++++++++++++++++- serverAppConfigs-docs/index.md | 12 +- shared/homepage/services.json | 4 +- tests/frank_large_unit.bats | 7 +- tests/frank_small_unit.bats | 135 ++--- 12 files changed, 1005 insertions(+), 328 deletions(-) create mode 100755 scripts/download-frank-small.sh create mode 100644 serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md diff --git a/AGENTS.md b/AGENTS.md index 1695d1e..874c100 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on - the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on + the gfx1151 iGPU) and `frank-small` (SGLang on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 1024, MoE gather 32×64, **4 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 8,311,013,376/rank, util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Five full-length contexts BOOT but leave ~0 bytes free at runtime — a real client request (995-token prompt, `max_tokens=261149`) killed the worker with `torch.OutOfMemoryError ... 0 bytes free` — so four slots carry the deployed shape and its VRAM headroom. Six do not fit at all (6×241664 asked 10.70 GiB vs 9.18 GiB free). fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 | | Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 | -| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 | +| Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 4 slots of 151,552, bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -243,7 +243,7 @@ merely wedged kernel usually still answers ARP. 1. Stop and mask (self-healing would restart it): - Dankclaw `frank-large` (root unit, gufo on the iGPU): `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - - Dankclaw `frank-small` (root unit, llama.cpp on the 3090): + - Dankclaw `frank-small` (root unit, SGLang on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` - Dankserver `frank-primary` (root unit, tcclaviger vLLM on the 2x R9700): `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary` @@ -277,19 +277,20 @@ When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As -of 2026-09-26 both `frank-small` and `frank-primary` accept images +of 2026-10-01 both `frank-small` and `frank-primary` accept images (`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly -(this build never autoloads projectors); `frank-large` remains text-only -because its stock llama.cpp deployment loads no -vision projector (the mmproj is not loaded). Note also that -the reasoning channel differs by runtime: llama.cpp endpoints emit -`reasoning_content`, vLLM emits `reasoning` — academic since the -2026-09-26 swap, as every endpoint in the fleet is llama.cpp now. +(this build never autoloads projectors), and frank-small's MiMo payload +bakes its vision tower INTO the checkpoint (no projector file at all); +`frank-large` remains text-only because its gufo deployment loads no +vision projector. Note also that +the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 +parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` — +models.yml sets `reasoningContentField` per endpoint to match. There is **no audio** -input anywhere in the fleet except frank-small: that capability left with the -FLM NPU endpoint on 2026-09-11 and returned on 2026-09-17 with the Gemma4 -projector's audio encoder (frank-small transcribes WAV verbatim even though -`audio` is omitted from its `input` list — OMP's schema rejects it). +input anywhere in the fleet: that capability left with the FLM NPU +endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's +audio encoder, and left again with the Gemma4 retirement on 2026-10-01 +(the MiMo tower is vision-only). ### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE) @@ -397,31 +398,46 @@ no longer a one-minute flip. **Engine** — Qwen-Sharp back. The DFlash2 drafter was never reclaimed and still lives in `Qwen3.8-27B-UD-Q8_K_XL/`. -### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) +### frank-small — SGLang on the RTX 3090 (Dankclaw, ACTIVE) -Mainline llama.cpp master `37b53fd` (CUDA build with -`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo -`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`), -serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M. -The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4 -nextn layers) loaded via `--model-draft`; the draftless **ngram-mod** -matcher chains behind it in the same `--spec-type` value. BF16 mmproj, -port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4` -(four FULL 262,144-token slots — the measured card ceiling), batch -**8192/1024**, greedy default. Deployed flags are in -`scripts/frank-small.env`; the serve script refuses to start if the -weights, the projector, the sidecar or the binary are missing. +SGLang (latest) in a **self-contained python3.12 venv** +(`/home/dank/GitPackages/sglang-cu128/`, torch 2.13.0+cu130) as a **root +systemd unit** (repo `scripts/frank-small.service`, `relink_unit`ed into +`/etc/systemd/system/`), serving +`letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound` (9B dense +qwen3_5 hybrid — 8 full-attention + 24 Gated DeltaNet layers — INT8 W8A16 +AutoRound, 12.0 GiB, vision tower baked into the checkpoint). Port 7771. -Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a -chat completion returns content plus `reasoning_content`; a tool call -returns `tool_calls`; an image request is answered from the projector; -an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at -max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft -acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why -ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll -100 is WORSE at long ctx) is tabled in the runbook. -Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. -Rollback: [[frank-small-qwen38-gsq-rco]]. +**The shape is memory-pinned, and the sharp edge is the mamba pool.** The +DeltaNet ssm state pool autofills to a fixed ~5.1 GB budget *regardless of +dtype* (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are +pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` (795 MiB). +The KV pool is then **pinned at 606,208 tokens** (`--max-total-tokens`) — +the engine's autofit picks 687,388 and leaves 0.53 GB, which Triton's +deferred kernel loads eat and the first long request OOMs on. Four +151,552-token slots; a single request may grow to the whole pool (170k +verified). Prefill runs **eager** (`--disable-prefill-cuda-graph`) — there +is no headroom for capture; decode graphs are on. + +**No speculative decode — a checkpoint fact.** `text_config` declares +`mtp_num_hidden_layers: 1`, but the shipped safetensors (base BF16 and this +INT8 conversion alike) carry zero `mtp.*` tensors; NEXTN would draft with +an uninitialised 4.24 GiB ghost model at 0% acceptance. The seam gates +that no `--speculative-*` flag reaches the engine. + +**The venv carries a CUDA 13.0 nvcc extracted from debs** (system nvcc is +12.4) so the sgl-kernel/flashinfer JIT compiles for sm86; three local +patches ride in the tree (GCC-16 shim, CCCL symlink, rsqrt noexcept). +Details in the runbook. + +Verified at deploy (2026-10-01): `/v1/models` reports `frank-small`; +chat returns content plus `reasoning_content` (SGLang's qwen3 parser uses +llama.cpp's field name); a red-square image answers "Red" +(`image_tokens: 64`); 4 concurrent 54k-token requests all answer; a +170,019-token prompt prefills in 50.5 s (~3,370 t/s eager). Decode +**83.9 tok/s**. +Runbook: `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md`. +Rollback: [[frank-small-gemma4-12b]] (payload and binary intact on disk). ### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE) @@ -803,7 +819,7 @@ Active: | Service | Runbook | |---------|---------| | frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` | -| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | +| frank-small (SGLang MiMo-V2.6 INT8, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md` | | frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -826,6 +842,7 @@ Retired (rollback targets and history): | Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` | | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` | | vLLM Gemma4-12B w4a16 (`frank-small`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-small-vllm-gemma4.md` | +| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-small`) | Dankclaw | 2026-09-17 → 10-01 | `frank-small-gemma4-12b.md` | | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` | | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` | diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 1825e46..97dca2c 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1177,23 +1177,39 @@ deploy_llm_large() { wait_for_health frank-large http://127.0.0.1:7770/health 1200 } -# deploy_llm_small — frank-small is mainline llama.cpp on the RTX 3090, as a -# ROOT unit: the runtime is a native CUDA binary, so there is no container -# image to install and no generated unit to avoid. relink_unit points -# /etc/systemd/system/frank-small.service at the repo file and boot persistence -# is that unit's [Install] section. +# deploy_llm_small — frank-small is SGLang (python3.12 venv) on the RTX 3090, +# as a ROOT unit: the runtime is a self-contained venv, so there is no +# container image to install and no generated unit to avoid. relink_unit +# points /etc/systemd/system/frank-small.service at the repo file and boot +# persistence is that unit's [Install] section. # # It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is # not shared: frank-large's engine runs on the gfx1151 iGPU. -LLM_SMALL_MODEL="${LLM_SMALL_MODEL:-/home/dank/models/Gemma4-12B-QAT/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}" +LLM_SMALL_MODEL_DIR="${LLM_SMALL_MODEL_DIR:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}" +LLM_SMALL_RUNTIME="${LLM_SMALL_RUNTIME:-/home/dank/GitPackages/sglang-cu128/bin/sglang}" deploy_llm_small() { - if [[ ! -f "$LLM_SMALL_MODEL" ]]; then - echo "ERROR: model absent at $LLM_SMALL_MODEL" >&2 - echo "frank-small: fetch HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M, the mmproj and the MTP sidecar) first;" >&2 + # Refuse rather than restart-loop: a unit whose runtime or payload is + # missing never becomes healthy, and a green deploy would leave the + # endpoint dark with nothing in the log to say why. + if [[ ! -x "$LLM_SMALL_RUNTIME" ]]; then + echo "ERROR: SGLang runtime absent at $LLM_SMALL_RUNTIME" >&2 + echo "frank-small: build the venv (python3.12 -m venv /home/dank/GitPackages/sglang-cu128" >&2 + echo "frank-small: and pip install 'sglang[all]' into it) first." >&2 + return 1 + fi + if [[ ! -f "$LLM_SMALL_MODEL_DIR/model.safetensors.index.json" ]]; then + echo "ERROR: payload index absent at $LLM_SMALL_MODEL_DIR" >&2 + echo "frank-small: run scripts/download-frank-small.sh first;" >&2 echo "frank-small: refusing to start a unit that cannot load." >&2 return 1 fi + local count + count="$(find "$LLM_SMALL_MODEL_DIR" -maxdepth 1 -name 'model-*-of-*.safetensors' | wc -l)" + if [[ "$count" -ne 8 ]]; then + echo "ERROR: payload shard count wrong: $LLM_SMALL_MODEL_DIR (found $count, want 8)" >&2 + return 1 + fi relink_unit frank-small run_privileged systemctl daemon-reload run_privileged systemctl enable frank-small.service diff --git a/scripts/download-frank-small.sh b/scripts/download-frank-small.sh new file mode 100755 index 0000000..d054864 --- /dev/null +++ b/scripts/download-frank-small.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-small +# (SGLang, RTX 3090). Public repo — no auth header on the command line. +# +# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB +# oversized file on a 31 GiB payload; a fresh -f fetch plus size+sha256 is +# the house rule. +# +# Usage: +# scripts/download-frank-small.sh [DEST_DIR] [--verify-only] +# +# LFS shards are verified against the Hub LFS sha256; non-LFS text files +# against their declared byte size. Idempotent: complete files are skipped. +set -euo pipefail + +REPO="letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound" +BASE="https://huggingface.co/${REPO}/resolve/main" + +DEST="" +VERIFY_ONLY=0 +for arg in "$@"; do + case "$arg" in + --verify-only) VERIFY_ONLY=1 ;; + *) DEST="$arg" ;; + esac +done +DEST="${DEST:-/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}" + +# name bytes sha256(- = non-LFS, size-only check) +FILES=( + "model-00001-of-00008.safetensors 2946296312 ddb6117c04cb6feaf7c265e5d75efadf21949037d73253a8e4521d40e164366d" + "model-00002-of-00008.safetensors 1402134792 2dc78d55d1d33d7b3b21dd6f8b4b817b62c2fb1376e6448b322428798c10e257" + "model-00003-of-00008.safetensors 1427365904 5c797d14e345309d67c78306b9de4aa01a21dbd92dd98e40b0ad69b2900e9203" + "model-00004-of-00008.safetensors 1420041856 7402f852e2c7a24ebf4694b98a220e71dfb2c93ba3776272cf2acfc65acd8126" + "model-00005-of-00008.safetensors 1410655648 991cbf2eda2ff91fc3ffbb15591d7fcac340fb10d9c33bd3da20010e2f3f4090" + "model-00006-of-00008.safetensors 1427627856 efc42448054cfb5e3061e154ba4539204088ce95176d689a95b58de885ef7eeb" + "model-00007-of-00008.safetensors 2035442408 75a101fc71ae6f2e1f5c4816f711d62cf1d7a0e02ef6e5182361b8c11e72b626" + "model-00008-of-00008.safetensors 822153664 0fb90cb50e2d336fbad58bd4c01880eeddbf537c1dca1cf0ae291195d5efbb4b" + "tokenizer.json 19989325 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523" + "model.safetensors.index.json 106278 -" + "config.json 4608 -" + "generation_config.json 161 -" + "quantization_config.json 1654 -" + "chat_template.jinja 3916 -" + "tokenizer_config.json 1168 -" + "preprocessor_config.json 443 -" + "processor_config.json 1191 -" + "README.md 8448 -" +) + +mkdir -p "$DEST" +status=0 +for entry in "${FILES[@]}"; do + read -r name size sha <<<"$entry" + f="$DEST/$name" + have=0 + if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then + if [[ "$sha" == "-" ]] || sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then + have=1 + fi + fi + if (( have )); then + echo "OK $name" + continue + fi + if (( VERIFY_ONLY )); then + echo "MISSING $name" + status=1 + continue + fi + echo "FETCHING $name" + tmp="$f.part" + # no -C resume; -f + --fail-with-body so an HTTP error is not a success + curl -fsSL --fail-with-body -o "$tmp" "$BASE/$name" + mv "$tmp" "$f" + if [[ "$(stat -c%s "$f")" != "$size" ]]; then + echo "SIZE FAIL $name ($(stat -c%s "$f") != $size)" + status=1 + continue + fi + if [[ "$sha" != "-" ]] && ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then + echo "SHA FAIL $name" + status=1 + continue + fi + echo "VERIFIED $name" +done +if (( status == 0 )); then + echo "DOWNLOAD_DONE $DEST" +else + echo "DOWNLOAD_FAILED $DEST" >&2 +fi +exit "$status" diff --git a/scripts/frank-small.env b/scripts/frank-small.env index 9304e47..0a0b00a 100644 --- a/scripts/frank-small.env +++ b/scripts/frank-small.env @@ -1,46 +1,36 @@ -# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS- -# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP -# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17. +# Tunables for frank-small (letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8- +# W8A16-AutoRound, qwen3_5 hybrid, in-checkpoint vision) on SGLang, +# Dankclaw's RTX 3090, since 2026-10-01. # -# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86: -# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \ -# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON -# cmake --build build-cuda --target llama-server -j -# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash -# attention, and without the flag quantized-KV prefill collapses ~12x on -# CUDA (see the CUDA lesson in AGENTS.md). +# Runtime: SGLang in a python3.12 venv (pip "sglang[all]"): +# python3.12 -m venv /home/dank/GitPackages/sglang-cu128 +# /home/dank/GitPackages/sglang-cu128/bin/pip install "sglang[all]" +# The venv carries its own CUDA torch + sgl-kernel + flashinfer; it is +# self-contained and does not touch the host Python (3.14) or the ROCm +# stack that serves frank-large. # -# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M -# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv -# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, -# per-layer token embeddings (PLE) inside the file, and a sliding-window -# attention pattern (attention.sliding_window = 1024) -- most layers cache -# only a 1024-position window, which is why the KV pool is a fraction of -# the full-ctx x all-layers number. +# Model: MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound — a Qwen3.5-9B +# distillation at INT8 W8A16 (AutoRound, group 128, symmetric). 8 +# safetensors shards, 12.0 GiB total. The vision tower +# (qwen3_5_vision, 27 layers) and embed_tokens stay fp16 inside the +# checkpoint, so the image path needs no separate projector file. +# Architecture: 32 layers, 8 full-attention (4 KV heads x 256 head_dim) +# + 24 Gated DeltaNet, native 262144 window. text_config declares +# mtp_num_hidden_layers = 1, but the shipped safetensors carry NO mtp.* +# tensors (verified against model.safetensors.index.json 2026-10-01 — +# base BF16 and this INT8 conversion both omit the head), so there is no +# spec decode: SGLang would draft with an uninitialised 4.24 GiB ghost +# model at 0% acceptance and pay the verify tax for nothing. # -# MTP: the Multi-Token Prediction head ships as a SEPARATE file -# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24 -# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one -# is passed via --model-draft with --spec-type draft-mtp, so the draft -# context loads the assistant model. The same --spec-type value chains the -# draftless ngram-mod matcher, which attaches unconditionally; two -# context-backed drafters cannot share one process, so draft-mtp,ngram-mod -# is the only legal shape that carries both. -# -# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four -# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346 -# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot -# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the -# card, and the KV pool allocates upfront -- so 4 is the max. Explicit -# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool, -# which would shrink each slot below the full 262144 window once -# parallel > 1. -# -# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M -# params; it carries an audio encoder too). The fleet's image path lives -# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio -# WORKS: input_audio content parts transcribe verbatim (verified -# 2026-09-17 with a 16k mono WAV). +# SIZING (RTX 3090), measured 2026-10-01: the 0.98 budget holds the +# checkpoint (11,354 MiB) + the DeltaNet ssm state pool (795 MiB at 32 +# bf16 slots) + the KV pool. The engine's autofit picks a 687,388-token +# pool and leaves ~0.53 GB — too little: Triton's deferred kernel loads +# take most of it after boot, and eager-prefill activations then OOM on +# the first long request. The pool is therefore PINNED at 606,208 tokens +# (4 x 151,552), which leaves ~0.8 GB for the prefill workspace; a 170k +# single request and 4 concurrent 54k requests are verified against it. +# The ssm pool is the 4.5 GiB the weights-only arithmetic misses. # # Sampling: greedy by default (the fleet contract); a client that sends a # temperature gets its own, and the harness applies per-model overrides @@ -50,69 +40,41 @@ # reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in # AGENTS.md). This box is CUDA, but the rule keeps env files portable. -# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86, -# FA_ALL_QUANTS=ON. -LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server +# SGLang runtime: python3.12 venv, self-contained CUDA stack. +SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang -# Model package. The MTP sidecar file is required for speculative decoding. -MODEL_DIR=/home/dank/models/Gemma4-12B-QAT -MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf -MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf -DRAFT_FILE=mtp-gemma-4-12B-it.gguf +# Model package (8 shards + configs, sha256-verified by +# scripts/download-frank-small.sh). +MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound # Fleet identity and network. 7771 is this service's own endpoint on -# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and -# what clients ask for -- the harness entry in models.yml names it. +# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models +# and what clients ask for -- the harness entry in models.yml names it. ALIAS=frank-small BIND_ADDR=0.0.0.0 LISTEN_PORT=7771 -# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total -# pool; --parallel divides it). See the SIZING note above for the fit -# arithmetic; a fifth slot does not fit. -CTX_SIZE=1048576 -PARALLEL=4 -CACHE_TYPE_K=q8_0 -CACHE_TYPE_V=q8_0 -# CPU RAM cache cap in MiB (build default 8192). This host is memory-tight -# (frank-large's 70 GiB GTT plus the rest of the stack), and Gemma4 runs -# fully on the 3090 (-ngl 999), so the CPU-side cache is not the hot path; -# cap it at 2048 to give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet -# model, so the >12.8 GiB CACHE_RAM floor does not apply. -CACHE_RAM=2048 +# Four slots on fp8 KV. The pool is pinned (see SIZING above): 606,208 +# tokens = 4 x 151,552, so the slot window is 151,552. A single request +# may still grow to the whole pool (a 170k prompt is verified); the grid +# is the four-way concurrency contract. +MEM_FRACTION=0.98 +MAX_TOTAL_TOKENS=606208 +CONTEXT_LENGTH=151552 +MAX_RUNNING=4 +KV_DTYPE=fp8_e4m3 +QUANTIZATION=auto-round +# DeltaNet (mamba) recurrent states: the ssm state pool is the single +# biggest hidden consumer on this card. The engine autofills the slot +# COUNT to spend a fixed ~5.1 GB budget regardless of dtype (fp32 at 110 +# slots, bf16 at 217 — same 5.1 GB), so BOTH dials are pinned below: +# bf16 dtype + 32 slots = 795 MiB. +MAMBA_SSM_DTYPE=bfloat16 +# 32 slots is 8x the 4-way concurrency: enough for the radix mamba cache +# to keep shared-prefix states, small enough to leave the KV pool room. +MAX_MAMBA_CACHE_SIZE=32 -# Batch geometry: 8192 logical / 1024 physical. Measured on this card at -# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV): -# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962, -# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp -# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be -# tried. Idle VRAM is 23,300 MiB at this geometry. -BATCH_SIZE=8192 -UBATCH_SIZE=1024 - -# CPU threads: this service runs entirely on the GPU (NGL=999), so threads -# cover tokenization, sampling and the HTTP layer only. Deliberately modest -# -- this host's CPU is shared with halogen's engine, which is CPU-hungry. -NUM_THREADS=8 -THREAD_BATCH=8 - -# Every layer on the 3090, with flash attention (required for q8 KV). -NGL=999 -FLASH_ATTN=on - -# Speculative decoding: the MTP sidecar (draft context) chained with the -# draftless ngram-mod matcher. Draft depth and busy-wait level are made -# explicit here (both were the implicit build defaults: 3 and 50). -# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s) -# than depth 3 + poll 50 (257-302 t/s) — the longer verification round -# costs more attention re-read at long context than the extra accepted -# token earns. Keep 3/50. -SPEC_TYPE=draft-mtp,ngram-mod -SPEC_DRAFT_N_MAX=3 -POLL_LEVEL=50 - -# Greedy default (fleet contract); reasoning is extracted into -# reasoning_content with an unlimited budget (the model decides). -TEMPERATURE=0.0 -REASONING_FORMAT=deepseek -REASONING_BUDGET=-1 +# Reasoning is extracted into reasoning_content; tools use the qwen3_coder +# grammar (both parsers per the SGLang Qwen3.5 cookbook). +REASONING_PARSER=qwen3 +TOOLCALL_PARSER=qwen3_coder diff --git a/scripts/frank-small.service b/scripts/frank-small.service index 14ddd9c..d7d1e31 100644 --- a/scripts/frank-small.service +++ b/scripts/frank-small.service @@ -1,5 +1,5 @@ [Unit] -Description=llama.cpp Qwen3.8-27B GSQ-RCO IQ3_S MTP+vision (dankclaw, frank-small) +Description=SGLang MiMo-V2.6-Distill-Qwen-9B-INT8 (qwen3_5 hybrid, MTP+vision) 4x172k fp8 KV (dankclaw, frank-small) After=network-online.target Wants=network-online.target @@ -8,8 +8,7 @@ Type=simple User=dank Group=dank # CUDA pins its staging buffers, so the process needs the memlock budget even -# though the weights themselves are copied into VRAM. Parity with the halogen -# quadlet's Ulimit=memlock=-1:-1 and the retired ik unit. +# though the weights themselves are copied into VRAM. LimitMEMLOCK=infinity Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env" ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index 21ebd58..03b537b 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -1,37 +1,28 @@ #!/usr/bin/env bash -# Serve HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced (Q4_K_M) as -# frank-small on Dankclaw's RTX 3090 via mainline llama.cpp. +# Serve letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound as +# frank-small on Dankclaw's RTX 3090 via SGLang. # -# Topology: the whole model lives in VRAM (-ngl 999) -- 6.87 GiB of Q4_K_M -# weights plus the BF16 vision projector and q8_0 KV inside the card's -# 24 GiB. The host CPU only tokenizes and samples here; this box's CPU -# belongs to the halogen engine that serves frank-large. +# Topology: the whole checkpoint (12.0 GiB, INT8 W8A16 AutoRound) lives in +# VRAM; the 3090's 24 GiB holds weights + a ~11.8 GiB fp8 KV pool at +# --mem-fraction-static 0.98. The host CPU only tokenizes. # -# MTP: this model's Multi-Token Prediction head ships as a SEPARATE file -# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers). It is -# passed via --model-draft with --spec-type draft-mtp: the draft context -# loads the assistant model. The same --spec-type value chains the -# draftless ngram-mod matcher, which attaches unconditionally; two -# context-backed drafters cannot share one process, so this chain is the -# only shape that carries both. +# Architecture: qwen3_5 hybrid — 32 layers, 8 full-attention + 24 Gated +# DeltaNet, native 262144 window, and a vision tower (qwen3_5_vision) +# baked into the safetensors: no separate projector file, the image path +# is in the checkpoint. The config declares an MTP layer, but the shipped +# weights carry none (no mtp.* tensors in the index), so this service runs +# WITHOUT speculative decoding — a NEXTN draft would be an uninitialised +# 4.24 GiB ghost model drafting at 0% acceptance. # -# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot -# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below) -# maximises the TOTAL pool instead, which would divide a smaller window -# across slots. The 4-slot layout is measured to fit the card (see the -# SIZING note in frank-small.env); more slots would fail the KV allocation. +# Context: CONTEXT_LENGTH x MAX_RUNNING=4 slots must fit the fp8 KV pool, +# which the engine autofits around the weights AND the DeltaNet ssm state +# pool (see frank-small.env). The mamba pool defaults to ~5.1 GB, so it is +# pinned (bf16, 32 slots). With the grid at its measured fit the card has +# no headroom for prefill CUDA-graph capture, so prefill runs eager +# (--disable-prefill-cuda-graph); decode graphs are untouched. # -# Vision + audio: --mmproj loads the BF16 projector, which also carries -# an audio encoder — input_audio content parts transcribe verbatim -# (verified 2026-09-17). The fleet's image path lives here, which is why -# halogen no longer sets HALOGEN_VISION_TOWER. -# -# Sampling: greedy by default (the fleet contract). Clients that send their own -# temperature get their own values, and the harness overrides per model from -# models.yml. --jinja is pinned even though this build defaults it on: the -# model's own template supplies reasoning_effort, preserve_thinking and the -# tool-call grammar, and a future build that flips that default must not -# silently take tools away. +# Sampling: greedy by default (the fleet contract); clients that send their +# own temperature get their own values. # # Tunables come from frank-small.env unless FRANK_SMALL_ENV_FILE overrides it. set -euo pipefail @@ -46,101 +37,61 @@ if [[ -f "$ENV_FILE" ]]; then fi # --- defaults (used when the env file does not set them) ------------------- -: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server}" -: "${MODEL_DIR:=/home/dank/models/Gemma4-12B-QAT}" -: "${MODEL_FILE:=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf}" -: "${MMPROJ_PATH:=$MODEL_DIR/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf}" -# NOTE: the fallback above was the retired ISTA-DASLab Qwen3.8-27B GSQ-RCO -# payload, which frank-small served on 2026-09-16/17 and which no longer -# exists on disk. frank-small.env has always overridden it, so the drift was -# invisible — but a missing or unreadable env file would have left this -# script pointing at a payload that is not there, turning a config problem -# into a "missing asset" that names the wrong model. -: "${DRAFT_FILE:=}" +: "${SGLANG_BIN:=/home/dank/GitPackages/sglang-cu128/bin/sglang}" +: "${MODEL_DIR:=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound}" : "${ALIAS:=frank-small}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7771}" -: "${CTX_SIZE:=}" -: "${PARALLEL:=1}" -: "${CACHE_TYPE_K:=f16}" -: "${CACHE_TYPE_V:=f16}" -: "${CACHE_RAM:=8192}" -: "${BATCH_SIZE:=2048}" -: "${UBATCH_SIZE:=512}" -# Decode-shape knobs: the MTP drafter's per-round proposal length (build -# default 3; the Gemma4 sidecar carries 4 nextn layers) and the engine's -# busy-wait level (build default 50). -: "${SPEC_DRAFT_N_MAX:=3}" -: "${POLL_LEVEL:=50}" -: "${NUM_THREADS:=8}" -: "${THREAD_BATCH:=8}" -: "${NGL:=999}" -: "${FLASH_ATTN:=on}" -: "${SPEC_TYPE:=draft-mtp}" -: "${TEMPERATURE:=0.0}" -: "${REASONING_FORMAT:=deepseek}" -: "${REASONING_BUDGET:=-1}" +: "${MEM_FRACTION:=0.98}" +: "${CONTEXT_LENGTH:=151552}" +: "${MAX_TOTAL_TOKENS:=606208}" +: "${MAX_RUNNING:=4}" +: "${KV_DTYPE:=fp8_e4m3}" +: "${QUANTIZATION:=auto-round}" +: "${REASONING_PARSER:=qwen3}" +: "${TOOLCALL_PARSER:=qwen3_coder}" +: "${MAMBA_SSM_DTYPE:=bfloat16}" +: "${MAX_MAMBA_CACHE_SIZE:=32}" # Fail loudly rather than half-loading a model into VRAM: a missing file here # is a download that has not finished, and the unit's restart policy would # otherwise turn that into a crash loop that reads like a runtime fault. -for path in "$LLAMA_BIN" "$MODEL_DIR/$MODEL_FILE" "$MMPROJ_PATH"; do +for path in "$SGLANG_BIN" "$MODEL_DIR/model.safetensors.index.json" \ + "$(cd -- "$(dirname -- "$SGLANG_BIN")/.." && pwd)/cuda-nvcc/usr/local/cuda-13.0/bin/nvcc"; do [[ -e "$path" ]] || { echo "frank-small: missing $path" >&2; exit 1; } done -if [[ -n "${DRAFT_FILE:-}" ]]; then - [[ -e "$MODEL_DIR/$DRAFT_FILE" ]] || { echo "frank-small: missing $MODEL_DIR/$DRAFT_FILE" >&2; exit 1; } +shard_count=$(find "$MODEL_DIR" -maxdepth 1 -name 'model-*.safetensors' | wc -l) +if [[ "$shard_count" -ne 8 ]]; then + echo "frank-small: expected 8 safetensors shards in $MODEL_DIR, found $shard_count" >&2 + exit 1 fi # The UVM device node is created lazily by nvidia-modprobe on the first CUDA # open after boot; make sure it exists before the server needs it. nvidia-modprobe -u -c=0 2>/dev/null || true -MMPROJ_ARGS=() -if [[ -n "${MMPROJ_PATH:-}" && -f "$MMPROJ_PATH" ]]; then - MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH") -fi +# FlashInfer JITs its attention kernels with nvcc at boot. The host's +# /usr/bin/nvcc is CUDA 12.4 and dies on `--compress-mode` (a 12.8+ +# option), so the venv carries a matching CUDA 13.0 nvcc (extracted from +# the cuda-nvcc-13-0 deb); it must shadow the system one on PATH. +export PATH="$(cd -- "$(dirname -- "$SGLANG_BIN")/.." && pwd)/cuda-nvcc/usr/local/cuda-13.0/bin:$PATH" -SPEC_ARGS=() -if [[ -n "${SPEC_TYPE:-}" && "$SPEC_TYPE" != "none" ]]; then - SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX") -fi - -DRAFT_ARGS=() -if [[ -n "${DRAFT_FILE:-}" ]]; then - DRAFT_ARGS=(--model-draft "$MODEL_DIR/$DRAFT_FILE") -fi - -# CTX_SIZE set -> pinned context. Unset -> --fit sizes the KV pool against -# whatever the weights, projector, draft context and compute buffers leave -# on the card; the engine prints the chosen value at startup. -if [[ -n "${CTX_SIZE:-}" ]]; then - CTX_ARGS=(--ctx-size "$CTX_SIZE") -else - CTX_ARGS=(--fit on) -fi - -exec "$LLAMA_BIN" \ - --model "$MODEL_DIR/$MODEL_FILE" \ - "${MMPROJ_ARGS[@]}" \ - "${DRAFT_ARGS[@]}" \ - --alias "$ALIAS" \ +# Sampling: SGLang has no server-wide sampling defaults; the harness sends +# its per-model recipe from models.yml (compat.extraBody), and other clients +# get the engine's own defaults. +exec "$SGLANG_BIN" serve \ + --model-path "$MODEL_DIR" \ + --served-model-name "$ALIAS" \ --host "$BIND_ADDR" \ --port "$LISTEN_PORT" \ - --gpu-layers "$NGL" \ - --flash-attn "$FLASH_ATTN" \ - "${CTX_ARGS[@]}" \ - --parallel "$PARALLEL" \ - --batch-size "$BATCH_SIZE" \ - --ubatch-size "$UBATCH_SIZE" \ - --threads "$NUM_THREADS" \ - --threads-batch "$THREAD_BATCH" \ - --poll "$POLL_LEVEL" \ - --cache-type-k "$CACHE_TYPE_K" \ - --cache-type-v "$CACHE_TYPE_V" \ - --cache-ram "$CACHE_RAM" \ - "${SPEC_ARGS[@]}" \ - --jinja \ - --metrics \ - --temp "$TEMPERATURE" \ - --reasoning-format "$REASONING_FORMAT" \ - --reasoning-budget "$REASONING_BUDGET" + --mem-fraction-static "$MEM_FRACTION" \ + --context-length "$CONTEXT_LENGTH" \ + --max-running-requests "$MAX_RUNNING" \ + --kv-cache-dtype "$KV_DTYPE" \ + --quantization "$QUANTIZATION" \ + --reasoning-parser "$REASONING_PARSER" \ + --tool-call-parser "$TOOLCALL_PARSER" \ + --mamba-ssm-dtype "$MAMBA_SSM_DTYPE" \ + --max-mamba-cache-size "$MAX_MAMBA_CACHE_SIZE" \ + --disable-prefill-cuda-graph \ + --max-total-tokens "$MAX_TOTAL_TOKENS" diff --git a/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md b/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md new file mode 100644 index 0000000..eca7d4e --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md @@ -0,0 +1,156 @@ +--- +type: reference +title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16" +description: "The active frank-small runbook — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode shape, the CUDA 13 nvcc toolchain the JIT needs, deployed flags, the client contract, and what to check when it misbehaves." +tags: + - llm + - frank-small + - dankclaw + - sglang + - rtx3090 + - qwen + - mtp + - vision + - int8 +timestamp: 2026-10-01 +--- + +# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP) + +> **ACTIVE since 2026-10-01.** Replaces [[frank-small-gemma4-12b]] +> (llama.cpp Gemma4-12B QAT Q4_K_M, now the rollback target). Same port +> (7771), same alias (`frank-small`), same client contract surface. + +## Identity + +- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw). +- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at + `/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130, + transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root + systemd unit `scripts/frank-small.service` (`User=dank`, + `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with + tunables from `scripts/frank-small.env`. +- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound` + (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric). + +## Shape + +- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention + (4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated + DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision` + tower (27 layers) in the safetensors — no separate mmproj file. +- **Context:** four 151,552-token slots (`--context-length 151552`, + `--max-running-requests 4`) against a **pinned** 606,208-token pool + (`--max-total-tokens`). A single request may grow to the whole pool — + a 170,019-token prompt is verified — the grid is the four-way + concurrency contract. +- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256 + dim x 1 B), pool 9.31 GiB. +- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet + ssm state pool is a first-class consumer. The engine autofills the + mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype* + (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are + pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795 + MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB + leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for + CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and + dies in capture, or later in the first long request once Triton's + deferred kernel loads eat the remainder). Hence + `--disable-prefill-cuda-graph`; decode graphs are untouched. +- **Quantization:** `--quantization auto-round` — the checkpoint's + `quantization_config.json` declares `quant_method: "auto-round"` and + SGLang loads it natively (vision tower, `embed_tokens` and + `linear_attn.in_proj` stay fp16). + +## Spec method + +**None — and this is a checkpoint fact, not a tuning choice.** The +`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped +safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against +`model.safetensors.index.json` for both this INT8 conversion and the base +`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no +`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang +boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and +decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected, +paying the 4-token verify tax for no token. The serve script passes no +`--speculative-*` flag, and the seam gates that. + +## CUDA 13 toolchain (the sharp edge) + +The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack, +flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the +venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no +`--compress-mode`) and its headers clash with the 13.0 device passes. + +Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed): + +| Package | Supplies | +|---|---| +| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … | +| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) | +| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` | +| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) | +| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` | +| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) | + +Three local modifications the tree carries: + +1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding + `-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13). +2. **`targets/x86_64-linux/include` and `lib` are symlinks** to + `../../../include` / `../../../lib64` (the deb ships them as real + near-empty dirs; the stub pass must see the 13.0 headers, not + `/usr/include` 12.4). +3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer + includes ``), and **`crt/math_functions.h`** patched with + `noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`; + the unpatched 13.0 declarations are incompatible in the C++ host stub). + +JIT caches: `~/.cache/sglang/jit/sm86/` and +`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain +change lands. + +## Payload + +Eight safetensors shards (12,891,718,440 B total) + tokenizer and config +files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`, +all LFS sha256-verified by `scripts/download-frank-small.sh` (`--verify-only` +re-checks). The serve script refuses to start with a missing shard. + +## Client contract + +- **Alias:** `frank-small` (the `id` at `/v1/models`). +- **Reasoning:** extracted to **`reasoning_content`** (live-verified + 2026-10-01 — this SGLang build's qwen3 parser uses the same field name + as llama.cpp; models.yml sets `reasoningContentField: + "reasoning_content"`). `--reasoning-parser qwen3`. +- **Tools:** `--tool-call-parser qwen3_coder`. +- **Vision:** supported (in-checkpoint tower). +- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/ + `--top-k` — sampling rides on the request; the harness sends the Qwen + recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`. +- **Reasoning effort:** the full OpenAI set is accepted (the chat template + has no effort gate). + +## Measured + +- Boot ~60 s with warm JIT caches (several minutes cold — the first boot + compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain + above is what makes that compile succeed). +- Decode **83.9 tok/s** (no spec decode — nothing to accept). +- Prefill ~2,950–3,370 t/s eager (a 170,019-token prompt in 50.5 s). +- 4 concurrent 54,027-token requests all answer in ~17 s (216k tokens of + live KV at once). +- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`. + +## Rollback + +[[frank-small-gemma4-12b]] — the llama.cpp Gemma4-12B QAT payload and binary +remain intact on disk; its unit/env/serve trio is in `old/` after the +cutover. + +# Citations + +- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound +- SGLang: https://github.com/sgl-project/sglang +- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env` diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md index fd537e6..8d7f374 100644 --- a/serverAppConfigs-docs/LLM/halogen-flash-server.md +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -1,7 +1,7 @@ --- type: reference -title: "halogen-flash-server on Dankclaw (frank-large)" -description: "The RETIRED frank-large runbook (2026-09-16 → 09-19, rollback target) — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. Replaced by [[frank-large-ciru-orca]] on 2026-09-19." +title: halogen-flash-server on Dankclaw (frank-large) +description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. tags: - llm - halogen @@ -13,3 +13,466 @@ tags: - rocm timestamp: 2026-09-13 --- + +# halogen-flash-server on Dankclaw (frank-large) + +> **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp / +> kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see +> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see +> [[frank-large-ik-uncensored]], the current rollback), and came back as the +> shape the fleet asked for: two 200k-token conversations rather than the +> native-context pair. Deployed as a rootless Quadlet container, +> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090 +> it does not touch now belongs to [[frank-small-qwen38-gsq-rco]]. + +The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next +architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs. +Upstream's own comparison table measures it at **roughly 4x the prefill +throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp, +ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any +prompt with real context. That, not decode, is the reason for the swap. + +## Shape + +One container runs both halves of the stack: + +| half | port | reachable | +|---|---|---| +| engine | 8730 | container loopback only — **unauthenticated, never publish it** | +| OpenAI front-end | 8731 | published as host **7770** | + +The `all` mode is upstream's default and the shape a single-container host +wants: the front-end talks to the engine over loopback inside the container, so +there is one process tree, one log, and one published port. + +`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in +`/health`, and in every response body. Requests are **never rejected for naming +a different model** — the server serves whatever asks. Keeping the fleet alias +is therefore free, and it means Hermes, the harness config, the OpenWebUI +discovery, the `services.json` probe and the docs needed no rename. + +## Weights + +`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated): + +| file | size | role | +|---|---|---| +| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint | +| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on | +| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) | +| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount | + +`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start +only** — the entrypoint acts only when the checkpoint is absent, so a restart +never re-downloads, and `huggingface_hub` resumes an interrupted pull. The +mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in +`/home/dank/models/halogen`. + +**Refreshing the weights.** Delete the mount's contents (or the whole +directory, then recreate it **empty**) and start the unit: the entrypoint sees +the checkpoint absent and re-fetches the repo, resuming if interrupted. Two +traps. First, the bind mount's source directory must *exist* — podman refuses +the container with `Error: statfs /home/dank/models/halogen: no such file or +directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty +directory and the fetch starts on the next start. Second, "refresh" only means +"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars +(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn` +2,383,306,048 → **2,478,095,488**) while the checkpoint returned +byte-identical in size (124,068,083,904) — so an update can live entirely in +the sidecars, and equal file sizes are not proof of an unchanged repo. + +`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16 +refresh) comes along with the repo but is unused: it trades the `o_proj` +calibration for about 2 % of decode, and the quality sidecar is the default for +a reason. + +## Deployed flags + +Everything else is configured by environment variable — there is no config +file, and the flags are read once at startup. These are the departures from +upstream's defaults: + +| flag | value | why | +|---|---|---| +| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename | +| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` | +| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column | +| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well | +| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces | +| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) | +| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match | +| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch | +| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug | + +Everything else is upstream's default, deliberately: prompt cache mode `2` +(two entries per conversation — end-of-system-prompt and end-of-history — which +already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot +stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at +temperature 0, and it stands down whenever a second conversation is +generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value +gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`. + +## Memory model + +Three numbers explain the whole box: + +- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the + GPU in place, never copied. The kernel counts the locked pages as reclaimable + file cache, so `free`, `MemAvailable` and every dashboard built on them + **overstate free memory by the size of the model**. The server prints the + corrected figure at startup; believe that line, not Grafana. +- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV, + ~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger + reads the two together in the engine's own words: `400384 positions, ~27.2 + GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so + ~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a + vision tower. The engine's device-side budget stops at roughly 47 GiB on a + 128 GB machine *with the weights pinned* — and lower on a machine carrying + more besides this server, which Dankclaw is. +- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup + table is read through. Too little and long prompts crawl with the disk busy + and the process in uninterruptible sleep — that is a cache problem, not a + memory problem, and the fix is a *smaller* pool, not a bigger one. + +`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers +the pool itself if the configured one will not fit, printing what it chose. +Read those lines on every restart; they are the ground truth for sizing. + +**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own +startup account, including three deliberate experiments: + +| pool | max_tok | reserve | KV | arena | need | budget | verdict | +|---|---|---|---|---|---|---|---| +| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines | +| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare | +| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green | +| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** | +| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** | +| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below | +| **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** | +| 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune | + +### The sizing rule + +One budget covers the pool, the arena **and** the vision tower together: + +``` +device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB + = 121.2 - 67.7 - reserve + reserve 24 -> 29.5 GiB + reserve 15 -> 38.5 GiB + +need = KV (28 KiB/position) + arena + vision (1.97 measured) +arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384) +``` + +The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB = +121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5. +Use the unrounded value. Two of the three terms are fixed hardware and +checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which +buys the *fit margin*, not the host's memory. It is a term in this inequality, +not an allocation, and the section at the end of this page shows the live test +that proves it. + +Trust the engine's printed requirement over this model wherever they disagree: +a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5 +of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close +enough to plan with, and the authoritative number to cut the other way. + +The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured +point to 0.1 GiB — including upstream's three published "on the device" figures +(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are +**pool + arena totals at the default `max_tok`**, not pool alone. That is why +their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be +49.4. Reading those figures as pool-only is the trap that makes a doubling look +affordable when it is not. + +At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 = +13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's +ceiling, not a conservative setting.** The corner check agrees: at 32768 only +7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward +262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4 += 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now +fits unmodified. + +### What the reserve actually costs + +The device budget is not the whole story. Memory position by configuration +(401,408 is the 2026-09-13 deployed row, read from its boot ledger): + +| config | held | host left | contiguous 2 MiB blocks | +|---|---|---|---| +| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 | +| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 | +| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 | +| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** | +| **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** | + +At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the +engine volunteered the stall note. It did not print it at the 401,408 trim, and +it prints it again after the 2026-09-16 retune. Three data points, and they do +not line up with a "free memory below X" reading: 310 blocks printed, 1396 did +not, 475 did. So treat the note as "the machine is tight" and read the +`host memory left` line rather than waiting to be told: + +``` +halogen: NOTE this server now holds most of the machine, and 310 contiguous +2 MiB blocks remain. It runs best on a host of its own. If other large +processes share this machine they will compete for what is left, and when it +runs out BOTH they and this server can stall for minutes at a time at full +CPU with no disk activity and no output, which is not a crash and needs no +restart. +``` + +That is the failure mode to watch, and it is the least diagnosable of the +three: not a silent clamp, not a boot refusal, but a minutes-long freeze that +looks like a hang, for the neighbours as much as for frank-large. `free` is +useless for spotting it — `MemAvailable` still reads ~74 GiB because the +page cache counts as reclaimable. Watch the `host memory left for everything +else` line and the contiguous-block count at boot instead. + +28 co-tenant containers (~4.3 GiB working set) stayed up through the change, +so it is survivable. + +### The reserve cannot buy air — ceiling 16 + +The last row above is the experiment that proves it. Raising the reserve from +15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1 +GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an +allocation; the actual footprint is `weights + KV + arena + vision`, none of +which the reserve touches. So the knob has exactly two regimes: + +| reserve | budget | vs required 36.5 | effect | +|---|---|---|---| +| 15 | 38.5 | +2.0 | deployed | +| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless | +| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term | +| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** | +| 24 | 29.5 | −7.0 | clamps the pool to 262144 | + +The 36.5 requirement is the engine's own, read from a clamped boot: +`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only +thing a bigger reserve can do is take the doubling away. + +**To actually free host memory, change the pool or the arena**, and the reserve +then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back +7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both +are one-line edits that the fit check will confirm on the next boot. The +2026-09-13 trim to 401,408 was this remedy applied at half strength — +`HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB +with a 4.5x improvement in contiguous blocks (310 → 1396), less than the +modelled 3.4 because the engine holds the freed KV as working slack. + +### The `WARNING` line only appears when the fit check had to move + +This one cost me a wrong conclusion, so it is worth stating plainly. When the +configured pool already fits, the engine reserves it **and prints nothing** — +the 09-11 baseline boot goes straight from the pre-flight line to +`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and +its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only +after a clamp**. + +So the absence of a warning in old logs is not evidence of headroom. Both +`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the +baseline has lived in that state since it was deployed without incident, +because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free +at that point). The engine's budget is a conservative floor, not a hard wall. + +### Two failure modes, and only one is dangerous + +Raising `max_tok` past the ceiling does not shrink the pool — the arena alone +outgrows the budget, and the server cannot allocate at all: + +``` +kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB). +kv pool: WARNING even 262144 positions (the context, the smallest pool that can + serve one request) needs ~44.5 GiB and host RAM cannot spare it. +dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory) +``` + +This is the quiet one: the unit stays `active`, the container reports +`Up (starting)`, and `deploy.sh` sits in its health wait for the whole +`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a +large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be +slower". + +Under-running the ceiling is far more benign and far easier to miss. At 524288 +with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and +`deploy.sh` still reported healthy in 12 seconds: + +``` +kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host + RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB). +``` + +A green health check does not mean the pool you asked for is the pool you got — +read the `kv pool:` lines after every restart. + +The 47.7 GiB n-gram lookup table is never held in RAM; it is read through +the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no +`/proc/meminfo` figure at all. + +**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published +decode numbers were measured with, and it means this container holds most of +the host. If a co-tenant then takes the remainder, both it and the engine can +stall for minutes at a time at full CPU with no disk activity and no output. +That is documented behaviour, not a crash, and it needs no restart. +`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the +decode speed — a last resort, not a tuning option. + +## Client contract + +The differences from llama.cpp that bite are all in the request, not the +deployment: + +- **No server-side sampler defaults.** `temperature` absent or `0` is + **greedy** decode; above 0 the request samples on the same drafter. Supported + fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`, + `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused + with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any + value outside its defined range (values are refused, never clamped). + The fleet runs greedy for clients that stay silent — deterministic, and + byte-identical with the drafter. +- **The token budget covers thinking.** The default is **8192**, and a budget + that runs out mid-thought returns `finish_reason: "length"` with an **empty + `content`** and the partial reasoning in `reasoning_content` — which most + clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a + request is a 400 rather than a silent truncation. `max_tokens`, + `max_completion_tokens` and `max_output_tokens` all work and must agree if + more than one is sent. +- **A budget above (window − compaction-threshold) is a guaranteed 400 zone.** + The fit check (`prompt + max_tokens <= HALOGEN_CTX`) refuses, and the harness + only compacts at its own threshold — with OMP at 70 % of a 200,192 window + (140,134), any prompt in (136,192, 140,134] with a 64,000 budget overflows + the window while sitting below the compaction trigger. That collided live on + 2026-09-17 (prompt 138,898 + 64,000 = 202,898 > 200,192). The fix is + harness-side: keep the declared `maxTokens` at or under + `window − threshold` (57,344 chosen) so no pre-compaction request can miss + the fit check. Do not fix it by raising `HALOGEN_CTX` past pool/2 — that + breaks the two-resident-conversations property. +- **`reasoning_effort`** is accepted at the top level and inside + `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The + model's default is `xhigh`. `minimal` aliases the template's `low` and still + reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is + honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key. +- `/v1/models`, `/health` and completions all report the served id, and + `reasoning_content` carries the reasoning trace. + +**Pool admission is the one arithmetic trap.** A request reserves its prompt +**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it +the request waits in arrival order, and after the queue timeout (3600 s) it +gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds +the pool can therefore never be admitted at all. The 2026-09-16 shape is a +400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two +conversations at the ceiling fit the pool exactly, so a pair that is both +near-full and both generating is pool-bound — the second waits for the first to +release its region. The pool binds **how many** conversations stay resident, +not how long one may be; the older 524,288 / 401,408 numbers live in the ledger +tables above. + +## Vision + +**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset: +the tower is a 1.97 GiB charge against the same device budget as the pool and +the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]], +which loads the BF16 projector alongside its own weights. This server refuses +image content parts and says why, rather than ignoring them: + +```json +{"error": {"message": "this server does not accept images: the engine was +started without a vision tower (HALOGEN_VISION_TOWER)", "type": +"invalid_request_error"}} +``` + +The llama.cpp side handles images the same way this engine used to: a `data:` +URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs +refused deliberately (fetching one would make the server issue outbound +requests to wherever a client pointed it). Audio and video have no path +anywhere in the fleet — that left with [[frank-npu]]. + +## Deploy and operate + +```bash +./scripts/deploy.sh dankclaw --service frank-large +``` + +The handler retires the ik-era **root** unit first (stop, disable, move +`/etc/systemd/system/frank-large.service` into +`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's +`disable` may delete the file outright, which the handler now reports), stops +the Quadlet's own unit if it is running, then pulls +`ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the +`localhost/` alias the unit references, and **waits for port 7770 to be free** +before installing and starting anything. + +That last gate is not paranoia. On the first cutover attempt the ik server +logged "Deactivated successfully", and pasta — rootless podman's userspace +network stack, which binds the published port at container CREATE — still +reported `Address already in use` a second later, exiting the unit with code +126. Only `Restart=on-failure` recovered it, minutes later. A deploy that +starts a doomed container has not deployed anything, so `wait_port_free` polls +every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with +the socket table if it never clears. + +Boot persistence comes from `[Install] WantedBy=default.target` in the unit +file plus linger on the deploy user — generated units **cannot** be +`systemctl --user enable`d on this host. + +```bash +systemctl --user status frank-large.service # the generated unit +podman logs frank-large | head -40 # version banner + memory ledger +curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs +curl -s http://127.0.0.1:7770/v1/models +``` + +`/health` asks the **engine** for a PONG before answering, so a 200 means +"ready to serve", not merely "process alive". During the first-start download +it stays unhealthy by design. + +## Troubleshooting + +| symptom | cause | +|---|---| +| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) | +| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` | +| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) | +| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above | +| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model | +| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable | + +`HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is +already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is +deliberately **not** set: it logs every allocation over 64 MB, which is +steady-state noise on a shared host — raise it only for a live OOM report. +The lines worth grepping first: + +```bash +podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):' +``` + +Retrieval: `podman logs frank-large` covers the **current** instance only — +the Quadlet runs `--replace --rm`, so everything before the last recreate is +gone from the container log. `journalctl --user -u frank-large.service` is +the durable record. Both are rootless: `sudo podman logs frank-large` looks +in root's storage and answers "no such container". + +## Rollback + +The predecessor is intact: the ik payload stays at +`/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the +ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env, +serve script and the Qwen-Sharp template are archived in the repo under +`scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back +is a `git revert` of this change set plus a deploy — **but the Quadlet must be +removed from `~/.config/containers/systemd/` first**, or the restarted root +unit and the still-installed container both claim port 7770. The port gate in +the handler now covers exactly that race. See [[frank-large-ik-uncensored]]. + +The `amd_iommu=off` kernel argument that came with this change is separate and +reverts on its own — see [[dankclaw-iommu-translation-off]]. + +## Citations + +- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8) +- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory +- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 0d5f724..b29de60 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -269,10 +269,16 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since - 2026-09-17: mainline llama.cpp serving +- [[frank-small-sglang-mimo-v26]] — the ACTIVE frank-small runbook since + 2026-10-01: SGLang in a self-contained python3.12 venv serving + letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 + hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on + Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool, + 4 x 151,552 slots, bf16 mamba states, eager prefill +- [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target): + mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M - (MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on + (MTP assistant sidecar + ngram-mod, q8_0 KV) on Dankclaw's RTX 3090, port 7771 - [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert, rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16 diff --git a/shared/homepage/services.json b/shared/homepage/services.json index 434a452..c412927 100644 --- a/shared/homepage/services.json +++ b/shared/homepage/services.json @@ -303,8 +303,8 @@ "section": "AI", "details": { "role": "Fast utility model", - "spec": "Gemma4-12B QAT (Q4_K_M, 7.4 GiB) — 1M-token pool across four conversations; fastest at images.", - "inputs": ["text", "image", "audio"], + "spec": "MiMo-V2.6-Distill-Qwen-9B (INT8 W8A16, 12 GiB) on SGLang — 606k-token pool across four 151k-token conversations; 84 tok/s decode; sees images.", + "inputs": ["text", "image"], "use": "Agentic search, RAG, research subagents, translation, roleplay, and summarization — any small task that doesn't need much intelligence." }, "url": "", diff --git a/tests/frank_large_unit.bats b/tests/frank_large_unit.bats index c430003..de27f7f 100755 --- a/tests/frank_large_unit.bats +++ b/tests/frank_large_unit.bats @@ -167,11 +167,12 @@ argv_value() { @test "frank-large stays text-only" { # The payload has no mmproj, so an --mmproj would point at nothing. The - # fleet's image path is frank-small; models.yml declares frank-large - # text-only. + # fleet's image path is frank-small: its SGLang MiMo payload carries the + # vision tower inside the checkpoint (no projector file at all), and + # models.yml declares frank-large text-only. serve_argv ! argv_has --mmproj - grep -q -- '--mmproj' "$REPO_ROOT/scripts/serve-frank-small.sh" + grep -q 'MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound' "$REPO_ROOT/scripts/frank-small.env" } @test "the engine is asked for greedy decoding and for preserved prior reasoning" { diff --git a/tests/frank_small_unit.bats b/tests/frank_small_unit.bats index 1f729f4..028f38a 100755 --- a/tests/frank_small_unit.bats +++ b/tests/frank_small_unit.bats @@ -1,25 +1,30 @@ #!/usr/bin/env bats -# Seam: the frank-small mainline-llama.cpp root unit on dankclaw. -# frank-small is the RTX 3090 endpoint created on 2026-09-16, when frank-large -# went back to the halogen engine on the iGPU. This seam pins what the flags -# alone do not say: boot persistence for a NATIVE runtime (no generated quadlet -# is involved), the repo-relative paths the unit executes, the separate client -# port, the MTP contract (the head is a separate sidecar file loaded as the -# draft model, chained behind ngram-mod), the vision handoff from frank-large, -# and the KV budget: q8_0 quantized KV whose context --fit sizes to the card. +# Seam: the frank-small SGLang root unit on dankclaw. +# frank-small is the RTX 3090 endpoint; since 2026-10-01 it serves +# letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, +# INT8 W8A16 AutoRound, in-checkpoint vision) via SGLang in a +# python3.12 venv. This seam pins what the flags alone do not say: boot +# persistence for a NATIVE runtime (no generated quadlet is involved), the +# repo-relative paths the unit executes, the separate client port, the +# no-spec-decode gate (the checkpoint ships NO mtp.* tensors, so a NEXTN +# draft would be an uninitialised ghost model), the 8-shard payload gate, +# and the KV budget: fp8 KV at 0.98 utilization whose 4-slot grid must fit +# the card's 24 GiB. load helpers UNIT="$REPO_ROOT/scripts/frank-small.service" SERVE="$REPO_ROOT/scripts/serve-frank-small.sh" ENV_FILE="$REPO_ROOT/scripts/frank-small.env" +DOWNLOAD="$REPO_ROOT/scripts/download-frank-small.sh" REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" ENV_LARGE="$REPO_ROOT/scripts/frank-large.env" -@test "frank-small unit, serve script and env exist in scripts/" { +@test "frank-small unit, serve script, env and download script exist in scripts/" { [ -f "$UNIT" ] [ -f "$SERVE" ] [ -f "$ENV_FILE" ] + [ -f "$DOWNLOAD" ] } @test "frank-small is boot-persistent, self-healing and unprivileged" { @@ -35,64 +40,67 @@ ENV_LARGE="$REPO_ROOT/scripts/frank-large.env" grep -q 'FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-small.env' "$UNIT" } -@test "frank-small offloads every layer to the 3090 with flash attention" { - grep -q 'llama.cpp-mainline/build-cuda/bin/llama-server' "$ENV_FILE" - grep -q '^NGL=999$' "$ENV_FILE" - grep -q -- '--gpu-layers "$NGL"' "$SERVE" - grep -q '^FLASH_ATTN=on$' "$ENV_FILE" - grep -q -- '--flash-attn "$FLASH_ATTN"' "$SERVE" - # Prometheus scrapes this endpoint; without --metrics there is no /metrics. - grep -q -- '--metrics' "$SERVE" +@test "frank-small serves the MiMo-V2.6 checkpoint from the self-contained SGLang venv" { + grep -q '^SGLANG_BIN=/home/dank/GitPackages/sglang-cu128/bin/sglang$' "$ENV_FILE" + grep -q '^MODEL_DIR=/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound$' "$ENV_FILE" + grep -q -- '--model-path "$MODEL_DIR"' "$SERVE" + # The AutoRound checkpoint declares its quant method; SGLang must load it + # as auto-round, not re-quantize online. + grep -q '^QUANTIZATION=auto-round$' "$ENV_FILE" + grep -q -- '--quantization "$QUANTIZATION"' "$SERVE" } -@test "frank-small caps its CPU RAM cache below the build default" { - # Without the flag the 8192 MiB default returns and the env cap is a - # silent no-op; this memory-tight host needs the cap to hold. - grep -q -- '--cache-ram "$CACHE_RAM"' "$SERVE" -} - -@test "frank-small drafts with the MTP sidecar chained behind ngram-mod" { - grep -q '^SPEC_TYPE=draft-mtp,ngram-mod$' "$ENV_FILE" - grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE" - # The Gemma4 MTP head is a separate assistant file (gemma4-assistant, 4 - # nextn layers); it loads as --model-draft and shares the target's last - # layers' KV. ngram-mod is draftless, so the chain is legal. - grep -q '^DRAFT_FILE=mtp-gemma-4-12B-it\.gguf$' "$ENV_FILE" - grep -q -- '--model-draft "$MODEL_DIR/$DRAFT_FILE"' "$SERVE" +@test "frank-small runs WITHOUT speculative decoding (no MTP weights shipped)" { + # The config declares mtp_num_hidden_layers=1 but the safetensors index + # carries zero mtp.* tensors (base and INT8 conversion alike, verified + # 2026-10-01). SGLang would draft with an uninitialised 4.24 GiB model + # at 0% acceptance; the serve script must not pass any speculative flag. + ! grep -q -- '--speculative-' "$SERVE" + ! grep -q '^SPEC_' "$ENV_FILE" } @test "frank-small carries the fleet's image path, and only it" { - grep -q '^MMPROJ_PATH=.*mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16\.gguf$' "$ENV_FILE" - grep -q -- '--mmproj "$MMPROJ_PATH"' "$SERVE" + # The vision tower is baked into the safetensors (qwen3_5_vision): no + # separate projector file, and the serve script must not look for one. + ! grep -q '^MMPROJ_PATH=' "$ENV_FILE" + ! grep -q -- '--mmproj' "$SERVE" ! grep -q '^ENABLE_VISION=1$' "$ENV_LARGE" } -@test "frank-small's slot grid is full-window and fits the 24 GiB card" { - local parallel cache_k cache_v ctx fixed per_slot - parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")" - cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")" - cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")" - ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" - [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$ctx" ] - # Every slot must be the model's FULL trained window (262144), not a - # divided one -- CTX_SIZE is the total pool and --parallel divides it. - [ $((ctx / parallel)) -eq 262144 ] - [ $((ctx % parallel)) -eq 0 ] - [ "$cache_k" = "q8_0" ] && [ "$cache_v" = "q8_0" ] - # Fit budget from the measured boot arithmetic on the 3090 (24576 - # MiB): one full-context slot idles at 11,996 MiB, four at 22,346, - # so each slot costs (22346-11996)/3 = 3450 MiB and the fixed part - # (weights, projector, sidecar draft, compute buffers, CUDA context) - # is 11996-3450 = 8546 MiB. If the config asks for more slots than - # the card holds, this inequality fails before deploy instead of the - # KV allocation failing at boot. - fixed=8546 - per_slot=3450 - [ $((fixed + parallel * per_slot)) -le 24576 ] - # Quantized KV on CUDA needs the GGML_CUDA_FA_ALL_QUANTS build option, - # without which prefill collapses about 12x (see the CUDA lesson in - # AGENTS.md). The build-cuda tree must carry it. - grep -q '^FLASH_ATTN=on$' "$ENV_FILE" +@test "frank-small's slot grid is fp8 KV and fits the measured card budget" { + local ctx running kv ssmdtype mambasize required + ctx="$(sed -nE 's/^CONTEXT_LENGTH=([0-9]+)$/\1/p' "$ENV_FILE")" + running="$(sed -nE 's/^MAX_RUNNING=([0-9]+)$/\1/p' "$ENV_FILE")" + kv="$(sed -nE 's/^KV_DTYPE=(.+)$/\1/p' "$ENV_FILE")" + ssmdtype="$(sed -nE 's/^MAMBA_SSM_DTYPE=(.+)$/\1/p' "$ENV_FILE")" + mambasize="$(sed -nE 's/^MAX_MAMBA_CACHE_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" + [ -n "$ctx" ] && [ -n "$running" ] && [ -n "$kv" ] + # fp8 KV is the only dtype whose 16 KiB/token makes four slots fit; + # 64 tokens per MiB of pool. + case "$kv" in fp8*) ;; *) return 1 ;; esac + # The DeltaNet ssm state pool is the hidden fit cost: the engine + # autofills mamba slots to spend a fixed ~5.1 GB budget, so dtype + # alone does not shrink it — the SLOT COUNT must be pinned (bf16, + # bounded). These two gates are what make the measured pool below + # exist; change either and re-measure the pool at boot. + [ "$ssmdtype" = "bfloat16" ] + [ -n "$mambasize" ] && [ "$mambasize" -ge "$running" ] && [ "$mambasize" -le 64 ] + grep -q -- '--mamba-ssm-dtype "$MAMBA_SSM_DTYPE"' "$SERVE" + grep -q -- '--max-mamba-cache-size "$MAX_MAMBA_CACHE_SIZE"' "$SERVE" + # The slot window is inside the model's native 262144 window. + [ "$ctx" -le 262144 ] + # The grid must equal the pinned pool. The engine's autofit picks a + # bigger pool (687,388 tokens) but leaves ~0.53 GB — Triton's + # deferred kernel loads take most of it and eager-prefill activations + # then OOM on the first long request. 606,208 (4 x 151,552) is the + # verified ceiling: 170k single request and 4 concurrent 54k requests + # run against it (2026-10-01). A weights or mamba-flag change moves + # this number; re-measure before trusting the grid. + pinned_pool=606208 + required=$((running * ctx)) + [ "$required" -le "$pinned_pool" ] + grep -q -- '--max-total-tokens "$MAX_TOTAL_TOKENS"' "$SERVE" + grep -q -- '--disable-prefill-cuda-graph' "$SERVE" } @test "frank-small runs on its own port beside frank-large's" { @@ -106,16 +114,21 @@ ENV_LARGE="$REPO_ROOT/scripts/frank-large.env" ! grep -q '^PORT=7771$' "$ENV_LARGE" } -@test "the deploy handler installs the root unit and refuses a missing model" { +@test "the deploy handler installs the root unit and refuses a missing payload" { grep -q 'deploy_llm_small' "$REMOTE_DEPLOY" grep -q 'relink_unit frank-small' "$REMOTE_DEPLOY" grep -q 'systemctl enable frank-small.service' "$REMOTE_DEPLOY" grep -q 'systemctl restart frank-small.service' "$REMOTE_DEPLOY" - grep -q 'Gemma4-12B-QAT' "$REMOTE_DEPLOY" + grep -q 'MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound' "$REMOTE_DEPLOY" grep -q 'refusing to start a unit that cannot load' "$REMOTE_DEPLOY" grep -q '127.0.0.1:7771/health' "$REMOTE_DEPLOY" } @test "frank-small's serve script refuses to start on a half-finished download" { + # The payload is 8 safetensors shards; the serve script counts them and + # the download script pins each one's LFS sha256. grep -q 'frank-small: missing' "$SERVE" + grep -q 'expected 8 safetensors shards' "$SERVE" + grep -q 'model-00001-of-00008.safetensors' "$DOWNLOAD" + grep -q 'model-00008-of-00008.safetensors' "$DOWNLOAD" }