frank-small: SGLang MiMo-V2.6-Distill-Qwen-9B INT8 (qwen3_5 hybrid, RTX 3090)

Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback). SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0 nvcc toolchain for the sgl-kernel/flashinfer JIT. Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba states pinned at 32 slots (the engine autofills the ssm pool to a fixed ~5.1 GB regardless of dtype), eager prefill (no headroom for graph capture), no spec decode (the checkpoint ships no mtp.* tensors despite mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB ghost at 0% acceptance). Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k requests, vision (in-checkpoint tower), reasoning_content field.
2026-10-01 06:55:48 -07:00
parent 7da504a651
commit 0c5e054fb8
12 changed files with 1005 additions and 328 deletions
@@ -0,0 +1,156 @@
+---
+type: reference
+title: "frank-small on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16"
+description: "The active frank-small runbook — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode shape, the CUDA 13 nvcc toolchain the JIT needs, deployed flags, the client contract, and what to check when it misbehaves."
+tags:
+  - llm
+  - frank-small
+  - dankclaw
+  - sglang
+  - rtx3090
+  - qwen
+  - mtp
+  - vision
+  - int8
+timestamp: 2026-10-01
+---
+
+# frank-small — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
+
+> **ACTIVE since 2026-10-01.** Replaces [[frank-small-gemma4-12b]]
+> (llama.cpp Gemma4-12B QAT Q4_K_M, now the rollback target). Same port
+> (7771), same alias (`frank-small`), same client contract surface.
+
+## Identity
+
+- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
+- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
+  `/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
+  transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
+  systemd unit `scripts/frank-small.service` (`User=dank`,
+  `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-small.sh` with
+  tunables from `scripts/frank-small.env`.
+- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
+  (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
+
+## Shape
+
+- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
+  (4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
+  DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
+  tower (27 layers) in the safetensors — no separate mmproj file.
+- **Context:** four 151,552-token slots (`--context-length 151552`,
+  `--max-running-requests 4`) against a **pinned** 606,208-token pool
+  (`--max-total-tokens`). A single request may grow to the whole pool —
+  a 170,019-token prompt is verified — the grid is the four-way
+  concurrency contract.
+- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
+  dim x 1 B), pool 9.31 GiB.
+- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
+  ssm state pool is a first-class consumer. The engine autofills the
+  mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
+  (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
+  pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
+  MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
+  leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
+  CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
+  dies in capture, or later in the first long request once Triton's
+  deferred kernel loads eat the remainder). Hence
+  `--disable-prefill-cuda-graph`; decode graphs are untouched.
+- **Quantization:** `--quantization auto-round` — the checkpoint's
+  `quantization_config.json` declares `quant_method: "auto-round"` and
+  SGLang loads it natively (vision tower, `embed_tokens` and
+  `linear_attn.in_proj` stay fp16).
+
+## Spec method
+
+**None — and this is a checkpoint fact, not a tuning choice.** The
+`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
+safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
+`model.safetensors.index.json` for both this INT8 conversion and the base
+`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
+`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
+boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
+decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
+paying the 4-token verify tax for no token. The serve script passes no
+`--speculative-*` flag, and the seam gates that.
+
+## CUDA 13 toolchain (the sharp edge)
+
+The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
+flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
+venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
+`--compress-mode`) and its headers clash with the 13.0 device passes.
+
+Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):
+
+| Package | Supplies |
+|---|---|
+| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
+| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
+| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
+| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
+| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
+| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |
+
+Three local modifications the tree carries:
+
+1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
+   `-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
+2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
+   `../../../include` / `../../../lib64` (the deb ships them as real
+   near-empty dirs; the stub pass must see the 13.0 headers, not
+   `/usr/include` 12.4).
+3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
+   includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
+   `noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
+   the unpatched 13.0 declarations are incompatible in the C++ host stub).
+
+JIT caches: `~/.cache/sglang/jit/sm86/` and
+`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
+change lands.
+
+## Payload
+
+Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
+files under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`,
+all LFS sha256-verified by `scripts/download-frank-small.sh` (`--verify-only`
+re-checks). The serve script refuses to start with a missing shard.
+
+## Client contract
+
+- **Alias:** `frank-small` (the `id` at `/v1/models`).
+- **Reasoning:** extracted to **`reasoning_content`** (live-verified
+  2026-10-01 — this SGLang build's qwen3 parser uses the same field name
+  as llama.cpp; models.yml sets `reasoningContentField:
+  "reasoning_content"`). `--reasoning-parser qwen3`.
+- **Tools:** `--tool-call-parser qwen3_coder`.
+- **Vision:** supported (in-checkpoint tower).
+- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
+  `--top-k` — sampling rides on the request; the harness sends the Qwen
+  recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
+- **Reasoning effort:** the full OpenAI set is accepted (the chat template
+  has no effort gate).
+
+## Measured
+
+- Boot ~60 s with warm JIT caches (several minutes cold — the first boot
+  compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
+  above is what makes that compile succeed).
+- Decode **83.9 tok/s** (no spec decode — nothing to accept).
+- Prefill ~2,950–3,370 t/s eager (a 170,019-token prompt in 50.5 s).
+- 4 concurrent 54,027-token requests all answer in ~17 s (216k tokens of
+  live KV at once).
+- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
+
+## Rollback
+
+[[frank-small-gemma4-12b]] — the llama.cpp Gemma4-12B QAT payload and binary
+remain intact on disk; its unit/env/serve trio is in `old/` after the
+cutover.
+
+# Citations
+
+- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
+- SGLang: https://github.com/sgl-project/sglang
+- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: "halogen-flash-server on Dankclaw (frank-large)"
-description: "The RETIRED frank-large runbook (2026-09-16 → 09-19, rollback target) — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. Replaced by [[frank-large-ciru-orca]] on 2026-09-19."
+title: halogen-flash-server on Dankclaw (frank-large)
+description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
 tags:
  - llm
  - halogen
@@ -13,3 +13,466 @@ tags:
  - rocm
 timestamp: 2026-09-13
 ---
+
+# halogen-flash-server on Dankclaw (frank-large)
+
+> **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp /
+> kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
+> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
+> [[frank-large-ik-uncensored]], the current rollback), and came back as the
+> shape the fleet asked for: two 200k-token conversations rather than the
+> native-context pair. Deployed as a rootless Quadlet container,
+> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090
+> it does not touch now belongs to [[frank-small-qwen38-gsq-rco]].
+
+The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
+architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
+Upstream's own comparison table measures it at **roughly 4x the prefill
+throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
+ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
+prompt with real context. That, not decode, is the reason for the swap.
+
+## Shape
+
+One container runs both halves of the stack:
+
+| half | port | reachable |
+|---|---|---|
+| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
+| OpenAI front-end | 8731 | published as host **7770** |
+
+The `all` mode is upstream's default and the shape a single-container host
+wants: the front-end talks to the engine over loopback inside the container, so
+there is one process tree, one log, and one published port.
+
+`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
+`/health`, and in every response body. Requests are **never rejected for naming
+a different model** — the server serves whatever asks. Keeping the fleet alias
+is therefore free, and it means Hermes, the harness config, the OpenWebUI
+discovery, the `services.json` probe and the docs needed no rename.
+
+## Weights
+
+`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):
+
+| file | size | role |
+|---|---|---|
+| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
+| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
+| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
+| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
+
+`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
+only** — the entrypoint acts only when the checkpoint is absent, so a restart
+never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
+mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
+`/home/dank/models/halogen`.
+
+**Refreshing the weights.** Delete the mount's contents (or the whole
+directory, then recreate it **empty**) and start the unit: the entrypoint sees
+the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
+traps. First, the bind mount's source directory must *exist* — podman refuses
+the container with `Error: statfs /home/dank/models/halogen: no such file or
+directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
+directory and the fetch starts on the next start. Second, "refresh" only means
+"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
+(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
+2,383,306,048 → **2,478,095,488**) while the checkpoint returned
+byte-identical in size (124,068,083,904) — so an update can live entirely in
+the sidecars, and equal file sizes are not proof of an unchanged repo.
+
+`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
+refresh) comes along with the repo but is unused: it trades the `o_proj`
+calibration for about 2 % of decode, and the quality sidecar is the default for
+a reason.
+
+## Deployed flags
+
+Everything else is configured by environment variable — there is no config
+file, and the flags are read once at startup. These are the departures from
+upstream's defaults:
+
+| flag | value | why |
+|---|---|---|
+| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
+| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
+| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
+| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
+| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
+| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
+| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
+| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
+| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
+
+Everything else is upstream's default, deliberately: prompt cache mode `2`
+(two entries per conversation — end-of-system-prompt and end-of-history — which
+already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
+stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
+temperature 0, and it stands down whenever a second conversation is
+generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
+gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
+
+## Memory model
+
+Three numbers explain the whole box:
+
+- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
+  GPU in place, never copied. The kernel counts the locked pages as reclaimable
+  file cache, so `free`, `MemAvailable` and every dashboard built on them
+  **overstate free memory by the size of the model**. The server prints the
+  corrected figure at startup; believe that line, not Grafana.
+- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
+  ~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger
+  reads the two together in the engine's own words: `400384 positions, ~27.2
+  GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so
+  ~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a
+  vision tower. The engine's device-side budget stops at roughly 47 GiB on a
+  128 GB machine *with the weights pinned* — and lower on a machine carrying
+  more besides this server, which Dankclaw is.
+- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
+  table is read through. Too little and long prompts crawl with the disk busy
+  and the process in uninterruptible sleep — that is a cache problem, not a
+  memory problem, and the fix is a *smaller* pool, not a bigger one.
+
+`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
+the pool itself if the configured one will not fit, printing what it chose.
+Read those lines on every restart; they are the ground truth for sizing.
+
+**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own
+startup account, including three deliberate experiments:
+
+| pool | max_tok | reserve | KV | arena | need | budget | verdict |
+|---|---|---|---|---|---|---|---|
+| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
+| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
+| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
+| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
+| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
+| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
+| **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** |
+| 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune |
+
+### The sizing rule
+
+One budget covers the pool, the arena **and** the vision tower together:
+
+```
+device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
+              = 121.2    - 67.7             - reserve
+                                    reserve 24 -> 29.5 GiB
+                                    reserve 15 -> 38.5 GiB
+
+need = KV (28 KiB/position) + arena + vision (1.97 measured)
+arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
+```
+
+The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
+121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
+Use the unrounded value. Two of the three terms are fixed hardware and
+checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which
+buys the *fit margin*, not the host's memory. It is a term in this inequality,
+not an allocation, and the section at the end of this page shows the live test
+that proves it.
+
+Trust the engine's printed requirement over this model wherever they disagree:
+a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5
+of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close
+enough to plan with, and the authoritative number to cut the other way.
+
+The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
+point to 0.1 GiB — including upstream's three published "on the device" figures
+(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
+**pool + arena totals at the default `max_tok`**, not pool alone. That is why
+their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
+49.4. Reading those figures as pool-only is the trap that makes a doubling look
+affordable when it is not.
+
+At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
+13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
+ceiling, not a conservative setting.** The corner check agrees: at 32768 only
+7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
+262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
+= 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
+fits unmodified.
+
+### What the reserve actually costs
+
+The device budget is not the whole story. Memory position by configuration
+(401,408 is the 2026-09-13 deployed row, read from its boot ledger):
+
+| config | held | host left | contiguous 2 MiB blocks |
+|---|---|---|---|
+| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
+| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
+| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
+| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** |
+| **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** |
+
+At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
+engine volunteered the stall note. It did not print it at the 401,408 trim, and
+it prints it again after the 2026-09-16 retune. Three data points, and they do
+not line up with a "free memory below X" reading: 310 blocks printed, 1396 did
+not, 475 did. So treat the note as "the machine is tight" and read the
+`host memory left` line rather than waiting to be told:
+
+```
+halogen: NOTE this server now holds most of the machine, and 310 contiguous
+2 MiB blocks remain. It runs best on a host of its own. If other large
+processes share this machine they will compete for what is left, and when it
+runs out BOTH they and this server can stall for minutes at a time at full
+CPU with no disk activity and no output, which is not a crash and needs no
+restart.
+```
+
+That is the failure mode to watch, and it is the least diagnosable of the
+three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
+looks like a hang, for the neighbours as much as for frank-large. `free` is
+useless for spotting it — `MemAvailable` still reads ~74 GiB because the
+page cache counts as reclaimable. Watch the `host memory left for everything
+else` line and the contiguous-block count at boot instead.
+
+28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
+so it is survivable.
+
+### The reserve cannot buy air — ceiling 16
+
+The last row above is the experiment that proves it. Raising the reserve from
+15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1
+GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an
+allocation; the actual footprint is `weights + KV + arena + vision`, none of
+which the reserve touches. So the knob has exactly two regimes:
+
+| reserve | budget | vs required 36.5 | effect |
+|---|---|---|---|
+| 15 | 38.5 | +2.0 | deployed |
+| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless |
+| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term |
+| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** |
+| 24 | 29.5 | −7.0 | clamps the pool to 262144 |
+
+The 36.5 requirement is the engine's own, read from a clamped boot:
+`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only
+thing a bigger reserve can do is take the doubling away.
+
+**To actually free host memory, change the pool or the arena**, and the reserve
+then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
+7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both
+are one-line edits that the fit check will confirm on the next boot. The
+2026-09-13 trim to 401,408 was this remedy applied at half strength —
+`HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB
+with a 4.5x improvement in contiguous blocks (310 → 1396), less than the
+modelled 3.4 because the engine holds the freed KV as working slack.
+
+### The `WARNING` line only appears when the fit check had to move
+
+This one cost me a wrong conclusion, so it is worth stating plainly. When the
+configured pool already fits, the engine reserves it **and prints nothing** —
+the 09-11 baseline boot goes straight from the pre-flight line to
+`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
+its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
+after a clamp**.
+
+So the absence of a warning in old logs is not evidence of headroom. Both
+`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
+baseline has lived in that state since it was deployed without incident,
+because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
+at that point). The engine's budget is a conservative floor, not a hard wall.
+
+### Two failure modes, and only one is dangerous
+
+Raising `max_tok` past the ceiling does not shrink the pool — the arena alone
+outgrows the budget, and the server cannot allocate at all:
+
+```
+kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB).
+kv pool: WARNING even 262144 positions (the context, the smallest pool that can
+         serve one request) needs ~44.5 GiB and host RAM cannot spare it.
+dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)
+```
+
+This is the quiet one: the unit stays `active`, the container reports
+`Up (starting)`, and `deploy.sh` sits in its health wait for the whole
+`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a
+large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be
+slower".
+
+Under-running the ceiling is far more benign and far easier to miss. At 524288
+with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and
+`deploy.sh` still reported healthy in 12 seconds:
+
+```
+kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
+         RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).
+```
+
+A green health check does not mean the pool you asked for is the pool you got —
+read the `kv pool:` lines after every restart.
+
+The 47.7 GiB n-gram lookup table is never held in RAM; it is read through
+the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no
+`/proc/meminfo` figure at all.
+
+**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
+decode numbers were measured with, and it means this container holds most of
+the host. If a co-tenant then takes the remainder, both it and the engine can
+stall for minutes at a time at full CPU with no disk activity and no output.
+That is documented behaviour, not a crash, and it needs no restart.
+`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
+decode speed — a last resort, not a tuning option.
+
+## Client contract
+
+The differences from llama.cpp that bite are all in the request, not the
+deployment:
+
+- **No server-side sampler defaults.** `temperature` absent or `0` is
+  **greedy** decode; above 0 the request samples on the same drafter. Supported
+  fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
+  `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
+  with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
+  value outside its defined range (values are refused, never clamped).
+  The fleet runs greedy for clients that stay silent — deterministic, and
+  byte-identical with the drafter.
+- **The token budget covers thinking.** The default is **8192**, and a budget
+  that runs out mid-thought returns `finish_reason: "length"` with an **empty
+  `content`** and the partial reasoning in `reasoning_content` — which most
+  clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
+  request is a 400 rather than a silent truncation. `max_tokens`,
+  `max_completion_tokens` and `max_output_tokens` all work and must agree if
+  more than one is sent.
+- **A budget above (window − compaction-threshold) is a guaranteed 400 zone.**
+  The fit check (`prompt + max_tokens <= HALOGEN_CTX`) refuses, and the harness
+  only compacts at its own threshold — with OMP at 70 % of a 200,192 window
+  (140,134), any prompt in (136,192, 140,134] with a 64,000 budget overflows
+  the window while sitting below the compaction trigger. That collided live on
+  2026-09-17 (prompt 138,898 + 64,000 = 202,898 > 200,192). The fix is
+  harness-side: keep the declared `maxTokens` at or under
+  `window − threshold` (57,344 chosen) so no pre-compaction request can miss
+  the fit check. Do not fix it by raising `HALOGEN_CTX` past pool/2 — that
+  breaks the two-resident-conversations property.
+- **`reasoning_effort`** is accepted at the top level and inside
+  `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
+  model's default is `xhigh`. `minimal` aliases the template's `low` and still
+  reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
+  honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
+- `/v1/models`, `/health` and completions all report the served id, and
+  `reasoning_content` carries the reasoning trace.
+
+**Pool admission is the one arithmetic trap.** A request reserves its prompt
+**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
+the request waits in arrival order, and after the queue timeout (3600 s) it
+gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
+the pool can therefore never be admitted at all. The 2026-09-16 shape is a
+400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two
+conversations at the ceiling fit the pool exactly, so a pair that is both
+near-full and both generating is pool-bound — the second waits for the first to
+release its region. The pool binds **how many** conversations stay resident,
+not how long one may be; the older 524,288 / 401,408 numbers live in the ledger
+tables above.
+
+## Vision
+
+**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
+the tower is a 1.97 GiB charge against the same device budget as the pool and
+the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]],
+which loads the BF16 projector alongside its own weights. This server refuses
+image content parts and says why, rather than ignoring them:
+
+```json
+{"error": {"message": "this server does not accept images: the engine was
+started without a vision tower (HALOGEN_VISION_TOWER)", "type":
+"invalid_request_error"}}
+```
+
+The llama.cpp side handles images the same way this engine used to: a `data:`
+URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs
+refused deliberately (fetching one would make the server issue outbound
+requests to wherever a client pointed it). Audio and video have no path
+anywhere in the fleet — that left with [[frank-npu]].
+
+## Deploy and operate
+
+```bash
+./scripts/deploy.sh dankclaw --service frank-large
+```
+
+The handler retires the ik-era **root** unit first (stop, disable, move
+`/etc/systemd/system/frank-large.service` into
+`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
+`disable` may delete the file outright, which the handler now reports), stops
+the Quadlet's own unit if it is running, then pulls
+`ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the
+`localhost/` alias the unit references, and **waits for port 7770 to be free**
+before installing and starting anything.
+
+That last gate is not paranoia. On the first cutover attempt the ik server
+logged "Deactivated successfully", and pasta — rootless podman's userspace
+network stack, which binds the published port at container CREATE — still
+reported `Address already in use` a second later, exiting the unit with code
+126. Only `Restart=on-failure` recovered it, minutes later. A deploy that
+starts a doomed container has not deployed anything, so `wait_port_free` polls
+every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with
+the socket table if it never clears.
+
+Boot persistence comes from `[Install] WantedBy=default.target` in the unit
+file plus linger on the deploy user — generated units **cannot** be
+`systemctl --user enable`d on this host.
+
+```bash
+systemctl --user status frank-large.service     # the generated unit
+podman logs frank-large | head -40              # version banner + memory ledger
+curl -s http://127.0.0.1:7770/health            # 200 only after the engine PONGs
+curl -s http://127.0.0.1:7770/v1/models
+```
+
+`/health` asks the **engine** for a PONG before answering, so a 200 means
+"ready to serve", not merely "process alive". During the first-start download
+it stays unhealthy by design.
+
+## Troubleshooting
+
+| symptom | cause |
+|---|---|
+| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
+| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
+| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
+| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
+| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
+| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |
+
+`HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is
+already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is
+deliberately **not** set: it logs every allocation over 64 MB, which is
+steady-state noise on a shared host — raise it only for a live OOM report.
+The lines worth grepping first:
+
+```bash
+podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
+```
+
+Retrieval: `podman logs frank-large` covers the **current** instance only —
+the Quadlet runs `--replace --rm`, so everything before the last recreate is
+gone from the container log. `journalctl --user -u frank-large.service` is
+the durable record. Both are rootless: `sudo podman logs frank-large` looks
+in root's storage and answers "no such container".
+
+## Rollback
+
+The predecessor is intact: the ik payload stays at
+`/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the
+ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env,
+serve script and the Qwen-Sharp template are archived in the repo under
+`scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back
+is a `git revert` of this change set plus a deploy — **but the Quadlet must be
+removed from `~/.config/containers/systemd/` first**, or the restarted root
+unit and the still-installed container both claim port 7770. The port gate in
+the handler now covers exactly that race. See [[frank-large-ik-uncensored]].
+
+The `amd_iommu=off` kernel argument that came with this change is separate and
+reverts on its own — see [[dankclaw-iommu-translation-off]].
+
+## Citations
+
+- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
+- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
+- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model
@@ -269,10 +269,16 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since
-  2026-09-17: mainline llama.cpp serving
+- [[frank-small-sglang-mimo-v26]] — the ACTIVE frank-small runbook since
+  2026-10-01: SGLang in a self-contained python3.12 venv serving
+  letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
+  hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
+  Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
+  4 x 151,552 slots, bf16 mamba states, eager prefill
+- [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
+  mainline llama.cpp serving
  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
-  (MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on
+  (MTP assistant sidecar + ngram-mod, q8_0 KV) on
  Dankclaw's RTX 3090, port 7771
 - [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
  rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16