diff --git a/AGENTS.md b/AGENTS.md index 3cb6a79..a4bbda9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,7 +12,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service. - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, - Alloy, Pi-hole) plus TWO LLM services: `frank-large` (stock llama.cpp on + Alloy, Pi-hole) plus TWO LLM services: `frank-large` (vLLM TheRock on the gfx1151 iGPU) and `frank-small` (SGLang on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack @@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 | -| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | `unsloth/Qwen3.8-27B-GGUF` **UD-Q4_K_XL** (27B dense, **vanilla — no longer abliterated**, 16.35 GiB, Apache-2.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), **4 sessions x 163840 ctx, unquantized (f16) KV** — gufo has no KV-quant lever — greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only. Swapped 2026-10-01 from ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB): ~13 GiB less weights is what funds four f16 sessions, at the cost of the uncensored behaviour | 7770 | +| Dankclaw | `frank-large` | vLLM 0.30 in the digest-pinned `docker.io/kyuz0/vllm-therock-gfx1151` image (TheRock: ROCm 10 / torch 2.11, Triton Gated DeltaNet kernels; plain `podman run`, root systemd unit) | `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` **full BF16** (9B dense, 17.53 GiB, 4 shards — the arch frank-small serves as INT8 on the 3090), **cross-model MTP spec decode from `Qwen/Qwen3.5-9B` (2 draft tokens; 19.8 t/s decode measured, +51% over the 13.1 t/s base, which is ~94% of the LPDDR5x BF16 roof)**, 4 seqs x 262144 native ctx at util 0.45, qwen3 reasoning + qwen3_coder tool parsers, text-only (`--language-model-only`). Cut over 2026-10-01 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 | | Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 2 slots of 262,144 (native window; 2026-10-01 reslice from 4 x 151,552 for compaction headroom), bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -209,14 +209,14 @@ archive. **STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: `llama-bench` for the llama.cpp -endpoints, `gufo bench` for frank-large. Rolling your own bench +endpoints; frank-large's vLLM shape is measured by replaying real text through the live server and reading `Server-Timing`. Rolling your own bench requires asking first. Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself. -**`gufo bench` cannot rank speculative-decode settings, and will invert the -ranking.** Measured 2026-09-28 on frank-large at depth 32768: its corpus +(A gufo-era law; frank-large is vLLM since 2026-10-01, but the rollback tag keeps it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.** +Measured 2026-09-28 on frank-large at depth 32768: its corpus reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across draft widths, i.e. the same highly-predictable text every run, and 93 % of every proposal accepted. It reported 29–37 t/s decode where real traffic @@ -241,7 +241,7 @@ one: `ip neigh show ` from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP. 1. Stop and mask (self-healing would restart it): - - Dankclaw `frank-large` (root unit, gufo on the iGPU): + - Dankclaw `frank-large` (root unit, vLLM podman on the iGPU): `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - Dankclaw `frank-small` (root unit, SGLang on the 3090): `sudo systemctl stop frank-small && sudo systemctl mask frank-small` @@ -281,8 +281,8 @@ of 2026-10-01 both `frank-small` and `frank-primary` accept images (`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly (this build never autoloads projectors), and frank-small's MiMo payload bakes its vision tower INTO the checkpoint (no projector file at all); -`frank-large` remains text-only because its gufo deployment loads no -vision projector. Note also that +`frank-large` stays text-only because its vLLM deployment passes +`--language-model-only` — the MiMo vision tower is in the checkpoint; the flag is the lever. Note also that the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` — models.yml sets `reasoningContentField` per endpoint to match. @@ -292,111 +292,75 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01 (the MiMo tower is vision-only). -### frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (Dankclaw, ACTIVE) +### frank-large — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (Dankclaw, ACTIVE) -**gufo** (`gufo-org/gufo`) v0.1.1 — "the Strix Halo inference engine", a -vertical engine built for exactly this hardware (Ryzen AI MAX+ 395 / -Radeon 8060S, `gfx1151`) — as a **root systemd unit** -(`scripts/frank-large.service`, `User=dank`), serving -`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s `UD-Q8_K_XL` -(27B dense, 29.3 GiB, sha256-verified; **abliterated/uncensored**, and under -the *non-OSI* SWIFT OPEN LICENSE 1.0) on port 7770 from the gfx1151 iGPU. -One session, full 262,144 ctx (the model's native window, `SESSIONS=1`), -DFlash2 spec decode (the z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` drafter, gufo's -**adaptive** block policy, `--draft-tokens 7`), greedy sampling, reasoning -extracted to `reasoning_content`, and **no vision projector**. +**vLLM 0.30** in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151` +image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — TheRock +build: ROCm 10.0.0 / torch 2.11.0, Triton Gated DeltaNet kernels, +`vllm/v1/spec_decode/dflash.py` present) — as a **root systemd unit** +(`scripts/frank-large.service`, `User=dank`, `Restart=always`, +`ExecStop=podman stop -t 120`, `TimeoutStopSec` 180 > the stop grace), +serving `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16** (9B dense +qwen3_5 hybrid, 17.53 GiB, 4 sha256-verified shards) on port 7770 from the +gfx1151 iGPU. Official vLLM images are CUDA-only; this fork is the fleet's +gfx1151 path. `scripts/serve-frank-large.sh` gates image-by-repo-digest → +payload (config + index + exact shard count) → spec draft (accepts BOTH +weight layouts: single `model.safetensors` for dflash, sharded index for +the MTP head) → whole `/dev/dri` (never individual renderD nodes), sweeps +the bench identity `vllm-mimo`, then `exec podman wait` so a container +death trips `Restart=always`. Boot ~200–240 s. -**Two swaps landed on 2026-09-28**, both same day: the engine (from the -unsloth llama.cpp b11030 prebuilt, which had replaced Kairic Edge ROCmFPX) -and then the payload (from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on -disk and the payload-only rollback). The two targets are structurally -interchangeable for gufo — same `qwen35` architecture, the same 866 tensors -in the same order, the same per-tensor Q8_0/F32/BF16 layout, the same MTP -head at `blk.64` — verified by parsing and diffing both GGUF headers before -either was loaded. **The DFlash2 drafter did not move**, so it has its own -`DRAFTER_DIR` and the deployer's stray-run sweep keys on the drafter's -filename, not the target's. +**The shape is chosen by roof math, and the spec decode is the +performance feature.** BF16 decode is LPDDR5x-bandwidth-bound here: +13.1 t/s measured = ~94% of the roof for 17.5 GiB of weights — a quant, +not a bug. The only way past it is multiple tokens per weight pass. +Measured 2026-10-01 (one session, temp 0, 28,647-token prefill / 512 +decode): base **17,876 t/s prefill / 13.1 decode**; +DFlash-8 7,976 / +17.4 (acceptance 59%/26% per position); **+MTP-2 7,380 / 19.8 +(acceptance 71%/44%, token rate 57.4%)** — deployed arm. +`num_speculative_tokens` beyond 2 is UNMEASURED; both spec arms pay +~55–59% of prefill. KV grows only on the 8 full-attention layers +(32 KiB/token bf16); the 24 GDN layers hold fixed ~25 MiB state per +sequence — so 4 x 262,144 at util 0.45 fits the GTT aperture easily. -**The runtime is built from source on the host** — -`/home/dank/GitPackages/gufo-v0.1.1/`, reproducible with -`scripts/build-gufo.sh` (`--verify` re-checks an installed tree). gufo ships -no prebuilt Linux binaries and the v0.1.0/v0.1.1 release assets are empty. -Unlike the retired llama.cpp prebuilt, which bundled its own ROCm, **gufo -links the host ROCm 7.2.4 through its RPATH**, so the HIP dev packages must -stay installed. Two host workarounds the build script encodes: +**The MTP head is CROSS-MODEL — the fleet's first.** MiMo ships ZERO +`mtp.*` tensors (the reason frank-small's SGLang NEXTN is a 0.00 ghost: +SGLang reads the head from the TARGET checkpoint); vLLM reads the draft +from a separate path, so `--speculative-config +'{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'` +drafts from the distill's SOURCE model, and 71%/44% acceptance is the +measured proof that the distillation preserved Qwen3.5 hidden-state +structure. The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, fetched; +its "PR #40898 unmerged" card claim is false — dflash ships in vLLM 0.30) +stays on disk: `FRANK_LARGE_SPEC=dflash` + repoint `SPEC_DIR`. -- **Ubuntu's `librocwmma-dev` must be absent.** It installs a truncated - rocWMMA header set into `/usr/include` that shadows AMD's complete one - and breaks the build with `'internal/accessors.hpp' file not found`. A - `-I` cannot fix it — in HIP mode the host pass searches the system - directories first. The package has no reverse dependencies. -- **`--gcc-install-dir` must name GCC 15.** Ubuntu 26.04 ships - `/usr/include/c++/16`, which the ROCm clang prefers; its `` - needs a `std::__detail::__notify_impl` symbol version the installed - `libstdc++.so.6` (GLIBCXX_3.4.35) does not export, and the link fails. +**Spec decode does NOT preserve greedy text here (measured).** +Temperature-0 completions differ across arms (bf16 batched verification +reorders reductions; near-tie argmaxes diverge and cascade). vLLM spec +decode is distribution-preserving, not bitwise-greedy-preserving — do not +chase text drift between arms as a regression. -`gufo diagnose` is the preflight and must report `[PASS]` with `gfx1151`. -It also self-caps compute queues (`GPU_MAX_HW_QUEUES`, 2 resident for -`serve llm`) after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`, which -matters because ComfyUI holds queues on the same device; four HIP processes -is the device ceiling. +**Client contract differs from every llama.cpp-era endpoint:** vLLM emits +thinking in **`reasoning`**, not `reasoning_content` — `models.yml` sets +`reasoningContentField: "reasoning"` for this provider (miss it and +thinking vanishes from transcripts). Tools ride +`--enable-auto-tool-choice --tool-call-parser qwen3_coder`. The harness +extraBody moved to the Qwen3.5-family recipe (0.7/0.8) with +`chat_template_kwargs.{enable_thinking,preserve_thinking}` — a top-level +`preserve_thinking` is ignored by vLLM. Env keys stay +`FRANK_LARGE_`-prefixed (HIP reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE`). -**The client contract changed, and the chat template is the sharp edge.** -**gufo has no Jinja engine at all** — `src/models/qwen/chat_template.cpp` is -587 lines of hand-written C++ that appends strings, with no -`--chat-template-file`, no env override, and no template body to point at. -`scripts/qwen-sharp-v22.4.0.jinja` therefore **cannot** be supplied to -frank-large: that is a port, not a config line. Given the choice on -2026-09-28 the decision was to ship on gufo's stock renderer and accept the -loss — the `terse` system-prompt injection, the in-band `<|think_*|>` -control, the `json|xml` `tool_call_format` switch, the tool-argument -truncation and `add_vision_id` prefixes are all gone. gufo's own renderer -does keep the effort instruction strings and `preserve_thinking`, which are -what the harness depends on. **This is the thing to watch**: the endpoint -now runs an *abliterated* model without the prompt that tells it to stop -waffling. The cheap fix, if verbosity shows up, is to send the `terse` text -as an ordinary system message from `models.yml` — not a template change. -`reasoning_effort` accepts the full OpenAI set (all six levels verified -200), so `models.yml`'s `reasoningEffortMap` fold is no longer *required* — -it is kept deliberately, so that removing it does not silently move every -`high` role from `xhigh` to `high` effort inside a swap. And there is no -KV-quantization or prompt-cache-RAM flag; prompt reuse is on by default and -`--cache-disk` is opt-in and off. +Fetch: `scripts/download-frank-large-mimo.sh` (target), +`scripts/download-frank-large-qwen35.sh` (MTP source), +`scripts/download-frank-large-dflash.sh` (dflash alternative) — all +`-f` + byte-count + LFS sha256 verified, `--verify-only` supported, no +token on the command line. -**Read the perf trade before assuming a win.** gufo's own Q8_K_XL tables -(vs llama.cpp b11069, DFlash2) show prefill +37 % to +80 %, cold load 4.5× -faster, decode +7 % at zero depth and +40 % at 4 users — but **−14 % at -65,536 tokens of depth and −24 % at 131,072**. On the 8-bit target the -crossover is ~32k of cached context, and frank-large runs one 262,144-token -conversation, so long agent sessions sit on the losing side of that line. -The Q4_K_XL target shows no such regression; revisiting the quantization is -a separate decision from the runtime. Peak memory is 50.58 GiB at the full -262,144 context against the iGPU's 114 GiB GTT aperture. - -The serve script (`scripts/serve-frank-large.sh`) validates runtime → target -GGUF → drafter and exits with a pointer message if any is missing, then -`exec`s `gufo serve --host 0.0.0.0 --port 7770 --sessions 1 llm --model -…--served-model-name frank-large --context 262144 --speculative dflash2 …`. -The deployer's `deploy_llm_large` gates on the **runtime binary** as well -as both payload paths, and returns before it stops anything. - -Fetch: `scripts/download-frank-large-swift.sh` (the target; per-file byte -count + sha256 against the Hub LFS oid, no auth header — the repo is public -and a token on the curl command line is world-readable) and -`scripts/download-frank-large.sh` (the DFlash2 drafter, which did not move). -Both take `--verify-only`. - -Runbook: `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`. -Rollback, two ways: **payload only** — re-fetch with -`scripts/download-frank-large.sh` and repoint `MODEL_FILE` at -`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`; that -29.3 GiB target was **reclaimed on 2026-09-28** (staged to -`~/.AGENTTRASH/20260928-retired-models/`, restorable from there), so it is -no longer a one-minute flip. **Engine** — -[[frank-large-unsloth-qwen38-27b-q8xl]], the b11030 prebuilt at -`/home/dank/GitPackages/unsloth-llama-b11030/`; the only way to get -Qwen-Sharp back. The DFlash2 drafter was never reclaimed and still lives in -`Qwen3.8-27B-UD-Q8_K_XL/`. +Runbook: `serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md`. +Rollback: git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 +(`scripts/build-gufo.sh` stays on disk for it), the Q4_K_XL Qwen3.8-27B +payload and the DFlash2 drafter all remain on the host; restore the tag's +unit/env/serve trio and redeploy. ### frank-small — SGLang on the RTX 3090 (Dankclaw, ACTIVE) @@ -827,7 +791,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-large (gufo Qwen3.8-27B UD-Q8_K_XL, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md` | +| frank-large (vLLM TheRock MiMo-V2.6 BF16 + cross-model MTP, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md` | | frank-small (SGLang MiMo-V2.6 INT8, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md` | | frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | @@ -837,6 +801,7 @@ Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| +| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01 | `frank-large-gufo-qwen38-27b.md` | |tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`| | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` | | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` | diff --git a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md index c53f1b9..8d3b8b8 100644 --- a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md +++ b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md @@ -1,7 +1,7 @@ --- type: reference title: "gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)" -description: "The active frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing." +description: "The RETIRED frank-large runbook (2026-09-28 → 2026-10-01, rollback target via tag frank-large-gufo-q4kxl-20261001) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving the abliterated ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU: the from-source build and its two host workarounds, the GGUF header evidence that made the payload swap safe, the one-session / full-window shape, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime and what that costs, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing." tags: - llm - gufo @@ -18,7 +18,11 @@ timestamp: 2026-09-28 # frank-large — gufo, Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (DFlash2) -> **ACTIVE since 2026-09-28.** Replaces +> **RETIRED 2026-10-01** — replaced by [[frank-large-vllm-mimo-v26-bf16]] +> (vLLM TheRock + MiMo-V2.6 BF16). Rollback: git tag +> `frank-large-gufo-q4kxl-20261001`; the gufo build (`build-gufo.sh`), the +> Q4_K_XL payload and the DFlash2 drafter all remain on the host. +> **Originally ACTIVE 2026-09-28 → 2026-10-01.** Replaces > [[frank-large-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth > b11030 prebuilt), which had replaced > [[frank-large-kairic-edge-qwen38-27b]]. Same host, same port (7770), same diff --git a/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md b/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md new file mode 100644 index 0000000..cef2dec --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md @@ -0,0 +1,176 @@ +--- +type: reference +title: "frank-large on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP" +description: "The active frank-large runbook — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU: identity, the measured roof math, the cross-model Qwen3.5-9B MTP drafter and its acceptance curve, the greedy-text numerics trap, the client-contract field change, deployed flags, and what to check when it misbehaves." +tags: + - llm + - frank-large + - dankclaw + - vllm + - strix-halo + - mimo + - mtp + - spec-decode + - bf16 +timestamp: 2026-10-01 +--- + +# frank-large — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP) + +> **ACTIVE since 2026-10-01.** Replaces [[frank-large-gufo-qwen38-27b]] +> (gufo + Qwen3.8-27B GGUF, now the rollback target, git tag +> `frank-large-gufo-q4kxl-20261001`). Same port (7770), same served alias +> (`frank-large`). This is the fleet's second vLLM endpoint and the first +> CROSS-MODEL speculative-decode deployment: the target is Xiaomi's distill, +> the draft head is Qwen's. + +## Identity + +- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw). +- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151` + image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 / + torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM + images are CUDA-only; this fork is the fleet's gfx1151 path. Plain + `podman run` driven by `scripts/serve-frank-large.sh` under a root unit + (`scripts/frank-large.service`, `User=dank`, `Restart=always`, + `ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace). +- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16** + (17.53 GiB, 4 shards) — the same checkpoint frank-small serves as an INT8 + W8A16 AutoRound conversion on the 3090, here unquantized. +- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors + are loaded into the draft pass). + +## Why this shape (the roof math) + +Decode speed for a dense model on Strix Halo is bandwidth-bound: +BF16 weights x one pass per token against the LPDDR5x aperture. Measured +2026-10-01 on this host: **13.1 t/s is ~94% of the roof** — the payload, not +a misconfiguration. The same INT8 payload reaches 83.9 t/s on the 3090 +because it moves ~12 GiB through ~936 GB/s. The only way past a bandwidth +wall is generating more than one token per weight pass: **speculative decode +is the performance feature of this endpoint**, not a garnish. + +The memory side is the opposite of a squeeze: KV grows ONLY on the 8 full +attention layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B); +the 24 Gated DeltaNet layers hold a FIXED ~25 MiB state per running +sequence. `--max-model-len 262144` (the native window) with +`--max-num-seqs 4` at `--gpu-memory-utilization 0.45` boots clean — the +four-session contract the gufo shape ran, with the pool shared by the +scheduler instead of carved per session. + +## Spec method: cross-model MTP — measured, not assumed + +| arm | prefill | decode | acceptance | +|---|---|---|---| +| BF16 base | 17,876 t/s | 13.1 t/s | — | +| BF16 + DFlash 8 tok | 7,976 t/s | 17.4 t/s | pos0 59% pos1 26% (token rate 14.8%) | +| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) | + +One session, temperature 0, 28,647-token prefill / 512-token decode, same +client for all three arms (2026-10-01 bench window, frank-large stopped per +Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft +tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve +decays 59/26/14/7, so 8 proposals are mostly wasted verification. +`num_speculative_tokens` beyond 2 is UNMEASURED; do not raise it without a +fresh A/B. Both spec arms pay ~55–59% of prefill (the draft passes share +the compute queues) — this endpoint's thinking-heavy traffic spends its wall +clock in decode, which is the regime where that trade wins. + +**Why a Qwen head accepts on a MiMo target:** MiMo-V2.6 is a distillation of +Qwen3.5-9B that preserves the hidden-state geometry — identical +4096-hidden/248320-vocab/32-layer/interval-4 structure, and the 71%/44% +per-position acceptance is the direct measurement that the distillation +preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors** +(verified in both the BF16 base and the INT8 conversion) — which is exactly +why frank-small's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang +reads the head from the TARGET checkpoint, vLLM reads the draft from a +separate path. That difference is the whole trick: + +``` +--speculative-config '{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}' +``` + +The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids +[1,5,…,29]) stays fetched on disk; flip `FRANK_LARGE_SPEC=dflash` and repoint +`FRANK_LARGE_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim +is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30. + +## The numerics trap (greedy text is NOT preserved by spec decode) + +Measured: temperature-0 completions DIFFER across arms (sha `1318aeb6` base, +`0208b45d` dflash, `20c41c38` mtp). Batched verification changes the bf16 +reduction order, argmax near-ties diverge, and the divergence cascades. +vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving. +The gufo-era "greedy == autoregressive text" law does not carry over; do not +chase text drift between spec and no-spec arms as a regression. + +## Payload + +- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under + `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard + byte-count + LFS sha256-verified by `scripts/download-frank-large-mimo.sh` + (`--verify-only` re-checks; no token on the curl command line). +- MTP source: `scripts/download-frank-large-qwen35.sh` → + `/home/dank/models/Qwen3.5-9B/` (4 shards behind + `model.safetensors.index.json`; the shards carry the odd HF asset names + `model.safetensors-0000N-of-00004.safetensors` — the index resolves them, + the engine boots from exactly this tree). +- DFlash drafter: `scripts/download-frank-large-dflash.sh` → + `/home/dank/models/Qwen3.5-9B-DFlash/`. +- The serve script gates the image by REPO DIGEST (rootless podman resolves + tags from the network; the pin exists to boot exactly these bytes + offline), then config + index + exact shard count, then the spec draft + (either weight layout: single `model.safetensors` or a sharded index), + then `/dev/kfd` + whole `/dev/dri` (never individual renderD nodes — + Dankserver 2026-09-29 lesson). + +## Client contract + +- **Alias:** `frank-large` (the `id` at `/v1/models`). +- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in + **`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness + `models.yml` compat block for this provider sets + `reasoningContentField: "reasoning"`. Miss that and thinking silently + vanishes from transcripts. +- **Tools:** `--enable-auto-tool-choice --tool-call-parser qwen3_coder`. +- **Vision:** the checkpoint carries the `qwen3_5_vision` tower but the + serve script passes `--language-model-only` — the endpoint stays + text-only by decision; dropping the flag brings vision back. +- **Sampling:** request-side; the harness sends the Qwen3.5-family recipe + (temp 0.7, top_p 0.8) via `compat.extraBody`, and + `chat_template_kwargs.{enable_thinking,preserve_thinking}` — the + `chat_template_kwargs` form is what vLLM reads; a top-level + `preserve_thinking` is ignored there. +- **Env hygiene:** every tunable is `FRANK_LARGE_`-prefixed because HIP + reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports + the env file with `set -a`. + +## Ops + +- Boot ~200–240 s warm; health 503 while loading (do not kill it). +- The serve script sweeps stale container identities before starting: + `frank-large-mimo` itself and the bench-window container `vllm-mimo`. +- Container death → `podman wait` returns non-zero from ExecStart → + `Restart=always` self-heals (a clean exit-0 drain also restarts, the + frank-primary lesson). +- Bench windows: `systemctl stop frank-large` is enough (mask fails on the + relinked unit; the deploy restores). +- First-deploy fact to record here: the boot-time KV pool line + (`Maximum concurrency for 262,144 tokens per prompt: N.Nx`). + +## Rollback + +Git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from +source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B +payload + DFlash2 drafter, all still on the host. Restoring the tag's +unit/env/serve trio and redeploying returns the 27B endpoint. + +# Citations + +- Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B +- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B +- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash +- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151 +- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env` +- Predecessor: [[frank-large-gufo-qwen38-27b]]; sibling on the same arch: + [[frank-small-sglang-mimo-v26]] diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 9240710..a3fc1be 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -10,12 +10,12 @@ headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, uti reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on, measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw -runs two endpoints — -gufo v0.1.1 (the Strix Halo engine, built from source; frank-large, -unsloth Qwen3.8-27B UD-Q4_K_XL (vanilla) + the z-lab DFlash2 Q4_K_M drafter -on the gfx1151 iGPU, 4 x 163840 ctx with f16 KV, port 7770) and mainline llama.cpp (frank-small, -HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090, -port 7771). +runs two endpoints on the same arch — +vLLM (digest-pinned kyuz0 TheRock gfx1151 image; frank-large, +XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 with cross-model MTP +spec decode from the Qwen3.5-9B head on the gfx1151 iGPU, 4 x 262144 ctx, +port 7770) and SGLang (frank-small, the same model as an INT8 W8A16 +AutoRound conversion on the RTX 3090, 2 x 262144 ctx, port 7771). > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -39,14 +39,14 @@ fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] E2[RTX 3090 (PCIe x4 dock) + gfx1151 iGPU 114 GiB GTT] - E[stock llama.cpp (unsloth b11030, gfx1151) + E[vLLM TheRock gfx1151, digest-pinned frank-large, port 7770] - F[Qwen3.8-27B UD-Q8_K_XL + DFlash2 Q4_K_M drafter -262144 ctx / 1 slot, DFlash2 depth 4, q8_0 KV] - G[mainline llama.cpp 37b53fd + F[MiMo-V2.6-Distill-Qwen-9B BF16 + Qwen3.5-9B MTP head +4 x 262144 ctx, spec 2 tok, qwen3/qwen3_coder parsers] + G[SGLang venv cu130 frank-small, port 7771] - H[Gemma4-12B QAT Q4_K_M -4 x 262144 ctx, MTP + vision + audio] + H[MiMo-V2.6 INT8 W8A16 AutoRound +2 x 262144 ctx, pinned pool 606208, in-checkpoint vision] E2 --> E --> F E2 --> G --> H end @@ -162,14 +162,23 @@ frank-small, port 7771] (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[frank-large-gufo-qwen38-27b]] — ACTIVE frank-large runbook since - 2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the - Strix Halo inference engine, built from source on Dankclaw, serving - unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2 - Q4_K_M drafter (adaptive block policy) on the gfx1151 iGPU — one full - 262,144-token conversation, greedy, text-only, the two from-source build - workarounds, the client-contract delta, the measured prefill/decode trade - against llama.cpp, troubleshooting +- [[frank-large-vllm-mimo-v26-bf16]] — ACTIVE frank-large runbook since + 2026-10-01 (root systemd unit, port 7770): the digest-pinned + kyuz0/vllm-therock-gfx1151 image serving + XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU, + 4 x 262144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B + (the fleet's first; 71%/44% per-position acceptance measured) — + the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured + MTP-vs-DFlash table (19.8 vs 17.4 t/s decode), the greedy-text numerics + trap, the `reasoning` field change, troubleshooting +- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-09-28 → 2026-10-01 + (replaced by [[frank-large-vllm-mimo-v26-bf16]]; rollback target via git + tag frank-large-gufo-q4kxl-20261001, engine + payload + drafter still on + the host): gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, + built from source on Dankclaw, serving Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — the two + from-source build workarounds, the client-contract delta, the measured + prefill/decode trade against llama.cpp - [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp (unsloth b11030, ROCm gfx1151) serving @@ -203,12 +212,18 @@ frank-small, port 7771] .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract, troubleshooting (2026-09-16 → 09-19) -- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17 - (root systemd unit, port 7771): mainline llama.cpp serving +- [[frank-small-sglang-mimo-v26]] — ACTIVE frank-small runbook since + 2026-10-01 (root systemd unit, port 7771): SGLang in a self-contained + python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16- + AutoRound on the RTX 3090 — the same architecture frank-large now serves + in BF16 on the iGPU; 2 x 262144 ctx on a pinned 606,208-token fp8 pool, + NO spec decode (zero mtp.* tensors shipped), in-checkpoint vision. +- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by + [[frank-small-sglang-mimo-v26]]; rollback target, payload and binary + intact on disk): mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX 3090, with the separate MTP assistant sidecar chained behind ngram-mod, - q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. A same-day - vLLM detour (2026-09-19) was reverted the same day; see below. + q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. - [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert, rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16 Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647- diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 455efd7..e06f2b0 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -875,3 +875,40 @@ any repo service. verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live contract probes (reasoning split, tool_calls). Rollback target for the 2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`. + +## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP + +- **Question that started it:** can a Qwen3.5 MTP drafter attach to + frank-small, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1: + not on frank-small — SGLang reads the MTP head from the TARGET checkpoint + and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the + digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 / + TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and + vLLM reads the draft from a SEPARATE path — so cross-model MTP from + `Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer + geometry) works. Frank's call: ship that as frank-large. +- **Bench window (frank-large stopped, one session, temp 0, 28,647-token + prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1 + is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug); + +DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 / + 19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec + arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured. +- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy + text (bf16 batched verification reorders reductions; near-tie argmaxes + cascade). vLLM spec decode is distribution-preserving, not + bitwise-greedy-preserving. +- **Client-contract delta wired:** vLLM emits `reasoning`, not + `reasoning_content` - `models.yml` frank-large block now sets + `reasoningContentField: "reasoning"`, and the extraBody moved to the + Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs` + (a top-level `preserve_thinking` is ignored by vLLM). +- Deployer gap found pre-deploy: the spec-draft gate demanded a single + `model.safetensors` (the dflash layout) and would have rejected the + deployed MTP arm (4 shards behind an index); fixed to accept both + layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898 + unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in + 0.30 (verified in-image); the drafter stays on disk as the alternative. +- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB, + DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo + rollback pinned at git tag `frank-large-gufo-q4kxl-20261001`; engine, + Q4_K_XL payload and DFlash2 drafter all remain on the host.