diff --git a/AGENTS.md b/AGENTS.md index a4bbda9..bcc538b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,8 +12,8 @@ merge absorbed the former `vllmScripts` repo, history preserved). Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service. - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, - Alloy, Pi-hole) plus TWO LLM services: `frank-large` (vLLM TheRock on - the gfx1151 iGPU) and `frank-small` (SGLang on the RTX 3090). Caddy on + Alloy, Pi-hole) plus TWO LLM services: `frank-large` (ROCmFPX llama.cpp + fork on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts. - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside @@ -25,8 +25,8 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 | -| Dankclaw | `frank-large` | vLLM 0.30 in the digest-pinned `docker.io/kyuz0/vllm-therock-gfx1151` image (TheRock: ROCm 10 / torch 2.11, Triton Gated DeltaNet kernels; plain `podman run`, root systemd unit) | `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` **full BF16** (9B dense, 17.53 GiB, 4 shards — the arch frank-small serves as INT8 on the 3090), **cross-model MTP spec decode from `Qwen/Qwen3.5-9B` (2 draft tokens; 19.8 t/s decode measured, +51% over the 13.1 t/s base, which is ~94% of the LPDDR5x BF16 roof)**, 4 seqs x 262144 native ctx at util 0.45, qwen3 reasoning + qwen3_coder tool parsers, text-only (`--language-model-only`). Cut over 2026-10-01 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 | -| Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 2 slots of 262,144 (native window; 2026-10-01 reslice from 4 x 151,552 for compaction headroom), bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 | +| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 | +| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) on the RTX 3090, **2 × 262,144** (native window) with **asymmetric KV f16 K + q8_0 V** under flash attention, 20,164/24,576 MiB VRAM, 100.4–101.8 tok/s decode, text-only. Swapped 2026-10-02 from Q6_K; rollback `models/MiMo-Q6K/` | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -209,13 +209,13 @@ archive. **STOP the production service BEFORE benchmarking** — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: `llama-bench` for the llama.cpp -endpoints; frank-large's vLLM shape is measured by replaying real text through the live server and reading `Server-Timing`. Rolling your own bench +endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-small were measured. Rolling your own bench requires asking first. Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself. -(A gufo-era law; frank-large is vLLM since 2026-10-01, but the rollback tag keeps it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.** +(A gufo-era law; frank-large is ROCmFPX llama.cpp since 2026-10-02, but the rollback tag keeps it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.** Measured 2026-09-28 on frank-large at depth 32768: its corpus reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across draft widths, i.e. the same highly-predictable text every run, and 93 % of @@ -277,12 +277,12 @@ When renaming a service, retiring an endpoint, or moving a model: update the provider block, model `id`, `contextWindow`, `maxTokens`, `input` in `models.yml`, then fix every `modelRoles` entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As -of 2026-10-01 both `frank-small` and `frank-primary` accept images -(`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly -(this build never autoloads projectors), and frank-small's MiMo payload -bakes its vision tower INTO the checkpoint (no projector file at all); -`frank-large` stays text-only because its vLLM deployment passes -`--language-model-only` — the MiMo vision tower is in the checkpoint; the flag is the lever. Note also that +of 2026-10-02 `frank-large` (the BF16 mmproj that ships inside its GGUF +payload) and `frank-primary` (loads the BF16 mmproj explicitly — this build +never autoloads projectors) accept images (`input: [text, image]`); +`frank-small` is **text-only** — the GGUF payload it moved to on 2026-10-02 +ships no projector, and the in-checkpoint MiMo vision tower exists only in the +safetensors layouts. Note also that the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` — models.yml sets `reasoningContentField` per endpoint to match. @@ -292,125 +292,114 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01 (the MiMo tower is vision-only). -### frank-large — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (Dankclaw, ACTIVE) +### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE) -**vLLM 0.30** in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151` -image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — TheRock -build: ROCm 10.0.0 / torch 2.11.0, Triton Gated DeltaNet kernels, -`vllm/v1/spec_decode/dflash.py` present) — as a **root systemd unit** -(`scripts/frank-large.service`, `User=dank`, `Restart=always`, -`ExecStop=podman stop -t 120`, `TimeoutStopSec` 180 > the stop grace), -serving `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16** (9B dense -qwen3_5 hybrid, 17.53 GiB, 4 sha256-verified shards) on port 7770 from the -gfx1151 iGPU. Official vLLM images are CUDA-only; this fork is the fleet's -gfx1151 path. `scripts/serve-frank-large.sh` gates image-by-repo-digest → -payload (config + index + exact shard count) → spec draft (accepts BOTH -weight layouts: single `model.safetensors` for dflash, sharded index for -the MTP head) → whole `/dev/dri` (never individual renderD nodes), sweeps -the bench identity `vllm-mimo`, then `exec podman wait` so a container -death trips `Restart=always`. Boot ~200–240 s. +The **kingjones30/ROCmFPX llama.cpp fork** at commit +`36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built for gfx1151 by +`scripts/build-rocmfpx.sh` into `/home/dank/GitPackages/kingjones-ROCmFPX/build`, +as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`, +`Restart=always`) running a bare `llama-server` (no container) on port 7770 +from the Strix Halo iGPU. Boot to healthy: **16 s**. -**The shape is chosen by roof math, and the spec decode is the -performance feature.** BF16 decode is LPDDR5x-bandwidth-bound here: -13.1 t/s measured = ~94% of the roof for 17.5 GiB of weights — a quant, -not a bug. The only way past it is multiple tokens per weight pass. -Measured 2026-10-01 (one session, temp 0, 28,647-token prefill / 512 -decode): base **17,876 t/s prefill / 13.1 decode**; +DFlash-8 7,976 / -17.4 (acceptance 59%/26% per position); **+MTP-2 7,380 / 19.8 -(acceptance 71%/44%, token rate 57.4%)** — deployed arm. -`num_speculative_tokens` beyond 2 is UNMEASURED; both spec arms pay -~55–59% of prefill. KV grows only on the 8 full-attention layers -(32 KiB/token bf16); the 24 GDN layers hold fixed ~25 MiB state per -sequence — so 4 x 262,144 at util 0.45 fits the GTT aperture easily. +**The runtime is not interchangeable — it is chosen by the FILE.** +`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF`'s best rung, +`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` (28,214,425,952 B, sha256 `f3ef2bda…`, +ftype 115 ≈ 8.39 bpw), stores **311 tensors of type 103, +`GGML_TYPE_Q8_0_ROCMFPX`** — a quantization that exists in no other loader: +stock llama.cpp and gufo both refuse it at load (`unknown tensor type`). The +file is 353 F32 + 187 plain-Q8_0 + 311 ROCmFPX; the AGENT routing variant +keeps the output-head tensors at plain Q8_0 (187 vs 2 in the non-AGENT file), +which is why the card reports MTP acceptance 0.953 vs 0.911 — highest quality +AND the better drafter base. Vision comes from the bundled +`mmproj-Qwen3.8-27B-BF16.gguf` (931,145,888 B). -**The MTP head is CROSS-MODEL — the fleet's first.** MiMo ships ZERO -`mtp.*` tensors (the reason frank-small's SGLang NEXTN is a 0.00 ghost: -SGLang reads the head from the TARGET checkpoint); vLLM reads the draft -from a separate path, so `--speculative-config -'{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'` -drafts from the distill's SOURCE model, and 71%/44% acceptance is the -measured proof that the distillation preserved Qwen3.5 hidden-state -structure. The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, fetched; -its "PR #40898 unmerged" card claim is false — dflash ships in vLLM 0.30) -stays on disk: `FRANK_LARGE_SPEC=dflash` + repoint `SPEC_DIR`. +**The pin is load-bearing.** The 2026-08-25 ROCmFPX builds segfaulted in +`ggml_compute_forward_set_rows` on exactly this file; `build-rocmfpx.sh` +checks the commit out by SHA, so a fork-side force-push cannot silently +re-break the endpoint. -**Spec decode does NOT preserve greedy text here (measured).** -Temperature-0 completions differ across arms (bf16 batched verification -reorders reductions; near-tie argmaxes diverge and cascade). vLLM spec -decode is distribution-preserving, not bitwise-greedy-preserving — do not -chase text drift between arms as a regression. +**Spec decode is the performance feature, and it is measured.** The draft is +the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (1,680,271,648 B, sha256 `051a1764…`, +`nextn_predict_layers = 1`), pinned to the same card: +`--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0 +--spec-draft-p-min 0.0 --spec-draft-device ROCm0 --spec-draft-ngl 99`. +Measured 2026-10-02 in one session, temp 0, 2,202-token prompt / 512 tokens, +`cache_prompt: false`, production service stopped for each arm: +**MTP on 17.00–17.14 t/s decode** (draft acceptance 0.462, mean acceptance +length ≈ 1.86) against **7.60–7.66 t/s with the draft removed — +123 %**; +prefill pays 8–15 % (381 → 325–351 t/s). Acceptance is far from the card's +0.95 (that number is the card's own text at a shallow prompt); the gain is +real and n-max 4 is where it was measured, never swept. This replaces the +gufo Q4_K_XL arm at ~15 t/s, so the endpoint got higher quality AND more +speed at once. -**Client contract differs from every llama.cpp-era endpoint:** vLLM emits -thinking in **`reasoning`**, not `reasoning_content` — `models.yml` sets -`reasoningContentField: "reasoning"` for this provider (miss it and -thinking vanishes from transcripts). Tools ride -`--enable-auto-tool-choice --tool-call-parser qwen3_coder`. The harness -extraBody moved to the Qwen3.5-family recipe (0.7/0.8) with -`chat_template_kwargs.{enable_thinking,preserve_thinking}` — a top-level -`preserve_thinking` is ignored by vLLM. Env keys stay -`FRANK_LARGE_`-prefixed (HIP reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE`). +**Shape and budget:** ONE session at the native 262,144 window, unquantized +f16 KV (~64 KiB/token → ~16 GiB, affordable inside the iGPU's ~114 GiB GTT +aperture), `--fit off --flash-attn on --direct-io --ctx-checkpoints 16 +--cache-ram 8192 --cache-idle-slots`. Measured GTT 64.5 GiB with model + +draft + projector resident. Prompt cache verified: a repeat pass over the +same 6k prompt replays in 0.5 s. -Fetch: `scripts/download-frank-large-mimo.sh` (target), -`scripts/download-frank-large-qwen35.sh` (MTP source), -`scripts/download-frank-large-dflash.sh` (dflash alternative) — all -`-f` + byte-count + LFS sha256 verified, `--verify-only` supported, no -token on the command line. +**Client contract (llama.cpp, live-verified 2026-10-02):** `--jinja` is on +(without it `chat_template_kwargs` are silently ignored), tools are native +(`get_weather` → `{"city":"Seattle"}`), thinking lands in +**`reasoning_content`**, `reasoning_effort` accepts ONLY +`xhigh|medium|low` (anything else is HTTP 500 — `models.yml` folds `high` and +`max` to `xhigh`), and thinking is enabled server-side with the card's +recipe (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0) plus +`--reasoning-format deepseek --reasoning-budget -1`. Two traps: a request +whose whole budget is consumed by reasoning returns `content: null` (bench +clients must fall back to `reasoning_content` or they crash *after* +measuring), and the fork logs +`MTP boundary missing for seq_id=… ; resyncing` under concurrent draft +resets — self-recovering, not a fault. -Runbook: `serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md`. +Fetch: `scripts/download-frank-large-kingjones.sh` (`-f` + byte count + LFS +sha256 verified, `--verify-only`, no token on the command line). + +Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md`. Rollback: git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1 (`scripts/build-gufo.sh` stays on disk for it), the Q4_K_XL Qwen3.8-27B payload and the DFlash2 drafter all remain on the host; restore the tag's unit/env/serve trio and redeploy. -### frank-small — SGLang on the RTX 3090 (Dankclaw, ACTIVE) +### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) -SGLang (latest) in a **self-contained python3.12 venv** -(`/home/dank/GitPackages/sglang-cu128/`, torch 2.13.0+cu130) as a **root -systemd unit** (repo `scripts/frank-small.service`, `relink_unit`ed into +**llama.cpp master** (CUDA sm_86 build +`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a +**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`), serving -`letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound` (9B dense -qwen3_5 hybrid — 8 full-attention + 24 Gated DeltaNet layers — INT8 W8A16 -AutoRound, 12.0 GiB, vision tower baked into the checkpoint). Port 7771. +`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B, +sha256 `04cf46f2…`) on port 7771. The 2026-10-02 swap was a **quant rung down +from Q6_K**, not a runtime change — the llama.cpp path arrived 2026-10-01 +night for one reason: **asymmetric KV**. `--cache-type-k f16` + +`--cache-type-v q8_0` is the fleet's only per-tensor KV lever; SGLang's +`--kv-cache-dtype` moves K and V together. -**The shape is memory-pinned, and the sharp edge is the mamba pool.** The -DeltaNet ssm state pool autofills to a fixed ~5.1 GB budget *regardless of -dtype* (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are -pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` (795 MiB). -The KV pool is then **pinned at 606,208 tokens** (`--max-total-tokens`) — -the engine's autofit picks 687,388 and leaves 0.53 GB, which Triton's -deferred kernel loads eat and the first long request OOMs on. Two -262,144-token slots at the model's NATIVE window (2026-10-01 reslice from -4 x 151,552: 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack, and -a compaction request — frank-primary's trigger is 183,501 tokens plus a -32,768-token summary budget = 216,269 — now fits ONE request); a single -request may grow to the whole pool. Prefill runs **eager** -(`--disable-prefill-cuda-graph`) — there is no headroom for capture; -decode graphs are on. +**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated +DeltaNet, native 262,144 window. KV exists only in the 8 attention layers +(4 KV heads × 256 dim × 2 B ≈ 2 KiB/token/layer, ~16 KiB/token for f16 K+V); +the DeltaNet state is a fixed per-slot buffer, so slot count — not context — +drives that term. -**No speculative decode — a checkpoint fact.** `text_config` declares -`mtp_num_hidden_layers: 1`, but the shipped safetensors (base BF16 and this -INT8 conversion alike) carry zero `mtp.*` tensors; NEXTN would draft with -an uninitialised 4.24 GiB ghost model at 0% acceptance. The seam gates -that no `--speculative-*` flag reaches the engine. +**The shape:** `--ctx-size 524288 --parallel 2` = TWO full native windows +(llama.cpp divides the pool: `n_ctx_slot = CTX_SIZE / PARALLEL`), +`--flash-attn on` (the q8_0-V path is only cheap under flash attention — the +seam pins it), f16 K + q8_0 V, all layers on the GPU. Measured VRAM +**20,164 / 24,576 MiB**; full f16 V would add ~3.96 GiB and bust the card, +which is the whole justification for the asymmetric contract. -**The venv carries a CUDA 13.0 nvcc extracted from debs** (system nvcc is -12.4) so the sgl-kernel/flashinfer JIT compiles for sm86; three local -patches ride in the tree (GCC-16 shim, CCCL symlink, rsqrt noexcept). -Details in the runbook. +Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; decode +**100.4–101.8 tok/s** at temp 0; two concurrent requests answered in +parallel; native `tool_calls`; thinking in `reasoning_content`. **Text-only** +— this GGUF ships no projector, so `models.yml` declares `input: [text]` +(the in-checkpoint vision tower exists only in the safetensors layouts the +SGLang run used). -Verified at deploy (2026-10-01): `/v1/models` reports `frank-small`; -chat returns content plus `reasoning_content` (SGLang's qwen3 parser uses -llama.cpp's field name); a red-square image answers "Red" -(`image_tokens: 64`). At the 4 x 151,552 shape: 4 concurrent 54k-token -requests answered; a 170,019-token prompt prefilled in 50.5 s (~3,370 t/s -eager). At the 2 x 262,144 shape (reslice, same day): boot flags read -`context_len=262144 max_running_requests=2 max_total_num_tokens=606208`; -a 200,000-token prompt — well past frank-primary's 183,501 compaction -trigger — prefilled and completed in ONE request (97.3 s wall, ~2,056 -tok/s eager); two concurrent 50k-token requests both answered. Decode -**83.9 tok/s**. -Runbook: `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md`. -Rollback: [[frank-small-gemma4-12b]] (payload and binary intact on disk). +Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`. +Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at +`/home/dank/models/MiMo-Q6K/`); before that [[frank-small-sglang-mimo-v26]] +(venv + INT8 payload still on the host). ### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE) @@ -619,6 +608,26 @@ linked runbooks. non-greedy requests and reasoning budgets. Re-test chat before production. - **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch; stock binaries fail the draft load with a tensor-shape mismatch. +- **A GGUF's custom tensor type picks the runtime — read the histogram before + promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is + ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock + llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but + the only door. Counting tensor types out of the GGUF KV section takes one + python read and settles a runtime argument in minutes. The build identity + matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in + `ggml_compute_forward_set_rows` on this same file, so + `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. +- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork + `--model-draft` with no `--spec-type` (the naive way to build a "no MTP" + bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A + baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND + `--spec-type`, or it measures a crash instead of the absence of drafting. +- **An orphaned scratch server taxes the whole host.** A bench child left + alive by a client-side exception kept ~26 GiB of weights resident on a + 121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect. + Record the child pid and kill it in a `finally`, then prove `free` came back + before believing a number. Kill by pid — never `pkill -f llama-server`, + which also hits frank-small. **llama.cpp serving** @@ -791,8 +800,8 @@ Active: | Service | Runbook | |---------|---------| -| frank-large (vLLM TheRock MiMo-V2.6 BF16 + cross-model MTP, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md` | -| frank-small (SGLang MiMo-V2.6 INT8, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md` | +| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md` | +| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q5_K_M, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md` | | frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -801,7 +810,10 @@ Retired (rollback targets and history): | Runtime | Host | Window | Runbook | |---------|------|--------|---------| -| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01 | `frank-large-gufo-qwen38-27b.md` | +| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (rollback target) | `frank-large-gufo-qwen38-27b.md` | +| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` | +| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (rollback target) | `frank-small-llama-mimo-q6k.md` | +| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` | |tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`| | llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` | | R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` | diff --git a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md index e0e8557..b509400 100644 --- a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md +++ b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md @@ -1,7 +1,7 @@ --- type: reference title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-large)" -description: "The ACTIVE frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing." +description: "The RETIRED frank-large runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing." tags: - llm - gufo @@ -18,9 +18,11 @@ timestamp: 2026-09-28 # frank-large — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2) -> **ACTIVE again since 2026-10-01 night** — Frank rolled frank-large back -> after one day of [[frank-large-vllm-mimo-v26-bf16]] (battery decode -> 6.19-9.01 t/s; every ≥6-bit vLLM quant path dead on gfx1151). The shape +> **RETIRED 2026-10-02 — the rollback target.** Superseded by +> [[frank-large-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with +> MTP4, 17.0 t/s measured). This era was itself a return after one day of +> [[frank-large-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every +> ≥6-bit vLLM quant path dead on gfx1151). The shape > moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF > UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was > reclaimed), and the session grid is **one session at the full native diff --git a/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md new file mode 100644 index 0000000..8d65f3c --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md @@ -0,0 +1,215 @@ +--- +type: reference +title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP" +description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, and the bundled BF16 projector for vision. Measured MTP-vs-none A/B, the client contract, and rollback to gufo." +tags: + - llm + - frank-large + - dankclaw + - llama-cpp + - rocmfpx + - gfx1151 + - gguf + - spec-decode + - mtp +timestamp: 2026-10-02 +--- + +# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151) + +> **ACTIVE since 2026-10-02.** Cut over from +> [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s). Both requests came from +> Frank: *highest quality that runs*, and *one full context*. This endpoint +> delivers 8.39 bpw weights — two quant rungs above what gufo could load — and +> measures **17.0 t/s** decode instead of 15, so quality was not bought with +> speed. +> +> The capability that forced this whole shape is a GGUF tensor type: +> **`GGML_TYPE_Q8_0_ROCMFPX` (103)**. It exists in exactly one loader on this +> host, so the runtime was decided by the file, not by preference. + +## Identity + +| | | +|---|---| +| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture | +| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface | +| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) → `/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server` | +| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` | +| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) | +| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` — 28,214,425,952 B, sha256 `f3ef2bdbf56e244f…` | +| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764cff8c4f3…` | +| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B, sha256 `de2a4986…` | +| Fetch | [`scripts/download-frank-large-kingjones.sh`](../../scripts/download-frank-large-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) | +| Seam | `tests/frank_large_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) | + +## Why this file, and why only this runtime + +`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` ships several rungs. Parsed +headers of the two candidates: + +| File | ftype | Tensor mix | Output-head tensors at plain Q8_0 | Card MTP acceptance | +|---|---|---|---|---| +| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | **115** | 353 F32 + 187 Q8_0 + **311 type-103** | **187** | 0.953 | +| `Qwen3.8-27B-Q8_0_ROCMFPX.gguf` | 111 | 2 Q8_0 + type-103 | 2 | 0.911 | + +Two consequences, both measured off the header rather than guessed: + +1. **Quality.** The AGENT routing variant keeps the output-head (and other + routing-sensitive) tensors at plain Q8_0 instead of folding them into the + ROCmFPX kernel layout — 187 vs 2 — so the logits the sampler reads are the + highest-precision tensors in the file. It is also the card's MTP-tuned + variant, so it is the *faster* drafter base at the same time. +2. **The runtime is forced.** Type 103 is a private extension: stock llama.cpp + and gufo both fail at load (`unknown tensor type`). Verified on the host + before deploying — the bake-off was not a horse race, it was a lock-in + check. + +**The build identity matters as much as the fork.** The 2026-08-25 ROCmFPX +builds read this file and then died in `ggml_compute_forward_set_rows` — the +exact failure documented in [[kingjones-q8agent-rocmfpx]] when this payload was +first attempted on Dankclaw and routed to Dankserver as +[[frank-medium-rocmfpx]] instead. `scripts/build-rocmfpx.sh` therefore checks +the commit out **by SHA** and refuses to build anything else; the pin is +load-bearing, not tidiness. + +## Shape and budget + +One session, the model's full native window, unquantized KV: + +| Dial | Value | Why | +|---|---|---| +| `--ctx-size` | 262,144 | "one full context" — the native window, no rope scaling | +| `--parallel` | 1 | one conversation, all memory to it | +| KV dtype | **f16** (both K and V) | ~64 KiB/token → ~16 GiB; free inside a 114 GiB aperture, so there is no reason to pay a quant | +| `--fit off` | on | the auto-fitter would silently resize the window we asked for | +| `--flash-attn on` | on | required for the long-window attention path to fit | +| `--direct-io` | on | bypasses page cache for the 26 GiB weight stream (mmap is disabled by it — deliberate) | +| `--ctx-checkpoints 16` | 16 | slot state for checkpoint/rollback | +| `--cache-ram 8192` + `--cache-idle-slots` | 8 GiB | prompt cache; verified a repeat pass over the same 6k prompt replays in **0.5 s** | +| `--jinja` | on | without it `chat_template_kwargs` are silently ignored | +| `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | thinking in its own channel, unbounded | + +Measured steady state (production, one model copy resident): +`mem_info_gtt_used` **63.1 GiB** of the ~114 GiB aperture, `mem_info_vram_used` +0.17 / 2.0 GiB (the iGPU's dedicated carve-out is a stub; everything real lives +in GTT). llama-server RSS 2.1 GiB. Boot to `/health` 200: **16 s** — the fastest +endpoint bring-up in the fleet, because there is no engine compile step. + +## Speculative decode — measured, not assumed + +``` +--spec-type draft-mtp --model-draft +--spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0 +--spec-draft-device ROCm0 --spec-draft-ngl 99 +``` + +MTP in llama.cpp is a property of the file, not of the flags: the head is a +trained tensor set (`nextn_predict_layers = 1` in this repo's head, arch +`qwen35`, 65 blocks) and the draft context runs on the main model's weights, so +there is no second 26 GiB copy in memory. See the fleet law in +`AGENTS.md → llama.cpp serving`. + +**A/B, 2026-10-02** — same session, temp 0, 2,202-token prompt, 512 tokens, +`cache_prompt: false`, `frank-large.service` stopped for every scratch arm (the +benchmarking law), baseline arm stripped of `--model-draft` **and** every +`--spec-*` flag: + +| Arm | Decode t/s (2 runs) | Prefill t/s | Draft acceptance | +|---|---|---|---| +| **MTP, n-max 4 (deployed)** | **17.00 / 17.14** | 350.6 / 324.9 | 0.462 (mean acceptance length ≈ 1.86) | +| no draft | 7.60 / 7.66 | 381.8 / 381.0 | — | + +**+123 % decode for an 8–15 % prefill tax.** Notes that matter before someone +re-litigates this: + +- Acceptance here (0.462) is nowhere near the card's 0.953. The card measures + its own short, repetitive text; this is a 2.2k-token prose prompt. Treat the + card's number as a file property, not a throughput promise. +- `n-max 4` is where the gain was measured; 2 and 8 are **unmeasured**. The + gufo-era law applies in spirit — never rank spec settings on a + self-predictable corpus. +- End-to-end on the chat path is slower than the `/completion` number because + thinking tokens are included: a 2,253-token chat prompt generating 512 + tokens took 43.8 s wall (11.7 t/s end-to-end) — that is the cost of an + unbounded reasoning budget, not a regression in decoding. + +## Client contract (live-verified 2026-10-02) + +| Behavior | Observation | +|---|---| +| Model id | `/v1/models` → `frank-large` | +| Tools | native: `get_weather` → `{"city": "Seattle"}` in `tool_calls` | +| Thinking | in **`reasoning_content`** (llama.cpp field — *not* vLLM's `reasoning`); `models.yml` sets `reasoningContentField` accordingly | +| `reasoning_effort` | accepts **only** `xhigh`, `medium`, `low`; anything else is HTTP 500 (the template raises, it does not clamp). `models.yml` folds `high`/`max` → `xhigh` | +| Sampling | server defaults = the card's thinking recipe (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0), mirrored in the harness `extraBody` | +| Vision | the BF16 projector loads (`loaded multimodal model`); a red square answers **`Red`** (`usage: 75 prompt / 71 cached`) | +| Prompt cache | second pass over the same 6k prompt: 0.5 s wall | + +Three traps, all hit for real: + +1. **`content: null` when reasoning eats the budget.** With + `--reasoning-parser`-style separation, a request whose `max_tokens` is + consumed entirely by thinking returns `message.content = None`. A bench + client that indexes `content` crashes *after* measuring and loses the + numbers — fall back to `reasoning_content`, or size the budget (200 was + enough for the vision probe). +2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`** + appears in the log under draft resets. Self-recovering; not an outage signal. +3. **Never `pkill -f llama-server` on Dankclaw** — frank-small is also a + `llama-server`. The sweep key here is the payload name: + `pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`. + +## Operations + +```bash +# deploy (planner selects this endpoint only via its own unit/env/serve trio) +./scripts/deploy.sh dankclaw --service frank-large + +# status / logs / cause of a crash loop +systemctl status frank-large +journalctl -u frank-large -n 200 --no-pager + +# rebuild the pinned fork (only when deliberately moving the pin) +scripts/build-rocmfpx.sh + +# re-verify the payload without re-downloading +scripts/download-frank-large-kingjones.sh --verify-only +``` + +`serve-frank-large.sh` refuses to start — before touching the port — unless the +fork binary, the weights, the MTP head and (unless `FRANK_LARGE_MMPROJ=""`) the +projector all exist; `scripts/deploy-remote.sh` mirrors the same predicate so a +broken payload never stops the running unit. Caller environment **overrides** +`frank-large.env`, and defaults use `:=` so an explicitly empty value surfaces +as the error it reads as. + +## Rollback + +git tag **`frank-large-gufo-q4kxl-20261001`**: gufo v0.1.1 +(`scripts/build-gufo.sh` still on disk), the Q4_K_XL Qwen3.8-27B payload and the +DFlash2 drafter all remain on the host. Restore that tag's unit/env/serve trio +and redeploy — nothing here deletes anything. + +Payloads retired alongside this swap (MiMo-V2.6 BF16, Qwen3.5-9B MTP source, +DFlash draft dirs) are still on disk under `/home/dank/models/` and are the +subject of the [[deployment]] disk-pressure note, not of a rollback plan. + +## When it misbehaves + +- **`unknown tensor type` / ftype 115 refused** — something moved off the + ROCmFPX binary. Check `readlink -f $(systemctl show frank-large -p ExecStart)` + and the fork commit; this file cannot run on stock llama.cpp or gufo. +- **SIGSEGV in `ggml_compute_forward_set_rows` during load** — a fork build + older than the `36e9acd` pin; rebuild with `scripts/build-rocmfpx.sh`. + History: [[kingjones-q8agent-rocmfpx]]. +- **Slow decode with no log error** — check whether the draft is engaged + (`/metrics`, `draft_n` / `draft_n_accepted`; a zero draft count means the head + did not load), and check GTT pressure with + `cat /sys/class/drm/card1/device/mem_info_gtt_used` — a second resident copy + of this model (an orphaned scratch server) drives the whole host into + eviction and halves the rate silently. +- **`500` naming Jinja** — a `chat_template_kwargs` field reached a server + without `--jinja`. +- **HTTP 500 naming the template on the first request of a role** — an + unmapped `reasoning_effort`; fix the fold in `models.yml`, not the model. diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md new file mode 100644 index 0000000..c18a35b --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md @@ -0,0 +1,113 @@ +--- +type: reference +title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF" +description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the VRAM fit, measured rates, and what to check when it misbehaves." +tags: + - llm + - frank-small + - dankclaw + - llama-cpp + - rtx3090 + - mimo + - gguf + - kv-cache +timestamp: 2026-10-02 +--- + +# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090) + +> **ACTIVE since 2026-10-02.** One quant step down from +> [[frank-small-llama-mimo-q6k]] on the same runtime, the same shape and the +> same KV contract. Frank's order: `bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M` +> on llama.cpp, two contexts, whatever context size fits the 3090. The answer +> to "whatever fits" is unchanged — **2 x 262,144**, because 262,144 is the +> model's native window and two of them already fit. + +## Identity + +|Field|Value| +|---|---| +|Endpoint|`192.168.0.143:7771`, served alias `frank-small`| +|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap| +|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f`, at `/home/dank/models/MiMo-Q5KM/` (fetched by `scripts/download-frank-small.sh`)| +|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`| +|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`| + +## Shape and the KV contract + +`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots, so +each session gets `n_ctx_slot = 262144`, the model's full native window (boot +log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`). + +The KV contract is Frank's: **K unquantized (f16), V `q8_0`** — +`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it: + +- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path. + Without the flag the engine falls back silently and the contract becomes a + lie. The serve script pins the trio; the bats seam pins the script. +- The VRAM budget: attention KV lives in the 8 full-attention layers only + (4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token + (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool ≈ + **12.5 GiB**. Weights 6.40 + KV 12.5 + Gated DeltaNet state (~25 MiB/slot) + + compute buffers/CUDA graphs ⇒ **20,164 MiB measured** against the card's + 24,576 MiB. + +**Why the quant step did not buy a bigger window.** The model cannot be run +past its trained 262,144 without scaling, and two of those windows are already +the pool's ceiling. The 0.86 GiB the smaller quant frees is margin, not +geometry. It is also not enough to un-quantize V: f16 V costs 16 KiB/token +instead of 8.5, i.e. +3.96 GiB on this pool, which lands past the card. That +arithmetic is a test (`the KV budget fits the 3090…`), so a future CTX_SIZE or +quant change that breaks the fit fails the suite instead of the boot. + +## Measured on this file (2026-10-02) + +|Check|Result| +|---|---| +|Boot to healthy|~20 s, `n_slots = 2`, `n_ctx_slot = 262144`| +|VRAM|20,164 MiB / 24,576 MiB (`nvidia-smi`)| +|Decode, temp 0, no thinking|**100.4–101.8 tok/s** (256–512 token completions at ~900-token depth)| +|Two concurrent requests|both answered, 256 tokens each in 3.2 s wall (slots are real)| +|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"c"}`)| +|Thinking|`reasoning_content` populated on a reasoning-requiring prompt| + +## Client contract + +- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`), + the field `models.yml` reads for this endpoint. +- **Thinking is off unless the request asks**: the MiMo GGUF template defaults + `enable_thinking` false, so `models.yml` ships + `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in + `extraBody`. A raw client that omits it gets a direct answer — measured, and + not a fault. Trivial questions produce no thinking even with the flag on. +- Tools: `--jinja` + server-side parsing, verified 2026-10-02. +- **Text-only.** This GGUF carries no vision tensors. The HF repo does ship a + paired `mmproj-MiMo-V2.6-Distill-Qwen-9B-bf16.gguf`, so vision is one + `--mmproj` away if it is ever wanted here (frank-primary is the fleet's + vision endpoint today). + +## No spec decode + +The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00 +ghost), and the cross-model DFlash draft +(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an +optimization candidate, not a default — acceptance has to be measured on this +traffic first. The bats seam fails if a draft flag ships unannounced. + +## Rollback + +|Target|How| +|---|---| +|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default| +|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and payload both remain on disk; the fetch recipe moved to `old/download-frank-small-sglang-int8.sh`| + +## When it misbehaves + +- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1 + naming the missing file if the binary or GGUF moved. +- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node + is missing — `nvidia-modprobe -u -c=0` (the serve script runs it). +- Requests queue behind each other: two concurrent long generations are the + slot ceiling by contract. +- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out + of VRAM into unified memory halves the rate silently. diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md index c2d0237..f2cd2ef 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k.md @@ -1,7 +1,7 @@ --- type: reference -title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF" -description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the flash-attention dependency that comes with quantized V, the VRAM fit, and what to check when it misbehaves." +title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)" +description: "The retired 2026-10-01 frank-small runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload." tags: - llm - frank-small @@ -16,11 +16,14 @@ timestamp: 2026-10-01 # frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090) -> **ACTIVE since 2026-10-01 night.** Replaces -> [[frank-small-sglang-mimo-v26]] (SGLang + INT8-W8A16-AutoRound safetensors, -> same architecture). Rolled in by Frank's night order: move this endpoint to -> the GGUF path for **asymmetric KV** — unquantized K + q8_0 V — which only -> llama.cpp exposes per-tensor. +> **RETIRED 2026-10-01 → 10-02** (one night): superseded by +> [[frank-small-llama-mimo-q5km]], the same runtime and shape one quant rung +> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback +> target. +> +> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF +> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp +> exposes per-tensor. That contract is what the current deployment kept. ## Identity diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index a07f003..151bf73 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,6 +1,6 @@ # LLM Fleet — Knowledge Base -Documentation for the LLM inference fleet on the two hosts (2026-10-01): +Documentation for the LLM inference fleet on the two hosts (2026-10-02): **Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s @@ -10,10 +10,11 @@ headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, uti reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on, measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw -runs two endpoints — **gufo v0.1.1 (frank-large, port 7770: Qwen3.8-27B -UD-Q4_K_XL + DFlash2, ONE session at 262,144, f16 KV on the gfx1151 iGPU, -rolled back from the one-day vLLM MiMo experiment 2026-10-01 night)** and -**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K +runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770: +kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115 +file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE +session at 262,144, f16 KV, vision via the bundled projector)** and +**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**. > 2026-08-22: the former `vllmScripts` repository is merged into the @@ -38,13 +39,13 @@ fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] E2[RTX 3090 (PCIe x4 dock) + gfx1151 iGPU 114 GiB GTT] - E[gufo v0.1.1 built from source + E[ROCmFPX llama.cpp gfx1151, pinned fork build frank-large, port 7770] - F[Qwen3.8-27B UD-Q4_K_XL + DFlash2 drafter -1 x 262144 ctx, f16 KV, greedy, adaptive draft 7] + F[Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP-Q4_0 head +1 x 262144 ctx, f16 KV, draft n-max 4, mmproj vision] G[llama.cpp master CUDA sm_86 frank-small, port 7771] - H[MiMo-V2.6 Q6_K GGUF + H[MiMo-V2.6 Q5_K_M GGUF 2 x 262144 ctx, f16 K + q8_0 V, flash-attn] E2 --> E --> F E2 --> G --> H @@ -64,6 +65,19 @@ frank-small, port 7771] ## Active Deployment +- [[frank-large-rocmfpx-q8agent-mtp]] — **ACTIVE since 2026-10-02** + (root systemd unit, port 7770): the pinned kingjones30/ROCmFPX fork serving + the repo's highest-quality rung `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` + (26.28 GiB, ftype 115 = 187 plain-Q8_0 + 311 GGML_TYPE_Q8_0_ROCMFPX + tensors) with the bundled `mtp-Qwen3.8-27B-Q4_0.gguf` head drafting 4 + tokens, one 262,144 window, f16 KV, the BF16 projector for vision. Why the + runtime is not interchangeable, the pinned-build set_rows history, the + measured envelope, rollback to gufo +- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root + systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving + bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090, + 2 x 262,144 with K f16 + V q8_0, 100.4-101.8 tok/s decode, 20,164 MiB + measured. The KV arithmetic, the flash-attn dependency, rollback to Q6_K - [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the - failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro): (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 0ccb4b0..0ab1c04 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -213,20 +213,27 @@ The LLM inference fleet on the two hosts. Full index: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record) -- [[frank-large-vllm-mimo-v26-bf16]] — the ACTIVE frank-large runbook - since 2026-10-01 (root systemd unit, port 7770): the digest-pinned +- [[frank-large-rocmfpx-q8agent-mtp]] — the ACTIVE frank-large runbook + since 2026-10-02 (root systemd unit, port 7770): the pinned kingjones30/ + ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/ + Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF **Q8_0_ROCMFPX_AGENT** — ftype 115, + 311 tensors of GGML_TYPE_Q8_0_ROCMFPX, a format no other loader on the + host reads — with the repo's Q4_0 MTP head drafting 4 tokens and its BF16 + projector, one session at the native 262,144 window +- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (rollback + target, payload and `scripts/build-gufo.sh` both on disk): gufo v0.1.1 + built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2 + drafter on the gfx1151 iGPU — the two from-source build workarounds, the + template-hash allowlist, the measured prefill/decode trade +- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (same day; + replaced by the gufo rollback, then by + [[frank-large-rocmfpx-q8agent-mtp]]): the digest-pinned kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU, 4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B — the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap, the `reasoning` field change, troubleshooting -- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-09-28 → 2026-10-01 - (replaced by [[frank-large-vllm-mimo-v26-bf16]]; rollback via tag - frank-large-gufo-q4kxl-20261001): gufo v0.1.1 built from source serving - Qwen3.8-27B GGUF + the z-lab DFlash2 drafter on the gfx1151 iGPU — the - two from-source build workarounds, the client-contract delta, the - measured prefill/decode trade - [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by [[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp (unsloth b11030, ROCm gfx1151) serving @@ -275,12 +282,21 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-small-sglang-mimo-v26]] — the ACTIVE frank-small runbook since - 2026-10-01: SGLang in a self-contained python3.12 venv serving +- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since + 2026-10-02: a CUDA (sm_86) build of llama.cpp master serving + bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX + 3090, port 7771 — 2 x 262,144 native windows, the asymmetric KV contract + (f16 K + q8_0 V) and the VRAM arithmetic behind it, ~100 tok/s decode +- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced + by [[frank-small-llama-mimo-q5km]], payload still on disk): the first + llama.cpp + GGUF shape for this endpoint, Q6_K at the same + 2 x 262,144 / f16-K + q8_0-V contract, 20.9 GiB measured +- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target): + SGLang in a self-contained python3.12 venv serving letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool, - 4 x 151,552 slots, bf16 mamba states, eager prefill + 2 x 262,144 slots, bf16 mamba states, eager prefill - [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target): mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M