docs: record the 2026-10-02 swaps — frank-large on ROCmFPX, frank-small on Q5_K_M
frank-large: new runbook frank-large-rocmfpx-q8agent-mtp.md — the ftype-115 tensor-type lock-in that forced the fork, the pinned build (set_rows history), the measured MTP A/B (17.0 vs 7.6 t/s decode, acceptance 0.462), the live client contract (reasoning_content, effort fold, vision restored) and rollback. frank-small: new runbook frank-small-llama-mimo-q5km.md; q6k marked retired (rollback target). gufo runbook marked retired (rollback target). Both indexes, the fleet topology table, the vision/models.yml notes, the benchmarking law and three new Key Lessons updated.
This commit is contained in:
Frank committed
1 parent
125d04967f
commit
72bd4c967e
7 files changed
+522
-147
No files matched your search
@@ -12,8 +12,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (vLLM TheRock on
|
||||
the gfx1151 iGPU) and `frank-small` (SGLang on the RTX 3090). Caddy on
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (ROCmFPX llama.cpp
|
||||
fork on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
(Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
|
||||
@@ -25,8 +25,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
|
||||
| Dankclaw | `frank-large` | vLLM 0.30 in the digest-pinned `docker.io/kyuz0/vllm-therock-gfx1151` image (TheRock: ROCm 10 / torch 2.11, Triton Gated DeltaNet kernels; plain `podman run`, root systemd unit) | `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` **full BF16** (9B dense, 17.53 GiB, 4 shards — the arch frank-small serves as INT8 on the 3090), **cross-model MTP spec decode from `Qwen/Qwen3.5-9B` (2 draft tokens; 19.8 t/s decode measured, +51% over the 13.1 t/s base, which is ~94% of the LPDDR5x BF16 roof)**, 4 seqs x 262144 native ctx at util 0.45, qwen3 reasoning + qwen3_coder tool parsers, text-only (`--language-model-only`). Cut over 2026-10-01 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
|
||||
| Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 2 slots of 262,144 (native window; 2026-10-01 reslice from 4 x 151,552 for compaction headroom), bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 |
|
||||
| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`** (26.28 GiB, ftype 115 ≈ 8.39 bpw: 353 F32 + 187 Q8_0 + 311 `GGML_TYPE_Q8_0_ROCMFPX` tensors — only this fork can read them), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the bundled BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-02: decode **17.0–17.1 t/s** with MTP vs **7.6 without (+123 %)**, draft acceptance 0.462, prefill 325–351 t/s, prompt-cache replay 0.5 s, boot→healthy 16 s, GTT 64.5 GiB. Cut over 2026-10-02 from gufo; rollback tag `frank-large-gufo-q4kxl-20261001` | 7770 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** (9B dense qwen3_5 hybrid, 6.4 GiB GGUF, no MTP weights so no spec decode) on the RTX 3090, **2 × 262,144** (native window) with **asymmetric KV f16 K + q8_0 V** under flash attention, 20,164/24,576 MiB VRAM, 100.4–101.8 tok/s decode, text-only. Swapped 2026-10-02 from Q6_K; rollback `models/MiMo-Q6K/` | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -209,13 +209,13 @@ archive.
|
||||
**STOP the production service BEFORE benchmarking** — these servers serve
|
||||
real users, and a bench beside a live service starves the GPU and corrupts
|
||||
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
|
||||
endpoints; frank-large's vLLM shape is measured by replaying real text through the live server and reading `Server-Timing`. Rolling your own bench
|
||||
endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-small were measured. Rolling your own bench
|
||||
requires asking first.
|
||||
Scratch ports are not used on LLM hosts — a
|
||||
test process cannot share the single GPU anyway; validate against the service
|
||||
itself.
|
||||
|
||||
(A gufo-era law; frank-large is vLLM since 2026-10-01, but the rollback tag keeps it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
|
||||
(A gufo-era law; frank-large is ROCmFPX llama.cpp since 2026-10-02, but the rollback tag keeps it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
|
||||
Measured 2026-09-28 on frank-large at depth 32768: its corpus
|
||||
reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
|
||||
draft widths, i.e. the same highly-predictable text every run, and 93 % of
|
||||
@@ -277,12 +277,12 @@ When renaming a service, retiring an endpoint, or moving a model: update the
|
||||
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-10-01 both `frank-small` and `frank-primary` accept images
|
||||
(`input: [text, image]`) — frank-primary loads the BF16 mmproj explicitly
|
||||
(this build never autoloads projectors), and frank-small's MiMo payload
|
||||
bakes its vision tower INTO the checkpoint (no projector file at all);
|
||||
`frank-large` stays text-only because its vLLM deployment passes
|
||||
`--language-model-only` — the MiMo vision tower is in the checkpoint; the flag is the lever. Note also that
|
||||
of 2026-10-02 `frank-large` (the BF16 mmproj that ships inside its GGUF
|
||||
payload) and `frank-primary` (loads the BF16 mmproj explicitly — this build
|
||||
never autoloads projectors) accept images (`input: [text, image]`);
|
||||
`frank-small` is **text-only** — the GGUF payload it moved to on 2026-10-02
|
||||
ships no projector, and the in-checkpoint MiMo vision tower exists only in the
|
||||
safetensors layouts. Note also that
|
||||
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
|
||||
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
|
||||
models.yml sets `reasoningContentField` per endpoint to match.
|
||||
@@ -292,125 +292,114 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
|
||||
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
|
||||
(the MiMo tower is vision-only).
|
||||
|
||||
### frank-large — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (Dankclaw, ACTIVE)
|
||||
### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE)
|
||||
|
||||
**vLLM 0.30** in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
|
||||
image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — TheRock
|
||||
build: ROCm 10.0.0 / torch 2.11.0, Triton Gated DeltaNet kernels,
|
||||
`vllm/v1/spec_decode/dflash.py` present) — as a **root systemd unit**
|
||||
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
|
||||
`ExecStop=podman stop -t 120`, `TimeoutStopSec` 180 > the stop grace),
|
||||
serving `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16** (9B dense
|
||||
qwen3_5 hybrid, 17.53 GiB, 4 sha256-verified shards) on port 7770 from the
|
||||
gfx1151 iGPU. Official vLLM images are CUDA-only; this fork is the fleet's
|
||||
gfx1151 path. `scripts/serve-frank-large.sh` gates image-by-repo-digest →
|
||||
payload (config + index + exact shard count) → spec draft (accepts BOTH
|
||||
weight layouts: single `model.safetensors` for dflash, sharded index for
|
||||
the MTP head) → whole `/dev/dri` (never individual renderD nodes), sweeps
|
||||
the bench identity `vllm-mimo`, then `exec podman wait` so a container
|
||||
death trips `Restart=always`. Boot ~200–240 s.
|
||||
The **kingjones30/ROCmFPX llama.cpp fork** at commit
|
||||
`36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built for gfx1151 by
|
||||
`scripts/build-rocmfpx.sh` into `/home/dank/GitPackages/kingjones-ROCmFPX/build`,
|
||||
as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`,
|
||||
`Restart=always`) running a bare `llama-server` (no container) on port 7770
|
||||
from the Strix Halo iGPU. Boot to healthy: **16 s**.
|
||||
|
||||
**The shape is chosen by roof math, and the spec decode is the
|
||||
performance feature.** BF16 decode is LPDDR5x-bandwidth-bound here:
|
||||
13.1 t/s measured = ~94% of the roof for 17.5 GiB of weights — a quant,
|
||||
not a bug. The only way past it is multiple tokens per weight pass.
|
||||
Measured 2026-10-01 (one session, temp 0, 28,647-token prefill / 512
|
||||
decode): base **17,876 t/s prefill / 13.1 decode**; +DFlash-8 7,976 /
|
||||
17.4 (acceptance 59%/26% per position); **+MTP-2 7,380 / 19.8
|
||||
(acceptance 71%/44%, token rate 57.4%)** — deployed arm.
|
||||
`num_speculative_tokens` beyond 2 is UNMEASURED; both spec arms pay
|
||||
~55–59% of prefill. KV grows only on the 8 full-attention layers
|
||||
(32 KiB/token bf16); the 24 GDN layers hold fixed ~25 MiB state per
|
||||
sequence — so 4 x 262,144 at util 0.45 fits the GTT aperture easily.
|
||||
**The runtime is not interchangeable — it is chosen by the FILE.**
|
||||
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF`'s best rung,
|
||||
`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` (28,214,425,952 B, sha256 `f3ef2bda…`,
|
||||
ftype 115 ≈ 8.39 bpw), stores **311 tensors of type 103,
|
||||
`GGML_TYPE_Q8_0_ROCMFPX`** — a quantization that exists in no other loader:
|
||||
stock llama.cpp and gufo both refuse it at load (`unknown tensor type`). The
|
||||
file is 353 F32 + 187 plain-Q8_0 + 311 ROCmFPX; the AGENT routing variant
|
||||
keeps the output-head tensors at plain Q8_0 (187 vs 2 in the non-AGENT file),
|
||||
which is why the card reports MTP acceptance 0.953 vs 0.911 — highest quality
|
||||
AND the better drafter base. Vision comes from the bundled
|
||||
`mmproj-Qwen3.8-27B-BF16.gguf` (931,145,888 B).
|
||||
|
||||
**The MTP head is CROSS-MODEL — the fleet's first.** MiMo ships ZERO
|
||||
`mtp.*` tensors (the reason frank-small's SGLang NEXTN is a 0.00 ghost:
|
||||
SGLang reads the head from the TARGET checkpoint); vLLM reads the draft
|
||||
from a separate path, so `--speculative-config
|
||||
'{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'`
|
||||
drafts from the distill's SOURCE model, and 71%/44% acceptance is the
|
||||
measured proof that the distillation preserved Qwen3.5 hidden-state
|
||||
structure. The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, fetched;
|
||||
its "PR #40898 unmerged" card claim is false — dflash ships in vLLM 0.30)
|
||||
stays on disk: `FRANK_LARGE_SPEC=dflash` + repoint `SPEC_DIR`.
|
||||
**The pin is load-bearing.** The 2026-08-25 ROCmFPX builds segfaulted in
|
||||
`ggml_compute_forward_set_rows` on exactly this file; `build-rocmfpx.sh`
|
||||
checks the commit out by SHA, so a fork-side force-push cannot silently
|
||||
re-break the endpoint.
|
||||
|
||||
**Spec decode does NOT preserve greedy text here (measured).**
|
||||
Temperature-0 completions differ across arms (bf16 batched verification
|
||||
reorders reductions; near-tie argmaxes diverge and cascade). vLLM spec
|
||||
decode is distribution-preserving, not bitwise-greedy-preserving — do not
|
||||
chase text drift between arms as a regression.
|
||||
**Spec decode is the performance feature, and it is measured.** The draft is
|
||||
the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (1,680,271,648 B, sha256 `051a1764…`,
|
||||
`nextn_predict_layers = 1`), pinned to the same card:
|
||||
`--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0
|
||||
--spec-draft-p-min 0.0 --spec-draft-device ROCm0 --spec-draft-ngl 99`.
|
||||
Measured 2026-10-02 in one session, temp 0, 2,202-token prompt / 512 tokens,
|
||||
`cache_prompt: false`, production service stopped for each arm:
|
||||
**MTP on 17.00–17.14 t/s decode** (draft acceptance 0.462, mean acceptance
|
||||
length ≈ 1.86) against **7.60–7.66 t/s with the draft removed — +123 %**;
|
||||
prefill pays 8–15 % (381 → 325–351 t/s). Acceptance is far from the card's
|
||||
0.95 (that number is the card's own text at a shallow prompt); the gain is
|
||||
real and n-max 4 is where it was measured, never swept. This replaces the
|
||||
gufo Q4_K_XL arm at ~15 t/s, so the endpoint got higher quality AND more
|
||||
speed at once.
|
||||
|
||||
**Client contract differs from every llama.cpp-era endpoint:** vLLM emits
|
||||
thinking in **`reasoning`**, not `reasoning_content` — `models.yml` sets
|
||||
`reasoningContentField: "reasoning"` for this provider (miss it and
|
||||
thinking vanishes from transcripts). Tools ride
|
||||
`--enable-auto-tool-choice --tool-call-parser qwen3_coder`. The harness
|
||||
extraBody moved to the Qwen3.5-family recipe (0.7/0.8) with
|
||||
`chat_template_kwargs.{enable_thinking,preserve_thinking}` — a top-level
|
||||
`preserve_thinking` is ignored by vLLM. Env keys stay
|
||||
`FRANK_LARGE_`-prefixed (HIP reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE`).
|
||||
**Shape and budget:** ONE session at the native 262,144 window, unquantized
|
||||
f16 KV (~64 KiB/token → ~16 GiB, affordable inside the iGPU's ~114 GiB GTT
|
||||
aperture), `--fit off --flash-attn on --direct-io --ctx-checkpoints 16
|
||||
--cache-ram 8192 --cache-idle-slots`. Measured GTT 64.5 GiB with model +
|
||||
draft + projector resident. Prompt cache verified: a repeat pass over the
|
||||
same 6k prompt replays in 0.5 s.
|
||||
|
||||
Fetch: `scripts/download-frank-large-mimo.sh` (target),
|
||||
`scripts/download-frank-large-qwen35.sh` (MTP source),
|
||||
`scripts/download-frank-large-dflash.sh` (dflash alternative) — all
|
||||
`-f` + byte-count + LFS sha256 verified, `--verify-only` supported, no
|
||||
token on the command line.
|
||||
**Client contract (llama.cpp, live-verified 2026-10-02):** `--jinja` is on
|
||||
(without it `chat_template_kwargs` are silently ignored), tools are native
|
||||
(`get_weather` → `{"city":"Seattle"}`), thinking lands in
|
||||
**`reasoning_content`**, `reasoning_effort` accepts ONLY
|
||||
`xhigh|medium|low` (anything else is HTTP 500 — `models.yml` folds `high` and
|
||||
`max` to `xhigh`), and thinking is enabled server-side with the card's
|
||||
recipe (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0) plus
|
||||
`--reasoning-format deepseek --reasoning-budget -1`. Two traps: a request
|
||||
whose whole budget is consumed by reasoning returns `content: null` (bench
|
||||
clients must fall back to `reasoning_content` or they crash *after*
|
||||
measuring), and the fork logs
|
||||
`MTP boundary missing for seq_id=… ; resyncing` under concurrent draft
|
||||
resets — self-recovering, not a fault.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md`.
|
||||
Fetch: `scripts/download-frank-large-kingjones.sh` (`-f` + byte count + LFS
|
||||
sha256 verified, `--verify-only`, no token on the command line).
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md`.
|
||||
Rollback: git tag **`frank-large-gufo-q4kxl-20261001`** — gufo v0.1.1
|
||||
(`scripts/build-gufo.sh` stays on disk for it), the Q4_K_XL Qwen3.8-27B
|
||||
payload and the DFlash2 drafter all remain on the host; restore the tag's
|
||||
unit/env/serve trio and redeploy.
|
||||
|
||||
### frank-small — SGLang on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
SGLang (latest) in a **self-contained python3.12 venv**
|
||||
(`/home/dank/GitPackages/sglang-cu128/`, torch 2.13.0+cu130) as a **root
|
||||
systemd unit** (repo `scripts/frank-small.service`, `relink_unit`ed into
|
||||
**llama.cpp master** (CUDA sm_86 build
|
||||
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`) as a
|
||||
**root systemd unit** (`scripts/frank-small.service`, `relink_unit`ed into
|
||||
`/etc/systemd/system/`), serving
|
||||
`letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound` (9B dense
|
||||
qwen3_5 hybrid — 8 full-attention + 24 Gated DeltaNet layers — INT8 W8A16
|
||||
AutoRound, 12.0 GiB, vision tower baked into the checkpoint). Port 7771.
|
||||
`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` at **Q5_K_M** (6,876,124,704 B,
|
||||
sha256 `04cf46f2…`) on port 7771. The 2026-10-02 swap was a **quant rung down
|
||||
from Q6_K**, not a runtime change — the llama.cpp path arrived 2026-10-01
|
||||
night for one reason: **asymmetric KV**. `--cache-type-k f16` +
|
||||
`--cache-type-v q8_0` is the fleet's only per-tensor KV lever; SGLang's
|
||||
`--kv-cache-dtype` moves K and V together.
|
||||
|
||||
**The shape is memory-pinned, and the sharp edge is the mamba pool.** The
|
||||
DeltaNet ssm state pool autofills to a fixed ~5.1 GB budget *regardless of
|
||||
dtype* (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
|
||||
pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` (795 MiB).
|
||||
The KV pool is then **pinned at 606,208 tokens** (`--max-total-tokens`) —
|
||||
the engine's autofit picks 687,388 and leaves 0.53 GB, which Triton's
|
||||
deferred kernel loads eat and the first long request OOMs on. Two
|
||||
262,144-token slots at the model's NATIVE window (2026-10-01 reslice from
|
||||
4 x 151,552: 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack, and
|
||||
a compaction request — frank-primary's trigger is 183,501 tokens plus a
|
||||
32,768-token summary budget = 216,269 — now fits ONE request); a single
|
||||
request may grow to the whole pool. Prefill runs **eager**
|
||||
(`--disable-prefill-cuda-graph`) — there is no headroom for capture;
|
||||
decode graphs are on.
|
||||
**Architecture: qwen3_5 hybrid** — 32 layers, 8 full-attention + 24 Gated
|
||||
DeltaNet, native 262,144 window. KV exists only in the 8 attention layers
|
||||
(4 KV heads × 256 dim × 2 B ≈ 2 KiB/token/layer, ~16 KiB/token for f16 K+V);
|
||||
the DeltaNet state is a fixed per-slot buffer, so slot count — not context —
|
||||
drives that term.
|
||||
|
||||
**No speculative decode — a checkpoint fact.** `text_config` declares
|
||||
`mtp_num_hidden_layers: 1`, but the shipped safetensors (base BF16 and this
|
||||
INT8 conversion alike) carry zero `mtp.*` tensors; NEXTN would draft with
|
||||
an uninitialised 4.24 GiB ghost model at 0% acceptance. The seam gates
|
||||
that no `--speculative-*` flag reaches the engine.
|
||||
**The shape:** `--ctx-size 524288 --parallel 2` = TWO full native windows
|
||||
(llama.cpp divides the pool: `n_ctx_slot = CTX_SIZE / PARALLEL`),
|
||||
`--flash-attn on` (the q8_0-V path is only cheap under flash attention — the
|
||||
seam pins it), f16 K + q8_0 V, all layers on the GPU. Measured VRAM
|
||||
**20,164 / 24,576 MiB**; full f16 V would add ~3.96 GiB and bust the card,
|
||||
which is the whole justification for the asymmetric contract.
|
||||
|
||||
**The venv carries a CUDA 13.0 nvcc extracted from debs** (system nvcc is
|
||||
12.4) so the sgl-kernel/flashinfer JIT compiles for sm86; three local
|
||||
patches ride in the tree (GCC-16 shim, CCCL symlink, rsqrt noexcept).
|
||||
Details in the runbook.
|
||||
Verified at deploy (2026-10-02): `/v1/models` reports `frank-small`; decode
|
||||
**100.4–101.8 tok/s** at temp 0; two concurrent requests answered in
|
||||
parallel; native `tool_calls`; thinking in `reasoning_content`. **Text-only**
|
||||
— this GGUF ships no projector, so `models.yml` declares `input: [text]`
|
||||
(the in-checkpoint vision tower exists only in the safetensors layouts the
|
||||
SGLang run used).
|
||||
|
||||
Verified at deploy (2026-10-01): `/v1/models` reports `frank-small`;
|
||||
chat returns content plus `reasoning_content` (SGLang's qwen3 parser uses
|
||||
llama.cpp's field name); a red-square image answers "Red"
|
||||
(`image_tokens: 64`). At the 4 x 151,552 shape: 4 concurrent 54k-token
|
||||
requests answered; a 170,019-token prompt prefilled in 50.5 s (~3,370 t/s
|
||||
eager). At the 2 x 262,144 shape (reslice, same day): boot flags read
|
||||
`context_len=262144 max_running_requests=2 max_total_num_tokens=606208`;
|
||||
a 200,000-token prompt — well past frank-primary's 183,501 compaction
|
||||
trigger — prefilled and completed in ONE request (97.3 s wall, ~2,056
|
||||
tok/s eager); two concurrent 50k-token requests both answered. Decode
|
||||
**83.9 tok/s**.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md`.
|
||||
Rollback: [[frank-small-gemma4-12b]] (payload and binary intact on disk).
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md`.
|
||||
Rollback: [[frank-small-llama-mimo-q6k]] (Q6_K file on disk at
|
||||
`/home/dank/models/MiMo-Q6K/`); before that [[frank-small-sglang-mimo-v26]]
|
||||
(venv + INT8 payload still on the host).
|
||||
|
||||
### frank-primary — tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4 (Dankserver, ACTIVE)
|
||||
|
||||
@@ -619,6 +608,26 @@ linked runbooks.
|
||||
non-greedy requests and reasoning budgets. Re-test chat before production.
|
||||
- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
|
||||
stock binaries fail the draft load with a tensor-shape mismatch.
|
||||
- **A GGUF's custom tensor type picks the runtime — read the histogram before
|
||||
promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
|
||||
ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
|
||||
llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
|
||||
the only door. Counting tensor types out of the GGUF KV section takes one
|
||||
python read and settles a runtime argument in minutes. The build identity
|
||||
matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
|
||||
`ggml_compute_forward_set_rows` on this same file, so
|
||||
`scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
|
||||
- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
|
||||
`--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
|
||||
bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
|
||||
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
|
||||
`--spec-type`, or it measures a crash instead of the absence of drafting.
|
||||
- **An orphaned scratch server taxes the whole host.** A bench child left
|
||||
alive by a client-side exception kept ~26 GiB of weights resident on a
|
||||
121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
|
||||
Record the child pid and kill it in a `finally`, then prove `free` came back
|
||||
before believing a number. Kill by pid — never `pkill -f llama-server`,
|
||||
which also hits frank-small.
|
||||
|
||||
**llama.cpp serving**
|
||||
|
||||
@@ -791,8 +800,8 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (vLLM TheRock MiMo-V2.6 BF16 + cross-model MTP, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-vllm-mimo-v26-bf16.md` |
|
||||
| frank-small (SGLang MiMo-V2.6 INT8, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md` |
|
||||
| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md` |
|
||||
| frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q5_K_M, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md` |
|
||||
| frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8 + MTP4, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
@@ -801,7 +810,10 @@ Retired (rollback targets and history):
|
||||
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01 | `frank-large-gufo-qwen38-27b.md` |
|
||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (rollback target) | `frank-large-gufo-qwen38-27b.md` |
|
||||
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-small`) | Dankclaw | 2026-10-01 → 10-02 (rollback target) | `frank-small-llama-mimo-q6k.md` |
|
||||
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-small`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-small-sglang-mimo-v26.md` |
|
||||
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
|
||||
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
|
||||
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
|
||||
|
||||
Reference in new issue
Block a user