frank-large: document the ik_llama.cpp deployment and the decode performance campaign
- Runbook with the measured envelope (prefill ~192 t/s at the VNNI ceiling; decode ~3.8-4.3 t/s gated by 100 backend splits/token, GPU 95% idle), the root-cause evidence chain, tuning levers, and co-tenancy traps - AGENTS.md: fleet topology row + Active Services section for the ik deployment; halogen demoted to rollback; bench policy unit type fixed - LLM + root indexes relinked (frank-large-ik-uncensored ACTIVE, halogen-flash-server retired)
This commit is contained in:
Frank
@@ -0,0 +1,120 @@
|
||||
---
|
||||
type: reference
|
||||
description: frank-large on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps
|
||||
tags: [llm, frank-large, ik_llama, dankclaw, rtx3090, strix-halo, performance]
|
||||
timestamp: 2026-09-16T03:30:00-07:00
|
||||
---
|
||||
|
||||
# frank-large — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M
|
||||
|
||||
**ACTIVE since 2026-09-15.** Replaced [[halogen-flash-server]] (rollback target;
|
||||
quadlet unit archived in `~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` on
|
||||
dankclaw). Serves `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`
|
||||
(111 GiB GGUF + mmproj-f16) at `192.168.0.143:7770`.
|
||||
|
||||
## Shape
|
||||
|
||||
- Runtime: ik_llama.cpp master `d85a1d7`, built at
|
||||
`/home/dank/GitPackages/ik_llama.cpp/build` with `-DGGML_CUDA=ON
|
||||
-DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON
|
||||
-DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF`.
|
||||
- Unit: **root systemd unit** `/etc/systemd/system/frank-large.service`
|
||||
symlinked to `scripts/frank-large.service` in the repo; enabled at boot;
|
||||
`Restart=always`. Serve script `scripts/serve-frank-large.sh` + env
|
||||
`scripts/frank-large.env`.
|
||||
- Topology: 3090 (PCIe x4 in the M.2 dock, link downgraded from x16) carries
|
||||
dense/attention/KV (~11.5 GiB); ALL MoE expert tensors (~84 GiB) and the
|
||||
~33 GiB PLE n-gram table stay in LPDDR via CPU buffers.
|
||||
- Client contract: greedy default (`--temp 0.0`), reasoning on (deepseek
|
||||
format, unlimited budget), vision via mmproj, slot-state persistence under
|
||||
`scripts/slot-state`, 262144 ctx, 1 slot.
|
||||
|
||||
## Measured performance envelope (2026-09-15/16 campaign, all single-instance verified)
|
||||
|
||||
| config | decode tok/s | prefill tok/s | verdict |
|
||||
|---|---|---|---|
|
||||
| shipped (T16, `-ot exps=CPU`, NO_PINNED_WEIGHTS) | **3.8-4.0** | ~193 at 1.8k / 191 at 32k | shipped |
|
||||
| T8 / T4 / T2 / T1 | 4.18 / 4.20 / 4.26 / 3.97 | small-prompt only | decode flat across threads |
|
||||
| `-ncmoe 42` (6 layers' experts on GPU) | 4.54 | 32 (small prompt) | VRAM 22.0/24.6 GiB — fragile |
|
||||
| `-ncmoe 48 -thp` | 4.60 | 18 | **110 GiB anon host buffer — co-tenant killer, never ship** |
|
||||
| `--prefetch-experts` | 4.03 | — | no effect (page cache already resident) |
|
||||
| `--defer-experts --defer-ple -thp` | crashes | — | experts under `-ngl 999` still target CUDA0 → cudaMalloc OOM |
|
||||
| mainline llama.cpp `37b53fd` (`--n-cpu-moe 48`) | **1.01** | 34 | ruled out — CPU MoE kernels 4x slower than ik's iqk |
|
||||
|
||||
- Large-prompt prefill ≈ 192 tok/s is the **CPU AVX512-VNNI compute ceiling**
|
||||
for CPU-resident experts (expert weight reads amortize across the batch;
|
||||
compute-bound). Halogen's "4x llama.cpp family" prefill claim is therefore
|
||||
unreachable with experts on CPU — the physics caps hybrid MoE prefill here.
|
||||
- Decode is gated by a **serial per-token critical path**: ~100 CPU↔GPU
|
||||
backend splits per token (`graph splits = 100` in the log); the 3090 sits at
|
||||
**5% SM utilization** during decode; thread count does not matter (T1 ≈
|
||||
T16 ≈ 4 tok/s). The GPU sections are launch/sync-bound, not compute-bound.
|
||||
|
||||
## Root cause chain (perf evidence)
|
||||
|
||||
1. `perf` at T16: ~53% cycles in `ggml_graph_compute_thread` + ~37% in
|
||||
`ggml_compute_forward_mul_mat_id`, of which **~98% is the `pause`
|
||||
instruction** (thread-pool barrier spin). Actual vec_dot math: ~4%.
|
||||
2. T1 = T16 decode (3.97 vs 3.8-4.0) ⇒ the barrier spin is threads waiting on
|
||||
a serial critical path, not parallel overhead.
|
||||
3. GPU 95% idle during decode ⇒ the critical path is the GPU-section
|
||||
launch/sync overhead per split (x4 dock link, eager kernel launches).
|
||||
4. ik's CUDA-graph capture would collapse those launches but **aborts** on
|
||||
qwen4exp: the per-layer hyper-connection broadcasts (`ggml_repeat_4d`,
|
||||
4 sites in `src/graphs/build_qwen4exp.cpp`) resolve to a REPEAT node whose
|
||||
CUDA path does a capture-illegal operation
|
||||
(`CUDA error: operation not permitted when stream is capturing` → abort).
|
||||
5. Memory placement is NOT the lever: mmap-off/anon/THP/deferred variants all
|
||||
land at 3.8-4.6 tok/s.
|
||||
|
||||
## Why the 30 tok/s halogen bar is out of reach (honest record)
|
||||
|
||||
Halogen's engine held the whole 115.55 GiB w4b checkpoint resident in LPDDR
|
||||
(`HALOGEN_FLASH_PIN_TRUNK=1`) and served from one engine. This deployment must
|
||||
split 84 GiB of expert tensors across a 24 GB GPU + CPU, and every public
|
||||
runtime pays the split penalty above. Paths to >30 tok/s, in order of
|
||||
realism: (a) a runtime with capture-safe CUDA graphs + dedicated hc ops —
|
||||
mainline is building exactly this (ggml-org PR #28901 "qwen4exp: add hc ops");
|
||||
re-evaluate mainline when its hc-ops + graph handling mature; (b) experts on
|
||||
GPU — impossible, 84 GiB > 24 GiB; (c) a smaller model.
|
||||
|
||||
## Upstream issues tracking the slow TG path
|
||||
|
||||
ik_llama.cpp: #2374 (QSA opt POC), #2406 (single-threaded sigmoid), #2433
|
||||
(cache-miss full reprocess), #2460 (PLE history correctness). MTP/speculative
|
||||
decoding is unavailable for this GGUF — mradermacher stripped the NextN head
|
||||
(verified: no `blk.48.*`/nextn tensors in the file).
|
||||
|
||||
## Traps (learned the hard way 2026-09-15/16)
|
||||
|
||||
- **Pinned host buffers strand the host**: a default-policy start
|
||||
page-locks ~108 GiB (`CUDA_Host buffer`), evicting co-tenants into swap and
|
||||
starving sshd until a power cycle. `GGML_CUDA_NO_PINNED_WEIGHTS=1` + the
|
||||
serve-script guard exist for this. `-ncmoe N` with default NO_PINNED
|
||||
settings produces the same 108 GiB pinned buffer — never run it without the
|
||||
guard.
|
||||
- **Any `-ot`/`-ncmoe` override disables mmap** for the whole model load
|
||||
(`create_tensors` returns false → `ml.use_mmap = false`): the "mmap-backed
|
||||
CPU buffer" comment in the serve script is wrong on the mmap word; the
|
||||
buffer is anonymous. All placement variants perform alike anyway.
|
||||
- **`--defer-experts` does not keep experts off the GPU** under
|
||||
`-ngl 999` — only PLE defers (32.78 GiB). With ngl 999 the load dies with
|
||||
`cudaMalloc failed: out of memory` at ~79.7 GiB on device 0.
|
||||
- **The systemd unit self-heals through manual tests**: `pkill` the server and
|
||||
the unit respawns it ~4 s later (Restart=always) — two instances, port
|
||||
races, phantom measurements. For manual experiments:
|
||||
`sudo systemctl stop frank-large.service` first (it is a root unit; plain
|
||||
`systemctl --user` lines in older docs are stale), and restore with
|
||||
`sudo systemctl start frank-large.service`.
|
||||
- Verify "HTTP server listening" in the CURRENT log and
|
||||
`pgrep -xc llama-server` == 1 before trusting any measurement.
|
||||
- FA must stay ON — FA-off crashes the loader on this arch.
|
||||
- Do not run builds concurrently with decode measurements (`-j16` starves the
|
||||
service and corrupts both results).
|
||||
|
||||
## Rollback
|
||||
|
||||
Stop and disable the root unit, trash it, restore the halogen quadlet from
|
||||
`~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to
|
||||
`~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start;
|
||||
the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.
|
||||
@@ -1,10 +1,10 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts
|
||||
(2026-09-15): charlie12345/ROCmFPX llama.cpp on Dankserver (frank-medium,
|
||||
(2026-09-16): charlie12345/ROCmFPX llama.cpp on Dankserver (frank-medium,
|
||||
kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT on 2x Radeon AI PRO R9700) and
|
||||
halogen-flash-server 0.5.8 on Dankclaw (frank-large, peonist-ai
|
||||
Qwen3.8-Flash-Next w4b).
|
||||
ik_llama.cpp on Dankclaw (frank-large, mradermacher Qwen3.8-Flash-Next-
|
||||
Uncensored i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -26,11 +26,12 @@ layer split, 131k ctx / 1 slot, MTP]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[halogen-flash-server 0.5.8
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 64 GiB unified memory]
|
||||
E[ik_llama.cpp master d85a1d7
|
||||
frank-large, port 7770]
|
||||
F[peonist-ai Qwen3.8-Flash-Next
|
||||
w4b .hgn + overlay, 262k ctx / 4 slots]
|
||||
F[mradermacher Qwen3.8-Flash-Next-Uncensored
|
||||
i1-Q4_K_M GGUF, 262k ctx / 1 slot, vision]
|
||||
E2 --> E --> F
|
||||
end
|
||||
|
||||
@@ -58,8 +59,13 @@ w4b .hgn + overlay, 262k ctx / 4 slots]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[halogen-flash-server]] — ACTIVE frank-large runbook since 2026-09-11:
|
||||
peonist-ai/halogen-flash-server 0.5.8 serving
|
||||
- [[frank-large-ik-uncensored]] — ACTIVE frank-large runbook since
|
||||
2026-09-15: ik_llama.cpp (root systemd unit) serving mradermacher/
|
||||
Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 (dense/attention/
|
||||
KV) + LPDDR (experts/PLE) hybrid; measured performance envelope, the
|
||||
decode root-cause chain, tuning levers, and the co-tenancy traps
|
||||
- [[halogen-flash-server]] — RETIRED 2026-09-15 (rollback target): the
|
||||
peonist-ai halogen-flash-server 0.5.8 runbook serving
|
||||
peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn +
|
||||
quality overlay), rootless Quadlet container on Dankclaw port 7770 —
|
||||
deployed flags, memory model, client contract, troubleshooting
|
||||
|
||||
Reference in New Issue
Block a user