Frank
33eae71d33
All ~173 managed skills read in full and merged, one polished reference page per skill-cluster: - LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds + deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up). - Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage. - Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access. - Infrastructure: Observability Provisioning + Homepage Status Probes; updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page preserved and extended). - FrankBook-Pro: ARR-Stack filled (API split, schema-first probing, auth-posture, auth-config trap); qBittorrent E2E verification. - Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and Edits, Command Center Dotfiles, PICO-8 Asset Standards. - Indexes: LLM Cross-Cutting References section, Operations/ and Agent/ index pages, root-index Skill Distillation section, AGENTS.md pointer. - log.md carries the wave plan. check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
150 lines
8.6 KiB
Markdown
150 lines
8.6 KiB
Markdown
---
|
||
type: reference
|
||
description: frank-large on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps
|
||
tags: [llm, frank-large, ik_llama, dankclaw, rtx3090, strix-halo, performance]
|
||
timestamp: 2026-09-16T03:30:00-07:00
|
||
---
|
||
|
||
# frank-large — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M
|
||
|
||
**RETIRED 2026-09-16, after one day.** Replaced by [[halogen-flash-server]],
|
||
which was the runtime before it as well — the fleet went ik → halogen → back.
|
||
Its unit, env, serve script and the Qwen-Sharp template are archived in the repo
|
||
under `scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`, and the
|
||
payload, build and chat template stay on disk. The measurements below are not a
|
||
stale plan: they are the complete account of what this model costs on this
|
||
hardware, and they are why the fleet went back. It served
|
||
`mradermacher/Qwen3.8-Flash-Next-Uncensored` (111 GiB GGUF + mmproj-f16) at
|
||
`192.168.0.143:7770` while it ran.
|
||
|
||
## Shape
|
||
|
||
- Runtime: ik_llama.cpp master `d85a1d7`, built at
|
||
`/home/dank/GitPackages/ik_llama.cpp/build` with `-DGGML_CUDA=ON
|
||
-DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON
|
||
-DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF`.
|
||
- Unit: **root systemd unit** `/etc/systemd/system/frank-large.service`
|
||
symlinked to `scripts/frank-large.service` in the repo; enabled at boot;
|
||
`Restart=always`. Serve script `scripts/serve-frank-large.sh` + env
|
||
`scripts/frank-large.env`.
|
||
- Topology: 3090 (PCIe x4 in the M.2 dock, link downgraded from x16) carries
|
||
dense/attention/KV (~11.5 GiB); ALL MoE expert tensors (~84 GiB) and the
|
||
~33 GiB PLE n-gram table stay in LPDDR via CPU buffers.
|
||
- Client contract: greedy default (`--temp 0.0`), reasoning on (deepseek
|
||
format, unlimited budget), vision via mmproj, slot-state persistence under
|
||
`scripts/slot-state`, 262144 ctx, 1 slot.
|
||
|
||
## Measured performance envelope (2026-09-15/16 campaign, all single-instance verified)
|
||
|
||
| config | decode tok/s | prefill tok/s | verdict |
|
||
|---|---|---|---|
|
||
| shipped (T16, `-ot exps=CPU`, NO_PINNED_WEIGHTS) | **3.8-4.0** | ~193 at 1.8k / 191 at 32k | shipped |
|
||
| T8 / T4 / T2 / T1 | 4.18 / 4.20 / 4.26 / 3.97 | small-prompt only | decode flat across threads |
|
||
| `-ncmoe 42` (6 layers' experts on GPU) | 4.54 | 32 (small prompt) | VRAM 22.0/24.6 GiB — fragile |
|
||
| `-ncmoe 48 -thp` | 4.60 | 18 | **110 GiB anon host buffer — co-tenant killer, never ship** |
|
||
| `--prefetch-experts` | 4.03 | — | no effect (page cache already resident) |
|
||
| `--defer-experts --defer-ple -thp` | crashes | — | experts under `-ngl 999` still target CUDA0 → cudaMalloc OOM |
|
||
| mainline llama.cpp `37b53fd` (`--n-cpu-moe 48`) | **1.01** | 34 | ruled out — CPU MoE kernels 4x slower than ik's iqk |
|
||
|
||
- Large-prompt prefill ≈ 192 tok/s is the **CPU AVX512-VNNI compute ceiling**
|
||
for CPU-resident experts (expert weight reads amortize across the batch;
|
||
compute-bound). Halogen's "4x llama.cpp family" prefill claim is therefore
|
||
unreachable with experts on CPU — the physics caps hybrid MoE prefill here.
|
||
- Decode is gated by a **serial per-token critical path**: ~100 CPU↔GPU
|
||
backend splits per token (`graph splits = 100` in the log); the 3090 sits at
|
||
**5% SM utilization** during decode; thread count does not matter (T1 ≈
|
||
T16 ≈ 4 tok/s). The GPU sections are launch/sync-bound, not compute-bound.
|
||
|
||
## Root cause chain (perf evidence)
|
||
|
||
1. `perf` at T16: ~53% cycles in `ggml_graph_compute_thread` + ~37% in
|
||
`ggml_compute_forward_mul_mat_id`, of which **~98% is the `pause`
|
||
instruction** (thread-pool barrier spin). Actual vec_dot math: ~4%.
|
||
2. T1 = T16 decode (3.97 vs 3.8-4.0) ⇒ the barrier spin is threads waiting on
|
||
a serial critical path, not parallel overhead.
|
||
3. GPU 95% idle during decode ⇒ the critical path is the GPU-section
|
||
launch/sync overhead per split (x4 dock link, eager kernel launches).
|
||
4. ik's CUDA-graph capture would collapse those launches but **aborts** on
|
||
qwen4exp: the per-layer hyper-connection broadcasts (`ggml_repeat_4d`,
|
||
4 sites in `src/graphs/build_qwen4exp.cpp`) resolve to a REPEAT node whose
|
||
CUDA path does a capture-illegal operation
|
||
(`CUDA error: operation not permitted when stream is capturing` → abort).
|
||
5. Memory placement is NOT the lever: mmap-off/anon/THP/deferred variants all
|
||
land at 3.8-4.6 tok/s.
|
||
|
||
## Why the 30 tok/s halogen bar is out of reach (honest record)
|
||
|
||
Halogen's engine held the whole 115.55 GiB w4b checkpoint resident in LPDDR
|
||
(`HALOGEN_FLASH_PIN_TRUNK=1`) and served from one engine. This deployment must
|
||
split 84 GiB of expert tensors across a 24 GB GPU + CPU, and every public
|
||
runtime pays the split penalty above. Paths to >30 tok/s, in order of
|
||
realism: (a) a runtime with capture-safe CUDA graphs + dedicated hc ops —
|
||
mainline is building exactly this (ggml-org PR #28901 "qwen4exp: add hc ops");
|
||
re-evaluate mainline when its hc-ops + graph handling mature; (b) experts on
|
||
GPU — impossible, 84 GiB > 24 GiB; (c) a smaller model.
|
||
|
||
## Upstream issues tracking the slow TG path
|
||
|
||
ik_llama.cpp: #2374 (QSA opt POC), #2406 (single-threaded sigmoid), #2433
|
||
(cache-miss full reprocess), #2460 (PLE history correctness). MTP/speculative
|
||
decoding is unavailable for this GGUF — mradermacher stripped the NextN head
|
||
(verified: no `blk.48.*`/nextn tensors in the file).
|
||
|
||
## Traps (learned the hard way 2026-09-15/16)
|
||
|
||
- **Pinned host buffers strand the host**: a default-policy start
|
||
page-locks ~108 GiB (`CUDA_Host buffer`), evicting co-tenants into swap and
|
||
starving sshd until a power cycle. `GGML_CUDA_NO_PINNED_WEIGHTS=1` + the
|
||
serve-script guard exist for this. `-ncmoe N` with default NO_PINNED
|
||
settings produces the same 108 GiB pinned buffer — never run it without the
|
||
guard.
|
||
- **Any `-ot`/`-ncmoe` override disables mmap** for the whole model load
|
||
(`create_tensors` returns false → `ml.use_mmap = false`): the "mmap-backed
|
||
CPU buffer" comment in the serve script is wrong on the mmap word; the
|
||
buffer is anonymous. All placement variants perform alike anyway.
|
||
- **`--defer-experts` does not keep experts off the GPU** under
|
||
`-ngl 999` — only PLE defers (32.78 GiB). With ngl 999 the load dies with
|
||
`cudaMalloc failed: out of memory` at ~79.7 GiB on device 0.
|
||
- **The systemd unit self-heals through manual tests**: `pkill` the server and
|
||
the unit respawns it ~4 s later (Restart=always) — two instances, port
|
||
races, phantom measurements. For manual experiments:
|
||
`sudo systemctl stop frank-large.service` first (it is a root unit; plain
|
||
`systemctl --user` lines in older docs are stale), and restore with
|
||
`sudo systemctl start frank-large.service`.
|
||
- Verify "HTTP server listening" in the CURRENT log and
|
||
`pgrep -xc llama-server` == 1 before trusting any measurement.
|
||
- FA must stay ON — FA-off crashes the loader on this arch.
|
||
- Do not run builds concurrently with decode measurements (`-j16` starves the
|
||
service and corrupts both results).
|
||
|
||
## Rollback
|
||
|
||
Stop and disable the root unit, trash it, restore the halogen quadlet from
|
||
`~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to
|
||
`~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start;
|
||
the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.
|
||
|
||
## Hybrid campaign envelope (2026-09-16, 3090 + Strix Halo)
|
||
|
||
Why this deployment could not be tuned into a production endpoint: decode
|
||
3.8–4.3 tok/s flat across thread counts (T1 ≈ T16); prefill ~192 tok/s at
|
||
1.8k–32k prompts = the **CPU AVX512-VNNI compute ceiling** for CPU experts.
|
||
The decode root cause is structural: ~100 CPU↔GPU backend splits per token
|
||
with the GPU at 5 % SM during decode — launch/sync-bound over the PCIe x4
|
||
dock, not compute- or memory-bound. T16 profiling shows ~90 % of cycles in
|
||
`pause` (ggml_barrier spin): threads wait on the serial critical path.
|
||
Memory placement (mmap/anon/THP/deferred) does not matter — all variants
|
||
landed 3.8–4.6.
|
||
|
||
- CUDA-graph capture aborts: qwen4exp's hyper-connection broadcasts use
|
||
per-layer `ggml_repeat_4d`, whose CUDA path is capture-illegal; until ik
|
||
gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs
|
||
stay dead.
|
||
- Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels
|
||
4× slower than ik's iqk) — do not switch engines for speed.
|
||
- MTP/speculative decoding is impossible for mradermacher i1 GGUFs: the
|
||
NextN head is stripped (no `blk.48` tensors).
|
||
- Tuning lever ranked first: `-ncmoe 42` (6 layers' experts on the 3090) →
|
||
decode 4.54, small-prompt prefill 32 — but VRAM 22/24.6 GiB at 262k ctx,
|
||
an operator OOM-margin decision, and it needs `GGML_CUDA_NO_PINNED_WEIGHTS=1`.
|