diff --git a/AGENTS.md b/AGENTS.md index 7b23194..a8ef820 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-medium` | charlie12345/ROCmFPX llama.cpp (system unit) | kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF, 2x Radeon AI PRO R9700 layer split, 131072 ctx / 1 slot, MTP draft on ROCm1 | 7770 | -| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai/halogen-qwen3.8-flash-next w4b .hgn, 262144 ctx / 3 slots, vision tower on | 7770 | +| Dankclaw | `frank-large` | ik_llama.cpp master d85a1d7 (root systemd unit) | mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF, RTX 3090 (dense/attn/KV) + LPDDR (experts/PLE) hybrid, 262144 ctx / 1 slot, vision via mmproj | 7770 | Retired endpoints (`frank-npu`, `frank-small`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -192,8 +192,8 @@ test process cannot share the single GPU anyway; validate against the service itself. 1. Stop and mask (self-healing would restart it): - - Dankclaw `frank-large` (rootless — no sudo): - `systemctl --user stop frank-large && systemctl --user mask frank-large` + - Dankclaw `frank-large` (root unit since the 2026-09-15 ik cutover): + `sudo systemctl stop frank-large && sudo systemctl mask frank-large` - Dankserver `frank-medium` (system unit): `sudo systemctl stop frank-medium && sudo systemctl mask frank-medium` 2. Bench. @@ -224,38 +224,29 @@ input anywhere in the fleet** (left with `frank-npu`). ## Active Services -### frank-large — halogen-flash-server (Dankclaw) +### frank-large — ik_llama.cpp (Dankclaw) -`ghcr.io/peonist-ai/halogen-flash-server:0.5.8` as a rootless Quadlet -(`dankclaw/frank-large/quadlet/frank-large.container` → `frank-large.service` -in the user manager), serving `peonist-ai/halogen-qwen3.8-flash-next`. -One container, two halves: engine on container loopback :8730 (unauthenticated, -NEVER published) and OpenAI front-end on :8731 published as host **7770**. +ik_llama.cpp master `d85a1d7` as a **root systemd unit** (repo +`scripts/frank-large.service` symlinked into `/etc/systemd/system/`; +`Restart=always`), serving `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M` +(111 GiB GGUF + mmproj-f16) on port 7770 with 262144 ctx / 1 slot, greedy +default, reasoning on (deepseek), vision via mmproj. Topology: the RTX 3090 +(PCIe x4 in the M.2 dock) carries dense/attention/KV; ALL MoE expert tensors +and the ~33 GiB PLE table stay in LPDDR via CPU buffers. Deployed flags in +`scripts/frank-large.env` / `scripts/serve-frank-large.sh`; the serve script +refuses to start without `GGML_CUDA_NO_PINNED_WEIGHTS=1` (the pinned-host +buffer incident starved co-tenants into unreachability until a power cycle). -Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=401408`, -`HALOGEN_KV_SLOTS=3`, `HALOGEN_MAX_TOK=32768`, `HALOGEN_HOST_RESERVE_GIB=15`, -`HALOGEN_VISION_TOWER=1`. Everything else upstream default. Sizing history -and the flag-by-flag rationale live in the runbook — do not re-derive here. +Measured envelope (2026-09-15/16 campaign): prefill ~192 tok/s (1.8k-32k +prompts, CPU VNNI compute ceiling), decode ~3.8-4.3 tok/s — gated by ~100 +CPU↔GPU backend splits per token (GPU 95% idle, launch/sync-bound; thread +count irrelevant). The halogen-era 30 tok/s decode bar is out of reach on +this model+hardware; the runbook carries the full evidence chain, the +mainline-llama.cpp comparison (1.0 tok/s — ruled out), and the tuning levers. -Invariants that survive any config change: - -- **Pool and reserve are a matched pair.** Budget = `121.2 host − 67.7 - weights − reserve`; that one budget carries KV pool + prefill arena + the - 1.97 GiB vision tower. The reserve is **not an allocation** — raising it - cannot free host RAM, it can only shrink the pool. To free host RAM, cut - `HALOGEN_KV_POOL_POSITIONS` or `HALOGEN_MAX_TOK`. -- **`free` lies on this host.** The ~68 GiB pinned weights count as - reclaimable page cache, so `free`/`MemAvailable`/Grafana overstate free - memory by the model size. Believe the server's startup ledger. A co-tenant - running the box dry looks like minutes of full-CPU stalls with no disk - activity — not a crash. -- **Client-visible contract:** no server-side sampler defaults (no - `temperature` ⇒ greedy); per-request budget defaults to 8192 and **includes - thinking** (running out mid-thought ⇒ `finish_reason: "length"`, empty - content); cap 65536; a request reserving prompt + `max_tokens` beyond the - pool queues, then fails `503 engine_busy` after the 3600 s timeout. - -Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`. +Runbook: `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md`. +Rollback: [[halogen-flash-server]] (quadlet archived in +`~/.AGENTTRASH/20260915-ik-swap/` on dankclaw). ### frank-medium — ROCmFPX llama.cpp (Dankserver) diff --git a/serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md b/serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md new file mode 100644 index 0000000..033636b --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md @@ -0,0 +1,120 @@ +--- +type: reference +description: frank-large on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps +tags: [llm, frank-large, ik_llama, dankclaw, rtx3090, strix-halo, performance] +timestamp: 2026-09-16T03:30:00-07:00 +--- + +# frank-large — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M + +**ACTIVE since 2026-09-15.** Replaced [[halogen-flash-server]] (rollback target; +quadlet unit archived in `~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` on +dankclaw). Serves `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M` +(111 GiB GGUF + mmproj-f16) at `192.168.0.143:7770`. + +## Shape + +- Runtime: ik_llama.cpp master `d85a1d7`, built at + `/home/dank/GitPackages/ik_llama.cpp/build` with `-DGGML_CUDA=ON + -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON + -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF`. +- Unit: **root systemd unit** `/etc/systemd/system/frank-large.service` + symlinked to `scripts/frank-large.service` in the repo; enabled at boot; + `Restart=always`. Serve script `scripts/serve-frank-large.sh` + env + `scripts/frank-large.env`. +- Topology: 3090 (PCIe x4 in the M.2 dock, link downgraded from x16) carries + dense/attention/KV (~11.5 GiB); ALL MoE expert tensors (~84 GiB) and the + ~33 GiB PLE n-gram table stay in LPDDR via CPU buffers. +- Client contract: greedy default (`--temp 0.0`), reasoning on (deepseek + format, unlimited budget), vision via mmproj, slot-state persistence under + `scripts/slot-state`, 262144 ctx, 1 slot. + +## Measured performance envelope (2026-09-15/16 campaign, all single-instance verified) + +| config | decode tok/s | prefill tok/s | verdict | +|---|---|---|---| +| shipped (T16, `-ot exps=CPU`, NO_PINNED_WEIGHTS) | **3.8-4.0** | ~193 at 1.8k / 191 at 32k | shipped | +| T8 / T4 / T2 / T1 | 4.18 / 4.20 / 4.26 / 3.97 | small-prompt only | decode flat across threads | +| `-ncmoe 42` (6 layers' experts on GPU) | 4.54 | 32 (small prompt) | VRAM 22.0/24.6 GiB — fragile | +| `-ncmoe 48 -thp` | 4.60 | 18 | **110 GiB anon host buffer — co-tenant killer, never ship** | +| `--prefetch-experts` | 4.03 | — | no effect (page cache already resident) | +| `--defer-experts --defer-ple -thp` | crashes | — | experts under `-ngl 999` still target CUDA0 → cudaMalloc OOM | +| mainline llama.cpp `37b53fd` (`--n-cpu-moe 48`) | **1.01** | 34 | ruled out — CPU MoE kernels 4x slower than ik's iqk | + +- Large-prompt prefill ≈ 192 tok/s is the **CPU AVX512-VNNI compute ceiling** + for CPU-resident experts (expert weight reads amortize across the batch; + compute-bound). Halogen's "4x llama.cpp family" prefill claim is therefore + unreachable with experts on CPU — the physics caps hybrid MoE prefill here. +- Decode is gated by a **serial per-token critical path**: ~100 CPU↔GPU + backend splits per token (`graph splits = 100` in the log); the 3090 sits at + **5% SM utilization** during decode; thread count does not matter (T1 ≈ + T16 ≈ 4 tok/s). The GPU sections are launch/sync-bound, not compute-bound. + +## Root cause chain (perf evidence) + +1. `perf` at T16: ~53% cycles in `ggml_graph_compute_thread` + ~37% in + `ggml_compute_forward_mul_mat_id`, of which **~98% is the `pause` + instruction** (thread-pool barrier spin). Actual vec_dot math: ~4%. +2. T1 = T16 decode (3.97 vs 3.8-4.0) ⇒ the barrier spin is threads waiting on + a serial critical path, not parallel overhead. +3. GPU 95% idle during decode ⇒ the critical path is the GPU-section + launch/sync overhead per split (x4 dock link, eager kernel launches). +4. ik's CUDA-graph capture would collapse those launches but **aborts** on + qwen4exp: the per-layer hyper-connection broadcasts (`ggml_repeat_4d`, + 4 sites in `src/graphs/build_qwen4exp.cpp`) resolve to a REPEAT node whose + CUDA path does a capture-illegal operation + (`CUDA error: operation not permitted when stream is capturing` → abort). +5. Memory placement is NOT the lever: mmap-off/anon/THP/deferred variants all + land at 3.8-4.6 tok/s. + +## Why the 30 tok/s halogen bar is out of reach (honest record) + +Halogen's engine held the whole 115.55 GiB w4b checkpoint resident in LPDDR +(`HALOGEN_FLASH_PIN_TRUNK=1`) and served from one engine. This deployment must +split 84 GiB of expert tensors across a 24 GB GPU + CPU, and every public +runtime pays the split penalty above. Paths to >30 tok/s, in order of +realism: (a) a runtime with capture-safe CUDA graphs + dedicated hc ops — +mainline is building exactly this (ggml-org PR #28901 "qwen4exp: add hc ops"); +re-evaluate mainline when its hc-ops + graph handling mature; (b) experts on +GPU — impossible, 84 GiB > 24 GiB; (c) a smaller model. + +## Upstream issues tracking the slow TG path + +ik_llama.cpp: #2374 (QSA opt POC), #2406 (single-threaded sigmoid), #2433 +(cache-miss full reprocess), #2460 (PLE history correctness). MTP/speculative +decoding is unavailable for this GGUF — mradermacher stripped the NextN head +(verified: no `blk.48.*`/nextn tensors in the file). + +## Traps (learned the hard way 2026-09-15/16) + +- **Pinned host buffers strand the host**: a default-policy start + page-locks ~108 GiB (`CUDA_Host buffer`), evicting co-tenants into swap and + starving sshd until a power cycle. `GGML_CUDA_NO_PINNED_WEIGHTS=1` + the + serve-script guard exist for this. `-ncmoe N` with default NO_PINNED + settings produces the same 108 GiB pinned buffer — never run it without the + guard. +- **Any `-ot`/`-ncmoe` override disables mmap** for the whole model load + (`create_tensors` returns false → `ml.use_mmap = false`): the "mmap-backed + CPU buffer" comment in the serve script is wrong on the mmap word; the + buffer is anonymous. All placement variants perform alike anyway. +- **`--defer-experts` does not keep experts off the GPU** under + `-ngl 999` — only PLE defers (32.78 GiB). With ngl 999 the load dies with + `cudaMalloc failed: out of memory` at ~79.7 GiB on device 0. +- **The systemd unit self-heals through manual tests**: `pkill` the server and + the unit respawns it ~4 s later (Restart=always) — two instances, port + races, phantom measurements. For manual experiments: + `sudo systemctl stop frank-large.service` first (it is a root unit; plain + `systemctl --user` lines in older docs are stale), and restore with + `sudo systemctl start frank-large.service`. +- Verify "HTTP server listening" in the CURRENT log and + `pgrep -xc llama-server` == 1 before trusting any measurement. +- FA must stay ON — FA-off crashes the loader on this arch. +- Do not run builds concurrently with decode measurements (`-j16` starves the + service and corrupts both results). + +## Rollback + +Stop and disable the root unit, trash it, restore the halogen quadlet from +`~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to +`~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start; +the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk. diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 0773c52..8de0379 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,10 +1,10 @@ # LLM Fleet — Knowledge Base Documentation for the LLM inference fleet on the two hosts -(2026-09-15): charlie12345/ROCmFPX llama.cpp on Dankserver (frank-medium, +(2026-09-16): charlie12345/ROCmFPX llama.cpp on Dankserver (frank-medium, kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT on 2x Radeon AI PRO R9700) and -halogen-flash-server 0.5.8 on Dankclaw (frank-large, peonist-ai -Qwen3.8-Flash-Next w4b). +ik_llama.cpp on Dankclaw (frank-large, mradermacher Qwen3.8-Flash-Next- +Uncensored i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid). > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -26,11 +26,12 @@ layer split, 131k ctx / 1 slot, MTP] end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] - E2[gfx1151 iGPU 64 GiB unified memory] - E[halogen-flash-server 0.5.8 + E2[RTX 3090 (PCIe x4 dock) + +gfx1151 iGPU 64 GiB unified memory] + E[ik_llama.cpp master d85a1d7 frank-large, port 7770] - F[peonist-ai Qwen3.8-Flash-Next -w4b .hgn + overlay, 262k ctx / 4 slots] + F[mradermacher Qwen3.8-Flash-Next-Uncensored +i1-Q4_K_M GGUF, 262k ctx / 1 slot, vision] E2 --> E --> F end @@ -58,8 +59,13 @@ w4b .hgn + overlay, 262k ctx / 4 slots] (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[halogen-flash-server]] — ACTIVE frank-large runbook since 2026-09-11: - peonist-ai/halogen-flash-server 0.5.8 serving +- [[frank-large-ik-uncensored]] — ACTIVE frank-large runbook since + 2026-09-15: ik_llama.cpp (root systemd unit) serving mradermacher/ + Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 (dense/attention/ + KV) + LPDDR (experts/PLE) hybrid; measured performance envelope, the + decode root-cause chain, tuning levers, and the co-tenancy traps +- [[halogen-flash-server]] — RETIRED 2026-09-15 (rollback target): the + peonist-ai halogen-flash-server 0.5.8 runbook serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn + quality overlay), rootless Quadlet container on Dankclaw port 7770 — deployed flags, memory model, client contract, troubleshooting diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index b7c9944..3a1625b 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -125,8 +125,12 @@ The LLM inference fleet on the two hosts. Full index: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record) -- [[halogen-flash-server]] — the ACTIVE frank-large runbook since - 2026-09-11: peonist-ai/halogen-flash-server 0.5.8 serving +- [[frank-large-ik-uncensored]] — the ACTIVE frank-large runbook since + 2026-09-15: ik_llama.cpp serving mradermacher/Qwen3.8-Flash-Next- + Uncensored-i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid (root systemd + unit, port 7770) — measured performance envelope and traps +- [[halogen-flash-server]] — the RETIRED 2026-09-15 halogen runbook and + rollback target: peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (w4b) as a rootless Quadlet container on Dankclaw, port 7770 — flags, memory model, client contract - [[dankclaw-iommu-translation-off]] — why Dankclaw boots with