From 6c9e71efd2445cf4a3e58accae600b61ec20d4ed Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 25 Aug 2026 12:16:47 -0700 Subject: [PATCH] Document Q8 AGENT set_rows serving blocker --- AGENTS.md | 10 ++++ docs/LLM/index.md | 2 +- docs/LLM/kingjones-q8agent-rocmfpx.md | 66 +++++++++++++++++++++++++-- 3 files changed, 72 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8437997..f9df80b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -196,6 +196,16 @@ budget (raised 2026-08-25). Env `scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`, runbook [docs/LLM/kingjones-q8agent-rocmfpx.md](docs/LLM/kingjones-q8agent-rocmfpx.md). +**Serving status (2026-08-25):** the model loads but crashes in +`ggml_compute_forward_set_rows` (NULL dispatch in `libggml-cpu.so.0`) before it +can serve. Tested on all five available ROCmFPX builds (build-kairic v81, +build-promptforge v3, build-rocmi4-w4a4 v257, build-strix-rocmfp4 v209) and the +all-CPU path; all crash. Not GPU-specific, not MTP-specific, not fixed by +`LLAMA_SET_ROWS=0`. Fix is a rebuild of charlie12345/ROCmFPX with a working +`set_rows`. The proven serving context is 131072 at `gl=45` (262144 OOMs on +device-local at `gl=999`). Config goals met: 114 GiB TTM/GTT, reasoning on, +`preserve_thinking`/`preserve_reasoning` on. + ### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25) The v4 runtime serves `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (`eemin` namespace, diff --git a/docs/LLM/index.md b/docs/LLM/index.md index a358555..607a245 100644 --- a/docs/LLM/index.md +++ b/docs/LLM/index.md @@ -26,7 +26,7 @@ ngram-mod, selected experts offloaded via override, 491520 ctx] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] D[gfx1151 iGPU 64 GiB unified memory] - E[CIRU ROCmFPX llama.cpp PromptForge v3 + E[charlie12345/ROCmFPX build-kairic (v81) frank-default-large, port 7770] F[Qwen3.8-27B ROCmFPX Q8_0 AGENT MTP draft + Turbo4 V cache, 262K ctx / 1 slot] diff --git a/docs/LLM/kingjones-q8agent-rocmfpx.md b/docs/LLM/kingjones-q8agent-rocmfpx.md index 1c71527..b731dbd 100644 --- a/docs/LLM/kingjones-q8agent-rocmfpx.md +++ b/docs/LLM/kingjones-q8agent-rocmfpx.md @@ -1,6 +1,9 @@ # frank-default-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT -**Status:** ACTIVE on Dankclaw (since 2026-08-25). +**Status:** Deployed on Dankclaw (since 2026-08-25). Serving is **BLOCKED by +the `set_rows` build bug** — the model loads but crashes in +`ggml_compute_forward_set_rows` before it can serve. All config goals are met; +a build fix is required for the model to run. **Alias:** `frank-default-large`, port **7770**. @@ -26,6 +29,49 @@ fit in the lifted **114 GiB TTM/GTT** budget (see the 114 GiB change in | Tunables | `scripts/frank-default-large.env` | | Serve script | `scripts/serve-frank-large.sh` | +## Serving status: the `set_rows` build bug (blocks serving) + +As of 2026-08-25 the Q8_0 AGENT model will not serve on Dankclaw. It loads, +reaches `initializing slots, n_slots = 1`, but never prints `start serving`. +Health stays `503` until the process exits. The crash is in +`ggml_compute_forward_set_rows` in `libggml-cpu.so.0`, a NULL function-pointer +jump (address `0x00000000`). + +The backtrace (gdb). The main thread and one OpenMP worker crash in the same +path: + +```mermaid +graph TD + A[llama_context::graph_compute] --> B[ggml_backend_cpu_graph_compute] + B --> C[ggml_graph_compute] + C --> D[GOMP_parallel] + D --> E[ggml_compute_forward_set_rows] + E --> F[NULL dispatch -> address 0x0] + D --> G[Thread 23 worker: same path] +``` + +Test matrix. Five ROCmFPX builds and the all-CPU path were tested on a scratch +port (`/tmp/g80bench/run.sh`), `gl=45`, `ctx=131072`. All crash in `set_rows`. + +| Build | Binary | Version | Result | +| ----- | ------ | ------- | ------ | +| build-kairic | `ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server` | v81 (205a3e5f4) | segfault in set_rows | +| build-promptforge | `ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` | v3 (caf-eemin lineage) | segfault in set_rows | +| build-rocmi4-w4a4 | `rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` | v257 (c49ebdb) | segfault in set_rows | +| build-strix-rocmfp4 | `GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server` | v209 (00d5452) | segfault in set_rows | +| all-CPU | build-kairic, `-ngl 0` | v81 | segfault in set_rows | + +The crash is not GPU-specific (the all-CPU path fails too), not MTP-specific +(`--spec-type none` fails), and not fixed by `LLAMA_SET_ROWS=0`. So it is a +build bug: the Q8_0 AGENT GGUF graph needs a `set_rows` dispatch the ROCmFPX +backends do not register. The older caf-eemin and Kairic Edge models served +fine on the same builds, so the Q8_0 AGENT graph is the trigger. + +**Fix path:** rebuild charlie12345/ROCmFPX from source with a working +`set_rows` for the ROCmFPX backend, or find a build where it is registered. +This unblocks the model. See the context budget reality below once `set_rows` +is fixed. + ## Model files Three files, from the HF repo above: @@ -93,9 +139,14 @@ Key choices: - **f16 K + TurboQuant V** cache. f16 K is the safe long-ctx default; TurboQuant halves the V cache at 262k while keeping quality (see the ubatch/threads-batch notes in [[qwen3.8-strix-optimizations]]). -- **Full 262144 context, 1 slot.** The whole context KV cache plus the ~30 GB - payload (main + draft + mmproj) fit in the 114 GiB budget. Concurrent - requests queue; the server never preempts a running generation. +- **262144 context, 1 slot — the target.** The whole context KV cache plus the + ~30 GB payload (main + draft + mmproj) are meant to fit in the 114 GiB + budget. Measured reality: `gl=999` (all layers on ROCm0) at 262144 **OOMs** + on device-local (`cudaMalloc failed`, the KV cache needs ~10-18 GiB that does + not fit once the payload is on the card). `gl=45` (partial offload) fits a + 131072 context. The env ships `CTX_SIZE=262144`; the proven serving context + today is **131072 at `gl=45`** (before the `set_rows` fix). `deploy.sh` waits + up to 20 minutes (see `LLM_WAIT`). - **Reasoning on.** The Sharp template drives thinking (medium effort); `--reasoning-format none` lets the template project the reasoning content into `reasoning_content`. `--reasoning-budget -1` = no budget cap. @@ -160,6 +211,11 @@ reads `--host/--port/--temp` as CLI flags, never as env vars. - **2026-08-25 (today):** switched the large service to the kingjones Q8_0-AGENT model with 262144 context and reasoning on. Downloaded the three payload files from the HF repo, raised the TTM/GTT to 114 GiB for the full - context, and persisted the env/serve script in this repo. + context, and persisted the env/serve script in this repo. Measured that the + model loads but crashes in `ggml_compute_forward_set_rows` on all five + available ROCmFPX builds; serving is BLOCKED pending a rebuild. The config + goals (GTT 114 GiB, reasoning on, preserve_thinking/preserve_reasoning on, + 262144 env context) are in place. Proven serving context today is 131072 at + `gl=45`. See the `set_rows` build bug section above. - Preceding model: Kairic Edge Qwen3.8-27B IU4 (deployed 2026-08-25 ~05:42, retired the same day for the AGENT model). See [[kairic-edge]].