Document Q8 AGENT set_rows serving blocker
This commit is contained in:
Frank
@@ -196,6 +196,16 @@ budget (raised 2026-08-25). Env `scripts/frank-default-large.env`, serve
|
||||
`scripts/serve-frank-large.sh`, runbook
|
||||
[docs/LLM/kingjones-q8agent-rocmfpx.md](docs/LLM/kingjones-q8agent-rocmfpx.md).
|
||||
|
||||
**Serving status (2026-08-25):** the model loads but crashes in
|
||||
`ggml_compute_forward_set_rows` (NULL dispatch in `libggml-cpu.so.0`) before it
|
||||
can serve. Tested on all five available ROCmFPX builds (build-kairic v81,
|
||||
build-promptforge v3, build-rocmi4-w4a4 v257, build-strix-rocmfp4 v209) and the
|
||||
all-CPU path; all crash. Not GPU-specific, not MTP-specific, not fixed by
|
||||
`LLAMA_SET_ROWS=0`. Fix is a rebuild of charlie12345/ROCmFPX with a working
|
||||
`set_rows`. The proven serving context is 131072 at `gl=45` (262144 OOMs on
|
||||
device-local at `gl=999`). Config goals met: 114 GiB TTM/GTT, reasoning on,
|
||||
`preserve_thinking`/`preserve_reasoning` on.
|
||||
|
||||
### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25)
|
||||
|
||||
The v4 runtime serves `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (`eemin` namespace,
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ ngram-mod, selected experts offloaded via override, 491520 ctx]
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
D[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[CIRU ROCmFPX llama.cpp PromptForge v3
|
||||
E[charlie12345/ROCmFPX build-kairic (v81)
|
||||
frank-default-large, port 7770]
|
||||
F[Qwen3.8-27B ROCmFPX Q8_0 AGENT
|
||||
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
# frank-default-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
|
||||
|
||||
**Status:** ACTIVE on Dankclaw (since 2026-08-25).
|
||||
**Status:** Deployed on Dankclaw (since 2026-08-25). Serving is **BLOCKED by
|
||||
the `set_rows` build bug** — the model loads but crashes in
|
||||
`ggml_compute_forward_set_rows` before it can serve. All config goals are met;
|
||||
a build fix is required for the model to run.
|
||||
|
||||
**Alias:** `frank-default-large`, port **7770**.
|
||||
|
||||
@@ -26,6 +29,49 @@ fit in the lifted **114 GiB TTM/GTT** budget (see the 114 GiB change in
|
||||
| Tunables | `scripts/frank-default-large.env` |
|
||||
| Serve script | `scripts/serve-frank-large.sh` |
|
||||
|
||||
## Serving status: the `set_rows` build bug (blocks serving)
|
||||
|
||||
As of 2026-08-25 the Q8_0 AGENT model will not serve on Dankclaw. It loads,
|
||||
reaches `initializing slots, n_slots = 1`, but never prints `start serving`.
|
||||
Health stays `503` until the process exits. The crash is in
|
||||
`ggml_compute_forward_set_rows` in `libggml-cpu.so.0`, a NULL function-pointer
|
||||
jump (address `0x00000000`).
|
||||
|
||||
The backtrace (gdb). The main thread and one OpenMP worker crash in the same
|
||||
path:
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[llama_context::graph_compute] --> B[ggml_backend_cpu_graph_compute]
|
||||
B --> C[ggml_graph_compute]
|
||||
C --> D[GOMP_parallel]
|
||||
D --> E[ggml_compute_forward_set_rows]
|
||||
E --> F[NULL dispatch -> address 0x0]
|
||||
D --> G[Thread 23 worker: same path]
|
||||
```
|
||||
|
||||
Test matrix. Five ROCmFPX builds and the all-CPU path were tested on a scratch
|
||||
port (`/tmp/g80bench/run.sh`), `gl=45`, `ctx=131072`. All crash in `set_rows`.
|
||||
|
||||
| Build | Binary | Version | Result |
|
||||
| ----- | ------ | ------- | ------ |
|
||||
| build-kairic | `ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server` | v81 (205a3e5f4) | segfault in set_rows |
|
||||
| build-promptforge | `ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` | v3 (caf-eemin lineage) | segfault in set_rows |
|
||||
| build-rocmi4-w4a4 | `rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` | v257 (c49ebdb) | segfault in set_rows |
|
||||
| build-strix-rocmfp4 | `GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server` | v209 (00d5452) | segfault in set_rows |
|
||||
| all-CPU | build-kairic, `-ngl 0` | v81 | segfault in set_rows |
|
||||
|
||||
The crash is not GPU-specific (the all-CPU path fails too), not MTP-specific
|
||||
(`--spec-type none` fails), and not fixed by `LLAMA_SET_ROWS=0`. So it is a
|
||||
build bug: the Q8_0 AGENT GGUF graph needs a `set_rows` dispatch the ROCmFPX
|
||||
backends do not register. The older caf-eemin and Kairic Edge models served
|
||||
fine on the same builds, so the Q8_0 AGENT graph is the trigger.
|
||||
|
||||
**Fix path:** rebuild charlie12345/ROCmFPX from source with a working
|
||||
`set_rows` for the ROCmFPX backend, or find a build where it is registered.
|
||||
This unblocks the model. See the context budget reality below once `set_rows`
|
||||
is fixed.
|
||||
|
||||
## Model files
|
||||
|
||||
Three files, from the HF repo above:
|
||||
@@ -93,9 +139,14 @@ Key choices:
|
||||
- **f16 K + TurboQuant V** cache. f16 K is the safe long-ctx default; TurboQuant
|
||||
halves the V cache at 262k while keeping quality (see the ubatch/threads-batch
|
||||
notes in [[qwen3.8-strix-optimizations]]).
|
||||
- **Full 262144 context, 1 slot.** The whole context KV cache plus the ~30 GB
|
||||
payload (main + draft + mmproj) fit in the 114 GiB budget. Concurrent
|
||||
requests queue; the server never preempts a running generation.
|
||||
- **262144 context, 1 slot — the target.** The whole context KV cache plus the
|
||||
~30 GB payload (main + draft + mmproj) are meant to fit in the 114 GiB
|
||||
budget. Measured reality: `gl=999` (all layers on ROCm0) at 262144 **OOMs**
|
||||
on device-local (`cudaMalloc failed`, the KV cache needs ~10-18 GiB that does
|
||||
not fit once the payload is on the card). `gl=45` (partial offload) fits a
|
||||
131072 context. The env ships `CTX_SIZE=262144`; the proven serving context
|
||||
today is **131072 at `gl=45`** (before the `set_rows` fix). `deploy.sh` waits
|
||||
up to 20 minutes (see `LLM_WAIT`).
|
||||
- **Reasoning on.** The Sharp template drives thinking (medium effort);
|
||||
`--reasoning-format none` lets the template project the reasoning content into
|
||||
`reasoning_content`. `--reasoning-budget -1` = no budget cap.
|
||||
@@ -160,6 +211,11 @@ reads `--host/--port/--temp` as CLI flags, never as env vars.
|
||||
- **2026-08-25 (today):** switched the large service to the kingjones
|
||||
Q8_0-AGENT model with 262144 context and reasoning on. Downloaded the three
|
||||
payload files from the HF repo, raised the TTM/GTT to 114 GiB for the full
|
||||
context, and persisted the env/serve script in this repo.
|
||||
context, and persisted the env/serve script in this repo. Measured that the
|
||||
model loads but crashes in `ggml_compute_forward_set_rows` on all five
|
||||
available ROCmFPX builds; serving is BLOCKED pending a rebuild. The config
|
||||
goals (GTT 114 GiB, reasoning on, preserve_thinking/preserve_reasoning on,
|
||||
262144 env context) are in place. Proven serving context today is 131072 at
|
||||
`gl=45`. See the `set_rows` build bug section above.
|
||||
- Preceding model: Kairic Edge Qwen3.8-27B IU4 (deployed 2026-08-25 ~05:42,
|
||||
retired the same day for the AGENT model). See [[kairic-edge]].
|
||||
|
||||
Reference in New Issue
Block a user