# frank-default-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT **Status:** Deployed on Dankclaw (since 2026-08-25). Serving is **BLOCKED by the `set_rows` build bug** — the model loads but crashes in `ggml_compute_forward_set_rows` before it can serve. All config goals are met; a build fix is required for the model to run. **Alias:** `frank-default-large`, port **7770**. The large service serves the kingjones `Qwen3.8-27B-ROCmFPX-Q8_0-AGENT` GGUF from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` with a charlie12345/ROCmFPX `llama-server` (build `build-kairic`). It runs a **standalone MTP draft head** (loaded separately, on ROCm0), **TurboQuant** V KV cache, the **Sharp** chat template with **reasoning on**, and a **full 262144 context / 1 slot** profile. The whole payload plus the 262k KV cache fit in the lifted **114 GiB TTM/GTT** budget (see the 114 GiB change in [[Dankclaw Improvements]] and [[qwen3.8-strix-optimizations]]). | Figure | Value | | ------ | ----- | | Model | kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT (`Q8_0_ROCMFPX`) | | HF repo | `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF` (public) | | Runtime | charlie12345/ROCmFPX `llama-server`, `build-kairic` | | Draft head | standalone `mtp-Qwen3.8-27B-Q4_0.gguf` on ROCm0 | | KV cache | f16 K, TurboQuant (`turbo4`) V, full 262144 context, 1 slot | | Reasoning | on, `deepseek`, budget `-1`, `--reasoning-format none` | | Chat template | Sharp (repo-tracked), `enable_thinking` + `preserve_thinking` on | | Host path | `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` | | Tunables | `scripts/frank-default-large.env` | | Serve script | `scripts/serve-frank-large.sh` | ## Serving status: the `set_rows` build bug (blocks serving) As of 2026-08-25 the Q8_0 AGENT model will not serve on Dankclaw. It loads, reaches `initializing slots, n_slots = 1`, but never prints `start serving`. Health stays `503` until the process exits. The crash is in `ggml_compute_forward_set_rows` in `libggml-cpu.so.0`, a NULL function-pointer jump (address `0x00000000`). The backtrace (gdb). The main thread and one OpenMP worker crash in the same path: ```mermaid graph TD A[llama_context::graph_compute] --> B[ggml_backend_cpu_graph_compute] B --> C[ggml_graph_compute] C --> D[GOMP_parallel] D --> E[ggml_compute_forward_set_rows] E --> F[NULL dispatch -> address 0x0] D --> G[Thread 23 worker: same path] ``` Test matrix. Five ROCmFPX builds and the all-CPU path were tested on a scratch port (`/tmp/g80bench/run.sh`), `gl=45`, `ctx=131072`. All crash in `set_rows`. | Build | Binary | Version | Result | | ----- | ------ | ------- | ------ | | build-kairic | `ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server` | v81 (205a3e5f4) | segfault in set_rows | | build-promptforge | `ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` | PromptForge/ActiveFPX lineage | segfault in set_rows | | build-rocmi4-w4a4 | `rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` | v257 (c49ebdb) | segfault in set_rows | | build-strix-rocmfp4 | `GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server` | v209 (00d5452) | segfault in set_rows | | all-CPU | build-kairic, `-ngl 0` | v81 | segfault in set_rows | The crash is not GPU-specific (the all-CPU path fails too), not MTP-specific (`--spec-type none` fails), and not fixed by `LLAMA_SET_ROWS=0`. So it is a build bug: the Q8_0 AGENT GGUF graph needs a `set_rows` dispatch the ROCmFPX backends do not register. The older caf-eemin ROCmI4 and Kairic Edge models served fine on related ROCmFPX builds. The caf-eemin payload was about 14.4 GiB; the separate PromptForge/ActiveFPX deployment used a much larger multi-file payload of about 36.7 GiB. These are different deployments. The Q8_0 AGENT graph is the trigger for the `set_rows` failure. **Fix path:** rebuild charlie12345/ROCmFPX from source with a working `set_rows` for the ROCmFPX backend, or find a build where it is registered. This unblocks the model. See the context budget reality below once `set_rows` is fixed. ## Model files Three files, from the HF repo above: | File | Size | Purpose | | ---- | ---- | ------- | | `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | 28.21 GB | main weights | | `mtp-Qwen3.8-27B-Q4_0.gguf` | 1.68 GB | standalone MTP draft head (loaded by the script) | | `mmproj-Qwen3.8-27B-BF16.gguf` | 0.93 GB | vision projector (`--mmproj`) | The repo also has a `mtp-Qwen3.8-27B-Q8_0.gguf` (3.16 GB) draft; the env uses the Q4_0 draft. Download them from the resolve URLs below (they are large — use the token in `~/.cache/huggingface/token`). ## Download ```bash # on Dankclaw TOKEN=$(cat ~/.cache/huggingface/token) REPO=kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF DIR=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT mkdir -p "$DIR"; cd "$DIR" dl(){ curl -fsSL -C - -o "$1" --retry 5 --retry-delay 3 --retry-max-time 3600 \ -H "Authorization: Bearer $TOKEN" \ "https://huggingface.co/$REPO/resolve/main/$1"; echo "done: $1"; } dl Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf dl mmproj-Qwen3.8-27B-BF16.gguf dl mtp-Qwen3.8-27B-Q4_0.gguf ``` The serve script checks each artifact is present and readable before launch (`scripts/frank-default-large.env` -> the four `MODEL_*` files plus the chat template). A missing file fails the artifact check with `missing or unreadable required artifact`. ## Serving profile `scripts/serve-frank-large.sh` builds the `llama-server` call from the env: ``` /dev/llama-server -m
--alias frank-default-large --host 0.0.0.0 --port 7770 --jinja -dev ROCm0 -ngl 999 -np 1 -c 262144 -b 512 -ub 512 -fa on -ctk f16 -ctv turbo4 -t 16 -tb 32 -ctxcp 32 --cache-ram 8192 --cache-prompt --cache-idle-slots --metrics --spec-type draft-mtp --model-draft --mmproj --spec-draft-device ROCm0 --spec-draft-ngl 999 --spec-draft-type-k f16 --spec-draft-type-v turbo4 --spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0 --spec-draft-p-split 0.10 --spec-draft-backend-sampling --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --reasoning on --reasoning-format none --reasoning-budget -1 ``` Key choices: - **Standalone MTP draft** (`--model-draft`, `--spec-type draft-mtp`). The AGENT GGUF has no embedded draft head; the draft head is a separate file on ROCm0. This matches the Qwen3.8-27B AGENT shape (`--spec-draft-type-k f16 --spec-draft-type-v turbo4`). - **f16 K + TurboQuant V** cache. f16 K is the safe long-ctx default; TurboQuant halves the V cache at 262k while keeping quality (see the ubatch/threads-batch notes in [[qwen3.8-strix-optimizations]]). - **262144 context, 1 slot — the target.** The whole context KV cache plus the ~30 GB payload (main + draft + mmproj) are meant to fit in the 114 GiB budget. Measured reality: `gl=999` (all layers on ROCm0) at 262144 **OOMs** on device-local (`cudaMalloc failed`, the KV cache needs ~10-18 GiB that does not fit once the payload is on the card). `gl=45` (partial offload) fits a 131072 context. The env ships `CTX_SIZE=262144`; the proven serving context today is **131072 at `gl=45`** (before the `set_rows` fix). `deploy.sh` waits up to 20 minutes (see `LLM_WAIT`). - **Reasoning on.** The Sharp template drives thinking (medium effort); `--reasoning-format none` lets the template project the reasoning content into `reasoning_content`. `--reasoning-budget -1` = no budget cap. ## Reasoning + chat template The Sharp template (`scripts/chat-template.jinja`, repo-tracked) is a Qwen-AGENT shape. Env `CHAT_TEMPLATE_KWARGS` keeps the reasoning trail in the conversation: ```json {"enable_thinking": true, "preserve_thinking": true, "preserve_reasoning": true} ``` `preserve_thinking: true` keeps the model's prior thinking in the next prompt (the user's "preserve thinking" request). `preserve_reasoning: true` keeps the model's prior reasoning trace across requests too (OpenWebUI sends `preserve_reasoning`/`preserve_thinking: true` when the conversation needs the full trail). For multi-turn reasoning, the template stays in the reasoning shape and `enable_thinking: true` per request. ## Memory: the 114 GiB TTM/GTT requirement The 262k f16 KV cache is large (~a few GB) and the payload is ~30 GB. Both live in the Strix Halo unified-memory GTT. The TTM `pages_limit` was raised to 114 GiB on 2026-08-25 (written live to `/sys/module/ttm/parameters/pages_limit`, then persisted in `/etc/modprobe.d/ttm.conf`). Without the raise the effective ceiling is ~15 GiB and a 28 GB + 262k-KV load stalls. The GTT page-limit change is per the `qwen3.8-strix-optimizations` table item 5. ## Health + troubleshooting - Health: `curl http://192.168.0.143:7770/health` → `{"status":"ok"}`. - `/api/models` reports `n_ctx:262144, n_ctx_train:262144, n_params:27.3B`. - Model load is slow (a few minutes cold for the ~30 GB payload with a fresh KV cache). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes (see `LLM_WAIT`). - Kernel spam about `amdgpu: SVM mapping failed` is the known documented symptom, not root cause (see the crash-loop procedure in the project AGENTS.md). Investigate other causes before assuming memory pressure. - A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop logs only `slot release` with `truncated = 0`. ## Env tunables `scripts/frank-default-large.env`. Serving: `CTX_SIZE=262144`, `PARALLEL=1`, `BATCH_SIZE=512`, `UBATCH_SIZE=512`, `NUM_THREADS=16`, `THREAD_BATCH=32`, `CACHE_TYPE_K=f16`, `CACHE_TYPE_V=turbo4`, `CTX_CHECKPOINTS=32`, `CACHE_RAM=8192`. Reasoning: `REASONING=on`, `REASONING_FORMAT=none`. Sampling: `TEMPERATURE=0.7`, `TOP_K=20`, `TOP_P=0.8`. MTP: `MTP_N_MAX=4`, `MTP_N_MIN=0`, `MTP_P_MIN=0.0`, `MTP_P_SPLIT=0.10`. Strix Halo: `HSA_OVERRIDE_GFX_VERSION=11.5.1`, `GGML_HIP_ENABLE_UNIFIED_MEMORY=1`. **Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`** — the ROCm HIP runtime reads those bare vars (see the 2026-08-13 TEMP incident). llama.cpp reads `--host/--port/--temp` as CLI flags, never as env vars. ## History - **2026-08-25 (today):** switched the large service to the kingjones Q8_0-AGENT model with 262144 context and reasoning on. Downloaded the three payload files from the HF repo, raised the TTM/GTT to 114 GiB for the full context, and persisted the env/serve script in this repo. Measured that the model loads but crashes in `ggml_compute_forward_set_rows` on all five available ROCmFPX builds; serving is BLOCKED pending a rebuild. The config goals (GTT 114 GiB, reasoning on, preserve_thinking/preserve_reasoning on, 262144 env context) are in place. Proven serving context today is 131072 at `gl=45`. See the `set_rows` build bug section above. - Preceding model: Kairic Edge Qwen3.8-27B IU4 (deployed 2026-08-25 ~05:42, retired the same day for the AGENT model). See [[kairic-edge]].