frank-default-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT

Status: Deployed on Dankclaw (since 2026-08-25). Serving is BLOCKED by the set_rows build bug — the model loads but crashes in ggml_compute_forward_set_rows before it can serve. All config goals are met; a build fix is required for the model to run.

Alias: frank-default-large, port 7770.

The large service serves the kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT GGUF from /home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/ with a charlie12345/ROCmFPX llama-server (build build-kairic). It runs a standalone MTP draft head (loaded separately, on ROCm0), TurboQuant V KV cache, the Sharp chat template with reasoning on, and a full 262144 context / 1 slot profile. The whole payload plus the 262k KV cache fit in the lifted 114 GiB TTM/GTT budget (see the 114 GiB change in Dankclaw Improvements and qwen3.8-strix-optimizations).

Figure	Value
Model	kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT (`Q8_0_ROCMFPX`)
HF repo	`kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF` (public)
Runtime	charlie12345/ROCmFPX `llama-server`, `build-kairic`
Draft head	standalone `mtp-Qwen3.8-27B-Q4_0.gguf` on ROCm0
KV cache	f16 K, TurboQuant (`turbo4`) V, full 262144 context, 1 slot
Reasoning	on, `deepseek`, budget `-1`, `--reasoning-format none`
Chat template	Sharp (repo-tracked), `enable_thinking` + `preserve_thinking` on
Host path	`/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/`
Tunables	`scripts/frank-default-large.env`
Serve script	`scripts/serve-frank-large.sh`

Serving status: the `set_rows` build bug (blocks serving)

As of 2026-08-25 the Q8_0 AGENT model will not serve on Dankclaw. It loads, reaches initializing slots, n_slots = 1, but never prints start serving. Health stays 503 until the process exits. The crash is in ggml_compute_forward_set_rows in libggml-cpu.so.0, a NULL function-pointer jump (address 0x00000000).

The backtrace (gdb). The main thread and one OpenMP worker crash in the same path:

graph TD
    A[llama_context::graph_compute] --> B[ggml_backend_cpu_graph_compute]
    B --> C[ggml_graph_compute]
    C --> D[GOMP_parallel]
    D --> E[ggml_compute_forward_set_rows]
    E --> F[NULL dispatch -> address 0x0]
    D --> G[Thread 23 worker: same path]

Test matrix. Five ROCmFPX builds and the all-CPU path were tested on a scratch port (/tmp/g80bench/run.sh), gl=45, ctx=131072. All crash in set_rows.

Build	Binary	Version	Result
build-kairic	`ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server`	v81 (205a3e5f4)	segfault in set_rows
build-promptforge	`ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`	PromptForge/ActiveFPX lineage	segfault in set_rows
build-rocmi4-w4a4	`rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server`	v257 (c49ebdb)	segfault in set_rows
build-strix-rocmfp4	`GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server`	v209 (00d5452)	segfault in set_rows
all-CPU	build-kairic, `-ngl 0`	v81	segfault in set_rows

The crash is not GPU-specific (the all-CPU path fails too), not MTP-specific (--spec-type none fails), and not fixed by LLAMA_SET_ROWS=0. So it is a build bug: the Q8_0 AGENT GGUF graph needs a set_rows dispatch the ROCmFPX backends do not register. The older caf-eemin ROCmI4 and Kairic Edge models served fine on related ROCmFPX builds. The caf-eemin payload was about 14.4 GiB; the separate PromptForge/ActiveFPX deployment used a much larger multi-file payload of about 36.7 GiB. These are different deployments. The Q8_0 AGENT graph is the trigger for the set_rows failure.

Fix path: rebuild charlie12345/ROCmFPX from source with a working set_rows for the ROCmFPX backend, or find a build where it is registered. This unblocks the model. See the context budget reality below once set_rows is fixed.

Model files

Three files, from the HF repo above:

File	Size	Purpose
`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`	28.21 GB	main weights
`mtp-Qwen3.8-27B-Q4_0.gguf`	1.68 GB	standalone MTP draft head (loaded by the script)
`mmproj-Qwen3.8-27B-BF16.gguf`	0.93 GB	vision projector (`--mmproj`)

The repo also has a mtp-Qwen3.8-27B-Q8_0.gguf (3.16 GB) draft; the env uses the Q4_0 draft. Download them from the resolve URLs below (they are large — use the token in ~/.cache/huggingface/token).

Download

# on Dankclaw
TOKEN=$(cat ~/.cache/huggingface/token)
REPO=kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF
DIR=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT
mkdir -p "$DIR"; cd "$DIR"
dl(){ curl -fsSL -C - -o "$1" --retry 5 --retry-delay 3 --retry-max-time 3600 \
  -H "Authorization: Bearer $TOKEN" \
  "https://huggingface.co/$REPO/resolve/main/$1"; echo "done: $1"; }
dl Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
dl mmproj-Qwen3.8-27B-BF16.gguf
dl mtp-Qwen3.8-27B-Q4_0.gguf

The serve script checks each artifact is present and readable before launch (scripts/frank-default-large.env -> the four MODEL_* files plus the chat template). A missing file fails the artifact check with missing or unreadable required artifact.

Serving profile

scripts/serve-frank-large.sh builds the llama-server call from the env:

/dev/llama-server -m <main> --alias frank-default-large
  --host 0.0.0.0 --port 7770 --jinja
  -dev ROCm0 -ngl 999 -np 1 -c 262144
  -b 512 -ub 512 -fa on
  -ctk f16 -ctv turbo4 -t 16 -tb 32
  -ctxcp 32 --cache-ram 8192 --cache-prompt --cache-idle-slots --metrics
  --spec-type draft-mtp --model-draft <mtp-Q4_0>
  --mmproj <mmproj-BF16>
  --spec-draft-device ROCm0 --spec-draft-ngl 999
  --spec-draft-type-k f16 --spec-draft-type-v turbo4
  --spec-draft-n-max 4 --spec-draft-n-min 0
  --spec-draft-p-min 0.0 --spec-draft-p-split 0.10
  --spec-draft-backend-sampling
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0
  --presence-penalty 1.5
  --reasoning on --reasoning-format none --reasoning-budget -1

Key choices:

Standalone MTP draft (--model-draft, --spec-type draft-mtp). The AGENT GGUF has no embedded draft head; the draft head is a separate file on ROCm0. This matches the Qwen3.8-27B AGENT shape (--spec-draft-type-k f16 --spec-draft-type-v turbo4).
f16 K + TurboQuant V cache. f16 K is the safe long-ctx default; TurboQuant halves the V cache at 262k while keeping quality (see the ubatch/threads-batch notes in qwen3.8-strix-optimizations).
262144 context, 1 slot — the target. The whole context KV cache plus the ~30 GB payload (main + draft + mmproj) are meant to fit in the 114 GiB budget. Measured reality: gl=999 (all layers on ROCm0) at 262144 OOMs on device-local (cudaMalloc failed, the KV cache needs ~10-18 GiB that does not fit once the payload is on the card). gl=45 (partial offload) fits a 131072 context. The env ships CTX_SIZE=262144; the proven serving context today is 131072 at gl=45 (before the set_rows fix). deploy.sh waits up to 20 minutes (see LLM_WAIT).
Reasoning on. The Sharp template drives thinking (medium effort); --reasoning-format none lets the template project the reasoning content into reasoning_content. --reasoning-budget -1 = no budget cap.

Reasoning + chat template

The Sharp template (scripts/chat-template.jinja, repo-tracked) is a Qwen-AGENT shape. Env CHAT_TEMPLATE_KWARGS keeps the reasoning trail in the conversation:

{"enable_thinking": true, "preserve_thinking": true, "preserve_reasoning": true}

preserve_thinking: true keeps the model's prior thinking in the next prompt (the user's "preserve thinking" request). preserve_reasoning: true keeps the model's prior reasoning trace across requests too (OpenWebUI sends preserve_reasoning/preserve_thinking: true when the conversation needs the full trail). For multi-turn reasoning, the template stays in the reasoning shape and enable_thinking: true per request.

Memory: the 114 GiB TTM/GTT requirement

The 262k f16 KV cache is large (~a few GB) and the payload is ~30 GB. Both live in the Strix Halo unified-memory GTT. The TTM pages_limit was raised to 114 GiB on 2026-08-25 (written live to /sys/module/ttm/parameters/pages_limit, then persisted in /etc/modprobe.d/ttm.conf). Without the raise the effective ceiling is ~15 GiB and a 28 GB + 262k-KV load stalls. The GTT page-limit change is per the qwen3.8-strix-optimizations table item 5.

Health + troubleshooting

Health: curl http://192.168.0.143:7770/health → {"status":"ok"}.
/api/models reports n_ctx:262144, n_ctx_train:262144, n_params:27.3B.
Model load is slow (a few minutes cold for the ~30 GB payload with a fresh KV cache). Health returns 503 while loading. Do not kill the process. deploy.sh waits up to 20 minutes (see LLM_WAIT).
Kernel spam about amdgpu: SVM mapping failed is the known documented symptom, not root cause (see the crash-loop procedure in the project AGENTS.md). Investigate other causes before assuming memory pressure.
A client abort logs W srv stop: cancel task, id_task = N; a natural stop logs only slot release with truncated = 0.

Env tunables

scripts/frank-default-large.env. Serving: CTX_SIZE=262144, PARALLEL=1, BATCH_SIZE=512, UBATCH_SIZE=512, NUM_THREADS=16, THREAD_BATCH=32, CACHE_TYPE_K=f16, CACHE_TYPE_V=turbo4, CTX_CHECKPOINTS=32, CACHE_RAM=8192. Reasoning: REASONING=on, REASONING_FORMAT=none. Sampling: TEMPERATURE=0.7, TOP_K=20, TOP_P=0.8. MTP: MTP_N_MAX=4, MTP_N_MIN=0, MTP_P_MIN=0.0, MTP_P_SPLIT=0.10. Strix Halo: HSA_OVERRIDE_GFX_VERSION=11.5.1, GGML_HIP_ENABLE_UNIFIED_MEMORY=1.

Never name a tunable TEMP, HOST, or GPU_MAX_HEAP_SIZE — the ROCm HIP runtime reads those bare vars (see the 2026-08-13 TEMP incident). llama.cpp reads --host/--port/--temp as CLI flags, never as env vars.

History

2026-08-25 (today): switched the large service to the kingjones Q8_0-AGENT model with 262144 context and reasoning on. Downloaded the three payload files from the HF repo, raised the TTM/GTT to 114 GiB for the full context, and persisted the env/serve script in this repo. Measured that the model loads but crashes in ggml_compute_forward_set_rows on all five available ROCmFPX builds; serving is BLOCKED pending a rebuild. The config goals (GTT 114 GiB, reasoning on, preserve_thinking/preserve_reasoning on, 262144 env context) are in place. Proven serving context today is 131072 at gl=45. See the set_rows build bug section above.
Preceding model: Kairic Edge Qwen3.8-27B IU4 (deployed 2026-08-25 ~05:42, retired the same day for the AGENT model). See kairic-edge.

11 KiB Raw Blame History