# frank-default-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT

**Status:** ACTIVE on Dankclaw (since 2026-08-25).

**Alias:** `frank-default-large`, port **7770**.

The large service serves the kingjones `Qwen3.8-27B-ROCmFPX-Q8_0-AGENT` GGUF
from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` with a
charlie12345/ROCmFPX `llama-server` (build `build-kairic`). It runs a
**standalone MTP draft head** (loaded separately, on ROCm0), **TurboQuant** V
KV cache, the **Sharp** chat template with **reasoning on**, and a **full
262144 context / 1 slot** profile. The whole payload plus the 262k KV cache
fit in the lifted **114 GiB TTM/GTT** budget (see the 114 GiB change in
[[Dankclaw Improvements]] and [[qwen3.8-strix-optimizations]]).

| Figure | Value |
| ------ | ----- |
| Model | kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT (`Q8_0_ROCMFPX`) |
| HF repo | `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF` (public) |
| Runtime | charlie12345/ROCmFPX `llama-server`, `build-kairic` |
| Draft head | standalone `mtp-Qwen3.8-27B-Q4_0.gguf` on ROCm0 |
| KV cache | f16 K, TurboQuant (`turbo4`) V, full 262144 context, 1 slot |
| Reasoning | on, `deepseek`, budget `-1`, `--reasoning-format none` |
| Chat template | Sharp (repo-tracked), `enable_thinking` + `preserve_thinking` on |
| Host path | `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` |
| Tunables | `scripts/frank-default-large.env` |
| Serve script | `scripts/serve-frank-large.sh` |

## Model files

Three files, from the HF repo above:

| File | Size | Purpose |
| ---- | ---- | ------- |
| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | 28.21 GB | main weights |
| `mtp-Qwen3.8-27B-Q4_0.gguf` | 1.68 GB | standalone MTP draft head (loaded by the script) |
| `mmproj-Qwen3.8-27B-BF16.gguf` | 0.93 GB | vision projector (`--mmproj`) |

The repo also has a `mtp-Qwen3.8-27B-Q8_0.gguf` (3.16 GB) draft; the env uses
the Q4_0 draft. Download them from the resolve URLs below (they are large —
use the token in `~/.cache/huggingface/token`).

## Download

```bash
# on Dankclaw
TOKEN=$(cat ~/.cache/huggingface/token)
REPO=kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF
DIR=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT
mkdir -p "$DIR"; cd "$DIR"
dl(){ curl -fsSL -C - -o "$1" --retry 5 --retry-delay 3 --retry-max-time 3600 \
  -H "Authorization: Bearer $TOKEN" \
  "https://huggingface.co/$REPO/resolve/main/$1"; echo "done: $1"; }
dl Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
dl mmproj-Qwen3.8-27B-BF16.gguf
dl mtp-Qwen3.8-27B-Q4_0.gguf
```

The serve script checks each artifact is present and readable before launch
(`scripts/frank-default-large.env` -> the four `MODEL_*` files plus the chat
template). A missing file fails the artifact check with `missing or
unreadable required artifact`.

## Serving profile

`scripts/serve-frank-large.sh` builds the `llama-server` call from the env:

```
/dev/llama-server -m <main> --alias frank-default-large
  --host 0.0.0.0 --port 7770 --jinja
  -dev ROCm0 -ngl 999 -np 1 -c 262144
  -b 512 -ub 512 -fa on
  -ctk f16 -ctv turbo4 -t 16 -tb 32
  -ctxcp 32 --cache-ram 8192 --cache-prompt --cache-idle-slots --metrics
  --spec-type draft-mtp --model-draft <mtp-Q4_0>
  --mmproj <mmproj-BF16>
  --spec-draft-device ROCm0 --spec-draft-ngl 999
  --spec-draft-type-k f16 --spec-draft-type-v turbo4
  --spec-draft-n-max 4 --spec-draft-n-min 0
  --spec-draft-p-min 0.0 --spec-draft-p-split 0.10
  --spec-draft-backend-sampling
  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0
  --presence-penalty 1.5
  --reasoning on --reasoning-format none --reasoning-budget -1
```

Key choices:

- **Standalone MTP draft** (`--model-draft`, `--spec-type draft-mtp`). The
  AGENT GGUF has no embedded draft head; the draft head is a separate file on
  ROCm0. This matches the Qwen3.8-27B AGENT shape (`--spec-draft-type-k
  f16 --spec-draft-type-v turbo4`).
- **f16 K + TurboQuant V** cache. f16 K is the safe long-ctx default; TurboQuant
  halves the V cache at 262k while keeping quality (see the ubatch/threads-batch
  notes in [[qwen3.8-strix-optimizations]]).
- **Full 262144 context, 1 slot.** The whole context KV cache plus the ~30 GB
  payload (main + draft + mmproj) fit in the 114 GiB budget. Concurrent
  requests queue; the server never preempts a running generation.
- **Reasoning on.** The Sharp template drives thinking (medium effort);
  `--reasoning-format none` lets the template project the reasoning content into
  `reasoning_content`. `--reasoning-budget -1` = no budget cap.

## Reasoning + chat template

The Sharp template (`scripts/chat-template.jinja`, repo-tracked) is a
Qwen-AGENT shape. Env `CHAT_TEMPLATE_KWARGS` keeps the reasoning trail in the
conversation:

```json
{"enable_thinking": true, "preserve_thinking": true, "preserve_reasoning": true}
```

`preserve_thinking: true` keeps the model's prior thinking in the next prompt
(the user's "preserve thinking" request). `preserve_reasoning: true` keeps the
model's prior reasoning trace across requests too (OpenWebUI sends
`preserve_reasoning`/`preserve_thinking: true` when the conversation needs the
full trail). For multi-turn reasoning, the template stays in the reasoning
shape and `enable_thinking: true` per request.

## Memory: the 114 GiB TTM/GTT requirement

The 262k f16 KV cache is large (~a few GB) and the payload is ~30 GB. Both live
in the Strix Halo unified-memory GTT. The TTM `pages_limit` was raised to
114 GiB on 2026-08-25 (written live to
`/sys/module/ttm/parameters/pages_limit`, then persisted in
`/etc/modprobe.d/ttm.conf`). Without the raise the effective ceiling is ~15 GiB
and a 28 GB + 262k-KV load stalls. The GTT page-limit change is per the
`qwen3.8-strix-optimizations` table item 5.

## Health + troubleshooting

- Health: `curl http://192.168.0.143:7770/health` → `{"status":"ok"}`.
- `/api/models` reports `n_ctx:262144, n_ctx_train:262144, n_params:27.3B`.
- Model load is slow (a few minutes cold for the ~30 GB payload with a fresh
  KV cache). Health returns 503 while loading. Do not kill the process.
  `deploy.sh` waits up to 20 minutes (see `LLM_WAIT`).
- Kernel spam about `amdgpu: SVM mapping failed` is the known documented
  symptom, not root cause (see the crash-loop procedure in the project
  AGENTS.md). Investigate other causes before assuming memory pressure.
- A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop
  logs only `slot release` with `truncated = 0`.

## Env tunables

`scripts/frank-default-large.env`. Serving: `CTX_SIZE=262144`,
`PARALLEL=1`, `BATCH_SIZE=512`, `UBATCH_SIZE=512`, `NUM_THREADS=16`,
`THREAD_BATCH=32`, `CACHE_TYPE_K=f16`, `CACHE_TYPE_V=turbo4`,
`CTX_CHECKPOINTS=32`, `CACHE_RAM=8192`. Reasoning: `REASONING=on`,
`REASONING_FORMAT=none`. Sampling: `TEMPERATURE=0.7`, `TOP_K=20`,
`TOP_P=0.8`. MTP: `MTP_N_MAX=4`, `MTP_N_MIN=0`, `MTP_P_MIN=0.0`,
`MTP_P_SPLIT=0.10`. Strix Halo: `HSA_OVERRIDE_GFX_VERSION=11.5.1`,
`GGML_HIP_ENABLE_UNIFIED_MEMORY=1`.

**Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`** — the ROCm HIP
runtime reads those bare vars (see the 2026-08-13 TEMP incident). llama.cpp
reads `--host/--port/--temp` as CLI flags, never as env vars.

## History

- **2026-08-25 (today):** switched the large service to the kingjones
  Q8_0-AGENT model with 262144 context and reasoning on. Downloaded the three
  payload files from the HF repo, raised the TTM/GTT to 114 GiB for the full
  context, and persisted the env/serve script in this repo.
- Preceding model: Kairic Edge Qwen3.8-27B IU4 (deployed 2026-08-25 ~05:42,
  retired the same day for the AGENT model). See [[kairic-edge]].