Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
233 lines
13 KiB
Markdown
233 lines
13 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-dragoon on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (retired)"
|
||
description: "The retired frank-dragoon runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head, f16 KV, and the BF16 projector. Measured MTP-vs-none A/B, the client contract, and the rollback state as of 2026-10-03."
|
||
tags:
|
||
- llm
|
||
- frank-dragoon
|
||
- dankclaw
|
||
- llama-cpp
|
||
- rocmfpx
|
||
- gfx1151
|
||
- gguf
|
||
- spec-decode
|
||
- mtp
|
||
timestamp: 2026-10-02
|
||
---
|
||
|
||
# frank-dragoon — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151, retired)
|
||
|
||
> **RETIRED 2026-10-03** — superseded by
|
||
> [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]]: Frank swapped to the
|
||
> uncensored Q6_0_ROCMFPX_AGENT file (protected heads, one rung down), which
|
||
> measures **27.6–28.1 t/s** at a deeper prompt than this file's 17.0. This
|
||
> runbook stays live as the rollback reference and the trap catalogue; the
|
||
> payload it documents remains on disk at
|
||
> `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`.
|
||
>
|
||
> Cut over 2026-10-02 from [[frank-dragoon-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s)
|
||
> on two requests from Frank: *highest quality that runs*, and *one full
|
||
> context*. It delivered 8.39 bpw weights — two quant rungs above what gufo
|
||
> could load — at 17.0 t/s, so quality was not bought with speed.
|
||
>
|
||
> The capability that forced this whole shape is a GGUF tensor type:
|
||
> **`GGML_TYPE_Q8_0_ROCMFPX` (103)**. It exists in exactly one loader on this
|
||
> host, so the runtime was decided by the file, not by preference.
|
||
|
||
## Identity
|
||
|
||
| | |
|
||
|---|---|
|
||
| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
|
||
| Endpoint | `http://192.168.0.143:7770` — alias `frank-dragoon`, `/v1` OpenAI surface |
|
||
| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) → `/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server` |
|
||
| Unit | root systemd `frank-dragoon.service` ([`scripts/frank-dragoon.service`](../../scripts/frank-dragoon.service)), `User=dank`, `Restart=always` |
|
||
| Serve script | [`scripts/serve-frank-dragoon.sh`](../../scripts/serve-frank-dragoon.sh) (owns the argv), tunables [`scripts/frank-dragoon.env`](../../scripts/frank-dragoon.env) |
|
||
| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` — 28,214,425,952 B, sha256 `f3ef2bdbf56e244f…` |
|
||
| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764cff8c4f3…` |
|
||
| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B, sha256 `de2a4986…` |
|
||
| Fetch | [`scripts/download-frank-dragoon-kingjones.sh`](../../scripts/download-frank-dragoon-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
|
||
| Seam | `tests/frank_dragoon_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
|
||
|
||
## Why this file, and why only this runtime
|
||
|
||
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` ships several rungs. Parsed
|
||
headers of the two candidates:
|
||
|
||
| File | ftype | Tensor mix | Output-head tensors at plain Q8_0 | Card MTP acceptance |
|
||
|---|---|---|---|---|
|
||
| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | **115** | 353 F32 + 187 Q8_0 + **311 type-103** | **187** | 0.953 |
|
||
| `Qwen3.8-27B-Q8_0_ROCMFPX.gguf` | 111 | 2 Q8_0 + type-103 | 2 | 0.911 |
|
||
|
||
Two consequences, both measured off the header rather than guessed:
|
||
|
||
1. **Quality.** The AGENT routing variant keeps the output-head (and other
|
||
routing-sensitive) tensors at plain Q8_0 instead of folding them into the
|
||
ROCmFPX kernel layout — 187 vs 2 — so the logits the sampler reads are the
|
||
highest-precision tensors in the file. It is also the card's MTP-tuned
|
||
variant, so it is the *faster* drafter base at the same time.
|
||
2. **The runtime is forced.** Type 103 is a private extension: stock llama.cpp
|
||
and gufo both fail at load (`unknown tensor type`). Verified on the host
|
||
before deploying — the bake-off was not a horse race, it was a lock-in
|
||
check.
|
||
|
||
**The build identity matters as much as the fork.** The 2026-08-25 ROCmFPX
|
||
builds read this file and then died in `ggml_compute_forward_set_rows` — the
|
||
exact failure documented in [[kingjones-q8agent-rocmfpx]] when this payload was
|
||
first attempted on Dankclaw and routed to Dankserver as
|
||
[[frank-medium-rocmfpx]] instead. `scripts/build-rocmfpx.sh` therefore checks
|
||
the commit out **by SHA** and refuses to build anything else; the pin is
|
||
load-bearing, not tidiness.
|
||
|
||
## Shape and budget
|
||
|
||
One session, the model's full native window, unquantized KV:
|
||
|
||
| Dial | Value | Why |
|
||
|---|---|---|
|
||
| `--ctx-size` | 262,144 | "one full context" — the native window, no rope scaling |
|
||
| `--parallel` | 1 | one conversation, all memory to it |
|
||
| KV dtype | **f16** (both K and V) | ~64 KiB/token → ~16 GiB; free inside a 114 GiB aperture, so there is no reason to pay a quant |
|
||
| `--fit off` | on | the auto-fitter would silently resize the window we asked for |
|
||
| `--flash-attn on` | on | required for the long-window attention path to fit |
|
||
| `--direct-io` | on | bypasses page cache for the 26 GiB weight stream (mmap is disabled by it — deliberate) |
|
||
| `--ctx-checkpoints 16` | 16 | slot state for checkpoint/rollback |
|
||
| `--cache-ram 8192` + `--cache-idle-slots` | 8 GiB | prompt cache; verified a repeat pass over the same 6k prompt replays in **0.5 s** |
|
||
| `--jinja` | on | without it `chat_template_kwargs` are silently ignored |
|
||
| `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | thinking in its own channel, unbounded |
|
||
|
||
Measured steady state (production, one model copy resident):
|
||
`mem_info_gtt_used` **63.1 GiB** of the ~114 GiB aperture, `mem_info_vram_used`
|
||
0.17 / 2.0 GiB (the iGPU's dedicated carve-out is a stub; everything real lives
|
||
in GTT). llama-server RSS 2.1 GiB. Boot to `/health` 200: **16 s** — the fastest
|
||
endpoint bring-up in the fleet, because there is no engine compile step.
|
||
|
||
## Speculative decode — measured, not assumed
|
||
|
||
```
|
||
--spec-type draft-mtp --model-draft <mtp-Qwen3.8-27B-Q4_0.gguf>
|
||
--spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0
|
||
--spec-draft-device ROCm0 --spec-draft-ngl 99
|
||
```
|
||
|
||
MTP in llama.cpp is a property of the file, not of the flags: the head is a
|
||
trained tensor set (`nextn_predict_layers = 1` in this repo's head, arch
|
||
`qwen35`, 65 blocks) and the draft context runs on the main model's weights, so
|
||
there is no second 26 GiB copy in memory. See the fleet law in
|
||
`AGENTS.md → llama.cpp serving`.
|
||
|
||
**A/B, 2026-10-02** — same session, temp 0, 2,202-token prompt, 512 tokens,
|
||
`cache_prompt: false`, `frank-dragoon.service` stopped for every scratch arm (the
|
||
benchmarking law), baseline arm stripped of `--model-draft` **and** every
|
||
`--spec-*` flag:
|
||
|
||
| Arm | Decode t/s (2 runs) | Prefill t/s | Draft acceptance |
|
||
|---|---|---|---|
|
||
| **MTP, n-max 4 (deployed)** | **17.00 / 17.14** | 350.6 / 324.9 | 0.462 (mean acceptance length ≈ 1.86) |
|
||
| no draft | 7.60 / 7.66 | 381.8 / 381.0 | — |
|
||
|
||
**+123 % decode for an 8–15 % prefill tax.** Notes that matter before someone
|
||
re-litigates this:
|
||
|
||
- Acceptance here (0.462) is nowhere near the card's 0.953. The card measures
|
||
its own short, repetitive text; this is a 2.2k-token prose prompt. Treat the
|
||
card's number as a file property, not a throughput promise.
|
||
- `n-max 4` is where the gain was measured; 2 and 8 are **unmeasured**. The
|
||
gufo-era law applies in spirit — never rank spec settings on a
|
||
self-predictable corpus.
|
||
- End-to-end on the chat path is slower than the `/completion` number because
|
||
thinking tokens are included: a 2,253-token chat prompt generating 512
|
||
tokens took 43.8 s wall (11.7 t/s end-to-end) — that is the cost of an
|
||
unbounded reasoning budget, not a regression in decoding.
|
||
|
||
## Client contract (live-verified 2026-10-02)
|
||
|
||
| Behavior | Observation |
|
||
|---|---|
|
||
| Model id | `/v1/models` → `frank-dragoon` |
|
||
| Tools | native: `get_weather` → `{"city": "Seattle"}` in `tool_calls` |
|
||
| Thinking | in **`reasoning_content`** (llama.cpp field — *not* vLLM's `reasoning`); `models.yml` sets `reasoningContentField` accordingly |
|
||
| `reasoning_effort` | accepts **only** `xhigh`, `medium`, `low`; anything else is HTTP 500 (the template raises, it does not clamp). `models.yml` folds `high`/`max` → `xhigh` |
|
||
| Sampling | server defaults = the card's thinking recipe (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0), mirrored in the harness `extraBody` |
|
||
| Vision | the BF16 projector loads (`loaded multimodal model`); a red square answers **`Red`** (`usage: 75 prompt / 71 cached`) |
|
||
| Prompt cache | second pass over the same 6k prompt: 0.5 s wall |
|
||
|
||
Three traps, all hit for real:
|
||
|
||
1. **`content: null` when reasoning eats the budget.** With
|
||
`--reasoning-parser`-style separation, a request whose `max_tokens` is
|
||
consumed entirely by thinking returns `message.content = None`. A bench
|
||
client that indexes `content` crashes *after* measuring and loses the
|
||
numbers — fall back to `reasoning_content`, or size the budget (200 was
|
||
enough for the vision probe).
|
||
2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
|
||
appears in the log under draft resets. Self-recovering; not an outage signal.
|
||
3. **Never `pkill -f llama-server` on Dankclaw** — frank-zealot is also a
|
||
`llama-server`. The sweep key here is the payload name:
|
||
`pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
|
||
|
||
## Operations
|
||
|
||
```bash
|
||
# deploy (planner selects this endpoint only via its own unit/env/serve trio)
|
||
./scripts/deploy.sh dankclaw --service frank-dragoon
|
||
|
||
# status / logs / cause of a crash loop
|
||
systemctl status frank-dragoon
|
||
journalctl -u frank-dragoon -n 200 --no-pager
|
||
|
||
# rebuild the pinned fork (only when deliberately moving the pin)
|
||
scripts/build-rocmfpx.sh
|
||
|
||
# re-verify the payload without re-downloading
|
||
scripts/download-frank-dragoon-kingjones.sh --verify-only
|
||
```
|
||
|
||
`serve-frank-dragoon.sh` refuses to start — before touching the port — unless the
|
||
fork binary, the weights, the MTP head and (unless `FRANK_DRAGOON_MMPROJ=""`) the
|
||
projector all exist; `scripts/deploy-remote.sh` mirrors the same predicate so a
|
||
broken payload never stops the running unit. Caller environment **overrides**
|
||
`frank-dragoon.env`, and defaults use `:=` so an explicitly empty value surfaces
|
||
as the error it reads as.
|
||
|
||
## Rollback (as of retirement, 2026-10-03)
|
||
|
||
**Primary rollback: the ALIGNED Q8 file itself is the previous shape's
|
||
payload and remains on disk** at
|
||
`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the
|
||
2026-10-03 swap commit (unit/env/serve trio) or repoint
|
||
`FRANK_DRAGOON_MODEL_FILE`/`LLM_DRAGOON_MODEL_FILE`, verify with
|
||
`scripts/download-frank-dragoon-kingjones.sh --verify-only`, redeploy.
|
||
|
||
git tag **`frank-dragoon-gufo-q4kxl-20261001`** (gufo v0.1.1, Q4_K_XL +
|
||
DFlash2) is now a **re-download** rollback: the gufo payloads
|
||
`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and
|
||
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/` (incl. the DFlash2 drafter) were
|
||
removed from disk on 2026-10-03 with Frank's explicit approval to free space.
|
||
`scripts/build-gufo.sh` remains on disk; re-fetch per
|
||
[[frank-dragoon-gufo-qwen38-27b]].
|
||
|
||
The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
|
||
Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
|
||
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 —
|
||
recoverable until the trash is purged, and never part of a rollback plan.
|
||
|
||
## When it misbehaves
|
||
|
||
- **`unknown tensor type` / ftype 115 refused** — something moved off the
|
||
ROCmFPX binary. Check `readlink -f $(systemctl show frank-dragoon -p ExecStart)`
|
||
and the fork commit; this file cannot run on stock llama.cpp or gufo.
|
||
- **SIGSEGV in `ggml_compute_forward_set_rows` during load** — a fork build
|
||
older than the `36e9acd` pin; rebuild with `scripts/build-rocmfpx.sh`.
|
||
History: [[kingjones-q8agent-rocmfpx]].
|
||
- **Slow decode with no log error** — check whether the draft is engaged
|
||
(`/metrics`, `draft_n` / `draft_n_accepted`; a zero draft count means the head
|
||
did not load), and check GTT pressure with
|
||
`cat /sys/class/drm/card1/device/mem_info_gtt_used` — a second resident copy
|
||
of this model (an orphaned scratch server) drives the whole host into
|
||
eviction and halves the rate silently.
|
||
- **`500` naming Jinja** — a `chat_template_kwargs` field reached a server
|
||
without `--jinja`.
|
||
- **HTTP 500 naming the template on the first request of a role** — an
|
||
unmapped `reasoning_effort`; fix the fold in `models.yml`, not the model.
|