serverAppConfigs/serverAppConfigs-docs/LLM/frank-large-rocmfpx-q8agent-mtp.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP	The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, and the bundled BF16 projector for vision. Measured MTP-vs-none A/B, the client contract, and rollback to gufo.	llm frank-large dankclaw llama-cpp rocmfpx gfx1151 gguf spec-decode mtp	2026-10-02

frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151)

ACTIVE since 2026-10-02. Cut over from frank-large-gufo-qwen38-27b (Q4_K_XL, ~15 t/s). Both requests came from Frank: highest quality that runs, and one full context. This endpoint delivers 8.39 bpw weights — two quant rungs above what gufo could load — and measures 17.0 t/s decode instead of 15, so quality was not bought with speed.

The capability that forced this whole shape is a GGUF tensor type: GGML_TYPE_Q8_0_ROCMFPX (103). It exists in exactly one loader on this host, so the runtime was decided by the file, not by preference.

Identity

Host / device	Dankclaw (192.168.0.143), Radeon 8060S iGPU gfx1151, ~114 GiB GTT aperture
Endpoint	http://192.168.0.143:7770 — alias frank-large, /v1 OpenAI surface
Runtime	kingjones30/ROCmFPX llama.cpp fork @ 36e9acd40e10a87cd3c3ef8ec734668757dc8520, built by scripts/build-rocmfpx.sh → /home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
Unit	root systemd frank-large.service (scripts/frank-large.service), User=dank, Restart=always
Serve script	scripts/serve-frank-large.sh (owns the argv), tunables scripts/frank-large.env
Weights	/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf — 28,214,425,952 B, sha256 f3ef2bdbf56e244f…
Draft	…/mtp-Qwen3.8-27B-Q4_0.gguf — 1,680,271,648 B, sha256 051a1764cff8c4f3…
Projector	…/mmproj-Qwen3.8-27B-BF16.gguf — 931,145,888 B, sha256 de2a4986…
Fetch	scripts/download-frank-large-kingjones.sh (-f, byte-count + LFS sha256 verified, --verify-only, token never on the command line)
Seam	tests/frank_large_unit.bats (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity)

Why this file, and why only this runtime

kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF ships several rungs. Parsed headers of the two candidates:

File	ftype	Tensor mix	Output-head tensors at plain Q8_0	Card MTP acceptance
Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf	115	353 F32 + 187 Q8_0 + 311 type-103	187	0.953
Qwen3.8-27B-Q8_0_ROCMFPX.gguf	111	2 Q8_0 + type-103	2	0.911

Two consequences, both measured off the header rather than guessed:

1. Quality. The AGENT routing variant keeps the output-head (and other routing-sensitive) tensors at plain Q8_0 instead of folding them into the ROCmFPX kernel layout — 187 vs 2 — so the logits the sampler reads are the highest-precision tensors in the file. It is also the card's MTP-tuned variant, so it is the faster drafter base at the same time.
2. The runtime is forced. Type 103 is a private extension: stock llama.cpp and gufo both fail at load (unknown tensor type). Verified on the host before deploying — the bake-off was not a horse race, it was a lock-in check.

The build identity matters as much as the fork. The 2026-08-25 ROCmFPX builds read this file and then died in ggml_compute_forward_set_rows — the exact failure documented in kingjones-q8agent-rocmfpx when this payload was first attempted on Dankclaw and routed to Dankserver as frank-medium-rocmfpx instead. scripts/build-rocmfpx.sh therefore checks the commit out by SHA and refuses to build anything else; the pin is load-bearing, not tidiness.

Shape and budget

One session, the model's full native window, unquantized KV:

Dial	Value	Why
--ctx-size	262,144	"one full context" — the native window, no rope scaling
--parallel	1	one conversation, all memory to it
KV dtype	f16 (both K and V)	~64 KiB/token → ~16 GiB; free inside a 114 GiB aperture, so there is no reason to pay a quant
--fit off	on	the auto-fitter would silently resize the window we asked for
--flash-attn on	on	required for the long-window attention path to fit
--direct-io	on	bypasses page cache for the 26 GiB weight stream (mmap is disabled by it — deliberate)
--ctx-checkpoints 16	16	slot state for checkpoint/rollback
--cache-ram 8192 + --cache-idle-slots	8 GiB	prompt cache; verified a repeat pass over the same 6k prompt replays in 0.5 s
--jinja	on	without it chat_template_kwargs are silently ignored
--reasoning on --reasoning-format deepseek --reasoning-budget -1		thinking in its own channel, unbounded

Measured steady state (production, one model copy resident): mem_info_gtt_used 63.1 GiB of the ~114 GiB aperture, mem_info_vram_used 0.17 / 2.0 GiB (the iGPU's dedicated carve-out is a stub; everything real lives in GTT). llama-server RSS 2.1 GiB. Boot to /health 200: 16 s — the fastest endpoint bring-up in the fleet, because there is no engine compile step.

Speculative decode — measured, not assumed

--spec-type draft-mtp --model-draft <mtp-Qwen3.8-27B-Q4_0.gguf>
--spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0
--spec-draft-device ROCm0 --spec-draft-ngl 99

MTP in llama.cpp is a property of the file, not of the flags: the head is a trained tensor set (nextn_predict_layers = 1 in this repo's head, arch qwen35, 65 blocks) and the draft context runs on the main model's weights, so there is no second 26 GiB copy in memory. See the fleet law in AGENTS.md → llama.cpp serving.

A/B, 2026-10-02 — same session, temp 0, 2,202-token prompt, 512 tokens, cache_prompt: false, frank-large.service stopped for every scratch arm (the benchmarking law), baseline arm stripped of --model-draft and every --spec-* flag:

Arm	Decode t/s (2 runs)	Prefill t/s	Draft acceptance
MTP, n-max 4 (deployed)	17.00 / 17.14	350.6 / 324.9	0.462 (mean acceptance length ≈ 1.86)
no draft	7.60 / 7.66	381.8 / 381.0	—

+123 % decode for an 8–15 % prefill tax. Notes that matter before someone re-litigates this:

• Acceptance here (0.462) is nowhere near the card's 0.953. The card measures its own short, repetitive text; this is a 2.2k-token prose prompt. Treat the card's number as a file property, not a throughput promise.
• n-max 4 is where the gain was measured; 2 and 8 are unmeasured. The gufo-era law applies in spirit — never rank spec settings on a self-predictable corpus.
• End-to-end on the chat path is slower than the /completion number because thinking tokens are included: a 2,253-token chat prompt generating 512 tokens took 43.8 s wall (11.7 t/s end-to-end) — that is the cost of an unbounded reasoning budget, not a regression in decoding.

Client contract (live-verified 2026-10-02)

Behavior	Observation
Model id	/v1/models → frank-large
Tools	native: get_weather → {"city": "Seattle"} in tool_calls
Thinking	in reasoning_content (llama.cpp field — not vLLM's reasoning); models.yml sets reasoningContentField accordingly
reasoning_effort	accepts only xhigh, medium, low; anything else is HTTP 500 (the template raises, it does not clamp). models.yml folds high/max → xhigh
Sampling	server defaults = the card's thinking recipe (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0), mirrored in the harness extraBody
Vision	the BF16 projector loads (loaded multimodal model); a red square answers Red (usage: 75 prompt / 71 cached)
Prompt cache	second pass over the same 6k prompt: 0.5 s wall

Three traps, all hit for real:

1. content: null when reasoning eats the budget. With --reasoning-parser-style separation, a request whose max_tokens is consumed entirely by thinking returns message.content = None. A bench client that indexes content crashes after measuring and loses the numbers — fall back to reasoning_content, or size the budget (200 was enough for the vision probe).
2. MTP boundary missing for seq_id=… ; resyncing after a non-token step appears in the log under draft resets. Self-recovering; not an outage signal.
3. Never pkill -f llama-server on Dankclaw — frank-small is also a llama-server. The sweep key here is the payload name: pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'.

Operations

# deploy (planner selects this endpoint only via its own unit/env/serve trio)
./scripts/deploy.sh dankclaw --service frank-large

# status / logs / cause of a crash loop
systemctl status frank-large
journalctl -u frank-large -n 200 --no-pager

# rebuild the pinned fork (only when deliberately moving the pin)
scripts/build-rocmfpx.sh

# re-verify the payload without re-downloading
scripts/download-frank-large-kingjones.sh --verify-only

serve-frank-large.sh refuses to start — before touching the port — unless the fork binary, the weights, the MTP head and (unless FRANK_LARGE_MMPROJ="") the projector all exist; scripts/deploy-remote.sh mirrors the same predicate so a broken payload never stops the running unit. Caller environment overrides frank-large.env, and defaults use := so an explicitly empty value surfaces as the error it reads as.

Rollback

git tag frank-large-gufo-q4kxl-20261001: gufo v0.1.1 (scripts/build-gufo.sh still on disk), the Q4_K_XL Qwen3.8-27B payload and the DFlash2 drafter all remain on the host. Restore that tag's unit/env/serve trio and redeploy — nothing here deletes anything.

Payloads retired alongside this swap (MiMo-V2.6 BF16, Qwen3.5-9B MTP source, DFlash draft dirs) are still on disk under /home/dank/models/ and are the subject of the deployment disk-pressure note, not of a rollback plan.

When it misbehaves

• unknown tensor type / ftype 115 refused — something moved off the ROCmFPX binary. Check readlink -f $(systemctl show frank-large -p ExecStart) and the fork commit; this file cannot run on stock llama.cpp or gufo.
• SIGSEGV in ggml_compute_forward_set_rows during load — a fork build older than the 36e9acd pin; rebuild with scripts/build-rocmfpx.sh. History: kingjones-q8agent-rocmfpx.
• Slow decode with no log error — check whether the draft is engaged (/metrics, draft_n / draft_n_accepted; a zero draft count means the head did not load), and check GTT pressure with cat /sys/class/drm/card1/device/mem_info_gtt_used — a second resident copy of this model (an orphaned scratch server) drives the whole host into eviction and halves the rate silently.
• 500 naming Jinja — a chat_template_kwargs field reached a server without --jinja.
• HTTP 500 naming the template on the first request of a role — an unmapped reasoning_effort; fix the fold in models.yml, not the model.