Frank
0900551317
New runbook frank-large-halogen27b-dflash2.md; LLM + vault indices re-pointed, ROCmFPX q6agent runbook marked RETIRED (rollback target); AGENTS.md topology row, detail section, image-capability paragraph (two endpoints accept images now), bench-law prose and runbook index updated.
178 lines
9.7 KiB
Markdown
178 lines
9.7 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP"
|
||
description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Uncensored Q6_0_ROCMFPX_AGENT GGUF (ftype 114, protected output heads), with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, the BF16 projector for vision, and the measured rejection of the ngram-mod chain."
|
||
tags:
|
||
- llm
|
||
- frank-large
|
||
- dankclaw
|
||
- llama-cpp
|
||
- rocmfpx
|
||
- gfx1151
|
||
- gguf
|
||
- spec-decode
|
||
- mtp
|
||
timestamp: 2026-10-03
|
||
---
|
||
|
||
# frank-large — ROCmFPX Uncensored Q6_0_ROCMFPX_AGENT + MTP (gfx1151)
|
||
|
||
> **RETIRED 2026-10-03 → 2026-10-05 (rollback target; payload, fork build and
|
||
> build script all on disk).** Replaced by [[frank-large-halogen27b-dflash2]]
|
||
> (DFlash2, publisher 31.7 t/s mean vs the 27.6–28.1 measured here). This page
|
||
> remains the contract record for the rollback. Original banner follows.
|
||
> ACTIVE since 2026-10-03. Cut over from
|
||
> [[frank-large-rocmfpx-q8agent-mtp]] (ALIGNED Q8_0_ROCMFPX_AGENT, 17.0 t/s)
|
||
> on Frank's request: the *uncensored* variant of the same protected-heads
|
||
> family, one quant rung down. The swap turned out to be a strict upgrade on
|
||
> the axis that matters here: **decode 27.6–28.1 t/s at a 4,813-token prompt
|
||
> against 17.0 at a 2,202-token prompt** — faster with a deeper context, not a
|
||
> same-depth comparison, but the sign is not in question.
|
||
|
||
## Identity
|
||
|
||
| | |
|
||
|---|---|
|
||
| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
|
||
| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
|
||
| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) — unchanged from the Q8 era; the fork already knew type 102 |
|
||
| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
|
||
| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
|
||
| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` — 24,856,607,008 B, sha256 `e966deb3…` |
|
||
| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764…` (byte-identical to the aligned repo's head; hardlinked, not re-downloaded) |
|
||
| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B (hardlinked from the aligned payload dir; the same file the Q8 era served) |
|
||
| Fetch | [`scripts/download-frank-large-kingjones-uncensored.sh`](../../scripts/download-frank-large-kingjones-uncensored.sh) — hardlinks the draft/projector from the aligned dir when present, otherwise fetches; `-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line |
|
||
| Seam | `tests/frank_large_unit.bats` (payload gates, deployer/serve parity, spec block, KV dtype) |
|
||
|
||
## Why this file
|
||
|
||
`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` ships the same
|
||
"protected heads" construction as the aligned repo, in the uncensored tuning.
|
||
Parsed header of the deployed file:
|
||
|
||
| Field | Value |
|
||
|---|---|
|
||
| ftype | **114** — the fork's 64-bit-block family, ≈7.44 bpw |
|
||
| Tensor mix | 353 F32 + **339 type-102 (`GGML_TYPE_Q6_0_ROCMFPX`)** + **159 type-103 (`Q8_0_ROCMFPX`)** |
|
||
| Protected heads | 64 output-head tensors kept at plain precision — the AGENT construction from the aligned repo, at the 6-bit rung |
|
||
| Arch | qwen35, 64 layers, `full_attention_interval 4`, `context_length 262144` |
|
||
| Embedded sampling recipe | temp 1.0 / top_p 0.95 / top_k 20 |
|
||
|
||
The same lock-in law as the Q8 era applies: types 102 and 103 exist in no
|
||
loader but this fork; stock llama.cpp and gufo refuse the file at load
|
||
(`unknown tensor type`). The loader was not re-litigated — the file picks the
|
||
runtime, and the pin in `scripts/build-rocmfpx.sh` (commit by SHA) stays.
|
||
|
||
Rejected sibling: `Q6_0_ROCMFPX_LEAN` (6.45 bpw, no protected heads) — the
|
||
protection is what keeps the output-head logits at full precision and the
|
||
drafter accurate, so it stays deployed.
|
||
|
||
## Shape and budget
|
||
|
||
Unchanged from the Q8 era — the swap was payload-only:
|
||
|
||
| Dial | Value | Why |
|
||
|---|---|---|
|
||
| `--ctx-size` | 262,144 | the native window, no rope scaling |
|
||
| `--parallel` | 1 | one conversation, all memory to it |
|
||
| KV dtype | **f16** (both K and V) | ~64 KiB/token; free inside a 114 GiB aperture |
|
||
| `--fit off` | on | the auto-fitter would silently resize the window |
|
||
| `--flash-attn on` | on | required for the long-window attention path |
|
||
| `--direct-io`, `--ctx-checkpoints 16`, `--cache-ram 8192`, `--cache-idle-slots`, `--jinja`, `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | same as [[frank-large-rocmfpx-q8agent-mtp]] |
|
||
|
||
Spec decode: `--spec-type draft-mtp`, the repo's `mtp-Qwen3.8-27B-Q4_0.gguf`,
|
||
`--spec-draft-n-max 4`, pinned `--spec-draft-device ROCm0`.
|
||
|
||
Measured steady state: GTT **~61 GiB** under live load (host baseline 16.7),
|
||
boot to `/health` **~15 s**.
|
||
|
||
## Measured on this payload (2026-10-03, production service, `/completion` timings, temp 0, `cache_prompt: false`)
|
||
|
||
| Traffic | Decode t/s | Prefill t/s | Draft acceptance |
|
||
|---|---|---|---|
|
||
| Prose, 4,813-token prompt, 512 tokens | **27.57 / 28.10** (two runs) | ~205–217 | **396/456 = 0.868** → mean acceptance length ≈ 4.4, i.e. drafting at nearly the full n-max 4 |
|
||
| Schema echo (structured repetition) | **31.44** | — | 639/639 = 1.000 |
|
||
| Block copy (text repeated verbatim) | ~28.0 | — | 0.848 |
|
||
|
||
vs the Q8 era's 0.462 acceptance at a 2,202-token prompt: the MTP head drafts
|
||
materially better on this file. The card's claimed 0.953-class acceptance is
|
||
no longer science fiction on real prose — but keep reading it as a file
|
||
property, not a throughput promise.
|
||
|
||
## The ngram-mod chain — measured and rejected
|
||
|
||
Frank's ask: layer `--spec-type draft-mtp,ngram-mod` on top of MTP for an
|
||
extra gain. Measured in three bench windows (production service stopped per
|
||
the benchmarking law; artifacts `/home/dank/fl-ngram-ab/` on Dankclaw),
|
||
comparing `draft-mtp` alone vs the chain on the same traffic:
|
||
|
||
| Traffic | MTP alone | MTP + ngram-mod | Delta |
|
||
|---|---|---|---|
|
||
| Prose | 27.57 t/s | 23.40 t/s | **−15 %** |
|
||
| Block copy | 28.0 t/s | 26.1 t/s | **−7 %** (with ngram-mod demonstrably firing: 138 proposals/run) |
|
||
|
||
Mechanism, straight from arm B's boot log:
|
||
|
||
```
|
||
recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support
|
||
```
|
||
|
||
On the qwen35 hybrid (Gated DeltaNet recurrent layers) adding the ngram layer
|
||
disables the rollback path MTP's own verification relies on, so acceptance —
|
||
and throughput — drop on *all* traffic, even where the ngram matcher fires.
|
||
**Deployed decision: `draft-mtp` alone.** Chaining is not free on recurrent
|
||
models; the fleet law in `AGENTS.md → llama.cpp serving` records it.
|
||
|
||
## Bench-window hardening laws (learned building the A/B)
|
||
|
||
1. **Health 200 is not proof of ownership.** After boot, verify the pid you
|
||
started owns the port (`ss -ltnp`); a zombie server from a prior window
|
||
answers `/health` too and silently serves your arm.
|
||
2. **Kill scratch servers with SIGKILL after a grace period.** llama-server's
|
||
graceful SIGTERM waits on open keep-alive connections — a harness holding
|
||
the socket leaves the process (and its ~24 GiB of GTT) alive after the
|
||
script exits.
|
||
3. **Prove the port is free and GTT dropped between arms**, or the second
|
||
arm measures a collision.
|
||
|
||
## Client contract
|
||
|
||
Unchanged from the Q8 era and re-verified on this payload 2026-10-03: model id
|
||
`frank-large`, native tools, thinking in `reasoning_content`,
|
||
`reasoning_effort` ∈ {xhigh, medium, low} only (anything else HTTP 500),
|
||
card-recipe sampling defaults server-side, vision through the BF16 projector
|
||
(red PNG → **`Red`**, 74 prompt / 165 completion tokens). The traps list —
|
||
`content: null` when reasoning eats the budget, the self-recovering MTP
|
||
boundary log line, `enable_thinking:false` not `reasoning_effort:"none"` — is
|
||
in [[frank-large-rocmfpx-q8agent-mtp]] and still applies.
|
||
|
||
Sweep key for scratch servers is now the payload name:
|
||
`pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f
|
||
llama-server` (frank-small is also one).
|
||
|
||
## Operations
|
||
|
||
```bash
|
||
./scripts/deploy.sh dankclaw --service frank-large
|
||
systemctl status frank-large
|
||
journalctl -u frank-large -n 200 --no-pager
|
||
scripts/download-frank-large-kingjones-uncensored.sh --verify-only
|
||
```
|
||
|
||
## Rollback
|
||
|
||
| Target | How |
|
||
|---|---|
|
||
| **ALIGNED Q8_0_ROCMFPX_AGENT** (previous shape, 17 t/s) | the payload dir `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` is untouched on disk; revert the swap commit (unit/env/serve trio) and redeploy, or point `FRANK_LARGE_MODEL_FILE` + `LLM_LARGE_MODEL_FILE` at the old paths. Verify with `scripts/download-frank-large-kingjones.sh --verify-only` |
|
||
| gufo Q4_K_XL era | git tag `frank-large-gufo-q4kxl-20261001`, **but the gufo payloads (UD-Q4_K_XL 16.35 GiB, UD-Q8_K_XL + DFlash2 drafter) were removed from disk 2026-10-03 with Frank's approval** — this rollback now requires re-download per [[frank-large-gufo-qwen38-27b]] |
|
||
|
||
## When it misbehaves
|
||
|
||
Same decision tree as [[frank-large-rocmfpx-q8agent-mtp]]:
|
||
`unknown tensor type` ⇒ off the pinned fork; SIGSEGV in
|
||
`ggml_compute_forward_set_rows` ⇒ pre-`36e9acd` build; slow decode with no
|
||
error ⇒ draft not engaged (`draft_n` in `/completion` timings) or an orphaned
|
||
second copy eating GTT (`/sys/class/drm/card1/device/mem_info_gtt_used`);
|
||
500 naming Jinja ⇒ `--jinja` lost; 500 naming the template ⇒ unmapped
|
||
`reasoning_effort` in `models.yml`.
|