docs: record the 2026-10-02 swaps — frank-large on ROCmFPX, frank-small on Q5_K_M
frank-large: new runbook frank-large-rocmfpx-q8agent-mtp.md — the ftype-115 tensor-type lock-in that forced the fork, the pinned build (set_rows history), the measured MTP A/B (17.0 vs 7.6 t/s decode, acceptance 0.462), the live client contract (reasoning_content, effort fold, vision restored) and rollback. frank-small: new runbook frank-small-llama-mimo-q5km.md; q6k marked retired (rollback target). gufo runbook marked retired (rollback target). Both indexes, the fleet topology table, the vision/models.yml notes, the benchmarking law and three new Key Lessons updated.
This commit is contained in:
Frank
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-large)"
|
||||
description: "The ACTIVE frank-large runbook — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
description: "The RETIRED frank-large runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
|
||||
tags:
|
||||
- llm
|
||||
- gufo
|
||||
@@ -18,9 +18,11 @@ timestamp: 2026-09-28
|
||||
|
||||
# frank-large — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)
|
||||
|
||||
> **ACTIVE again since 2026-10-01 night** — Frank rolled frank-large back
|
||||
> after one day of [[frank-large-vllm-mimo-v26-bf16]] (battery decode
|
||||
> 6.19-9.01 t/s; every ≥6-bit vLLM quant path dead on gfx1151). The shape
|
||||
> **RETIRED 2026-10-02 — the rollback target.** Superseded by
|
||||
> [[frank-large-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with
|
||||
> MTP4, 17.0 t/s measured). This era was itself a return after one day of
|
||||
> [[frank-large-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every
|
||||
> ≥6-bit vLLM quant path dead on gfx1151). The shape
|
||||
> moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF
|
||||
> UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was
|
||||
> reclaimed), and the session grid is **one session at the full native
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-large on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP"
|
||||
description: "The active frank-large runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, and the bundled BF16 projector for vision. Measured MTP-vs-none A/B, the client contract, and rollback to gufo."
|
||||
tags:
|
||||
- llm
|
||||
- frank-large
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rocmfpx
|
||||
- gfx1151
|
||||
- gguf
|
||||
- spec-decode
|
||||
- mtp
|
||||
timestamp: 2026-10-02
|
||||
---
|
||||
|
||||
# frank-large — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151)
|
||||
|
||||
> **ACTIVE since 2026-10-02.** Cut over from
|
||||
> [[frank-large-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s). Both requests came from
|
||||
> Frank: *highest quality that runs*, and *one full context*. This endpoint
|
||||
> delivers 8.39 bpw weights — two quant rungs above what gufo could load — and
|
||||
> measures **17.0 t/s** decode instead of 15, so quality was not bought with
|
||||
> speed.
|
||||
>
|
||||
> The capability that forced this whole shape is a GGUF tensor type:
|
||||
> **`GGML_TYPE_Q8_0_ROCMFPX` (103)**. It exists in exactly one loader on this
|
||||
> host, so the runtime was decided by the file, not by preference.
|
||||
|
||||
## Identity
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
|
||||
| Endpoint | `http://192.168.0.143:7770` — alias `frank-large`, `/v1` OpenAI surface |
|
||||
| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) → `/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server` |
|
||||
| Unit | root systemd `frank-large.service` ([`scripts/frank-large.service`](../../scripts/frank-large.service)), `User=dank`, `Restart=always` |
|
||||
| Serve script | [`scripts/serve-frank-large.sh`](../../scripts/serve-frank-large.sh) (owns the argv), tunables [`scripts/frank-large.env`](../../scripts/frank-large.env) |
|
||||
| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` — 28,214,425,952 B, sha256 `f3ef2bdbf56e244f…` |
|
||||
| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764cff8c4f3…` |
|
||||
| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B, sha256 `de2a4986…` |
|
||||
| Fetch | [`scripts/download-frank-large-kingjones.sh`](../../scripts/download-frank-large-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
|
||||
| Seam | `tests/frank_large_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
|
||||
|
||||
## Why this file, and why only this runtime
|
||||
|
||||
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` ships several rungs. Parsed
|
||||
headers of the two candidates:
|
||||
|
||||
| File | ftype | Tensor mix | Output-head tensors at plain Q8_0 | Card MTP acceptance |
|
||||
|---|---|---|---|---|
|
||||
| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | **115** | 353 F32 + 187 Q8_0 + **311 type-103** | **187** | 0.953 |
|
||||
| `Qwen3.8-27B-Q8_0_ROCMFPX.gguf` | 111 | 2 Q8_0 + type-103 | 2 | 0.911 |
|
||||
|
||||
Two consequences, both measured off the header rather than guessed:
|
||||
|
||||
1. **Quality.** The AGENT routing variant keeps the output-head (and other
|
||||
routing-sensitive) tensors at plain Q8_0 instead of folding them into the
|
||||
ROCmFPX kernel layout — 187 vs 2 — so the logits the sampler reads are the
|
||||
highest-precision tensors in the file. It is also the card's MTP-tuned
|
||||
variant, so it is the *faster* drafter base at the same time.
|
||||
2. **The runtime is forced.** Type 103 is a private extension: stock llama.cpp
|
||||
and gufo both fail at load (`unknown tensor type`). Verified on the host
|
||||
before deploying — the bake-off was not a horse race, it was a lock-in
|
||||
check.
|
||||
|
||||
**The build identity matters as much as the fork.** The 2026-08-25 ROCmFPX
|
||||
builds read this file and then died in `ggml_compute_forward_set_rows` — the
|
||||
exact failure documented in [[kingjones-q8agent-rocmfpx]] when this payload was
|
||||
first attempted on Dankclaw and routed to Dankserver as
|
||||
[[frank-medium-rocmfpx]] instead. `scripts/build-rocmfpx.sh` therefore checks
|
||||
the commit out **by SHA** and refuses to build anything else; the pin is
|
||||
load-bearing, not tidiness.
|
||||
|
||||
## Shape and budget
|
||||
|
||||
One session, the model's full native window, unquantized KV:
|
||||
|
||||
| Dial | Value | Why |
|
||||
|---|---|---|
|
||||
| `--ctx-size` | 262,144 | "one full context" — the native window, no rope scaling |
|
||||
| `--parallel` | 1 | one conversation, all memory to it |
|
||||
| KV dtype | **f16** (both K and V) | ~64 KiB/token → ~16 GiB; free inside a 114 GiB aperture, so there is no reason to pay a quant |
|
||||
| `--fit off` | on | the auto-fitter would silently resize the window we asked for |
|
||||
| `--flash-attn on` | on | required for the long-window attention path to fit |
|
||||
| `--direct-io` | on | bypasses page cache for the 26 GiB weight stream (mmap is disabled by it — deliberate) |
|
||||
| `--ctx-checkpoints 16` | 16 | slot state for checkpoint/rollback |
|
||||
| `--cache-ram 8192` + `--cache-idle-slots` | 8 GiB | prompt cache; verified a repeat pass over the same 6k prompt replays in **0.5 s** |
|
||||
| `--jinja` | on | without it `chat_template_kwargs` are silently ignored |
|
||||
| `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | thinking in its own channel, unbounded |
|
||||
|
||||
Measured steady state (production, one model copy resident):
|
||||
`mem_info_gtt_used` **63.1 GiB** of the ~114 GiB aperture, `mem_info_vram_used`
|
||||
0.17 / 2.0 GiB (the iGPU's dedicated carve-out is a stub; everything real lives
|
||||
in GTT). llama-server RSS 2.1 GiB. Boot to `/health` 200: **16 s** — the fastest
|
||||
endpoint bring-up in the fleet, because there is no engine compile step.
|
||||
|
||||
## Speculative decode — measured, not assumed
|
||||
|
||||
```
|
||||
--spec-type draft-mtp --model-draft <mtp-Qwen3.8-27B-Q4_0.gguf>
|
||||
--spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0
|
||||
--spec-draft-device ROCm0 --spec-draft-ngl 99
|
||||
```
|
||||
|
||||
MTP in llama.cpp is a property of the file, not of the flags: the head is a
|
||||
trained tensor set (`nextn_predict_layers = 1` in this repo's head, arch
|
||||
`qwen35`, 65 blocks) and the draft context runs on the main model's weights, so
|
||||
there is no second 26 GiB copy in memory. See the fleet law in
|
||||
`AGENTS.md → llama.cpp serving`.
|
||||
|
||||
**A/B, 2026-10-02** — same session, temp 0, 2,202-token prompt, 512 tokens,
|
||||
`cache_prompt: false`, `frank-large.service` stopped for every scratch arm (the
|
||||
benchmarking law), baseline arm stripped of `--model-draft` **and** every
|
||||
`--spec-*` flag:
|
||||
|
||||
| Arm | Decode t/s (2 runs) | Prefill t/s | Draft acceptance |
|
||||
|---|---|---|---|
|
||||
| **MTP, n-max 4 (deployed)** | **17.00 / 17.14** | 350.6 / 324.9 | 0.462 (mean acceptance length ≈ 1.86) |
|
||||
| no draft | 7.60 / 7.66 | 381.8 / 381.0 | — |
|
||||
|
||||
**+123 % decode for an 8–15 % prefill tax.** Notes that matter before someone
|
||||
re-litigates this:
|
||||
|
||||
- Acceptance here (0.462) is nowhere near the card's 0.953. The card measures
|
||||
its own short, repetitive text; this is a 2.2k-token prose prompt. Treat the
|
||||
card's number as a file property, not a throughput promise.
|
||||
- `n-max 4` is where the gain was measured; 2 and 8 are **unmeasured**. The
|
||||
gufo-era law applies in spirit — never rank spec settings on a
|
||||
self-predictable corpus.
|
||||
- End-to-end on the chat path is slower than the `/completion` number because
|
||||
thinking tokens are included: a 2,253-token chat prompt generating 512
|
||||
tokens took 43.8 s wall (11.7 t/s end-to-end) — that is the cost of an
|
||||
unbounded reasoning budget, not a regression in decoding.
|
||||
|
||||
## Client contract (live-verified 2026-10-02)
|
||||
|
||||
| Behavior | Observation |
|
||||
|---|---|
|
||||
| Model id | `/v1/models` → `frank-large` |
|
||||
| Tools | native: `get_weather` → `{"city": "Seattle"}` in `tool_calls` |
|
||||
| Thinking | in **`reasoning_content`** (llama.cpp field — *not* vLLM's `reasoning`); `models.yml` sets `reasoningContentField` accordingly |
|
||||
| `reasoning_effort` | accepts **only** `xhigh`, `medium`, `low`; anything else is HTTP 500 (the template raises, it does not clamp). `models.yml` folds `high`/`max` → `xhigh` |
|
||||
| Sampling | server defaults = the card's thinking recipe (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0), mirrored in the harness `extraBody` |
|
||||
| Vision | the BF16 projector loads (`loaded multimodal model`); a red square answers **`Red`** (`usage: 75 prompt / 71 cached`) |
|
||||
| Prompt cache | second pass over the same 6k prompt: 0.5 s wall |
|
||||
|
||||
Three traps, all hit for real:
|
||||
|
||||
1. **`content: null` when reasoning eats the budget.** With
|
||||
`--reasoning-parser`-style separation, a request whose `max_tokens` is
|
||||
consumed entirely by thinking returns `message.content = None`. A bench
|
||||
client that indexes `content` crashes *after* measuring and loses the
|
||||
numbers — fall back to `reasoning_content`, or size the budget (200 was
|
||||
enough for the vision probe).
|
||||
2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
|
||||
appears in the log under draft resets. Self-recovering; not an outage signal.
|
||||
3. **Never `pkill -f llama-server` on Dankclaw** — frank-small is also a
|
||||
`llama-server`. The sweep key here is the payload name:
|
||||
`pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
|
||||
|
||||
## Operations
|
||||
|
||||
```bash
|
||||
# deploy (planner selects this endpoint only via its own unit/env/serve trio)
|
||||
./scripts/deploy.sh dankclaw --service frank-large
|
||||
|
||||
# status / logs / cause of a crash loop
|
||||
systemctl status frank-large
|
||||
journalctl -u frank-large -n 200 --no-pager
|
||||
|
||||
# rebuild the pinned fork (only when deliberately moving the pin)
|
||||
scripts/build-rocmfpx.sh
|
||||
|
||||
# re-verify the payload without re-downloading
|
||||
scripts/download-frank-large-kingjones.sh --verify-only
|
||||
```
|
||||
|
||||
`serve-frank-large.sh` refuses to start — before touching the port — unless the
|
||||
fork binary, the weights, the MTP head and (unless `FRANK_LARGE_MMPROJ=""`) the
|
||||
projector all exist; `scripts/deploy-remote.sh` mirrors the same predicate so a
|
||||
broken payload never stops the running unit. Caller environment **overrides**
|
||||
`frank-large.env`, and defaults use `:=` so an explicitly empty value surfaces
|
||||
as the error it reads as.
|
||||
|
||||
## Rollback
|
||||
|
||||
git tag **`frank-large-gufo-q4kxl-20261001`**: gufo v0.1.1
|
||||
(`scripts/build-gufo.sh` still on disk), the Q4_K_XL Qwen3.8-27B payload and the
|
||||
DFlash2 drafter all remain on the host. Restore that tag's unit/env/serve trio
|
||||
and redeploy — nothing here deletes anything.
|
||||
|
||||
Payloads retired alongside this swap (MiMo-V2.6 BF16, Qwen3.5-9B MTP source,
|
||||
DFlash draft dirs) are still on disk under `/home/dank/models/` and are the
|
||||
subject of the [[deployment]] disk-pressure note, not of a rollback plan.
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
- **`unknown tensor type` / ftype 115 refused** — something moved off the
|
||||
ROCmFPX binary. Check `readlink -f $(systemctl show frank-large -p ExecStart)`
|
||||
and the fork commit; this file cannot run on stock llama.cpp or gufo.
|
||||
- **SIGSEGV in `ggml_compute_forward_set_rows` during load** — a fork build
|
||||
older than the `36e9acd` pin; rebuild with `scripts/build-rocmfpx.sh`.
|
||||
History: [[kingjones-q8agent-rocmfpx]].
|
||||
- **Slow decode with no log error** — check whether the draft is engaged
|
||||
(`/metrics`, `draft_n` / `draft_n_accepted`; a zero draft count means the head
|
||||
did not load), and check GTT pressure with
|
||||
`cat /sys/class/drm/card1/device/mem_info_gtt_used` — a second resident copy
|
||||
of this model (an orphaned scratch server) drives the whole host into
|
||||
eviction and halves the rate silently.
|
||||
- **`500` naming Jinja** — a `chat_template_kwargs` field reached a server
|
||||
without `--jinja`.
|
||||
- **HTTP 500 naming the template on the first request of a role** — an
|
||||
unmapped `reasoning_effort`; fix the fold in `models.yml`, not the model.
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF"
|
||||
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the VRAM fit, measured rates, and what to check when it misbehaves."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama-cpp
|
||||
- rtx3090
|
||||
- mimo
|
||||
- gguf
|
||||
- kv-cache
|
||||
timestamp: 2026-10-02
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090)
|
||||
|
||||
> **ACTIVE since 2026-10-02.** One quant step down from
|
||||
> [[frank-small-llama-mimo-q6k]] on the same runtime, the same shape and the
|
||||
> same KV contract. Frank's order: `bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF:Q5_K_M`
|
||||
> on llama.cpp, two contexts, whatever context size fits the 3090. The answer
|
||||
> to "whatever fits" is unchanged — **2 x 262,144**, because 262,144 is the
|
||||
> model's native window and two of them already fit.
|
||||
|
||||
## Identity
|
||||
|
||||
|Field|Value|
|
||||
|---|---|
|
||||
|Endpoint|`192.168.0.143:7771`, served alias `frank-small`|
|
||||
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|
||||
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f`, at `/home/dank/models/MiMo-Q5KM/` (fetched by `scripts/download-frank-small.sh`)|
|
||||
|Unit|root `frank-small.service` (repo `scripts/frank-small.service`, `User=dank`, relinked), env `scripts/frank-small.env`, serve `scripts/serve-frank-small.sh`|
|
||||
|Repo pins|`tests/frank_small_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_small` gates in `scripts/deploy-remote.sh`|
|
||||
|
||||
## Shape and the KV contract
|
||||
|
||||
`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots, so
|
||||
each session gets `n_ctx_slot = 262144`, the model's full native window (boot
|
||||
log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
|
||||
|
||||
The KV contract is Frank's: **K unquantized (f16), V `q8_0`** —
|
||||
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
|
||||
|
||||
- **`--flash-attn on` is mandatory**: quantized V exists only on the FA path.
|
||||
Without the flag the engine falls back silently and the contract becomes a
|
||||
lie. The serve script pins the trio; the bats seam pins the script.
|
||||
- The VRAM budget: attention KV lives in the 8 full-attention layers only
|
||||
(4 KV heads × 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
|
||||
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool ≈
|
||||
**12.5 GiB**. Weights 6.40 + KV 12.5 + Gated DeltaNet state (~25 MiB/slot) +
|
||||
compute buffers/CUDA graphs ⇒ **20,164 MiB measured** against the card's
|
||||
24,576 MiB.
|
||||
|
||||
**Why the quant step did not buy a bigger window.** The model cannot be run
|
||||
past its trained 262,144 without scaling, and two of those windows are already
|
||||
the pool's ceiling. The 0.86 GiB the smaller quant frees is margin, not
|
||||
geometry. It is also not enough to un-quantize V: f16 V costs 16 KiB/token
|
||||
instead of 8.5, i.e. +3.96 GiB on this pool, which lands past the card. That
|
||||
arithmetic is a test (`the KV budget fits the 3090…`), so a future CTX_SIZE or
|
||||
quant change that breaks the fit fails the suite instead of the boot.
|
||||
|
||||
## Measured on this file (2026-10-02)
|
||||
|
||||
|Check|Result|
|
||||
|---|---|
|
||||
|Boot to healthy|~20 s, `n_slots = 2`, `n_ctx_slot = 262144`|
|
||||
|VRAM|20,164 MiB / 24,576 MiB (`nvidia-smi`)|
|
||||
|Decode, temp 0, no thinking|**100.4–101.8 tok/s** (256–512 token completions at ~900-token depth)|
|
||||
|Two concurrent requests|both answered, 256 tokens each in 3.2 s wall (slots are real)|
|
||||
|Tools|native `tool_calls` with correct arguments (`get_weather {"city":"Seattle","unit":"c"}`)|
|
||||
|Thinking|`reasoning_content` populated on a reasoning-requiring prompt|
|
||||
|
||||
## Client contract
|
||||
|
||||
- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
|
||||
the field `models.yml` reads for this endpoint.
|
||||
- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
|
||||
`enable_thinking` false, so `models.yml` ships
|
||||
`chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
|
||||
`extraBody`. A raw client that omits it gets a direct answer — measured, and
|
||||
not a fault. Trivial questions produce no thinking even with the flag on.
|
||||
- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
|
||||
- **Text-only.** This GGUF carries no vision tensors. The HF repo does ship a
|
||||
paired `mmproj-MiMo-V2.6-Distill-Qwen-9B-bf16.gguf`, so vision is one
|
||||
`--mmproj` away if it is ever wanted here (frank-primary is the fleet's
|
||||
vision endpoint today).
|
||||
|
||||
## No spec decode
|
||||
|
||||
The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
|
||||
ghost), and the cross-model DFlash draft
|
||||
(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`) is an
|
||||
optimization candidate, not a default — acceptance has to be measured on this
|
||||
traffic first. The bats seam fails if a draft flag ships unannounced.
|
||||
|
||||
## Rollback
|
||||
|
||||
|Target|How|
|
||||
|---|---|
|
||||
|Q6_K, same runtime|`scripts/frank-small.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` (7,793,710,624 B, still on disk) and the deployer's `LLM_SMALL_MODEL_FILE` default|
|
||||
|SGLang + INT8|[[frank-small-sglang-mimo-v26]] — venv and payload both remain on disk; the fetch recipe moved to `old/download-frank-small-sglang-int8.sh`|
|
||||
|
||||
## When it misbehaves
|
||||
|
||||
- Port 7771 dark: `journalctl -u frank-small -n 50`; the serve script exits 1
|
||||
naming the missing file if the binary or GGUF moved.
|
||||
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
|
||||
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
|
||||
- Requests queue behind each other: two concurrent long generations are the
|
||||
slot ceiling by contract.
|
||||
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
|
||||
of VRAM into unified memory halves the rate silently.
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF"
|
||||
description: "The active frank-small runbook — a CUDA (sm_86) build of llama.cpp master serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V), the flash-attention dependency that comes with quantized V, the VRAM fit, and what to check when it misbehaves."
|
||||
title: "frank-small on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
|
||||
description: "The retired 2026-10-01 frank-small runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
@@ -16,11 +16,14 @@ timestamp: 2026-10-01
|
||||
|
||||
# frank-small — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
|
||||
|
||||
> **ACTIVE since 2026-10-01 night.** Replaces
|
||||
> [[frank-small-sglang-mimo-v26]] (SGLang + INT8-W8A16-AutoRound safetensors,
|
||||
> same architecture). Rolled in by Frank's night order: move this endpoint to
|
||||
> the GGUF path for **asymmetric KV** — unquantized K + q8_0 V — which only
|
||||
> llama.cpp exposes per-tensor.
|
||||
> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
|
||||
> [[frank-small-llama-mimo-q5km]], the same runtime and shape one quant rung
|
||||
> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
|
||||
> target.
|
||||
>
|
||||
> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF
|
||||
> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp
|
||||
> exposes per-tensor. That contract is what the current deployment kept.
|
||||
|
||||
## Identity
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-01):
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-02):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
@@ -10,10 +10,11 @@ headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, uti
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
runs two endpoints — **gufo v0.1.1 (frank-large, port 7770: Qwen3.8-27B
|
||||
UD-Q4_K_XL + DFlash2, ONE session at 262,144, f16 KV on the gfx1151 iGPU,
|
||||
rolled back from the one-day vLLM MiMo experiment 2026-10-01 night)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
|
||||
runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
|
||||
kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115
|
||||
file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE
|
||||
session at 262,144, f16 KV, vision via the bundled projector)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
|
||||
GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
@@ -38,13 +39,13 @@ fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[gufo v0.1.1 built from source
|
||||
E[ROCmFPX llama.cpp gfx1151, pinned fork build
|
||||
frank-large, port 7770]
|
||||
F[Qwen3.8-27B UD-Q4_K_XL + DFlash2 drafter
|
||||
1 x 262144 ctx, f16 KV, greedy, adaptive draft 7]
|
||||
F[Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP-Q4_0 head
|
||||
1 x 262144 ctx, f16 KV, draft n-max 4, mmproj vision]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[MiMo-V2.6 Q6_K GGUF
|
||||
H[MiMo-V2.6 Q5_K_M GGUF
|
||||
2 x 262144 ctx, f16 K + q8_0 V, flash-attn]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
@@ -64,6 +65,19 @@ frank-small, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-large-rocmfpx-q8agent-mtp]] — **ACTIVE since 2026-10-02**
|
||||
(root systemd unit, port 7770): the pinned kingjones30/ROCmFPX fork serving
|
||||
the repo's highest-quality rung `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`
|
||||
(26.28 GiB, ftype 115 = 187 plain-Q8_0 + 311 GGML_TYPE_Q8_0_ROCMFPX
|
||||
tensors) with the bundled `mtp-Qwen3.8-27B-Q4_0.gguf` head drafting 4
|
||||
tokens, one 262,144 window, f16 KV, the BF16 projector for vision. Why the
|
||||
runtime is not interchangeable, the pinned-build set_rows history, the
|
||||
measured envelope, rollback to gufo
|
||||
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
|
||||
systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
|
||||
2 x 262,144 with K f16 + V q8_0, 100.4-101.8 tok/s decode, 20,164 MiB
|
||||
measured. The KV arithmetic, the flash-attn dependency, rollback to Q6_K
|
||||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
|
||||
@@ -213,20 +213,27 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
|
||||
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
|
||||
rollback record)
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — the ACTIVE frank-large runbook
|
||||
since 2026-10-01 (root systemd unit, port 7770): the digest-pinned
|
||||
- [[frank-large-rocmfpx-q8agent-mtp]] — the ACTIVE frank-large runbook
|
||||
since 2026-10-02 (root systemd unit, port 7770): the pinned kingjones30/
|
||||
ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
|
||||
Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF **Q8_0_ROCMFPX_AGENT** — ftype 115,
|
||||
311 tensors of GGML_TYPE_Q8_0_ROCMFPX, a format no other loader on the
|
||||
host reads — with the repo's Q4_0 MTP head drafting 4 tokens and its BF16
|
||||
projector, one session at the native 262,144 window
|
||||
- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (rollback
|
||||
target, payload and `scripts/build-gufo.sh` both on disk): gufo v0.1.1
|
||||
built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2
|
||||
drafter on the gfx1151 iGPU — the two from-source build workarounds, the
|
||||
template-hash allowlist, the measured prefill/decode trade
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (same day;
|
||||
replaced by the gufo rollback, then by
|
||||
[[frank-large-rocmfpx-q8agent-mtp]]): the digest-pinned
|
||||
kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
|
||||
4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B —
|
||||
the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
|
||||
MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap,
|
||||
the `reasoning` field change, troubleshooting
|
||||
- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-09-28 → 2026-10-01
|
||||
(replaced by [[frank-large-vllm-mimo-v26-bf16]]; rollback via tag
|
||||
frank-large-gufo-q4kxl-20261001): gufo v0.1.1 built from source serving
|
||||
Qwen3.8-27B GGUF + the z-lab DFlash2 drafter on the gfx1151 iGPU — the
|
||||
two from-source build workarounds, the client-contract delta, the
|
||||
measured prefill/decode trade
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
@@ -275,12 +282,21 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
|
||||
- [[frank-small-sglang-mimo-v26]] — the ACTIVE frank-small runbook since
|
||||
2026-10-01: SGLang in a self-contained python3.12 venv serving
|
||||
- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since
|
||||
2026-10-02: a CUDA (sm_86) build of llama.cpp master serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX
|
||||
3090, port 7771 — 2 x 262,144 native windows, the asymmetric KV contract
|
||||
(f16 K + q8_0 V) and the VRAM arithmetic behind it, ~100 tok/s decode
|
||||
- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced
|
||||
by [[frank-small-llama-mimo-q5km]], payload still on disk): the first
|
||||
llama.cpp + GGUF shape for this endpoint, Q6_K at the same
|
||||
2 x 262,144 / f16-K + q8_0-V contract, 20.9 GiB measured
|
||||
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
|
||||
SGLang in a self-contained python3.12 venv serving
|
||||
letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
|
||||
hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
|
||||
Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
|
||||
4 x 151,552 slots, bf16 mamba states, eager prefill
|
||||
2 x 262,144 slots, bf16 mamba states, eager prefill
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
|
||||
mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
||||
|
||||
Reference in New Issue
Block a user