serverAppConfigs/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md

---
type: configuration
title: frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
description: ACTIVE frank-primary runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment.
tags:
  - frank-primary
  - dankserver
  - llamacpp
  - unsloth
  - qwen38-flash-next
  - q8_0
  - mtp
  - ngram-mod
  - spec-decode
  - vision
  - rocm
  - gfx1201
  - r9700
timestamp: 2026-09-26
---

# frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook

The ACTIVE dankserver LLM endpoint (:7770, alias **frank-primary**):
unsloth llama.cpp serving `unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8**
(the 6-part `Q8_0/` main, 175.30 GiB) on **both** Radeon AI PRO R9700s,
with the **shared Q_8 MTP drafter** (`draft-mtp`, depth 2) layered with
**`ngram-mod`**, and **vision** through `mmproj-BF16.gguf`. It replaces the
Lucebox DS4.1 quadlet that held the port for hours earlier the same day
([[frank-primary-lucebox-ds4]]), which itself replaced
[[frank-medium-radiance-mxfp4]].

## Why this shape

- **One runtime for the fleet.** All three endpoints are llama.cpp now —
  one metric namespace (`llamacpp:*`), one reasoning contract
  (`reasoning_content`), one tool contract (`--jinja`). The two-day
  experiment with a single-vendor engine (lucebox) ended because Qwen
  Flash-Next at Q8 on llama.cpp delivers better quality than the DS4.1
  fp2/fp3 stream, plus vision, plus the full 262k window, at comparable
  speed.
- **The placement law, not a preference:** the routed experts and the
  51B-param n-gram (PLE) embedding table are **computed in host RAM**
  (`-ot exps=CPU,per_layer_token_embd.weight=CPU`) and never streamed over PCIe. The
  x4-pair measurement on the same model shape: streaming loses (3.7 t/s),
  computing in RAM does not care (20+ t/s). The dense weights, attention,
  projector and MTP head sit on the two cards (~29 GiB); the ~100 GiB of
  expert pages live in mmap page cache — `--no-mmap` is forbidden (it
  forces the table into anonymous memory; the cgroup kill is silent) and
  pinning the PLE table with a GPU target cost 23 → 13.4 t/s when tried.
  The tensor names are verified, not guessed: the GGUF v3 header dump
  (`dump-gguf.py`, run on the fetched parts 2026-09-26) shows the PLE
  table is one tensor — `per_layer_token_embd.weight`, dims
  `[160, 320001536]`, Q8_0, 54.4 GiB, exactly the whole of part 3 — and
  the model is `qwen4exp` with 512 experts top-10
  (`qwen4exp.ple.ngram_size=3`, `heads_per_ngram=8`). The same tensor is
  the MTP head's borrow target, so the override is required even if the
  placement were optional.
- **Two cards, layer split** (`-ts 1,1`): both R9700s are on CPU root
  ports with full PCIe atomics (the ProArt B850-Creator swap,
  2026-09-20), but this deployment uses llama.cpp's layer split —
  `-ot` tensor overrides do not compose with tensor-parallel mode, and
  `exps=CPU` is the override the placement law needs. The 27B-dense
  equivalent on this host measured 1991 t/s prefill / 18.8 t/s decode in
  this shape; Flash-Next Q8 with CPU experts lands in the same class
  (measured numbers below).
- **Spec chain: `draft-mtp,ngram-mod`.** MTP is a property of the FILE:
  the shared-Q8_0 head (`mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf`,
  2,786,568,256 B) is loaded via `--model-draft` and borrows the target's
  embedding tensors — which is why `per_layer_token_embd.weight` must be
  CPU-resident in the override set (a GPU-resident borrow crashes the
  draft load with `invalid vector subscript`). `ngram-mod` is draftless,
  so it chains legally behind the drafter. Depth 2: each longer
  verification round re-batches the CPU expert reads, so depth buys less
  than bandwidth here. **The drafter must sit on ROCm1** — the head also
  borrows the target's `output.weight`, which a `-ts 1,1` split places on
  the last-layers card; with tensor overrides set the draft scheduler runs
  `n_copies==1`, and a draft context elsewhere aborts at load
  (`ggml-backend.cpp:941`, "pre-allocated tensor (output.weight) in a
  buffer (ROCm1) that cannot run the operation"). The 2026-09-26 cutover
  crash-looped 4× on `ROCm0` before the probe confirmed `ROCm1` loads —
  the frank-medium precedent (drafter on the last-layers card) was right.
- **Vision, explicitly.** This build never autoloads projectors — without
  `--mmproj` the server comes up healthy and text-only and answers image
  requests with 500 (the frank-medium 2026-09-17 trap). The BF16 projector
  (907,542,944 B) is named in the serve script.

## Deployment

|Piece|Value|
|---|---|
|Unit|`scripts/frank-primary.service` — root systemd unit, `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity`, relinked to `/etc/systemd/system/`|
|Serve script|`scripts/serve-frank-primary.sh` — validates binary + all 6 parts + drafter + mmproj before exec|
|Env tunables|`scripts/frank-primary.env` — paths and numbers only; bind is `BIND_HOST` (never a bare `HOST`: HIP reads it)|
|Runtime|`/home/dank/GitPackages/unsloth-llama-b11030/llama-server` — unsloth b11030-mix prebuilt (commit 6ba30d05b), ROCm gfx120X, bundled runtime via `LD_LIBRARY_PATH`|
|Payload|`/home/dank/models/Qwen3.8-Flash-Next-GGUF/` — `Q8_0/` (6 parts), `MTP/mtp-...-shared-Q8_0.gguf`, `mmproj-BF16.gguf`; fetched by `scripts/download-frank-primary.sh` (sha256 + byte-count verified)|
|Port / alias|7770 / `frank-primary` — unchanged from the lucebox and radiance eras; no client moves|
|Context|262,144 × 1 slot (`PARALLEL_SLOTS=1`, one stream, the model's full native window)|
|Sampling|Greedy server default (deterministic e2e); Qwen-recommended sampling rides `models.yml` `compat.extraBody` per request|
|Template|The GGUF's embedded template via `--jinja`; reasoning → `reasoning_content` (`--reasoning-format deepseek`)|
|Observability|`frank-primary` Prometheus job at `192.168.0.235:7770` — cross-host LAN scrape (the netavark published-port quirk that forces `host.containers.internal` is same-host only); exports `llamacpp:*`|

### Cutover mechanics

The deployer (`deploy_llm_primary` in `scripts/deploy-remote.sh`) gates on
every artifact at its declared byte count (`stat -c %s`, never `wc -c` —
which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash
(`~/.AGENTTRASH/20260926-llamacpp-cutover/`), waits for port 7770 to be
truly free, relinks + enables + restarts the root unit, and waits up to
1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs).

### Preflight the GGUF without downloading it

GGUF metadata lives in the header and HF honours ranged GETs, so a few MB
answers every sizing question before a 13 GB transfer
(`Range: bytes=0-8000000`). Parse: magic `GGUF`, u32 version, u64
n_tensors, u64 n_kv, then key/value pairs (string = u64 len + bytes; ARRAY
= u32 elem type + u64 count; BOOL is 1 byte — the type hand-written parsers
forget). Walk arrays by byte size even for a summary: a naive parser
desynchronises on `tokenizer.ggml.tokens` and lands gigabytes off. Read:
`general.architecture`, `<arch>.block_count`, `<arch>.context_length`,
`<arch>.nextn_predict_layers` (MTP is a property of the FILE — the `-mtp`
build carries `blk.<n>.nextn.*`; `--spec-type draft-mtp` then loads the
head against the MAIN model's weights, and `--model-draft` would load the
file twice), `<arch>.head_count_kv`, `<arch>.key_length`/`value_length`,
`<arch>.full_attention_interval`, `<arch>.ssm.state_size`/`group_count`/
`time_step_rank`, and for the projector `clip.projector_type` +
`clip.has_vision_encoder`.

KV arithmetic for the hybrid (16 × (3 Gated DeltaNet + 1 Gated Attention)):
only full-attention layers take KV
(`attn_layers × n_head_kv × head_dim × 2 × bytes_per_elem`; f16 = 2 B,
q8_0 = 1.0625 B, q4_0 = 0.5625 B); DeltaNet state is a fixed per-slot
buffer (~0.16 GiB/sequence at 48 layers/128 d_state/48 v-heads f32); the
MTP draft context costs ≈ 4 KiB/token. `--parallel N` DIVIDES `--ctx-size`
between slots — it does not multiply.

## Verification (2026-09-26 deploy)

- `/v1/models` reports `frank-primary`.
- Chat completion returns `content` + `reasoning_content`.
- Tool call returns well-formed `tool_calls` (`--jinja`).
- Real image request answered from the projector (not a health check —
  health cannot see vision).
- Spec chain live: `draft acceptance` lines in the unit journal; ngram-mod
  has no startup banner (identify behaviourally: `draft_n` > 0 where text
  repeats).
- `reasoning_effort: high` → 200 (the template's `{xhigh, medium, low}`
  gate is folded by `reasoningEffortMap` in `models.yml`).
- VRAM/RAM headroom: `rocm-smi` per card + `free -g` after the first long
  completion.

_**Measured (2026-09-26 battery, 6/6 pass).** Load: ~7 s warm (mmap from
page cache; the fetch's own verify pass pre-warms it — a cold first load
pages in 175 GiB). Decode: **11.8 t/s** `predicted_per_second` at
depth-0 prose, 128-token generation, draft_n 109 / accepted 72 (0.66 on
reasoning-heavy prose; **0.864 mean len 2.73** on the tool-call task).
Vision: the probe image (green square centered on a red field) described
exactly. VRAM: GPU0 29% / GPU1 36% of 32 GiB (~9.3 / ~11.5 GiB — the
output-head card carries the drafter, as designed; ~22 GiB free per
card). RAM: ~11 GiB used, ~226 GiB page cache (the payload itself),
available ~235 of 246 GiB. `reasoning_effort: high` → 200. Two cutover
traps for the record: a LAST-ACK tail from a LAN client held
`wait_port_free` until it cleared (a foreign process on the port gates
the deploy correctly — but the deployer's `/health` wait cannot tell WHO
answers; after any aborted deploy, confirm the responder is the unit's
own MainPID), and llama-server's SIGTERM cleanup hung after
spec-decode service (logged `cleaning up before exit` and never exited —
`kill -9` was needed; `systemctl stop` relies on TimeoutStopSec)._
**KV law (from the -lv 4 load log, cross-checked by the GGUF header):**
29,696 B/token total ctx-scaled VRAM — 24,576 B main KV (only 12 of 48
layers carry attention, `full_attention_interval=4`; 2 KV heads x 256,
f16) + 3,072 B indexer + 2,048 B draft KV. Everything else (weights
~6.7 GiB, recurrent state 0.33 GiB, compute buffers ~4.1 GiB) is
ctx-independent. At 262,144: 7.25 GiB scaled, 18.75 GiB total, 9.37
GiB/card. At 100,000: 2.77 GiB scaled, 14.3 GiB total, 7.1 GiB/card.
`--parallel N` splits the same pool — slots do not multiply KV.

**Reshape (later the same day): four full slots, V q8_0, hot expert
blocks on-GPU.** `CONTEXT_SIZE=1048576` / `PARALLEL_SLOTS=4` — four
conversations, each the FULL native 262,144 window (per-slot window
unchanged, so no client or models.yml implication; slots divide the
pool, never multiply it). `CACHE_TYPE_V=q8_0` with K held at f16: main
KV drops 24,576 → 18,816 B/token (−23.4%), pool law 23,936 B/token.
The shared MTP drafter's KV ignores the V type (measured: draft pool
2,048 MiB = the f16 law at 1,048,576 cells — one layer's worth, the
miss costs ~0.5 GiB on ROCm1). At 4×262,144 the pool is ~23 GiB across
both cards: main 9,408 MiB + indexer 1,536 MiB per card, draft
2,048 MiB on ROCm1.

Expert blocks stay GPU-resident — and the measurement below shows it
cannot matter *which* ones. `-ot` granularity is the whole
per-block expert tensor (512 experts share one tensor), so "critical
experts" = blocks co-located on the card that holds their layers:
blocks 0-4 on ROCm0, 45-47 on ROCm1 (measured 2.42/2.49 GiB per
block). The placement law that makes it work: b11030 wants ONE
comma-joined `--override-tensor` (repeats log DEPRECATED), and the
loader applies the FIRST matching pattern per tensor — the
block-specific patterns MUST precede the broad `exps=CPU` sweep, or
the sweep silently swallows them (verified live: broad-first left
every expert on CPU with no warning anywhere in the log).

Measured headroom, deployed fill **5/3** (blocks 0-4 on ROCm0, 45-47
on ROCm1 — the operator directed the fill past the 3-4 GB checkpoint
floor the same day, "shift a few more experts"): card0 2.53 GiB free
at boot, 2.23 GiB under load with the vision graph exercised; card1
3.31 GiB at boot, 3.15 GiB under load. Card1 cannot take another
block — 3.31 − 2.49 = 0.82 GiB, and the vision graph (~0.43 GiB) plus
boot drift alone exceed that — so 5/3 is the physical limit at the
4×262144 / V-q8_0 shape. Earlier 4/3 state for reference: card0 5.02
GiB boot free. Live numbers (4/3 era): content decode 13.05 t/s
(acceptance 0.58, mean len 2.16), reasoning phase 7.24 t/s
(acceptance 0.74, mean len 2.48), two concurrent full generations both
delivered, prompt cache reusing 42/64 tokens across requests. All
three Prometheus fleet targets up; the 5/3 deploy re-verified chat,
vision and the target on the same battery class.

**Which blocks, measured (same day):** asked whether the `-ot` regex
could target the *most active* blocks, I instrumented the live server
instead of guessing — `scripts/frank-primary-expert-activity.py`
samples llama-server with AMD IBS (`perf record -e ibs_op// --data`)
while 4 mixed concurrent generations run, maps each sampled load
address through `/proc/<pid>/maps` back to its backing tensor (extents
derived from consecutive GGUF header offsets — never from a quant
type-size table, whose errors silently mis-bucket sample tails), and
aggregates per block and per expert. Verdict, 75 s window / 2.7M
expert-tensor samples: **per-block read volume is FLAT, 0.93×–1.08× of
mean across all 40 CPU-resident blocks** (0-4 and 45-47 absent —
GPU-resident). MoE top-k routing fires every block on every token, so
"most active layers" do not exist at `-ot` granularity: the regex can
express any 8-block set, and any set is equivalent to any other — the
lever is COUNT, already at the 5/3 VRAM ceiling. The real skew is
per-EXPERT (top 10% of the 48×512 (block,expert) cells carry 40.5% of
samples; hottest cell 0.1%), below the reach of a tensor-level
override. Not a benchmark: it instruments production serving in place
at negligible IBS overhead; re-measure any future checkpoint or
traffic mix with `frank-primary-expert-activity.py <pid>` (or
`--analyze-only` against the saved sample file in `/tmp`).

## Rollback

The lucebox DS4.1 material is fully retained on the host: image
`localhost/lucebox-rocm:gfx1201`, the 378 GiB payload, and the quadlet-era
repo files in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/` (see
[[frank-primary-lucebox-ds4]]). Restore = move the `.container` back,
uninstall the root unit (`systemctl disable --now frank-primary` — the
link, not the file; never `systemctl disable` a unit whose file must
survive), `daemon-reload`, start the user quadlet. The Qwen payload
(~179 GiB) stays fetched, so a forward re-swap is a redeploy, not a
refetch.

## Fleet context

- [[frank-large-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same
  b11030-mix binary family on gfx1151.
- [[frank-small-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the
  3090; the only audio-capable endpoint.
- The harness contract lives in `/home/blank/.omp/agent/models.yml`
  (`llamacpp (dankserver)` provider, `contextWindow 262144`,
  `maxTokens 65536`, `input: [text, image]`).