--- type: configuration title: frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook description: ACTIVE frank-primary runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment. tags: - frank-primary - dankserver - llamacpp - unsloth - qwen38-flash-next - q8_0 - mtp - ngram-mod - spec-decode - vision - rocm - gfx1201 - r9700 timestamp: 2026-09-26 --- # frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook The ACTIVE dankserver LLM endpoint (:7770, alias **frank-primary**): unsloth llama.cpp serving `unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8** (the 6-part `Q8_0/` main, 175.30 GiB) on **both** Radeon AI PRO R9700s, with the **shared Q_8 MTP drafter** (`draft-mtp`, depth 2) layered with **`ngram-mod`**, and **vision** through `mmproj-BF16.gguf`. It replaces the Lucebox DS4.1 quadlet that held the port for hours earlier the same day ([[frank-primary-lucebox-ds4]]), which itself replaced [[frank-medium-radiance-mxfp4]]. ## Why this shape - **One runtime for the fleet.** All three endpoints are llama.cpp now — one metric namespace (`llamacpp:*`), one reasoning contract (`reasoning_content`), one tool contract (`--jinja`). The two-day experiment with a single-vendor engine (lucebox) ended because Qwen Flash-Next at Q8 on llama.cpp delivers better quality than the DS4.1 fp2/fp3 stream, plus vision, plus the full 262k window, at comparable speed. - **The placement law, not a preference:** the routed experts and the 51B-param n-gram (PLE) embedding table are **computed in host RAM** (`-ot exps=CPU,ngram_embeddings=CPU`) and never streamed over PCIe. The x4-pair measurement on the same model shape: streaming loses (3.7 t/s), computing in RAM does not care (20+ t/s). The dense weights, attention, projector and MTP head sit on the two cards (~29 GiB); the ~100 GiB of expert pages live in mmap page cache — `--no-mmap` is forbidden (it forces the table into anonymous memory; the cgroup kill is silent) and pinning the PLE table with a GPU target cost 23 → 13.4 t/s when tried. - **Two cards, layer split** (`-ts 1,1`): both R9700s are on CPU root ports with full PCIe atomics (the ProArt B850-Creator swap, 2026-09-20), but this deployment uses llama.cpp's layer split — `-ot` tensor overrides do not compose with tensor-parallel mode, and `exps=CPU` is the override the placement law needs. The 27B-dense equivalent on this host measured 1991 t/s prefill / 18.8 t/s decode in this shape; Flash-Next Q8 with CPU experts lands in the same class (measured numbers below). - **Spec chain: `draft-mtp,ngram-mod`.** MTP is a property of the FILE: the shared-Q8_0 head (`mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf`, 2,786,568,256 B) is loaded via `--model-draft` and borrows the target's embedding tensors — which is why `per_layer_token_embd.weight` must be CPU-resident in the override set (a GPU-resident borrow crashes the draft load with `invalid vector subscript`). `ngram-mod` is draftless, so it chains legally behind the drafter. Depth 2: each longer verification round re-batches the CPU expert reads, so depth buys less than bandwidth here. - **Vision, explicitly.** This build never autoloads projectors — without `--mmproj` the server comes up healthy and text-only and answers image requests with 500 (the frank-medium 2026-09-17 trap). The BF16 projector (907,542,944 B) is named in the serve script. ## Deployment |Piece|Value| |---|---| |Unit|`scripts/frank-primary.service` — root systemd unit, `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity`, relinked to `/etc/systemd/system/`| |Serve script|`scripts/serve-frank-primary.sh` — validates binary + all 6 parts + drafter + mmproj before exec| |Env tunables|`scripts/frank-primary.env` — paths and numbers only; bind is `BIND_HOST` (never a bare `HOST`: HIP reads it)| |Runtime|`/home/dank/GitPackages/unsloth-llama-b11030/llama-server` — unsloth b11030-mix prebuilt (commit 6ba30d05b), ROCm gfx120X, bundled runtime via `LD_LIBRARY_PATH`| |Payload|`/home/dank/models/Qwen3.8-Flash-Next-GGUF/` — `Q8_0/` (6 parts), `MTP/mtp-...-shared-Q8_0.gguf`, `mmproj-BF16.gguf`; fetched by `scripts/download-frank-primary.sh` (sha256 + byte-count verified)| |Port / alias|7770 / `frank-primary` — unchanged from the lucebox and radiance eras; no client moves| |Context|262,144 × 1 slot (`PARALLEL_SLOTS=1`, one stream, the model's full native window)| |Sampling|Greedy server default (deterministic e2e); Qwen-recommended sampling rides `models.yml` `compat.extraBody` per request| |Template|The GGUF's embedded template via `--jinja`; reasoning → `reasoning_content` (`--reasoning-format deepseek`)| |Observability|`frank-primary` Prometheus job at `192.168.0.235:7770` — cross-host LAN scrape (the netavark published-port quirk that forces `host.containers.internal` is same-host only); exports `llamacpp:*`| ### Cutover mechanics The deployer (`deploy_llm_primary` in `scripts/deploy-remote.sh`) gates on every artifact at its declared byte count (`stat -c %s`, never `wc -c` — which read the whole 377 GiB lucebox file for minutes on 2026-09-26), retires the lucebox user quadlet to dated home trash (`~/.AGENTTRASH/20260926-llamacpp-cutover/`), waits for port 7770 to be truly free, relinks + enables + restarts the root unit, and waits up to 1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs). ## Verification (2026-09-26 deploy) - `/v1/models` reports `frank-primary`. - Chat completion returns `content` + `reasoning_content`. - Tool call returns well-formed `tool_calls` (`--jinja`). - Real image request answered from the projector (not a health check — health cannot see vision). - Spec chain live: `draft acceptance` lines in the unit journal; ngram-mod has no startup banner (identify behaviourally: `draft_n` > 0 where text repeats). - `reasoning_effort: high` → 200 (the template's `{xhigh, medium, low}` gate is folded by `reasoningEffortMap` in `models.yml`). - VRAM/RAM headroom: `rocm-smi` per card + `free -g` after the first long completion. _Measured numbers land here after the deploy battery._ ## Rollback The lucebox DS4.1 material is fully retained on the host: image `localhost/lucebox-rocm:gfx1201`, the 378 GiB payload, and the quadlet-era repo files in `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/` (see [[frank-primary-lucebox-ds4]]). Restore = move the `.container` back, uninstall the root unit (`systemctl disable --now frank-primary` — the link, not the file; never `systemctl disable` a unit whose file must survive), `daemon-reload`, start the user quadlet. The Qwen payload (~179 GiB) stays fetched, so a forward re-swap is a redeploy, not a refetch. ## Fleet context - [[frank-large-unsloth-qwen38-27b-q8xl]] — Dankclaw :7770, the same b11030-mix binary family on gfx1151. - [[frank-small-gemma4-12b]] — Dankclaw :7771, mainline llama.cpp on the 3090; the only audio-capable endpoint. - The harness contract lives in `/home/blank/.omp/agent/models.yml` (`llamacpp (dankserver)` provider, `contextWindow 262144`, `maxTokens 65536`, `input: [text, image]`).