serverAppConfigs/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
configuration	frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook	ACTIVE frank-primary runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment.	frank-primary dankserver llamacpp unsloth qwen38-flash-next q8_0 mtp ngram-mod spec-decode vision rocm gfx1201 r9700	2026-09-26

frank-primary llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook

The ACTIVE dankserver LLM endpoint (:7770, alias frank-primary): unsloth llama.cpp serving unsloth/Qwen3.8-Flash-Next-GGUF at Q_8 (the 6-part Q8_0/ main, 175.30 GiB) on both Radeon AI PRO R9700s, with the shared Q_8 MTP drafter (draft-mtp, depth 2) layered with ngram-mod, and vision through mmproj-BF16.gguf. It replaces the Lucebox DS4.1 quadlet that held the port for hours earlier the same day (frank-primary-lucebox-ds4), which itself replaced frank-medium-radiance-mxfp4.

Why this shape

• One runtime for the fleet. All three endpoints are llama.cpp now — one metric namespace (llamacpp:*), one reasoning contract (reasoning_content), one tool contract (--jinja). The two-day experiment with a single-vendor engine (lucebox) ended because Qwen Flash-Next at Q8 on llama.cpp delivers better quality than the DS4.1 fp2/fp3 stream, plus vision, plus the full 262k window, at comparable speed.
• The placement law, not a preference: the routed experts and the 51B-param n-gram (PLE) embedding table are computed in host RAM (-ot exps=CPU,per_layer_token_embd.weight=CPU) and never streamed over PCIe. The x4-pair measurement on the same model shape: streaming loses (3.7 t/s), computing in RAM does not care (20+ t/s). The dense weights, attention, projector and MTP head sit on the two cards (~29 GiB); the ~100 GiB of expert pages live in mmap page cache — --no-mmap is forbidden (it forces the table into anonymous memory; the cgroup kill is silent) and pinning the PLE table with a GPU target cost 23 → 13.4 t/s when tried. The tensor names are verified, not guessed: the GGUF v3 header dump (dump-gguf.py, run on the fetched parts 2026-09-26) shows the PLE table is one tensor — per_layer_token_embd.weight, dims [160, 320001536], Q8_0, 54.4 GiB, exactly the whole of part 3 — and the model is qwen4exp with 512 experts top-10 (qwen4exp.ple.ngram_size=3, heads_per_ngram=8). The same tensor is the MTP head's borrow target, so the override is required even if the placement were optional.
• Two cards, layer split (-ts 1,1): both R9700s are on CPU root ports with full PCIe atomics (the ProArt B850-Creator swap, 2026-09-20), but this deployment uses llama.cpp's layer split — -ot tensor overrides do not compose with tensor-parallel mode, and exps=CPU is the override the placement law needs. The 27B-dense equivalent on this host measured 1991 t/s prefill / 18.8 t/s decode in this shape; Flash-Next Q8 with CPU experts lands in the same class (measured numbers below).
• Spec chain: draft-mtp,ngram-mod. MTP is a property of the FILE: the shared-Q8_0 head (mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf, 2,786,568,256 B) is loaded via --model-draft and borrows the target's embedding tensors — which is why per_layer_token_embd.weight must be CPU-resident in the override set (a GPU-resident borrow crashes the draft load with invalid vector subscript). ngram-mod is draftless, so it chains legally behind the drafter. Depth 2: each longer verification round re-batches the CPU expert reads, so depth buys less than bandwidth here. The drafter must sit on ROCm1 — the head also borrows the target's output.weight, which a -ts 1,1 split places on the last-layers card; with tensor overrides set the draft scheduler runs n_copies==1, and a draft context elsewhere aborts at load (ggml-backend.cpp:941, "pre-allocated tensor (output.weight) in a buffer (ROCm1) that cannot run the operation"). The 2026-09-26 cutover crash-looped 4× on ROCm0 before the probe confirmed ROCm1 loads — the frank-medium precedent (drafter on the last-layers card) was right.
• Vision, explicitly. This build never autoloads projectors — without --mmproj the server comes up healthy and text-only and answers image requests with 500 (the frank-medium 2026-09-17 trap). The BF16 projector (907,542,944 B) is named in the serve script.

Deployment

Piece	Value
Unit	scripts/frank-primary.service — root systemd unit, User=dank, Restart=on-failure, LimitMEMLOCK=infinity, relinked to /etc/systemd/system/
Serve script	scripts/serve-frank-primary.sh — validates binary + all 6 parts + drafter + mmproj before exec
Env tunables	scripts/frank-primary.env — paths and numbers only; bind is BIND_HOST (never a bare HOST: HIP reads it)
Runtime	/home/dank/GitPackages/unsloth-llama-b11030/llama-server — unsloth b11030-mix prebuilt (commit 6ba30d05b), ROCm gfx120X, bundled runtime via LD_LIBRARY_PATH
Payload	/home/dank/models/Qwen3.8-Flash-Next-GGUF/ — Q8_0/ (6 parts), MTP/mtp-...-shared-Q8_0.gguf, mmproj-BF16.gguf; fetched by scripts/download-frank-primary.sh (sha256 + byte-count verified)
Port / alias	7770 / frank-primary — unchanged from the lucebox and radiance eras; no client moves
Context	262,144 × 1 slot (PARALLEL_SLOTS=1, one stream, the model's full native window)
Sampling	Greedy server default (deterministic e2e); Qwen-recommended sampling rides models.yml compat.extraBody per request
Template	The GGUF's embedded template via --jinja; reasoning → reasoning_content (--reasoning-format deepseek)
Observability	frank-primary Prometheus job at 192.168.0.235:7770 — cross-host LAN scrape (the netavark published-port quirk that forces host.containers.internal is same-host only); exports llamacpp:*

Cutover mechanics

The deployer (deploy_llm_primary in scripts/deploy-remote.sh) gates on every artifact at its declared byte count (stat -c %s, never wc -c — which read the whole 377 GiB lucebox file for minutes on 2026-09-26), retires the lucebox user quadlet to dated home trash (~/.AGENTTRASH/20260926-llamacpp-cutover/), waits for port 7770 to be truly free, relinks + enables + restarts the root unit, and waits up to 1800 s for /health (cold start page-ins ~175 GiB and builds the graphs).

Preflight the GGUF without downloading it

GGUF metadata lives in the header and HF honours ranged GETs, so a few MB answers every sizing question before a 13 GB transfer (Range: bytes=0-8000000). Parse: magic GGUF, u32 version, u64 n_tensors, u64 n_kv, then key/value pairs (string = u64 len + bytes; ARRAY = u32 elem type + u64 count; BOOL is 1 byte — the type hand-written parsers forget). Walk arrays by byte size even for a summary: a naive parser desynchronises on tokenizer.ggml.tokens and lands gigabytes off. Read: general.architecture, <arch>.block_count, <arch>.context_length, <arch>.nextn_predict_layers (MTP is a property of the FILE — the -mtp build carries blk.<n>.nextn.*; --spec-type draft-mtp then loads the head against the MAIN model's weights, and --model-draft would load the file twice), <arch>.head_count_kv, <arch>.key_length/value_length, <arch>.full_attention_interval, <arch>.ssm.state_size/group_count/ time_step_rank, and for the projector clip.projector_type + clip.has_vision_encoder.

KV arithmetic for the hybrid (16 × (3 Gated DeltaNet + 1 Gated Attention)): only full-attention layers take KV (attn_layers × n_head_kv × head_dim × 2 × bytes_per_elem; f16 = 2 B, q8_0 = 1.0625 B, q4_0 = 0.5625 B); DeltaNet state is a fixed per-slot buffer (~0.16 GiB/sequence at 48 layers/128 d_state/48 v-heads f32); the MTP draft context costs ≈ 4 KiB/token. --parallel N DIVIDES --ctx-size between slots — it does not multiply.

Verification (2026-09-26 deploy)

• /v1/models reports frank-primary.
• Chat completion returns content + reasoning_content.
• Tool call returns well-formed tool_calls (--jinja).
• Real image request answered from the projector (not a health check — health cannot see vision).
• Spec chain live: draft acceptance lines in the unit journal; ngram-mod has no startup banner (identify behaviourally: draft_n > 0 where text repeats).
• reasoning_effort: high → 200 (the template's {xhigh, medium, low} gate is folded by reasoningEffortMap in models.yml).
• VRAM/RAM headroom: rocm-smi per card + free -g after the first long completion.

Measured (2026-09-26 battery, 6/6 pass). Load: ~7 s warm (mmap from page cache; the fetch's own verify pass pre-warms it — a cold first load pages in 175 GiB). Decode: 11.8 t/s predicted_per_second at depth-0 prose, 128-token generation, draft_n 109 / accepted 72 (0.66 on reasoning-heavy prose; 0.864 mean len 2.73 on the tool-call task). Vision: the probe image (green square centered on a red field) described exactly. VRAM: GPU0 29% / GPU1 36% of 32 GiB (~9.3 / ~11.5 GiB — the output-head card carries the drafter, as designed; ~22 GiB free per card). RAM: ~11 GiB used, ~226 GiB page cache (the payload itself), available ~235 of 246 GiB. reasoning_effort: high → 200. Two cutover traps for the record: a LAST-ACK tail from a LAN client held wait_port_free until it cleared (a foreign process on the port gates the deploy correctly — but the deployer's /health wait cannot tell WHO answers; after any aborted deploy, confirm the responder is the unit's own MainPID), and llama-server's SIGTERM cleanup hung after spec-decode service (logged cleaning up before exit and never exited — kill -9 was needed; systemctl stop relies on TimeoutStopSec). KV law (from the -lv 4 load log, cross-checked by the GGUF header): 29,696 B/token total ctx-scaled VRAM — 24,576 B main KV (only 12 of 48 layers carry attention, full_attention_interval=4; 2 KV heads x 256, f16) + 3,072 B indexer + 2,048 B draft KV. Everything else (weights ~6.7 GiB, recurrent state 0.33 GiB, compute buffers ~4.1 GiB) is ctx-independent. At 262,144: 7.25 GiB scaled, 18.75 GiB total, 9.37 GiB/card. At 100,000: 2.77 GiB scaled, 14.3 GiB total, 7.1 GiB/card. --parallel N splits the same pool — slots do not multiply KV.

Reshape (later the same day): four full slots, V q8_0, hot expert blocks on-GPU. CONTEXT_SIZE=1048576 / PARALLEL_SLOTS=4 — four conversations, each the FULL native 262,144 window (per-slot window unchanged, so no client or models.yml implication; slots divide the pool, never multiply it). CACHE_TYPE_V=q8_0 with K held at f16: main KV drops 24,576 → 18,816 B/token (−23.4%), pool law 23,936 B/token. The shared MTP drafter's KV ignores the V type (measured: draft pool 2,048 MiB = the f16 law at 1,048,576 cells — one layer's worth, the miss costs ~0.5 GiB on ROCm1). At 4×262,144 the pool is ~23 GiB across both cards: main 9,408 MiB + indexer 1,536 MiB per card, draft 2,048 MiB on ROCm1.

Expert blocks stay GPU-resident — and the measurement below shows it cannot matter which ones. -ot granularity is the whole per-block expert tensor (512 experts share one tensor), so "critical experts" = blocks co-located on the card that holds their layers: blocks 0-4 on ROCm0, 45-47 on ROCm1 (measured 2.42/2.49 GiB per block). The placement law that makes it work: b11030 wants ONE comma-joined --override-tensor (repeats log DEPRECATED), and the loader applies the FIRST matching pattern per tensor — the block-specific patterns MUST precede the broad exps=CPU sweep, or the sweep silently swallows them (verified live: broad-first left every expert on CPU with no warning anywhere in the log).

Measured headroom, deployed fill 5/3 (blocks 0-4 on ROCm0, 45-47 on ROCm1 — the operator directed the fill past the 3-4 GB checkpoint floor the same day, "shift a few more experts"): card0 2.53 GiB free at boot, 2.23 GiB under load with the vision graph exercised; card1 3.31 GiB at boot, 3.15 GiB under load. Card1 cannot take another block — 3.31 − 2.49 = 0.82 GiB, and the vision graph (~0.43 GiB) plus boot drift alone exceed that — so 5/3 is the physical limit at the 4×262144 / V-q8_0 shape. Earlier 4/3 state for reference: card0 5.02 GiB boot free. Live numbers (4/3 era): content decode 13.05 t/s (acceptance 0.58, mean len 2.16), reasoning phase 7.24 t/s (acceptance 0.74, mean len 2.48), two concurrent full generations both delivered, prompt cache reusing 42/64 tokens across requests. All three Prometheus fleet targets up; the 5/3 deploy re-verified chat, vision and the target on the same battery class.

Which blocks, measured (same day): asked whether the -ot regex could target the most active blocks, I instrumented the live server instead of guessing — scripts/frank-primary-expert-activity.py samples llama-server with AMD IBS (perf record -e ibs_op// --data) while 4 mixed concurrent generations run, maps each sampled load address through /proc/<pid>/maps back to its backing tensor (extents derived from consecutive GGUF header offsets — never from a quant type-size table, whose errors silently mis-bucket sample tails), and aggregates per block and per expert. Verdict, 75 s window / 2.7M expert-tensor samples: per-block read volume is FLAT, 0.93×–1.08× of mean across all 40 CPU-resident blocks (0-4 and 45-47 absent — GPU-resident). MoE top-k routing fires every block on every token, so "most active layers" do not exist at -ot granularity: the regex can express any 8-block set, and any set is equivalent to any other — the lever is COUNT, already at the 5/3 VRAM ceiling. The real skew is per-EXPERT (top 10% of the 48×512 (block,expert) cells carry 40.5% of samples; hottest cell 0.1%), below the reach of a tensor-level override. Not a benchmark: it instruments production serving in place at negligible IBS overhead; re-measure any future checkpoint or traffic mix with frank-primary-expert-activity.py <pid> (or --analyze-only against the saved sample file in /tmp).

Rollback

The lucebox DS4.1 material is fully retained on the host: image localhost/lucebox-rocm:gfx1201, the 378 GiB payload, and the quadlet-era repo files in dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ (see frank-primary-lucebox-ds4). Restore = move the .container back, uninstall the root unit (systemctl disable --now frank-primary — the link, not the file; never systemctl disable a unit whose file must survive), daemon-reload, start the user quadlet. The Qwen payload (~179 GiB) stays fetched, so a forward re-swap is a redeploy, not a refetch.

Fleet context

• frank-large-unsloth-qwen38-27b-q8xl — Dankclaw :7770, the same b11030-mix binary family on gfx1151.
• frank-small-gemma4-12b — Dankclaw :7771, mainline llama.cpp on the 3090; the only audio-capable endpoint.
• The harness contract lives in /home/blank/.omp/agent/models.yml (llamacpp (dankserver) provider, contextWindow 262144, maxTokens 65536, input: [text, image]).