# Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix

This page records the speculative-decoding benchmark and the production
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
(`frank-default-large`), now a pipeline-parallel head with a Dankserver
worker.

## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)

Current production, verified against the live process and journal
(2026-08-17, load 02:06 + smoke tests). Topology detail:
[[Pipeline Parallelism/Qwen3.8 PP Deployment]].

- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
  passes no `--device` flag. Default selection puts RPC first and picks
  ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
  an option via `HEAD_BIN=.../build-vulkan-rpc`.
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti, 22.43 GB,
  imatrix, 65 blocks, 866 tensors, NO bundled MTP head). The repo's
  standalone draft sidecar `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
  (3.16 GB) is loaded via `--spec-draft-model`; it runs on ROCm0 only
  (1718 MiB, zero worker impact).
- Spec: **MTP** (standalone draft) `--spec-draft-n-max 4` + the
  `--spec-default` ngram-mod fallback (48/64/24). Measured acceptance
  0.83 then 0.75, mean len 4.0-4.3, per-position ~0.86/0.71/0.71/0.71 —
  stronger than the old bundled head (0.739/0.554/0.427/0.340).
- KV: K **q8_0** + V **f16** (mismatched). 3136 B/token per FA layer =
  784 MiB at 262144 (K 1088 + V 2048). RPC0 8624 MiB (11 FA), ROCm0
  3920 MiB (5 FA); total 12544 = 16 x 784. **Flash Attention stays
  enabled with the mismatch** (`resolve_fused_ops: Flash Attention
  enabled`) via the GGML_CUDA_FA_ALL_QUANTS rebuild of both binaries
  at 9b05354ec — see the PP doc.
- Split: `--tensor-split 0.68,0.32` = 45 of 65 blocks (0-44) on the
  3090, 69% of the layer weight. The maximum: 46 blocks -> ~0.3 GiB
  margin; 48 blocks (f >= 0.7197) adds FA block 47 -> KV 9408 -> OOM by
  ~0.85 GiB. Boundary math: the PP doc.
- Context: pool 262144, 1 slot, `n_ctx_slot = 262144`. The pi
  `models.json` declares 262144 (contextWindow = maxTokens, house
  convention).
- Worker steady state: 23854/24576 MiB used on the 3090, ~273 MiB free
  (tight but stable; all allocations happen at load).
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
  `--gpu-layers all`.

## Result (superseded 2026-08-17: Q8_K_XL, f16/f16 KV, split 0.53)

Superseded for topology on 2026-08-16, when the PP split became
production. The ngram-mod preset survives as the fallback layer under
MTP. The Vulkan0 verdict in this section applies to the standalone
configuration only.

- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
- Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24,
  n_min 48, n_max 64).  No draft model.  Chosen 2026-08-15 after DSpark
  crashed under parallel load (see below).
- **PARALLEL=2** — passes the 2x32k parallel stream test (both streams
  complete 256/256 concurrently, no crash).  ngram-mod is draftless and
  parallel-safe.
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes
  5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm only wins prefill (119 vs
  106 t/s).  See the high-context bench table below.
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).

## Result (superseded 2026-08-15: DSpark)

- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
- Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
  (`Qwen3.8-27B-DSpark-Q8_0.gguf`, 1.36B, block_size 7), `--spec-type
  draft-dspark`, `--spec-draft-n-max 7` (clamps to block_size).
- **PARALLEL=1.**  Two concurrent 16k prose prompts crashed the ROCm backend
  with `ROCm error: an illegal memory access` during per-sequence block
  sampling.  See the DSpark parallel incident below.
- Retired the same day in favor of `--spec-default`.
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).

## Result (superseded 2026-08-14: bundled MTP)

- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
- Spec: `draft-mtp`, `--spec-draft-n-max 5`, no `--spec-draft-p-min`.
- Batch 8192 / ubatch 256, parallel 2, ctx 524288 (262144 per slot).
- Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under
  2 concurrent streams. No premature stop.

## Incident: DSpark ROCm Crash Under Parallel Load (2026-08-15)

A two-stream parallel acceptance test crashed `frank-default-large`.

```mermaid
graph TD
    A[Two concurrent 16k prose prompts<br>high prompt-cache reuse] --> B[ROCm illegal memory access<br>in common_speculative_impl_draft_dflash::draft]
    B --> C[status 6/ABRT core dump]
    C --> D[systemd auto-restart, 503 during reload]
    D --> E[pinned PARALLEL=1]
```

Trigger: two distinct-but-overlapping ~16k-token realistic prose prompts
started simultaneously on PARALLEL=2.  The ROCm backend aborted at
`ggml-cuda.cu:106` with `ROCm error: an illegal memory access was
encountered`, surfacing in `hipStreamSynchronize` inside
`common_speculative_impl_draft_dflash::draft` (the class used by
`draft-dspark`).  `systemd` caught it (status 6/ABRT) and restarted the
unit; in-flight streams saw `Remote end closed connection without response`
until the ~reload finished.

Mechanism: DSpark reuses the DFlash block-diffusion draft machinery.  The
DFlash draft model uses one shared draft context (`ctx_dft`) across all
parallel slots; under concurrent block drafting on ROCm it performs an
illegal memory access (HIP stream sync).  This is the same multi-slot
class of bug that retired `draft-dflash` for premature EOS; here it is a
hard ROCm crash instead of a bad sample.

Fix: pin **PARALLEL=1** (single slot).  With one slot, concurrent requests
queue safely and each completes to `max_tokens`.  Revisit after an upstream
multi-slot DSpark fix (DSpark is known-good single-slot).  Single-slot 16k
prose generation was ~steady (measuring only the correct-size signal, not
this page).

## Parallel results by stream count (2026-08-15, correct-size tests)

| Parallel | Two 16k prose prompts | Two 67k prose prompts |
| -------- | -------------------- | --------------------- |
| 2 | crashes (illegal mem access) | passes |
| 1 | queues, both complete | n/a |

Note: the crash manifests on the smaller (16k) concurrent streams, not the
larger 67k pair.  Size is not the trigger; concurrent block drafting on the
shared draft context is.

## How MTP works on Qwen3.8

The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
the model file. llama.cpp b10411+ auto-detects it and creates the draft
context from the target model. Do not pass `--spec-draft-model`.

The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.

## Benchmark Method

The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`,
`draft-mtp`, `ngram-mod`) x `--spec-draft-n-max` 2..10 x
`--spec-draft-p-min` {off, 0.8}. Each config loaded `llama-server`,
then ran 3 generations. The tool is `scripts/spec-matrix-bench.sh`;
analysis is `scripts/analyze-spec-matrix.py`.

## Matrix Caveat: Benchmark Prompt

The matrix used a repetitive 1100-token prompt
(`scripts/bench_live.sh` style). MTP drafts that repeated sentence almost
perfectly, which inflated the numbers for long drafts. The matrix reported
`draft-mtp` n_max=10 at ~30 t/s. A realistic story prompt gave very
different results:

| n_max | median t/s (realistic) |
| ----- | ----------------------- |
| 4 | 12.5 |
| 5 | 13.3 (peak) |
| 6 | 10.6 |
| 7 | 10.1 |
| 10 | 8.6 |
| none (baseline) | 7.2 |

Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances
acceptance against draft cost.

## Premature Stop Investigation

Two failure patterns appeared during early production tests:

1. Speed collapse. `ubatch >= 512` reduced MTP acceptance to ~25% and
   generation to ~9 t/s. `ubatch 128` with `batch >= 2048` triggered a
   premature instant-EOS under 2 concurrent slots. `ubatch 256` avoids
   both: ~30 t/s single, concurrent-safe.
2. Apparent premature-EOS. The repetitive benchmark prompt with `temp 1.0`
   sometimes stops after 1-2 tokens because the model legitimately emits an
   EOS on a wall of repeated text. This is sampling variance, not a
   corruption bug. Realistic prompts never stop early.

## High-Context Backend Comparison (2026-08-15)

Tool: `scripts/bench-high-ctx.sh <device> <ctx>`.  Both runs at ctx 262144
with a ~200k-token realistic-prose prompt, `--spec-default`, batch 8192 /
ubatch 256, single slot, on scratch port 7892.

| Device | Prompt tokens | Prefill (t/s) | Generation (t/s) | Wall (single req) |
| ------ | ------------- | ------------- | ---------------- | ----------------- |
| ROCm0 | 202822 | **119.1** | 3.83 | **1770 s** |
| Vulkan0 | 200235 | 106.3 | **5.68** | 1929 s |

Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
3.83 t/s); ROCm0 prefills 12% faster.  Generation is the interactive cost
(users stream tokens), so **Vulkan0** is the production backend.  Both
`tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
that is the long-context attention penalty, not a spec artifact.

## Production Configuration

`scripts/frank-default-large.env` (live 2026-08-16):

```ini
RPC_SERVERS=192.168.0.235:50052   # Dankserver worker
TENSOR_SPLIT=0.40,0.60            # [RPC0, iGPU]
CTX_SIZE=401400                   # pool = 2 slots x 200700
PARALLEL=2
SPEC_NONE=0
SPEC_DEFAULT=1                    # ngram-mod fallback (48/64/24)
MTP=1                             # draft-mtp, bundled head
SPEC_DRAFT_N_MAX=4
SPEC_NGRAM_MOD_N_MIN=48
SPEC_NGRAM_MOD_N_MAX=64
SPEC_NGRAM_MOD_N_MATCH=24
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
BATCH_SIZE=8192
UBATCH_SIZE=256
```

There is no `DEVICE` variable. The head must not pass `--device` (the RPC
device would be silently dropped). Live slot context: `n_ctx_slot =
200704`. The pi `models.json` contextWindow for `frank-default-large`
declares 200000, a safe under-estimate of the live slot.

## See Also

- [[frank-default-large-benchmark]]
- [[ling-3.0-flash-trial]]
- [[deployment]]