Frank
7fe271e65d
- spec matrix: new 'Production state' section — PP head (RPC0+ROCm0, split 0.40/0.60), pool 401400 / n_ctx_slot 200704 live, MTP n_max 4 over ngram-mod fallback (journal acceptance stats), f16 KV. 2026-08-15 standalone state marked superseded (Vulkan0 verdict was standalone-only; PP head runs ROCm0 by default). - deployment.md: fix frank-default-large section (MTP not 'no draft', ROCm0 not Vulkan0, 200704/slot not 262144, f16 not q8_0 KV, serve-pp-head.sh not serve-llama-frank-large.sh). - PP deployment doc: production-state section; test-day open questions resolved with journal evidence. - index.md: NPU line now qwen3.6-moe:35b-a3b @262k ctx; Sibling projects section -> serverAppConfigs. - AGENTS.md: overview reflects PP+MTP reality; sibling pointer.
217 lines
9.1 KiB
Markdown
217 lines
9.1 KiB
Markdown
# Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix
|
|
|
|
This page records the speculative-decoding benchmark and the production
|
|
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
|
|
(`frank-default-large`), now a pipeline-parallel head with a Dankserver
|
|
worker.
|
|
|
|
## Production state (2026-08-16: PP split + MTP)
|
|
|
|
Current production, verified against the live process and journal
|
|
(2026-08-16, service start 15:03). Topology detail:
|
|
[[Pipeline Parallelism/Qwen3.8 PP Deployment]].
|
|
|
|
- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
|
|
passes no `--device` flag. Default selection puts RPC first and picks
|
|
ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
|
|
an option via `HEAD_BIN=.../build-vulkan-rpc`.
|
|
- Split: `--tensor-split 0.40,0.60` — 40% of the layers on the 3090
|
|
(its maximum under the KV budget), 60% on the iGPU.
|
|
- Context: pool 401400, 2 slots. The journal reports
|
|
`n_ctx_slot = 200704`. The pi `models.json` declares 200000
|
|
(conservative).
|
|
- Spec: **MTP re-enabled** — `--spec-type draft-mtp --spec-draft-n-max 4`
|
|
(bundled head) with the `--spec-default` ngram-mod fallback (48/64/24).
|
|
Journal acceptance over one production window: MTP mean len 3.00
|
|
(per-position 0.730/0.541/0.411/0.323, 58934 drafts); ngram-mod as the
|
|
rare fallback (423 drafts in the same window).
|
|
- KV: `f16` K / `f16` V on both devices (15680 MiB ROCm0 + 9408 MiB RPC0
|
|
at load).
|
|
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
|
|
`--gpu-layers all`.
|
|
|
|
## Result (superseded 2026-08-16: standalone --spec-default)
|
|
|
|
Superseded for topology on 2026-08-16, when the PP split became
|
|
production. The ngram-mod preset survives as the fallback layer under
|
|
MTP. The Vulkan0 verdict in this section applies to the standalone
|
|
configuration only.
|
|
|
|
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
|
- Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24,
|
|
n_min 48, n_max 64). No draft model. Chosen 2026-08-15 after DSpark
|
|
crashed under parallel load (see below).
|
|
- **PARALLEL=2** — passes the 2x32k parallel stream test (both streams
|
|
complete 256/256 concurrently, no crash). ngram-mod is draftless and
|
|
parallel-safe.
|
|
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes
|
|
5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm only wins prefill (119 vs
|
|
106 t/s). See the high-context bench table below.
|
|
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
|
|
|
|
## Result (superseded 2026-08-15: DSpark)
|
|
|
|
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
|
- Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
|
|
(`Qwen3.8-27B-DSpark-Q8_0.gguf`, 1.36B, block_size 7), `--spec-type
|
|
draft-dspark`, `--spec-draft-n-max 7` (clamps to block_size).
|
|
- **PARALLEL=1.** Two concurrent 16k prose prompts crashed the ROCm backend
|
|
with `ROCm error: an illegal memory access` during per-sequence block
|
|
sampling. See the DSpark parallel incident below.
|
|
- Retired the same day in favor of `--spec-default`.
|
|
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
|
|
|
|
## Result (superseded 2026-08-14: bundled MTP)
|
|
|
|
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
|
- Spec: `draft-mtp`, `--spec-draft-n-max 5`, no `--spec-draft-p-min`.
|
|
- Batch 8192 / ubatch 256, parallel 2, ctx 524288 (262144 per slot).
|
|
- Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under
|
|
2 concurrent streams. No premature stop.
|
|
|
|
## Incident: DSpark ROCm Crash Under Parallel Load (2026-08-15)
|
|
|
|
A two-stream parallel acceptance test crashed `frank-default-large`.
|
|
|
|
```mermaid
|
|
graph TD
|
|
A[Two concurrent 16k prose prompts<br>high prompt-cache reuse] --> B[ROCm illegal memory access<br>in common_speculative_impl_draft_dflash::draft]
|
|
B --> C[status 6/ABRT core dump]
|
|
C --> D[systemd auto-restart, 503 during reload]
|
|
D --> E[pinned PARALLEL=1]
|
|
```
|
|
|
|
Trigger: two distinct-but-overlapping ~16k-token realistic prose prompts
|
|
started simultaneously on PARALLEL=2. The ROCm backend aborted at
|
|
`ggml-cuda.cu:106` with `ROCm error: an illegal memory access was
|
|
encountered`, surfacing in `hipStreamSynchronize` inside
|
|
`common_speculative_impl_draft_dflash::draft` (the class used by
|
|
`draft-dspark`). `systemd` caught it (status 6/ABRT) and restarted the
|
|
unit; in-flight streams saw `Remote end closed connection without response`
|
|
until the ~reload finished.
|
|
|
|
Mechanism: DSpark reuses the DFlash block-diffusion draft machinery. The
|
|
DFlash draft model uses one shared draft context (`ctx_dft`) across all
|
|
parallel slots; under concurrent block drafting on ROCm it performs an
|
|
illegal memory access (HIP stream sync). This is the same multi-slot
|
|
class of bug that retired `draft-dflash` for premature EOS; here it is a
|
|
hard ROCm crash instead of a bad sample.
|
|
|
|
Fix: pin **PARALLEL=1** (single slot). With one slot, concurrent requests
|
|
queue safely and each completes to `max_tokens`. Revisit after an upstream
|
|
multi-slot DSpark fix (DSpark is known-good single-slot). Single-slot 16k
|
|
prose generation was ~steady (measuring only the correct-size signal, not
|
|
this page).
|
|
|
|
## Parallel results by stream count (2026-08-15, correct-size tests)
|
|
|
|
| Parallel | Two 16k prose prompts | Two 67k prose prompts |
|
|
| -------- | -------------------- | --------------------- |
|
|
| 2 | crashes (illegal mem access) | passes |
|
|
| 1 | queues, both complete | n/a |
|
|
|
|
Note: the crash manifests on the smaller (16k) concurrent streams, not the
|
|
larger 67k pair. Size is not the trigger; concurrent block drafting on the
|
|
shared draft context is.
|
|
|
|
## How MTP works on Qwen3.8
|
|
|
|
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
|
|
the model file. llama.cpp b10411+ auto-detects it and creates the draft
|
|
context from the target model. Do not pass `--spec-draft-model`.
|
|
|
|
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
|
|
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.
|
|
|
|
## Benchmark Method
|
|
|
|
The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`,
|
|
`draft-mtp`, `ngram-mod`) x `--spec-draft-n-max` 2..10 x
|
|
`--spec-draft-p-min` {off, 0.8}. Each config loaded `llama-server`,
|
|
then ran 3 generations. The tool is `scripts/spec-matrix-bench.sh`;
|
|
analysis is `scripts/analyze-spec-matrix.py`.
|
|
|
|
## Matrix Caveat: Benchmark Prompt
|
|
|
|
The matrix used a repetitive 1100-token prompt
|
|
(`scripts/bench_live.sh` style). MTP drafts that repeated sentence almost
|
|
perfectly, which inflated the numbers for long drafts. The matrix reported
|
|
`draft-mtp` n_max=10 at ~30 t/s. A realistic story prompt gave very
|
|
different results:
|
|
|
|
| n_max | median t/s (realistic) |
|
|
| ----- | ----------------------- |
|
|
| 4 | 12.5 |
|
|
| 5 | 13.3 (peak) |
|
|
| 6 | 10.6 |
|
|
| 7 | 10.1 |
|
|
| 10 | 8.6 |
|
|
| none (baseline) | 7.2 |
|
|
|
|
Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances
|
|
acceptance against draft cost.
|
|
|
|
## Premature Stop Investigation
|
|
|
|
Two failure patterns appeared during early production tests:
|
|
|
|
1. Speed collapse. `ubatch >= 512` reduced MTP acceptance to ~25% and
|
|
generation to ~9 t/s. `ubatch 128` with `batch >= 2048` triggered a
|
|
premature instant-EOS under 2 concurrent slots. `ubatch 256` avoids
|
|
both: ~30 t/s single, concurrent-safe.
|
|
2. Apparent premature-EOS. The repetitive benchmark prompt with `temp 1.0`
|
|
sometimes stops after 1-2 tokens because the model legitimately emits an
|
|
EOS on a wall of repeated text. This is sampling variance, not a
|
|
corruption bug. Realistic prompts never stop early.
|
|
|
|
## High-Context Backend Comparison (2026-08-15)
|
|
|
|
Tool: `scripts/bench-high-ctx.sh <device> <ctx>`. Both runs at ctx 262144
|
|
with a ~200k-token realistic-prose prompt, `--spec-default`, batch 8192 /
|
|
ubatch 256, single slot, on scratch port 7892.
|
|
|
|
| Device | Prompt tokens | Prefill (t/s) | Generation (t/s) | Wall (single req) |
|
|
| ------ | ------------- | ------------- | ---------------- | ----------------- |
|
|
| ROCm0 | 202822 | **119.1** | 3.83 | **1770 s** |
|
|
| Vulkan0 | 200235 | 106.3 | **5.68** | 1929 s |
|
|
|
|
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
|
|
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
|
|
(users stream tokens), so **Vulkan0** is the production backend. Both
|
|
`tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
|
|
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
|
|
that is the long-context attention penalty, not a spec artifact.
|
|
|
|
## Production Configuration
|
|
|
|
`scripts/frank-default-large.env` (live 2026-08-16):
|
|
|
|
```ini
|
|
RPC_SERVERS=192.168.0.235:50052 # Dankserver worker
|
|
TENSOR_SPLIT=0.40,0.60 # [RPC0, iGPU]
|
|
CTX_SIZE=401400 # pool = 2 slots x 200700
|
|
PARALLEL=2
|
|
SPEC_NONE=0
|
|
SPEC_DEFAULT=1 # ngram-mod fallback (48/64/24)
|
|
MTP=1 # draft-mtp, bundled head
|
|
SPEC_DRAFT_N_MAX=4
|
|
SPEC_NGRAM_MOD_N_MIN=48
|
|
SPEC_NGRAM_MOD_N_MAX=64
|
|
SPEC_NGRAM_MOD_N_MATCH=24
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
BATCH_SIZE=8192
|
|
UBATCH_SIZE=256
|
|
```
|
|
|
|
There is no `DEVICE` variable. The head must not pass `--device` (the RPC
|
|
device would be silently dropped). Live slot context: `n_ctx_slot =
|
|
200704`. The pi `models.json` contextWindow for `frank-default-large`
|
|
declares 200000, a safe under-estimate of the live slot.
|
|
|
|
## See Also
|
|
|
|
- [[frank-default-large-benchmark]]
|
|
- [[ling-3.0-flash-trial]]
|
|
- [[deployment]]
|