Files

T

Frank aa1941d592 docs + guard: single 262k slot state, split-limits analysis, cache incident

- spec matrix + PP doc: production state = 1 slot @ 262144, split 0.53
  (34 blocks), KV 8192/8192, 3090 at 23460/24576. New 'Split limits
  (measured 2026-08-17)' section: 0.50/0.54/0.55 boundary data, the
  KV-per-device rounding rule (round(f x 66)), and why 35 blocks is
  unreachable (9-FA flip at f~0.538).
- PP doc: cache incident writeup - full Dankserver disk -> 0-byte cache
  entries -> corrupt weights -> degenerate output; fix + signature.
- deployment.md + AGENTS.md + index.md: numbers brought current.
- scripts/prune-rpc-cache.sh: cron guard capping the worker cache at
  40 GiB (oldest first) to prevent the disk-full recurrence.

2026-08-17 01:24:06 -07:00

9.5 KiB

Raw Blame History

Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix

This page records the speculative-decoding benchmark and the production tuning for the unsloth Qwen3.8-27B UD-Q8_K_XL service (frank-default-large), now a pipeline-parallel head with a Dankserver worker.

Production state (2026-08-17: single 262k slot, split 0.53)

Current production, verified against the live process and journal (2026-08-17, load 01:21, output smoke test). Topology detail: Pipeline Parallelism/Qwen3.8 PP Deployment.

Devices: RPC0 (Dankserver RTX 3090) + ROCm0 (Strix iGPU). The head passes no --device flag. Default selection puts RPC first and picks ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays an option via HEAD_BIN=.../build-vulkan-rpc.
Split: --tensor-split 0.53,0.47 — 34 of 65 blocks (0-33) on the 3090, about 52% of the layer weight. This is the maximum provably-safe value (measured 2026-08-17): 0.54 jumped the KV to 9 full-attention layers -> cudaMalloc OOM. The KV-per-device count rounds the fraction over 66 layers, independent of the weight boundary; the 9-FA flip sits at frac ~= 0.538. Boundary analysis: the PP deployment doc.
Context: pool 262144, 1 slot. n_ctx_slot = 262144 exactly. The pi models.json declares 262144 (contextWindow = maxTokens, house convention).
Spec: MTP — --spec-type draft-mtp --spec-draft-n-max 4 (bundled head) with the --spec-default ngram-mod fallback (48/64/24). Smoke acceptance 0.83 (50/60, 2026-08-17); full production-window stats in the superseded section below.
KV: f16 K / f16 V. At the 262144 pool: 8192 MiB per device (8 full-attention layers each; FA at blocks 3,7,...,63).
Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
Batch 8192 / ubatch 256, --direct-io, --fit off, --gpu-layers all.

Result (superseded 2026-08-17: 2 slots x 200704, split 0.40)

Superseded for topology on 2026-08-16, when the PP split became production. The ngram-mod preset survives as the fallback layer under MTP. The Vulkan0 verdict in this section applies to the standalone configuration only.

Model: Qwen3.8-27B-UD-Q8_K_XL.gguf (29.29 GiB, bundled MTP).
Spec: llama.cpp --spec-default preset = ngram-mod (n_match 24, n_min 48, n_max 64). No draft model. Chosen 2026-08-15 after DSpark crashed under parallel load (see below).
PARALLEL=2 — passes the 2x32k parallel stream test (both streams complete 256/256 concurrently, no crash). ngram-mod is draftless and parallel-safe.
Backend: Vulkan0 — at 262144 ctx (~200k prompt) Vulkan decodes 5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm only wins prefill (119 vs 106 t/s). See the high-context bench table below.
Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).

Result (superseded 2026-08-15: DSpark)

Model: Qwen3.8-27B-UD-Q8_K_XL.gguf (29.29 GiB, bundled MTP).
Spec: DSpark sidecar draft magnitudedev/Qwen3.8-27B-DSpark-GGUF (Qwen3.8-27B-DSpark-Q8_0.gguf, 1.36B, block_size 7), --spec-type draft-dspark, --spec-draft-n-max 7 (clamps to block_size).
PARALLEL=1. Two concurrent 16k prose prompts crashed the ROCm backend with ROCm error: an illegal memory access during per-sequence block sampling. See the DSpark parallel incident below.
Retired the same day in favor of --spec-default.
Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).

Result (superseded 2026-08-14: bundled MTP)

Model: Qwen3.8-27B-UD-Q8_K_XL.gguf (29.29 GiB, bundled MTP).
Spec: draft-mtp, --spec-draft-n-max 5, no --spec-draft-p-min.
Batch 8192 / ubatch 256, parallel 2, ctx 524288 (262144 per slot).
Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under 2 concurrent streams. No premature stop.

Incident: DSpark ROCm Crash Under Parallel Load (2026-08-15)

A two-stream parallel acceptance test crashed frank-default-large.

graph TD
    A[Two concurrent 16k prose prompts<br>high prompt-cache reuse] --> B[ROCm illegal memory access<br>in common_speculative_impl_draft_dflash::draft]
    B --> C[status 6/ABRT core dump]
    C --> D[systemd auto-restart, 503 during reload]
    D --> E[pinned PARALLEL=1]

Trigger: two distinct-but-overlapping ~16k-token realistic prose prompts started simultaneously on PARALLEL=2. The ROCm backend aborted at ggml-cuda.cu:106 with ROCm error: an illegal memory access was encountered, surfacing in hipStreamSynchronize inside common_speculative_impl_draft_dflash::draft (the class used by draft-dspark). systemd caught it (status 6/ABRT) and restarted the unit; in-flight streams saw Remote end closed connection without response until the ~reload finished.

Mechanism: DSpark reuses the DFlash block-diffusion draft machinery. The DFlash draft model uses one shared draft context (ctx_dft) across all parallel slots; under concurrent block drafting on ROCm it performs an illegal memory access (HIP stream sync). This is the same multi-slot class of bug that retired draft-dflash for premature EOS; here it is a hard ROCm crash instead of a bad sample.

Fix: pin PARALLEL=1 (single slot). With one slot, concurrent requests queue safely and each completes to max_tokens. Revisit after an upstream multi-slot DSpark fix (DSpark is known-good single-slot). Single-slot 16k prose generation was ~steady (measuring only the correct-size signal, not this page).

Parallel results by stream count (2026-08-15, correct-size tests)

Parallel	Two 16k prose prompts	Two 67k prose prompts
2	crashes (illegal mem access)	passes
1	queues, both complete	n/a

Note: the crash manifests on the smaller (16k) concurrent streams, not the larger 67k pair. Size is not the trigger; concurrent block drafting on the shared draft context is.

How MTP works on Qwen3.8

The unsloth UD-* GGUFs keep the MTP (Multi-Token Prediction) head inside the model file. llama.cpp b10411+ auto-detects it and creates the draft context from the target model. Do not pass --spec-draft-model.

The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild on Dankclaw is b10433 (9b05354ec) with GGML_HIP + gfx1151 + Vulkan.

Benchmark Method

The matrix ran 54 configurations: 3 spec types (draft-mtp,ngram-mod, draft-mtp, ngram-mod) x --spec-draft-n-max 2..10 x --spec-draft-p-min {off, 0.8}. Each config loaded llama-server, then ran 3 generations. The tool is scripts/spec-matrix-bench.sh; analysis is scripts/analyze-spec-matrix.py.

Matrix Caveat: Benchmark Prompt

The matrix used a repetitive 1100-token prompt (scripts/bench_live.sh style). MTP drafts that repeated sentence almost perfectly, which inflated the numbers for long drafts. The matrix reported draft-mtp n_max=10 at ~30 t/s. A realistic story prompt gave very different results:

n_max	median t/s (realistic)
4	12.5
5	13.3 (peak)
6	10.6
7	10.1
10	8.6
none (baseline)	7.2

Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances acceptance against draft cost.

Premature Stop Investigation

Two failure patterns appeared during early production tests:

Speed collapse. ubatch >= 512 reduced MTP acceptance to ~25% and generation to ~9 t/s. ubatch 128 with batch >= 2048 triggered a premature instant-EOS under 2 concurrent slots. ubatch 256 avoids both: ~30 t/s single, concurrent-safe.
Apparent premature-EOS. The repetitive benchmark prompt with temp 1.0 sometimes stops after 1-2 tokens because the model legitimately emits an EOS on a wall of repeated text. This is sampling variance, not a corruption bug. Realistic prompts never stop early.

High-Context Backend Comparison (2026-08-15)

Tool: scripts/bench-high-ctx.sh <device> <ctx>. Both runs at ctx 262144 with a ~200k-token realistic-prose prompt, --spec-default, batch 8192 / ubatch 256, single slot, on scratch port 7892.

Device	Prompt tokens	Prefill (t/s)	Generation (t/s)	Wall (single req)
ROCm0	202822	119.1	3.83	1770 s
Vulkan0	200235	106.3	5.68	1929 s

Verdict: at very high context, Vulkan0 decodes 48% faster (5.68 vs 3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost (users stream tokens), so Vulkan0 is the production backend. Both tg_3s values were rock-steady across the run (3.83 and 5.67 t/s). High-context decode collapses on both backends versus ~25 t/s at 16k ctx; that is the long-context attention penalty, not a spec artifact.

Production Configuration

scripts/frank-default-large.env (live 2026-08-16):

RPC_SERVERS=192.168.0.235:50052   # Dankserver worker
TENSOR_SPLIT=0.40,0.60            # [RPC0, iGPU]
CTX_SIZE=401400                   # pool = 2 slots x 200700
PARALLEL=2
SPEC_NONE=0
SPEC_DEFAULT=1                    # ngram-mod fallback (48/64/24)
MTP=1                             # draft-mtp, bundled head
SPEC_DRAFT_N_MAX=4
SPEC_NGRAM_MOD_N_MIN=48
SPEC_NGRAM_MOD_N_MAX=64
SPEC_NGRAM_MOD_N_MATCH=24
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
BATCH_SIZE=8192
UBATCH_SIZE=256

There is no DEVICE variable. The head must not pass --device (the RPC device would be silently dropped). Live slot context: n_ctx_slot = 200704. The pi models.json contextWindow for frank-default-large declares 200000, a safe under-estimate of the live slot.

9.5 KiB Raw Blame History