Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
142 lines
6.3 KiB
Markdown
142 lines
6.3 KiB
Markdown
---
|
|
type: reference
|
|
description: "The speculative-decoding benchmark history for this endpoint, arm by arm, and what each arm ruled out."
|
|
tags:
|
|
- llm
|
|
- speculative
|
|
timestamp: 2026-09-03
|
|
---
|
|
|
|
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
|
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
|
|
> [deployment.md](deployment.md) for the active configuration.
|
|
|
|
# Qwen3.8-27B Speculative-Decoding Matrix
|
|
|
|
This page records the speculative-decoding benchmark history and the
|
|
production tuning for the then-Qwen3.8-27B service (`frank-dragoon`), at the
|
|
time a pipeline-parallel head with a Dankserver worker.
|
|
|
|
## Production state (2026-08-17)
|
|
|
|
The pipeline-parallel shape this matrix was tuned for — Q6_K target, standalone
|
|
Q8_0 draft on ROCm0 only, `--tensor-split 0.68,0.32` (45 of 65 blocks), K `q8_0`
|
|
/ V `f16`, pool 262144 at one slot, worker steady at 23854/24576 MiB — is
|
|
recorded once, in [[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Production
|
|
state, together with the split-limit arithmetic (why 45 blocks is the maximum),
|
|
the KV-per-layer bill and the FA-all-quants rebuild the mismatch needed. That
|
|
era's endpoint identity is an [[frank-dragoon]] era. This page keeps only the
|
|
speculative-decoding measurements.
|
|
|
|
## How MTP works on Qwen3.8 (bundled-head era)
|
|
|
|
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
|
|
the model file. llama.cpp b10411+ auto-detects it and creates the draft
|
|
context from the target model. Do not pass `--spec-draft-model` for those
|
|
files. This is superseded: the production target of 2026-08-17 carried no
|
|
bundled head, and the serve script passed the standalone draft sidecar with
|
|
`--spec-draft-model` (shape: [[Pipeline Parallelism/Qwen3.8 PP Deployment]]).
|
|
|
|
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
|
|
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.
|
|
|
|
## History (superseded configurations, 2026-08-14 to 08-16)
|
|
|
|
All superseded configs used `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB,
|
|
bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the
|
|
fallback layer under MTP. The Vulkan0 verdict from this era applies to the
|
|
standalone configuration only; the PP head runs ROCm0.
|
|
|
|
- **2026-08-14: bundled MTP** — `draft-mtp`, n_max 5, parallel 2, ctx
|
|
524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2
|
|
concurrent streams. No premature stop.
|
|
- **2026-08-15: DSpark sidecar** — `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
|
|
(Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k
|
|
prose prompts crashed the ROCm backend with `ROCm error: an illegal
|
|
memory access` in `common_speculative_impl_draft_dflash::draft` (status
|
|
6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash
|
|
block-diffusion machinery, which shares one draft context across
|
|
parallel slots; concurrent block drafting faults in HIP stream sync.
|
|
This is the same multi-slot bug class that retired `draft-dflash` (premature
|
|
EOS, retired 2026-08-14). The crash shows on 16k concurrent streams, not on a
|
|
67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1 (requests
|
|
queue safely). Retired the same day.
|
|
- **2026-08-15 to 08-16: `--spec-default` (ngram-mod), Vulkan0** — n_match
|
|
24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream
|
|
test passes). Chosen for parallel safety after DSpark; kept as the
|
|
fallback under MTP in the 2026-08-17 shape.
|
|
|
|
## Benchmark Method
|
|
|
|
The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`,
|
|
`draft-mtp`, `ngram-mod`) x `--spec-draft-n-max` 2..10 x
|
|
`--spec-draft-p-min` {off, 0.8}. Each config loaded `llama-server`,
|
|
then ran 3 generations. The tool is `scripts/spec-matrix-bench.sh`;
|
|
analysis is `scripts/analyze-spec-matrix.py`.
|
|
|
|
## Matrix Caveat: Benchmark Prompt
|
|
|
|
The matrix used a repetitive 1100-token prompt
|
|
(`scripts/bench_live.sh` style). MTP drafts that repeated sentence almost
|
|
perfectly, which inflated the numbers for long drafts. The matrix reported
|
|
`draft-mtp` n_max=10 at ~30 t/s. A realistic story prompt gave very
|
|
different results:
|
|
|
|
| n_max | median t/s (realistic) |
|
|
| ----- | ----------------------- |
|
|
| 4 | 12.5 |
|
|
| 5 | 13.3 (peak) |
|
|
| 6 | 10.6 |
|
|
| 7 | 10.1 |
|
|
| 10 | 8.6 |
|
|
| none (baseline) | 7.2 |
|
|
|
|
Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances
|
|
acceptance against draft cost.
|
|
|
|
## Premature Stop Investigation
|
|
|
|
Two failure patterns appeared during early production tests:
|
|
|
|
1. Speed collapse. `ubatch >= 512` reduced MTP acceptance to ~25% and
|
|
generation to ~9 t/s. `ubatch 128` with `batch >= 2048` triggered a
|
|
premature instant-EOS under 2 concurrent slots. `ubatch 256` avoids
|
|
both: ~30 t/s single, concurrent-safe.
|
|
2. Apparent premature-EOS. The repetitive benchmark prompt with `temp 1.0`
|
|
sometimes stops after 1-2 tokens because the model legitimately emits an
|
|
EOS on a wall of repeated text. This is sampling variance, not a
|
|
corruption bug. Realistic prompts never stop early.
|
|
|
|
## High-Context Backend Comparison (2026-08-15)
|
|
|
|
Tool: `scripts/bench-high-ctx.sh <device> <ctx>`. Both runs at ctx 262144
|
|
with a ~200k-token realistic-prose prompt, `--spec-default`, batch 8192 /
|
|
ubatch 256, single slot, on scratch port 7892.
|
|
|
|
| Device | Prompt tokens | Prefill (t/s) | Generation (t/s) | Wall (single req) |
|
|
| ------ | ------------- | ------------- | ---------------- | ----------------- |
|
|
| ROCm0 | 202822 | **119.1** | 3.83 | **1770 s** |
|
|
| Vulkan0 | 200235 | 106.3 | **5.68** | 1929 s |
|
|
|
|
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
|
|
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
|
|
(users stream tokens). This verdict applies to the standalone configuration; the PP
|
|
head runs ROCm0 — HIP registers before Vulkan and the iGPU dedup rule (the llama.cpp
|
|
PR #23897 workaround) keeps the first backend's device.
|
|
Both `tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
|
|
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
|
|
that is the long-context attention penalty, not a spec artifact.
|
|
|
|
## Where the tunables live
|
|
|
|
That era's head and worker env/units are listed (and marked deleted) in
|
|
[[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Files. `scripts/frank-dragoon.env`
|
|
is no longer a PP-head env at all — it is the halogen-flash-server trio of the
|
|
current era, which [[frank-dragoon]] owns.
|
|
|
|
## See Also
|
|
|
|
- [[frank-dragoon]]
|
|
- [[ling-3.0-flash-trial]]
|
|
- [[deployment]]
|