Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

6.3 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Qwen3.8-27B Speculative-Decoding Matrix

This page records the speculative-decoding benchmark history and the production tuning for the then-Qwen3.8-27B service (frank-dragoon), at the time a pipeline-parallel head with a Dankserver worker.

Production state (2026-08-17)

The pipeline-parallel shape this matrix was tuned for — Q6_K target, standalone Q8_0 draft on ROCm0 only, --tensor-split 0.68,0.32 (45 of 65 blocks), K q8_0 / V f16, pool 262144 at one slot, worker steady at 23854/24576 MiB — is recorded once, in Pipeline Parallelism/Qwen3.8 PP Deployment §Production state, together with the split-limit arithmetic (why 45 blocks is the maximum), the KV-per-layer bill and the FA-all-quants rebuild the mismatch needed. That era's endpoint identity is an frank-dragoon era. This page keeps only the speculative-decoding measurements.

How MTP works on Qwen3.8 (bundled-head era)

The unsloth UD-* GGUFs keep the MTP (Multi-Token Prediction) head inside the model file. llama.cpp b10411+ auto-detects it and creates the draft context from the target model. Do not pass --spec-draft-model for those files. This is superseded: the production target of 2026-08-17 carried no bundled head, and the serve script passed the standalone draft sidecar with --spec-draft-model (shape: Pipeline Parallelism/Qwen3.8 PP Deployment).

The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild on Dankclaw is b10433 (9b05354ec) with GGML_HIP + gfx1151 + Vulkan.

History (superseded configurations, 2026-08-14 to 08-16)

All superseded configs used Qwen3.8-27B-UD-Q8_K_XL.gguf (29.29 GiB, bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the fallback layer under MTP. The Vulkan0 verdict from this era applies to the standalone configuration only; the PP head runs ROCm0.

2026-08-14: bundled MTP — draft-mtp, n_max 5, parallel 2, ctx 524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2 concurrent streams. No premature stop.
2026-08-15: DSpark sidecar — magnitudedev/Qwen3.8-27B-DSpark-GGUF (Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k prose prompts crashed the ROCm backend with ROCm error: an illegal memory access in common_speculative_impl_draft_dflash::draft (status 6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash block-diffusion machinery, which shares one draft context across parallel slots; concurrent block drafting faults in HIP stream sync. This is the same multi-slot bug class that retired draft-dflash (premature EOS, retired 2026-08-14). The crash shows on 16k concurrent streams, not on a 67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1 (requests queue safely). Retired the same day.
2026-08-15 to 08-16: --spec-default (ngram-mod), Vulkan0 — n_match 24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream test passes). Chosen for parallel safety after DSpark; kept as the fallback under MTP in the 2026-08-17 shape.

Benchmark Method

The matrix ran 54 configurations: 3 spec types (draft-mtp,ngram-mod, draft-mtp, ngram-mod) x --spec-draft-n-max 2..10 x --spec-draft-p-min {off, 0.8}. Each config loaded llama-server, then ran 3 generations. The tool is scripts/spec-matrix-bench.sh; analysis is scripts/analyze-spec-matrix.py.

Matrix Caveat: Benchmark Prompt

The matrix used a repetitive 1100-token prompt (scripts/bench_live.sh style). MTP drafts that repeated sentence almost perfectly, which inflated the numbers for long drafts. The matrix reported draft-mtp n_max=10 at ~30 t/s. A realistic story prompt gave very different results:

n_max	median t/s (realistic)
4	12.5
5	13.3 (peak)
6	10.6
7	10.1
10	8.6
none (baseline)	7.2

Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances acceptance against draft cost.

Premature Stop Investigation

Two failure patterns appeared during early production tests:

Speed collapse. ubatch >= 512 reduced MTP acceptance to ~25% and generation to ~9 t/s. ubatch 128 with batch >= 2048 triggered a premature instant-EOS under 2 concurrent slots. ubatch 256 avoids both: ~30 t/s single, concurrent-safe.
Apparent premature-EOS. The repetitive benchmark prompt with temp 1.0 sometimes stops after 1-2 tokens because the model legitimately emits an EOS on a wall of repeated text. This is sampling variance, not a corruption bug. Realistic prompts never stop early.

High-Context Backend Comparison (2026-08-15)

Tool: scripts/bench-high-ctx.sh <device> <ctx>. Both runs at ctx 262144 with a ~200k-token realistic-prose prompt, --spec-default, batch 8192 / ubatch 256, single slot, on scratch port 7892.

Device	Prompt tokens	Prefill (t/s)	Generation (t/s)	Wall (single req)
ROCm0	202822	119.1	3.83	1770 s
Vulkan0	200235	106.3	5.68	1929 s

Verdict: at very high context, Vulkan0 decodes 48% faster (5.68 vs 3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost (users stream tokens). This verdict applies to the standalone configuration; the PP head runs ROCm0 — HIP registers before Vulkan and the iGPU dedup rule (the llama.cpp PR #23897 workaround) keeps the first backend's device. Both tg_3s values were rock-steady across the run (3.83 and 5.67 t/s). High-context decode collapses on both backends versus ~25 t/s at 16k ctx; that is the long-context attention penalty, not a spec artifact.

Where the tunables live

That era's head and worker env/units are listed (and marked deleted) in Pipeline Parallelism/Qwen3.8 PP Deployment §Files. scripts/frank-dragoon.env is no longer a PP-head env at all — it is the halogen-flash-server trio of the current era, which frank-dragoon owns.

6.3 KiB Raw Blame History