Reorganize: llm/docs -> docs/LLM (LLM fleet knowledge base)
The LLM fleet documentation (deployment runbook, model runbooks, benchmarks, research archive, retired-topology history) joins the project knowledge base as docs/LLM/. Wikilinks resolve by filename, so no link updates are needed for the move.
This commit is contained in:
Frank committed
1 parent
918fe4eaec
commit
017d3704d2
36 files changed
No files matched your search
@@ -0,0 +1,158 @@
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
|
||||
> [deployment.md](deployment.md) for the active configuration.
|
||||
|
||||
# Qwen3.8-27B Speculative-Decoding Matrix
|
||||
|
||||
This page records the speculative-decoding benchmark history and the
|
||||
production tuning for the Qwen3.8-27B service (`frank-default-large`),
|
||||
now a pipeline-parallel head with a Dankserver worker.
|
||||
|
||||
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
|
||||
|
||||
Current production, verified against the live process and journal
|
||||
(2026-08-17, load 02:06 + smoke tests). Topology detail:
|
||||
[[Pipeline Parallelism/Qwen3.8 PP Deployment]].
|
||||
|
||||
- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
|
||||
passes no `--device` flag. Default selection puts RPC first and picks
|
||||
ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
|
||||
an option via `HEAD_BIN=.../build-vulkan-rpc`.
|
||||
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti, 22.43 GB,
|
||||
imatrix, 65 blocks, 866 tensors, NO bundled MTP head). The repo's
|
||||
standalone draft sidecar `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
|
||||
(3.16 GB) is loaded via `--spec-draft-model`; it runs on ROCm0 only
|
||||
(1718 MiB, zero worker impact).
|
||||
- Spec: **MTP** (standalone draft) `--spec-draft-n-max 4` + the
|
||||
`--spec-default` ngram-mod fallback (48/64/24). Measured acceptance
|
||||
0.83 then 0.75, mean len 4.0-4.3, per-position ~0.86/0.71/0.71/0.71 —
|
||||
stronger than the old bundled head (0.739/0.554/0.427/0.340).
|
||||
- KV: K **q8_0** + V **f16** (mismatched). 3136 B/token per FA layer =
|
||||
784 MiB at 262144 (K 1088 + V 2048). RPC0 8624 MiB (11 FA), ROCm0
|
||||
3920 MiB (5 FA); total 12544 = 16 x 784. **Flash Attention stays
|
||||
enabled with the mismatch** (`resolve_fused_ops: Flash Attention
|
||||
enabled`) via the GGML_CUDA_FA_ALL_QUANTS rebuild of both binaries
|
||||
at 9b05354ec — see the PP doc.
|
||||
- Split: `--tensor-split 0.68,0.32` = 45 of 65 blocks (0-44) on the
|
||||
3090, 69% of the layer weight. The maximum: 46 blocks -> ~0.3 GiB
|
||||
margin; 48 blocks (f >= 0.7197) adds FA block 47 -> KV 9408 -> OOM by
|
||||
~0.85 GiB. Boundary math: the PP doc.
|
||||
- Context: pool 262144, 1 slot, `n_ctx_slot = 262144`. The pi
|
||||
`models.json` declares 262144 (contextWindow = maxTokens, house
|
||||
convention).
|
||||
- Worker steady state: 23854/24576 MiB used on the 3090, ~273 MiB free
|
||||
(tight but stable; all allocations happen at load).
|
||||
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
|
||||
`--gpu-layers all`.
|
||||
|
||||
## How MTP works on Qwen3.8 (bundled-head era)
|
||||
|
||||
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
|
||||
the model file. llama.cpp b10411+ auto-detects it and creates the draft
|
||||
context from the target model. Do not pass `--spec-draft-model` for those
|
||||
files. This is superseded: the production `Qwen3.8-27B-Uncensored-Q6_K.gguf`
|
||||
has NO bundled head, and the serve script passes the standalone draft
|
||||
sidecar with `--spec-draft-model` (see Production state above).
|
||||
|
||||
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
|
||||
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.
|
||||
|
||||
## History (superseded configurations, 2026-08-14 to 08-16)
|
||||
|
||||
All superseded configs used `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB,
|
||||
bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the
|
||||
fallback layer under MTP. The Vulkan0 verdict from this era applies to the
|
||||
standalone configuration only; the PP head runs ROCm0.
|
||||
|
||||
- **2026-08-14: bundled MTP** — `draft-mtp`, n_max 5, parallel 2, ctx
|
||||
524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2
|
||||
concurrent streams. No premature stop.
|
||||
- **2026-08-15: DSpark sidecar** — `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
|
||||
(Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k
|
||||
prose prompts crashed the ROCm backend with `ROCm error: an illegal
|
||||
memory access` in `common_speculative_impl_draft_dflash::draft` (status
|
||||
6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash
|
||||
block-diffusion machinery, which shares one draft context across
|
||||
parallel slots; concurrent block drafting faults in HIP stream sync.
|
||||
This is the same multi-slot bug class that retired `draft-dflash`
|
||||
(premature EOS). The crash shows on 16k concurrent streams, not on a
|
||||
67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1
|
||||
(requests queue safely). Retired the same day.
|
||||
- **2026-08-15 to 08-16: `--spec-default` (ngram-mod), Vulkan0** — n_match
|
||||
24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream
|
||||
test passes). Chosen for parallel safety after DSpark; kept as the
|
||||
fallback under MTP in the current config.
|
||||
|
||||
## Benchmark Method
|
||||
|
||||
The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`,
|
||||
`draft-mtp`, `ngram-mod`) x `--spec-draft-n-max` 2..10 x
|
||||
`--spec-draft-p-min` {off, 0.8}. Each config loaded `llama-server`,
|
||||
then ran 3 generations. The tool is `scripts/spec-matrix-bench.sh`;
|
||||
analysis is `scripts/analyze-spec-matrix.py`.
|
||||
|
||||
## Matrix Caveat: Benchmark Prompt
|
||||
|
||||
The matrix used a repetitive 1100-token prompt
|
||||
(`scripts/bench_live.sh` style). MTP drafts that repeated sentence almost
|
||||
perfectly, which inflated the numbers for long drafts. The matrix reported
|
||||
`draft-mtp` n_max=10 at ~30 t/s. A realistic story prompt gave very
|
||||
different results:
|
||||
|
||||
| n_max | median t/s (realistic) |
|
||||
| ----- | ----------------------- |
|
||||
| 4 | 12.5 |
|
||||
| 5 | 13.3 (peak) |
|
||||
| 6 | 10.6 |
|
||||
| 7 | 10.1 |
|
||||
| 10 | 8.6 |
|
||||
| none (baseline) | 7.2 |
|
||||
|
||||
Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances
|
||||
acceptance against draft cost.
|
||||
|
||||
## Premature Stop Investigation
|
||||
|
||||
Two failure patterns appeared during early production tests:
|
||||
|
||||
1. Speed collapse. `ubatch >= 512` reduced MTP acceptance to ~25% and
|
||||
generation to ~9 t/s. `ubatch 128` with `batch >= 2048` triggered a
|
||||
premature instant-EOS under 2 concurrent slots. `ubatch 256` avoids
|
||||
both: ~30 t/s single, concurrent-safe.
|
||||
2. Apparent premature-EOS. The repetitive benchmark prompt with `temp 1.0`
|
||||
sometimes stops after 1-2 tokens because the model legitimately emits an
|
||||
EOS on a wall of repeated text. This is sampling variance, not a
|
||||
corruption bug. Realistic prompts never stop early.
|
||||
|
||||
## High-Context Backend Comparison (2026-08-15)
|
||||
|
||||
Tool: `scripts/bench-high-ctx.sh <device> <ctx>`. Both runs at ctx 262144
|
||||
with a ~200k-token realistic-prose prompt, `--spec-default`, batch 8192 /
|
||||
ubatch 256, single slot, on scratch port 7892.
|
||||
|
||||
| Device | Prompt tokens | Prefill (t/s) | Generation (t/s) | Wall (single req) |
|
||||
| ------ | ------------- | ------------- | ---------------- | ----------------- |
|
||||
| ROCm0 | 202822 | **119.1** | 3.83 | **1770 s** |
|
||||
| Vulkan0 | 200235 | 106.3 | **5.68** | 1929 s |
|
||||
|
||||
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
|
||||
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
|
||||
(users stream tokens). This verdict applies to the standalone
|
||||
configuration; the PP head runs ROCm0 (HIP registers before Vulkan).
|
||||
Both `tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
|
||||
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
|
||||
that is the long-context attention penalty, not a spec artifact.
|
||||
|
||||
## Production Configuration
|
||||
|
||||
The live tunables live in `scripts/frank-default-large.env` (head) and
|
||||
`scripts/frank-pp-worker.env` (worker) in this repo. The Production
|
||||
state section at the top of this page records the current values: Q6_K
|
||||
model + standalone MTP draft, split 0.68/0.32, 262144 ctx, PARALLEL=1,
|
||||
q8_0/f16 KV.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-default-large-benchmark]]
|
||||
- [[ling-3.0-flash-trial]]
|
||||
- [[deployment]]
|
||||
Reference in new issue
Block a user