dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record --spec-default + Vulkan0 + PARALLEL=2 final state; 262k-ctx ROCm-vs-Vulkan bench

Browse Source
This commit is contained in:
Frank
2026-08-15 17:13:18 -07:00
parent 5296cf11d7
commit b6f2ee2945
6 changed files with 77 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/DFlash DSpark Speculative Decoding.md
+6 -7
View File
@@ -197,18 +197,17 @@ docker run --rm --name vllm-qwen36-dflash \
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with a **DSpark sidecar draft** on the ROCm (HIP) backend.
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with **no dedicated draft** on the **Vulkan0** backend.
Current facts (2026-08-15):
- Main model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (bundled MTP, unused).
- Draft model: `magnitudedev/Qwen3.8-27B-DSpark-GGUF` `Qwen3.8-27B-DSpark-Q8_0.gguf` (DSpark, 1.36B, block_size 7, loaded as a sidecar via `--spec-draft-model`).
- Spec command: `--spec-type draft-dspark` with `--spec-draft-n-max 7`.
- **PARALLEL=1.** DSpark multi-slot crashes the ROCm backend (`ROCm error: an illegal memory access`) under two concurrent 16k prose prompts. See [[qwen3.8-spec-matrix]] for the incident.
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V.
- Context: 1 slot x 262144; batch 8192 / ubatch 256.
- Spec: llama.cpp **`--spec-default`** preset = ngram-mod (n_match 24 / n_min 48 / n_max 64). No draft model.
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes 5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm wins prefill 119 vs 106 t/s. See [[qwen3.8-spec-matrix]] for the bench table.
- **PARALLEL=2** — passes the 2x32k parallel stream test.
- KV `q8_0` K / `q8_0` V; context 2 x 262144; batch 8192 / ubatch 256.
History: before DSpark, the box ran bundled MTP (`draft-mtp`, auto-detected b10411+, retired 2026-08-15). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) and the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
## Notes
docs/deployment.md
+14 -15
View File
@@ -139,22 +139,21 @@ curl -sf http://127.0.0.1:7770/health
The active service on Dankclaw runs llama.cpp serving the unsloth
Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port
7770. It uses a **DSpark sidecar draft model** (`draft-dspark`,
`magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max 7`)
for speculative decoding on the **ROCm (HIP)** backend, with **1 parallel
slot** at 262144 context, KV cache `q8_0` K / `q8_0` V on GPU, full layer
offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 256.
The unit file is `scripts/frank-default-large.service`. It launches through
`scripts/serve-llama-frank-large.sh`, which reads tunables from
`scripts/frank-default-large.env`.
7770. It uses llama.cpp's **`--spec-default`** preset (ngram-mod, no draft
model) for speculative decoding on the **Vulkan0** backend, with **2
parallel slots** at 262144 context each, KV cache `q8_0` K / `q8_0` V on
GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 /
ubatch 256. The unit file is `scripts/frank-default-large.service`. It
launches through `scripts/serve-llama-frank-large.sh`, which reads tunables
from `scripts/frank-default-large.env`.
> Note: DSpark replaced **bundled MTP** (`draft-mtp`, in-model, auto-detected
> b10411+) on 2026-08-15 at the user's request. **DSpark is single-slot
> only**: two concurrent 16k prose prompts crash the ROCm backend with
> `ROCm error: an illegal memory access` (status 6/ABRT) during per-sequence
> block sampling, so `PARALLEL=1`. DSpark is the semi-autoregressive
> extension of DFlash; DFlash was previously retired here for a
> shared-draft-context premature-EOS bug. See [[qwen3.8-spec-matrix]].
> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
> prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) →
> `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went
> ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster
> (5.68 vs 3.83 t/s). See [[qwen3.8-spec-matrix]] for both decisions and
> the bench table. Also retired earlier: bundled MTP, the vLLM-ROCm CIRU
> INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw.
### Dankserver — frank-default-small
docs/frank-default-large-benchmark.md
+11 -9
View File
@@ -6,20 +6,22 @@ llama.cpp build on that machine (upstream, b10133+).
## Summary
- **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline.
- **Spec (2026-08-15)**: DSpark sidecar draft (`draft-dspark`,
`magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max
7`). Replaced bundled MTP. **Single-slot (PARALLEL=1)** because DSpark
crashes the ROCm backend under two concurrent 16k prompts (see
- **Backend**: **Vulkan0** (2026-08-15). At 262k ctx Vulkan decodes 48%
faster than ROCm (5.68 vs 3.83 t/s); ROCm only wins prefill 119 vs 106
t/s. Generation is the interactive cost, so Vulkan0.
- **Spec (2026-08-15)**: llama.cpp `--spec-default` preset (ngram-mod,
n_match 24 / n_min 48 / n_max 64, draftless). Replaced DSpark, which
crashed the ROCm backend under two concurrent 16k prompts (see
[[qwen3.8-spec-matrix]]).
- **Context**: 1 slot x 262144.
- **Parallel**: 2 slots — passes the 2x32k parallel stream test.
- **Context**: 2 slots x 262144.
- **KV**: `q8_0` K / `q8_0` V on GPU.
- **Batch**: 8192 / ubatch 256.
> The A/B tables below are retained for history but predate the 2026-08-14
> Qwen3.8-27B `UD-Q8_K_XL` switch to bundled MTP and the 2026-08-15 DSpark
> switch. The current production spec decision and the DSpark parallel crash
> are recorded on [[qwen3.8-spec-matrix]].
> Qwen3.8-27B `UD-Q8_K_XL` switch. The current production spec decision,
> the DSpark parallel crash, and the high-context ROCm-vs-Vulkan bench are
> recorded on [[qwen3.8-spec-matrix]].
## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)
docs/index.md
+3 -3
View File
@@ -17,10 +17,10 @@ MoE experts on CPU, port 7770]
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
E[llama.cpp DSpark sidecar draft
E[llama.cpp --spec-default ngram-mod
frank-default-large]
F[Qwen3.8-27B UD-Q8_K_XL
DSpark Q8_0 draft, port 7770, PARALLEL=1]
Vulkan0 backend, port 7770, PARALLEL=2]
D --> E --> F
end
@@ -52,7 +52,7 @@ DSpark Q8_0 draft, port 7770, PARALLEL=1]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
- [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = DSpark sidecar draft, PARALLEL=1 (2026-08-15 DSpark ROCm parallel crash)
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = --spec-default (ngram-mod), PARALLEL=2, Vulkan0 backend (2026-08-15)
- [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
- [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
- [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings
docs/qwen3.8-spec-matrix.md
+40 -7
View File
@@ -4,7 +4,21 @@ This page records the speculative-decoding benchmark and the production
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw
(`frank-default-large`).
## Result (applied, 2026-08-15: DSpark)
## Result (applied, 2026-08-15: --spec-default)
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
- Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24,
n_min 48, n_max 64). No draft model. Chosen 2026-08-15 after DSpark
crashed under parallel load (see below).
- **PARALLEL=2** — passes the 2x32k parallel stream test (both streams
complete 256/256 concurrently, no crash). ngram-mod is draftless and
parallel-safe.
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes
5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm only wins prefill (119 vs
106 t/s). See the high-context bench table below.
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
## Result (superseded 2026-08-15: DSpark)
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
- Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
@@ -13,6 +27,7 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw
- **PARALLEL=1.** Two concurrent 16k prose prompts crashed the ROCm backend
with `ROCm error: an illegal memory access` during per-sequence block
sampling. See the DSpark parallel incident below.
- Retired the same day in favor of `--spec-default`.
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
## Result (superseded 2026-08-14: bundled MTP)
@@ -118,24 +133,42 @@ Two failure patterns appeared during early production tests:
EOS on a wall of repeated text. This is sampling variance, not a
corruption bug. Realistic prompts never stop early.
## High-Context Backend Comparison (2026-08-15)
Tool: `scripts/bench-high-ctx.sh <device> <ctx>`. Both runs at ctx 262144
with a ~200k-token realistic-prose prompt, `--spec-default`, batch 8192 /
ubatch 256, single slot, on scratch port 7892.
| Device | Prompt tokens | Prefill (t/s) | Generation (t/s) | Wall (single req) |
| ------ | ------------- | ------------- | ---------------- | ----------------- |
| ROCm0 | 202822 | **119.1** | 3.83 | **1770 s** |
| Vulkan0 | 200235 | 106.3 | **5.68** | 1929 s |
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
(users stream tokens), so **Vulkan0** is the production backend. Both
`tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
that is the long-context attention penalty, not a spec artifact.
## Production Configuration
`scripts/frank-default-large.env`:
```ini
SPEC_TYPE=draft-dspark
DRAFT_FILE=Qwen3.8-27B-DSpark-Q8_0.gguf
SPEC_DRAFT_N_MAX=7
SPEC_DRAFT_P_MIN=
SPEC_DEFAULT=1 # llama.cpp preset = ngram-mod (48/64/24)
SPEC_TYPE=
DRAFT_FILE=
BATCH_SIZE=8192
UBATCH_SIZE=256
CTX_SIZE=524288
PARALLEL=1 # DSpark multi-slot crashes on this box (2026-08-15)
PARALLEL=2 # passes 2x32k parallel stream test (2026-08-15)
DEVICE=Vulkan0 # 48% faster decode at 262k ctx
```
Window: computed at 262144 per slot (the pi `models.json` contextWindow for
`frank-default-large`). The unit reports `n_ctx_slot = 262144`; CTX_SIZE is
a pool ceiling and per-slot stays 262144 regardless of the single slot.
a pool ceiling and per-slot stays 262144 regardless of the slot count.
## See Also