docs: record --spec-default + Vulkan0 + PARALLEL=2 final state; 262k-ctx ROCm-vs-Vulkan bench
This commit is contained in:
Frank
@@ -2,7 +2,7 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with a DSpark sidecar draft (`draft-dspark`, single parallel slot, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (DSpark single-slot, 2026-08-15; DFlash retired earlier for premature EOS).
|
||||
This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with llama.cpp's `--spec-default` (ngram-mod, no draft model, Vulkan0 backend, 2 parallel slots, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (--spec-default + Vulkan0, 2026-08-15; DSpark retired same day for a ROCm parallel crash; DFlash retired earlier for premature EOS).
|
||||
|
||||
## Benchmarking Policy (MANDATORY)
|
||||
|
||||
@@ -61,7 +61,7 @@ Target map:
|
||||
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankclaw | dank@dankclaw | llama.cpp (UD-Q8_K_XL + DSpark sidecar, `draft-dspark`, 1 slot) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | llama.cpp (UD-Q8_K_XL, `--spec-default` ngram-mod, Vulkan0, 2 slots) | frank-default-large | 7770 |
|
||||
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 |
|
||||
|
||||
## Runtime & Troubleshooting
|
||||
@@ -108,7 +108,7 @@ graph TD
|
||||
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
|
||||
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
|
||||
- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding.
|
||||
- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also **single-slot only** on this box: two concurrent 16k prose prompts crash the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md).
|
||||
- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also unsafe here: two concurrent 16k prose prompts crashed the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). Production uses `--spec-default` (ngram-mod, draftless) at PARALLEL=2 on the Vulkan0 backend. See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md).
|
||||
|
||||
### Dankclaw memory model
|
||||
|
||||
|
||||
@@ -197,18 +197,17 @@ docker run --rm --name vllm-qwen36-dflash \
|
||||
|
||||
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
|
||||
|
||||
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with a **DSpark sidecar draft** on the ROCm (HIP) backend.
|
||||
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with **no dedicated draft** on the **Vulkan0** backend.
|
||||
|
||||
Current facts (2026-08-15):
|
||||
|
||||
- Main model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (bundled MTP, unused).
|
||||
- Draft model: `magnitudedev/Qwen3.8-27B-DSpark-GGUF` `Qwen3.8-27B-DSpark-Q8_0.gguf` (DSpark, 1.36B, block_size 7, loaded as a sidecar via `--spec-draft-model`).
|
||||
- Spec command: `--spec-type draft-dspark` with `--spec-draft-n-max 7`.
|
||||
- **PARALLEL=1.** DSpark multi-slot crashes the ROCm backend (`ROCm error: an illegal memory access`) under two concurrent 16k prose prompts. See [[qwen3.8-spec-matrix]] for the incident.
|
||||
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V.
|
||||
- Context: 1 slot x 262144; batch 8192 / ubatch 256.
|
||||
- Spec: llama.cpp **`--spec-default`** preset = ngram-mod (n_match 24 / n_min 48 / n_max 64). No draft model.
|
||||
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes 5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm wins prefill 119 vs 106 t/s. See [[qwen3.8-spec-matrix]] for the bench table.
|
||||
- **PARALLEL=2** — passes the 2x32k parallel stream test.
|
||||
- KV `q8_0` K / `q8_0` V; context 2 x 262144; batch 8192 / ubatch 256.
|
||||
|
||||
History: before DSpark, the box ran bundled MTP (`draft-mtp`, auto-detected b10411+, retired 2026-08-15). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) and the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
|
||||
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
+14
-15
@@ -139,22 +139,21 @@ curl -sf http://127.0.0.1:7770/health
|
||||
|
||||
The active service on Dankclaw runs llama.cpp serving the unsloth
|
||||
Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port
|
||||
7770. It uses a **DSpark sidecar draft model** (`draft-dspark`,
|
||||
`magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max 7`)
|
||||
for speculative decoding on the **ROCm (HIP)** backend, with **1 parallel
|
||||
slot** at 262144 context, KV cache `q8_0` K / `q8_0` V on GPU, full layer
|
||||
offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 256.
|
||||
The unit file is `scripts/frank-default-large.service`. It launches through
|
||||
`scripts/serve-llama-frank-large.sh`, which reads tunables from
|
||||
`scripts/frank-default-large.env`.
|
||||
7770. It uses llama.cpp's **`--spec-default`** preset (ngram-mod, no draft
|
||||
model) for speculative decoding on the **Vulkan0** backend, with **2
|
||||
parallel slots** at 262144 context each, KV cache `q8_0` K / `q8_0` V on
|
||||
GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 /
|
||||
ubatch 256. The unit file is `scripts/frank-default-large.service`. It
|
||||
launches through `scripts/serve-llama-frank-large.sh`, which reads tunables
|
||||
from `scripts/frank-default-large.env`.
|
||||
|
||||
> Note: DSpark replaced **bundled MTP** (`draft-mtp`, in-model, auto-detected
|
||||
> b10411+) on 2026-08-15 at the user's request. **DSpark is single-slot
|
||||
> only**: two concurrent 16k prose prompts crash the ROCm backend with
|
||||
> `ROCm error: an illegal memory access` (status 6/ABRT) during per-sequence
|
||||
> block sampling, so `PARALLEL=1`. DSpark is the semi-autoregressive
|
||||
> extension of DFlash; DFlash was previously retired here for a
|
||||
> shared-draft-context premature-EOS bug. See [[qwen3.8-spec-matrix]].
|
||||
> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
|
||||
> prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) →
|
||||
> `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went
|
||||
> ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster
|
||||
> (5.68 vs 3.83 t/s). See [[qwen3.8-spec-matrix]] for both decisions and
|
||||
> the bench table. Also retired earlier: bundled MTP, the vLLM-ROCm CIRU
|
||||
> INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw.
|
||||
|
||||
### Dankserver — frank-default-small
|
||||
|
||||
|
||||
@@ -6,20 +6,22 @@ llama.cpp build on that machine (upstream, b10133+).
|
||||
|
||||
## Summary
|
||||
|
||||
- **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline.
|
||||
- **Spec (2026-08-15)**: DSpark sidecar draft (`draft-dspark`,
|
||||
`magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max
|
||||
7`). Replaced bundled MTP. **Single-slot (PARALLEL=1)** because DSpark
|
||||
crashes the ROCm backend under two concurrent 16k prompts (see
|
||||
- **Backend**: **Vulkan0** (2026-08-15). At 262k ctx Vulkan decodes 48%
|
||||
faster than ROCm (5.68 vs 3.83 t/s); ROCm only wins prefill 119 vs 106
|
||||
t/s. Generation is the interactive cost, so Vulkan0.
|
||||
- **Spec (2026-08-15)**: llama.cpp `--spec-default` preset (ngram-mod,
|
||||
n_match 24 / n_min 48 / n_max 64, draftless). Replaced DSpark, which
|
||||
crashed the ROCm backend under two concurrent 16k prompts (see
|
||||
[[qwen3.8-spec-matrix]]).
|
||||
- **Context**: 1 slot x 262144.
|
||||
- **Parallel**: 2 slots — passes the 2x32k parallel stream test.
|
||||
- **Context**: 2 slots x 262144.
|
||||
- **KV**: `q8_0` K / `q8_0` V on GPU.
|
||||
- **Batch**: 8192 / ubatch 256.
|
||||
|
||||
> The A/B tables below are retained for history but predate the 2026-08-14
|
||||
> Qwen3.8-27B `UD-Q8_K_XL` switch to bundled MTP and the 2026-08-15 DSpark
|
||||
> switch. The current production spec decision and the DSpark parallel crash
|
||||
> are recorded on [[qwen3.8-spec-matrix]].
|
||||
> Qwen3.8-27B `UD-Q8_K_XL` switch. The current production spec decision,
|
||||
> the DSpark parallel crash, and the high-context ROCm-vs-Vulkan bench are
|
||||
> recorded on [[qwen3.8-spec-matrix]].
|
||||
|
||||
## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)
|
||||
|
||||
|
||||
+3
-3
@@ -17,10 +17,10 @@ MoE experts on CPU, port 7770]
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
|
||||
E[llama.cpp DSpark sidecar draft
|
||||
E[llama.cpp --spec-default ngram-mod
|
||||
frank-default-large]
|
||||
F[Qwen3.8-27B UD-Q8_K_XL
|
||||
DSpark Q8_0 draft, port 7770, PARALLEL=1]
|
||||
Vulkan0 backend, port 7770, PARALLEL=2]
|
||||
D --> E --> F
|
||||
end
|
||||
|
||||
@@ -52,7 +52,7 @@ DSpark Q8_0 draft, port 7770, PARALLEL=1]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = DSpark sidecar draft, PARALLEL=1 (2026-08-15 DSpark ROCm parallel crash)
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = --spec-default (ngram-mod), PARALLEL=2, Vulkan0 backend (2026-08-15)
|
||||
- [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
|
||||
- [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
|
||||
- [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings
|
||||
|
||||
@@ -4,7 +4,21 @@ This page records the speculative-decoding benchmark and the production
|
||||
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw
|
||||
(`frank-default-large`).
|
||||
|
||||
## Result (applied, 2026-08-15: DSpark)
|
||||
## Result (applied, 2026-08-15: --spec-default)
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24,
|
||||
n_min 48, n_max 64). No draft model. Chosen 2026-08-15 after DSpark
|
||||
crashed under parallel load (see below).
|
||||
- **PARALLEL=2** — passes the 2x32k parallel stream test (both streams
|
||||
complete 256/256 concurrently, no crash). ngram-mod is draftless and
|
||||
parallel-safe.
|
||||
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes
|
||||
5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm only wins prefill (119 vs
|
||||
106 t/s). See the high-context bench table below.
|
||||
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
|
||||
|
||||
## Result (superseded 2026-08-15: DSpark)
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
|
||||
@@ -13,6 +27,7 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw
|
||||
- **PARALLEL=1.** Two concurrent 16k prose prompts crashed the ROCm backend
|
||||
with `ROCm error: an illegal memory access` during per-sequence block
|
||||
sampling. See the DSpark parallel incident below.
|
||||
- Retired the same day in favor of `--spec-default`.
|
||||
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
|
||||
|
||||
## Result (superseded 2026-08-14: bundled MTP)
|
||||
@@ -118,24 +133,42 @@ Two failure patterns appeared during early production tests:
|
||||
EOS on a wall of repeated text. This is sampling variance, not a
|
||||
corruption bug. Realistic prompts never stop early.
|
||||
|
||||
## High-Context Backend Comparison (2026-08-15)
|
||||
|
||||
Tool: `scripts/bench-high-ctx.sh <device> <ctx>`. Both runs at ctx 262144
|
||||
with a ~200k-token realistic-prose prompt, `--spec-default`, batch 8192 /
|
||||
ubatch 256, single slot, on scratch port 7892.
|
||||
|
||||
| Device | Prompt tokens | Prefill (t/s) | Generation (t/s) | Wall (single req) |
|
||||
| ------ | ------------- | ------------- | ---------------- | ----------------- |
|
||||
| ROCm0 | 202822 | **119.1** | 3.83 | **1770 s** |
|
||||
| Vulkan0 | 200235 | 106.3 | **5.68** | 1929 s |
|
||||
|
||||
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
|
||||
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
|
||||
(users stream tokens), so **Vulkan0** is the production backend. Both
|
||||
`tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
|
||||
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
|
||||
that is the long-context attention penalty, not a spec artifact.
|
||||
|
||||
## Production Configuration
|
||||
|
||||
`scripts/frank-default-large.env`:
|
||||
|
||||
```ini
|
||||
SPEC_TYPE=draft-dspark
|
||||
DRAFT_FILE=Qwen3.8-27B-DSpark-Q8_0.gguf
|
||||
SPEC_DRAFT_N_MAX=7
|
||||
SPEC_DRAFT_P_MIN=
|
||||
SPEC_DEFAULT=1 # llama.cpp preset = ngram-mod (48/64/24)
|
||||
SPEC_TYPE=
|
||||
DRAFT_FILE=
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=256
|
||||
CTX_SIZE=524288
|
||||
PARALLEL=1 # DSpark multi-slot crashes on this box (2026-08-15)
|
||||
PARALLEL=2 # passes 2x32k parallel stream test (2026-08-15)
|
||||
DEVICE=Vulkan0 # 48% faster decode at 262k ctx
|
||||
```
|
||||
|
||||
Window: computed at 262144 per slot (the pi `models.json` contextWindow for
|
||||
`frank-default-large`). The unit reports `n_ctx_slot = 262144`; CTX_SIZE is
|
||||
a pool ceiling and per-slot stays 262144 regardless of the single slot.
|
||||
a pool ceiling and per-slot stays 262144 regardless of the slot count.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
Reference in New Issue
Block a user