docs: record DSpark single-slot switch + ROCm parallel crash (2026-08-15) across KB; refs AGENTS/deployment/benchmark/matrix/index
This commit is contained in:
Frank committed
1 parent
d04cc69e09
commit
020e874a5c
6 files changed
+108
-25
No files matched your search
@@ -2,7 +2,7 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.6-27B (HauhauCS uncensored Q8_K_P) with an MTP sidecar drafter (`draft-mtp`, 3 parallel slots, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md) for the spec decision (DFlash retired 2026-08-14 due to a parallel premature-EOS bug).
|
||||
This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with a DSpark sidecar draft (`draft-dspark`, single parallel slot, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (DSpark single-slot, 2026-08-15; DFlash retired earlier for premature EOS).
|
||||
|
||||
## Benchmarking Policy (MANDATORY)
|
||||
|
||||
@@ -61,7 +61,7 @@ Target map:
|
||||
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankclaw | dank@dankclaw | llama.cpp (Q8_K_P + MTP sidecar, `draft-mtp`, 3 slots) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | llama.cpp (UD-Q8_K_XL + DSpark sidecar, `draft-dspark`, 1 slot) | frank-default-large | 7770 |
|
||||
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 |
|
||||
|
||||
## Runtime & Troubleshooting
|
||||
@@ -108,7 +108,7 @@ graph TD
|
||||
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
|
||||
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
|
||||
- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding.
|
||||
- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw; use `draft-mtp` (n_max 3) or `ngram-mod`, both parallel-safe. See [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md).
|
||||
- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also **single-slot only** on this box: two concurrent 16k prose prompts crash the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md).
|
||||
|
||||
### Dankclaw memory model
|
||||
|
||||
|
||||
@@ -197,17 +197,18 @@ docker run --rm --name vllm-qwen36-dflash \
|
||||
|
||||
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
|
||||
|
||||
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with an **MTP sidecar draft** on the ROCm (HIP) backend. DFlash was retired on 2026-08-14.
|
||||
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with a **DSpark sidecar draft** on the ROCm (HIP) backend.
|
||||
|
||||
Current facts:
|
||||
Current facts (2026-08-15):
|
||||
|
||||
- Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive `Q8_K_P` (dense 27B).
|
||||
- Draft model: a4lg/Qwen3.6-27B-MTP-ONLY `Q8_0` (MTP head from the base model, loaded as a sidecar).
|
||||
- Spec command: `--spec-type draft-mtp` with `--spec-draft-n-max 3`.
|
||||
- Main model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (bundled MTP, unused).
|
||||
- Draft model: `magnitudedev/Qwen3.8-27B-DSpark-GGUF` `Qwen3.8-27B-DSpark-Q8_0.gguf` (DSpark, 1.36B, block_size 7, loaded as a sidecar via `--spec-draft-model`).
|
||||
- Spec command: `--spec-type draft-dspark` with `--spec-draft-n-max 7`.
|
||||
- **PARALLEL=1.** DSpark multi-slot crashes the ROCm backend (`ROCm error: an illegal memory access`) under two concurrent 16k prose prompts. See [[qwen3.8-spec-matrix]] for the incident.
|
||||
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V.
|
||||
- Context: 3 x 200700 = 602100; batch 8192 / ubatch 8192.
|
||||
- Context: 1 slot x 262144; batch 8192 / ubatch 256.
|
||||
|
||||
Why DFlash was retired: DFlash causes premature EOS on a generating slot when another task lands with high prompt-cache reuse (LCP ~0.99). See [[frank-default-large-benchmark]] for the incident writeup and the MTP-vs-ngram A/B. The DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) remains in `/home/dank/models/` as reference only.
|
||||
History: before DSpark, the box ran bundled MTP (`draft-mtp`, auto-detected b10411+, retired 2026-08-15). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) and the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
+17
-2
@@ -137,9 +137,24 @@ curl -sf http://127.0.0.1:7770/health
|
||||
|
||||
### Dankclaw — frank-default-large
|
||||
|
||||
The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses an **MTP sidecar drafter** (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `q8_0` V on GPU (pairing required for Flash Attention on this build), full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 8192 (peak ~65.5 GiB VRAM). The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`.
|
||||
The active service on Dankclaw runs llama.cpp serving the unsloth
|
||||
Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port
|
||||
7770. It uses a **DSpark sidecar draft model** (`draft-dspark`,
|
||||
`magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max 7`)
|
||||
for speculative decoding on the **ROCm (HIP)** backend, with **1 parallel
|
||||
slot** at 262144 context, KV cache `q8_0` K / `q8_0` V on GPU, full layer
|
||||
offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 256.
|
||||
The unit file is `scripts/frank-default-large.service`. It launches through
|
||||
`scripts/serve-llama-frank-large.sh`, which reads tunables from
|
||||
`scripts/frank-default-large.env`.
|
||||
|
||||
> Note: MTP replaced DFlash (`draft-dflash`) on 2026-08-14. DFlash retired because it releases a generating slot prematurely under parallel load when another task lands with high prompt-cache reuse. See [[frank-default-large-benchmark]]. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the llama-serve/vLLM units on Dankclaw.
|
||||
> Note: DSpark replaced **bundled MTP** (`draft-mtp`, in-model, auto-detected
|
||||
> b10411+) on 2026-08-15 at the user's request. **DSpark is single-slot
|
||||
> only**: two concurrent 16k prose prompts crash the ROCm backend with
|
||||
> `ROCm error: an illegal memory access` (status 6/ABRT) during per-sequence
|
||||
> block sampling, so `PARALLEL=1`. DSpark is the semi-autoregressive
|
||||
> extension of DFlash; DFlash was previously retired here for a
|
||||
> shared-draft-context premature-EOS bug. See [[qwen3.8-spec-matrix]].
|
||||
|
||||
### Dankserver — frank-default-small
|
||||
|
||||
|
||||
@@ -7,10 +7,19 @@ llama.cpp build on that machine (upstream, b10133+).
|
||||
## Summary
|
||||
|
||||
- **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline.
|
||||
- **Spec (2026-08-14)**: MTP sidecar drafter (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`). Replaced DFlash, which was retired because it releases a generating slot prematurely under parallel load (see below).
|
||||
- **Context**: 3 parallel slots x 200700 (pool 602100).
|
||||
- **Spec (2026-08-15)**: DSpark sidecar draft (`draft-dspark`,
|
||||
`magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max
|
||||
7`). Replaced bundled MTP. **Single-slot (PARALLEL=1)** because DSpark
|
||||
crashes the ROCm backend under two concurrent 16k prompts (see
|
||||
[[qwen3.8-spec-matrix]]).
|
||||
- **Context**: 1 slot x 262144.
|
||||
- **KV**: `q8_0` K / `q8_0` V on GPU.
|
||||
- **Batch**: 8192 / ubatch 8192 (2026-08-14, uses the VRAM headroom; peaks ~65.5 GiB of 96 GiB).
|
||||
- **Batch**: 8192 / ubatch 256.
|
||||
|
||||
> The A/B tables below are retained for history but predate the 2026-08-14
|
||||
> Qwen3.8-27B `UD-Q8_K_XL` switch to bundled MTP and the 2026-08-15 DSpark
|
||||
> switch. The current production spec decision and the DSpark parallel crash
|
||||
> are recorded on [[qwen3.8-spec-matrix]].
|
||||
|
||||
## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)
|
||||
|
||||
|
||||
+4
-4
@@ -17,10 +17,10 @@ MoE experts on CPU, port 7770]
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
|
||||
E[llama.cpp MTP sidecar
|
||||
E[llama.cpp DSpark sidecar draft
|
||||
frank-default-large]
|
||||
F[Qwen3.6-27B uncensored Q8_K_P
|
||||
MTP-ONLY draft, port 7770]
|
||||
F[Qwen3.8-27B UD-Q8_K_XL
|
||||
DSpark Q8_0 draft, port 7770, PARALLEL=1]
|
||||
D --> E --> F
|
||||
end
|
||||
|
||||
@@ -52,7 +52,7 @@ MTP-ONLY draft, port 7770]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL bundled-MTP spec matrix; winner draft-mtp n_max=5
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = DSpark sidecar draft, PARALLEL=1 (2026-08-15 DSpark ROCm parallel crash)
|
||||
- [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
|
||||
- [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
|
||||
- [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings
|
||||
|
||||
@@ -4,7 +4,18 @@ This page records the speculative-decoding benchmark and the production
|
||||
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw
|
||||
(`frank-default-large`).
|
||||
|
||||
## Result (applied)
|
||||
## Result (applied, 2026-08-15: DSpark)
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
|
||||
(`Qwen3.8-27B-DSpark-Q8_0.gguf`, 1.36B, block_size 7), `--spec-type
|
||||
draft-dspark`, `--spec-draft-n-max 7` (clamps to block_size).
|
||||
- **PARALLEL=1.** Two concurrent 16k prose prompts crashed the ROCm backend
|
||||
with `ROCm error: an illegal memory access` during per-sequence block
|
||||
sampling. See the DSpark parallel incident below.
|
||||
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
|
||||
|
||||
## Result (superseded 2026-08-14: bundled MTP)
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: `draft-mtp`, `--spec-draft-n-max 5`, no `--spec-draft-p-min`.
|
||||
@@ -12,6 +23,51 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw
|
||||
- Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under
|
||||
2 concurrent streams. No premature stop.
|
||||
|
||||
## Incident: DSpark ROCm Crash Under Parallel Load (2026-08-15)
|
||||
|
||||
A two-stream parallel acceptance test crashed `frank-default-large`.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Two concurrent 16k prose prompts<br>high prompt-cache reuse] --> B[ROCm illegal memory access<br>in common_speculative_impl_draft_dflash::draft]
|
||||
B --> C[status 6/ABRT core dump]
|
||||
C --> D[systemd auto-restart, 503 during reload]
|
||||
D --> E[pinned PARALLEL=1]
|
||||
```
|
||||
|
||||
Trigger: two distinct-but-overlapping ~16k-token realistic prose prompts
|
||||
started simultaneously on PARALLEL=2. The ROCm backend aborted at
|
||||
`ggml-cuda.cu:106` with `ROCm error: an illegal memory access was
|
||||
encountered`, surfacing in `hipStreamSynchronize` inside
|
||||
`common_speculative_impl_draft_dflash::draft` (the class used by
|
||||
`draft-dspark`). `systemd` caught it (status 6/ABRT) and restarted the
|
||||
unit; in-flight streams saw `Remote end closed connection without response`
|
||||
until the ~reload finished.
|
||||
|
||||
Mechanism: DSpark reuses the DFlash block-diffusion draft machinery. The
|
||||
DFlash draft model uses one shared draft context (`ctx_dft`) across all
|
||||
parallel slots; under concurrent block drafting on ROCm it performs an
|
||||
illegal memory access (HIP stream sync). This is the same multi-slot
|
||||
class of bug that retired `draft-dflash` for premature EOS; here it is a
|
||||
hard ROCm crash instead of a bad sample.
|
||||
|
||||
Fix: pin **PARALLEL=1** (single slot). With one slot, concurrent requests
|
||||
queue safely and each completes to `max_tokens`. Revisit after an upstream
|
||||
multi-slot DSpark fix (DSpark is known-good single-slot). Single-slot 16k
|
||||
prose generation was ~steady (measuring only the correct-size signal, not
|
||||
this page).
|
||||
|
||||
## Parallel results by stream count (2026-08-15, correct-size tests)
|
||||
|
||||
| Parallel | Two 16k prose prompts | Two 67k prose prompts |
|
||||
| -------- | -------------------- | --------------------- |
|
||||
| 2 | crashes (illegal mem access) | passes |
|
||||
| 1 | queues, both complete | n/a |
|
||||
|
||||
Note: the crash manifests on the smaller (16k) concurrent streams, not the
|
||||
larger 67k pair. Size is not the trigger; concurrent block drafting on the
|
||||
shared draft context is.
|
||||
|
||||
## How MTP works on Qwen3.8
|
||||
|
||||
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
|
||||
@@ -67,17 +123,19 @@ Two failure patterns appeared during early production tests:
|
||||
`scripts/frank-default-large.env`:
|
||||
|
||||
```ini
|
||||
SPEC_TYPE=draft-mtp
|
||||
SPEC_DRAFT_N_MAX=5
|
||||
SPEC_TYPE=draft-dspark
|
||||
DRAFT_FILE=Qwen3.8-27B-DSpark-Q8_0.gguf
|
||||
SPEC_DRAFT_N_MAX=7
|
||||
SPEC_DRAFT_P_MIN=
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=256
|
||||
CTX_SIZE=524288
|
||||
PARALLEL=2
|
||||
PARALLEL=1 # DSpark multi-slot crashes on this box (2026-08-15)
|
||||
```
|
||||
|
||||
Window: 524288 total / 2 = 262144 per slot, matching the pi
|
||||
`models.json` contextWindow for `frank-default-large`.
|
||||
Window: computed at 262144 per slot (the pi `models.json` contextWindow for
|
||||
`frank-default-large`). The unit reports `n_ctx_slot = 262144`; CTX_SIZE is
|
||||
a pool ceiling and per-slot stays 262144 regardless of the single slot.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
Reference in new issue
Block a user