diff --git a/AGENTS.md b/AGENTS.md
index df34e86..55c730b 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -2,7 +2,7 @@
## Overview
-This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.6-27B (HauhauCS uncensored Q8_K_P) with an MTP sidecar drafter (`draft-mtp`, 3 parallel slots, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md) for the spec decision (DFlash retired 2026-08-14 due to a parallel premature-EOS bug).
+This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with a DSpark sidecar draft (`draft-dspark`, single parallel slot, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (DSpark single-slot, 2026-08-15; DFlash retired earlier for premature EOS).
## Benchmarking Policy (MANDATORY)
@@ -61,7 +61,7 @@ Target map:
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
-| Dankclaw | dank@dankclaw | llama.cpp (Q8_K_P + MTP sidecar, `draft-mtp`, 3 slots) | frank-default-large | 7770 |
+| Dankclaw | dank@dankclaw | llama.cpp (UD-Q8_K_XL + DSpark sidecar, `draft-dspark`, 1 slot) | frank-default-large | 7770 |
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 |
## Runtime & Troubleshooting
@@ -108,7 +108,7 @@ graph TD
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding.
-- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw; use `draft-mtp` (n_max 3) or `ngram-mod`, both parallel-safe. See [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md).
+- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also **single-slot only** on this box: two concurrent 16k prose prompts crash the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md).
### Dankclaw memory model
diff --git a/docs/DFlash DSpark Speculative Decoding.md b/docs/DFlash DSpark Speculative Decoding.md
index aab53b3..225aa6c 100644
--- a/docs/DFlash DSpark Speculative Decoding.md
+++ b/docs/DFlash DSpark Speculative Decoding.md
@@ -197,17 +197,18 @@ docker run --rm --name vllm-qwen36-dflash \
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
-Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with an **MTP sidecar draft** on the ROCm (HIP) backend. DFlash was retired on 2026-08-14.
+Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with a **DSpark sidecar draft** on the ROCm (HIP) backend.
-Current facts:
+Current facts (2026-08-15):
-- Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive `Q8_K_P` (dense 27B).
-- Draft model: a4lg/Qwen3.6-27B-MTP-ONLY `Q8_0` (MTP head from the base model, loaded as a sidecar).
-- Spec command: `--spec-type draft-mtp` with `--spec-draft-n-max 3`.
+- Main model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (bundled MTP, unused).
+- Draft model: `magnitudedev/Qwen3.8-27B-DSpark-GGUF` `Qwen3.8-27B-DSpark-Q8_0.gguf` (DSpark, 1.36B, block_size 7, loaded as a sidecar via `--spec-draft-model`).
+- Spec command: `--spec-type draft-dspark` with `--spec-draft-n-max 7`.
+- **PARALLEL=1.** DSpark multi-slot crashes the ROCm backend (`ROCm error: an illegal memory access`) under two concurrent 16k prose prompts. See [[qwen3.8-spec-matrix]] for the incident.
- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V.
-- Context: 3 x 200700 = 602100; batch 8192 / ubatch 8192.
+- Context: 1 slot x 262144; batch 8192 / ubatch 256.
-Why DFlash was retired: DFlash causes premature EOS on a generating slot when another task lands with high prompt-cache reuse (LCP ~0.99). See [[frank-default-large-benchmark]] for the incident writeup and the MTP-vs-ngram A/B. The DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) remains in `/home/dank/models/` as reference only.
+History: before DSpark, the box ran bundled MTP (`draft-mtp`, auto-detected b10411+, retired 2026-08-15). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) and the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
## Notes
diff --git a/docs/deployment.md b/docs/deployment.md
index 2fcd671..ca7c9ac 100644
--- a/docs/deployment.md
+++ b/docs/deployment.md
@@ -137,9 +137,24 @@ curl -sf http://127.0.0.1:7770/health
### Dankclaw — frank-default-large
-The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses an **MTP sidecar drafter** (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `q8_0` V on GPU (pairing required for Flash Attention on this build), full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 8192 (peak ~65.5 GiB VRAM). The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`.
+The active service on Dankclaw runs llama.cpp serving the unsloth
+Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port
+7770. It uses a **DSpark sidecar draft model** (`draft-dspark`,
+`magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max 7`)
+for speculative decoding on the **ROCm (HIP)** backend, with **1 parallel
+slot** at 262144 context, KV cache `q8_0` K / `q8_0` V on GPU, full layer
+offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 256.
+The unit file is `scripts/frank-default-large.service`. It launches through
+`scripts/serve-llama-frank-large.sh`, which reads tunables from
+`scripts/frank-default-large.env`.
-> Note: MTP replaced DFlash (`draft-dflash`) on 2026-08-14. DFlash retired because it releases a generating slot prematurely under parallel load when another task lands with high prompt-cache reuse. See [[frank-default-large-benchmark]]. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the llama-serve/vLLM units on Dankclaw.
+> Note: DSpark replaced **bundled MTP** (`draft-mtp`, in-model, auto-detected
+> b10411+) on 2026-08-15 at the user's request. **DSpark is single-slot
+> only**: two concurrent 16k prose prompts crash the ROCm backend with
+> `ROCm error: an illegal memory access` (status 6/ABRT) during per-sequence
+> block sampling, so `PARALLEL=1`. DSpark is the semi-autoregressive
+> extension of DFlash; DFlash was previously retired here for a
+> shared-draft-context premature-EOS bug. See [[qwen3.8-spec-matrix]].
### Dankserver — frank-default-small
diff --git a/docs/frank-default-large-benchmark.md b/docs/frank-default-large-benchmark.md
index 0d1e146..be994cf 100644
--- a/docs/frank-default-large-benchmark.md
+++ b/docs/frank-default-large-benchmark.md
@@ -7,10 +7,19 @@ llama.cpp build on that machine (upstream, b10133+).
## Summary
- **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline.
-- **Spec (2026-08-14)**: MTP sidecar drafter (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`). Replaced DFlash, which was retired because it releases a generating slot prematurely under parallel load (see below).
-- **Context**: 3 parallel slots x 200700 (pool 602100).
+- **Spec (2026-08-15)**: DSpark sidecar draft (`draft-dspark`,
+ `magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max
+ 7`). Replaced bundled MTP. **Single-slot (PARALLEL=1)** because DSpark
+ crashes the ROCm backend under two concurrent 16k prompts (see
+ [[qwen3.8-spec-matrix]]).
+- **Context**: 1 slot x 262144.
- **KV**: `q8_0` K / `q8_0` V on GPU.
-- **Batch**: 8192 / ubatch 8192 (2026-08-14, uses the VRAM headroom; peaks ~65.5 GiB of 96 GiB).
+- **Batch**: 8192 / ubatch 256.
+
+> The A/B tables below are retained for history but predate the 2026-08-14
+> Qwen3.8-27B `UD-Q8_K_XL` switch to bundled MTP and the 2026-08-15 DSpark
+> switch. The current production spec decision and the DSpark parallel crash
+> are recorded on [[qwen3.8-spec-matrix]].
## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)
diff --git a/docs/index.md b/docs/index.md
index 4ea2cc1..8d55444 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -17,10 +17,10 @@ MoE experts on CPU, port 7770]
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
- E[llama.cpp MTP sidecar
+ E[llama.cpp DSpark sidecar draft
frank-default-large]
- F[Qwen3.6-27B uncensored Q8_K_P
-MTP-ONLY draft, port 7770]
+ F[Qwen3.8-27B UD-Q8_K_XL
+DSpark Q8_0 draft, port 7770, PARALLEL=1]
D --> E --> F
end
@@ -52,7 +52,7 @@ MTP-ONLY draft, port 7770]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
- [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
-- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL bundled-MTP spec matrix; winner draft-mtp n_max=5
+- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = DSpark sidecar draft, PARALLEL=1 (2026-08-15 DSpark ROCm parallel crash)
- [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
- [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
- [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings
diff --git a/docs/qwen3.8-spec-matrix.md b/docs/qwen3.8-spec-matrix.md
index e2f1802..bbbf571 100644
--- a/docs/qwen3.8-spec-matrix.md
+++ b/docs/qwen3.8-spec-matrix.md
@@ -4,7 +4,18 @@ This page records the speculative-decoding benchmark and the production
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw
(`frank-default-large`).
-## Result (applied)
+## Result (applied, 2026-08-15: DSpark)
+
+- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
+- Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
+ (`Qwen3.8-27B-DSpark-Q8_0.gguf`, 1.36B, block_size 7), `--spec-type
+ draft-dspark`, `--spec-draft-n-max 7` (clamps to block_size).
+- **PARALLEL=1.** Two concurrent 16k prose prompts crashed the ROCm backend
+ with `ROCm error: an illegal memory access` during per-sequence block
+ sampling. See the DSpark parallel incident below.
+- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
+
+## Result (superseded 2026-08-14: bundled MTP)
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
- Spec: `draft-mtp`, `--spec-draft-n-max 5`, no `--spec-draft-p-min`.
@@ -12,6 +23,51 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw
- Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under
2 concurrent streams. No premature stop.
+## Incident: DSpark ROCm Crash Under Parallel Load (2026-08-15)
+
+A two-stream parallel acceptance test crashed `frank-default-large`.
+
+```mermaid
+graph TD
+ A[Two concurrent 16k prose prompts
high prompt-cache reuse] --> B[ROCm illegal memory access
in common_speculative_impl_draft_dflash::draft]
+ B --> C[status 6/ABRT core dump]
+ C --> D[systemd auto-restart, 503 during reload]
+ D --> E[pinned PARALLEL=1]
+```
+
+Trigger: two distinct-but-overlapping ~16k-token realistic prose prompts
+started simultaneously on PARALLEL=2. The ROCm backend aborted at
+`ggml-cuda.cu:106` with `ROCm error: an illegal memory access was
+encountered`, surfacing in `hipStreamSynchronize` inside
+`common_speculative_impl_draft_dflash::draft` (the class used by
+`draft-dspark`). `systemd` caught it (status 6/ABRT) and restarted the
+unit; in-flight streams saw `Remote end closed connection without response`
+until the ~reload finished.
+
+Mechanism: DSpark reuses the DFlash block-diffusion draft machinery. The
+DFlash draft model uses one shared draft context (`ctx_dft`) across all
+parallel slots; under concurrent block drafting on ROCm it performs an
+illegal memory access (HIP stream sync). This is the same multi-slot
+class of bug that retired `draft-dflash` for premature EOS; here it is a
+hard ROCm crash instead of a bad sample.
+
+Fix: pin **PARALLEL=1** (single slot). With one slot, concurrent requests
+queue safely and each completes to `max_tokens`. Revisit after an upstream
+multi-slot DSpark fix (DSpark is known-good single-slot). Single-slot 16k
+prose generation was ~steady (measuring only the correct-size signal, not
+this page).
+
+## Parallel results by stream count (2026-08-15, correct-size tests)
+
+| Parallel | Two 16k prose prompts | Two 67k prose prompts |
+| -------- | -------------------- | --------------------- |
+| 2 | crashes (illegal mem access) | passes |
+| 1 | queues, both complete | n/a |
+
+Note: the crash manifests on the smaller (16k) concurrent streams, not the
+larger 67k pair. Size is not the trigger; concurrent block drafting on the
+shared draft context is.
+
## How MTP works on Qwen3.8
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
@@ -67,17 +123,19 @@ Two failure patterns appeared during early production tests:
`scripts/frank-default-large.env`:
```ini
-SPEC_TYPE=draft-mtp
-SPEC_DRAFT_N_MAX=5
+SPEC_TYPE=draft-dspark
+DRAFT_FILE=Qwen3.8-27B-DSpark-Q8_0.gguf
+SPEC_DRAFT_N_MAX=7
SPEC_DRAFT_P_MIN=
BATCH_SIZE=8192
UBATCH_SIZE=256
CTX_SIZE=524288
-PARALLEL=2
+PARALLEL=1 # DSpark multi-slot crashes on this box (2026-08-15)
```
-Window: 524288 total / 2 = 262144 per slot, matching the pi
-`models.json` contextWindow for `frank-default-large`.
+Window: computed at 262144 per slot (the pi `models.json` contextWindow for
+`frank-default-large`). The unit reports `n_ctx_slot = 262144`; CTX_SIZE is
+a pool ceiling and per-slot stays 262144 regardless of the single slot.
## See Also