From 020e874a5c87b851d6224c9e0137dd079ab6a0fd Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 15 Aug 2026 14:36:34 -0700 Subject: [PATCH] docs: record DSpark single-slot switch + ROCm parallel crash (2026-08-15) across KB; refs AGENTS/deployment/benchmark/matrix/index --- AGENTS.md | 6 +- docs/DFlash DSpark Speculative Decoding.md | 15 ++--- docs/deployment.md | 19 +++++- docs/frank-default-large-benchmark.md | 15 ++++- docs/index.md | 8 +-- docs/qwen3.8-spec-matrix.md | 70 ++++++++++++++++++++-- 6 files changed, 108 insertions(+), 25 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index df34e86..55c730b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ ## Overview -This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.6-27B (HauhauCS uncensored Q8_K_P) with an MTP sidecar drafter (`draft-mtp`, 3 parallel slots, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md) for the spec decision (DFlash retired 2026-08-14 due to a parallel premature-EOS bug). +This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with a DSpark sidecar draft (`draft-dspark`, single parallel slot, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (DSpark single-slot, 2026-08-15; DFlash retired earlier for premature EOS). ## Benchmarking Policy (MANDATORY) @@ -61,7 +61,7 @@ Target map: | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| -| Dankclaw | dank@dankclaw | llama.cpp (Q8_K_P + MTP sidecar, `draft-mtp`, 3 slots) | frank-default-large | 7770 | +| Dankclaw | dank@dankclaw | llama.cpp (UD-Q8_K_XL + DSpark sidecar, `draft-dspark`, 1 slot) | frank-default-large | 7770 | | Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 | ## Runtime & Troubleshooting @@ -108,7 +108,7 @@ graph TD - Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first. - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). - Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding. -- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw; use `draft-mtp` (n_max 3) or `ngram-mod`, both parallel-safe. See [docs/frank-default-large-benchmark.md](docs/frank-default-large-benchmark.md). +- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also **single-slot only** on this box: two concurrent 16k prose prompts crash the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). ### Dankclaw memory model diff --git a/docs/DFlash DSpark Speculative Decoding.md b/docs/DFlash DSpark Speculative Decoding.md index aab53b3..225aa6c 100644 --- a/docs/DFlash DSpark Speculative Decoding.md +++ b/docs/DFlash DSpark Speculative Decoding.md @@ -197,17 +197,18 @@ docker run --rm --name vllm-qwen36-dflash \ ## Deployment on Dankclaw (AMD Strix Halo gfx1151) -Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model with an **MTP sidecar draft** on the ROCm (HIP) backend. DFlash was retired on 2026-08-14. +Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with a **DSpark sidecar draft** on the ROCm (HIP) backend. -Current facts: +Current facts (2026-08-15): -- Main model: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive `Q8_K_P` (dense 27B). -- Draft model: a4lg/Qwen3.6-27B-MTP-ONLY `Q8_0` (MTP head from the base model, loaded as a sidecar). -- Spec command: `--spec-type draft-mtp` with `--spec-draft-n-max 3`. +- Main model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (bundled MTP, unused). +- Draft model: `magnitudedev/Qwen3.8-27B-DSpark-GGUF` `Qwen3.8-27B-DSpark-Q8_0.gguf` (DSpark, 1.36B, block_size 7, loaded as a sidecar via `--spec-draft-model`). +- Spec command: `--spec-type draft-dspark` with `--spec-draft-n-max 7`. +- **PARALLEL=1.** DSpark multi-slot crashes the ROCm backend (`ROCm error: an illegal memory access`) under two concurrent 16k prose prompts. See [[qwen3.8-spec-matrix]] for the incident. - Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V. -- Context: 3 x 200700 = 602100; batch 8192 / ubatch 8192. +- Context: 1 slot x 262144; batch 8192 / ubatch 256. -Why DFlash was retired: DFlash causes premature EOS on a generating slot when another task lands with high prompt-cache reuse (LCP ~0.99). See [[frank-default-large-benchmark]] for the incident writeup and the MTP-vs-ngram A/B. The DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) remains in `/home/dank/models/` as reference only. +History: before DSpark, the box ran bundled MTP (`draft-mtp`, auto-detected b10411+, retired 2026-08-15). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) and the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`) remain in `/home/dank/models/` as reference only. ## Notes diff --git a/docs/deployment.md b/docs/deployment.md index 2fcd671..ca7c9ac 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -137,9 +137,24 @@ curl -sf http://127.0.0.1:7770/health ### Dankclaw — frank-default-large -The active service on Dankclaw runs llama.cpp serving the HauhauCS Qwen3.6-27B uncensored `Q8_K_P` model under the alias `frank-default-large` on port 7770. It uses an **MTP sidecar drafter** (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`) for speculative decoding on the **ROCm (HIP)** backend, with **3 parallel slots at 200700 context each** (pool 602100), KV cache `q8_0` K / `q8_0` V on GPU (pairing required for Flash Attention on this build), full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 8192 (peak ~65.5 GiB VRAM). The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-llama-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`. +The active service on Dankclaw runs llama.cpp serving the unsloth +Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port +7770. It uses a **DSpark sidecar draft model** (`draft-dspark`, +`magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max 7`) +for speculative decoding on the **ROCm (HIP)** backend, with **1 parallel +slot** at 262144 context, KV cache `q8_0` K / `q8_0` V on GPU, full layer +offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 256. +The unit file is `scripts/frank-default-large.service`. It launches through +`scripts/serve-llama-frank-large.sh`, which reads tunables from +`scripts/frank-default-large.env`. -> Note: MTP replaced DFlash (`draft-dflash`) on 2026-08-14. DFlash retired because it releases a generating slot prematurely under parallel load when another task lands with high prompt-cache reuse. See [[frank-default-large-benchmark]]. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash runtime and the llama-serve/vLLM units on Dankclaw. +> Note: DSpark replaced **bundled MTP** (`draft-mtp`, in-model, auto-detected +> b10411+) on 2026-08-15 at the user's request. **DSpark is single-slot +> only**: two concurrent 16k prose prompts crash the ROCm backend with +> `ROCm error: an illegal memory access` (status 6/ABRT) during per-sequence +> block sampling, so `PARALLEL=1`. DSpark is the semi-autoregressive +> extension of DFlash; DFlash was previously retired here for a +> shared-draft-context premature-EOS bug. See [[qwen3.8-spec-matrix]]. ### Dankserver — frank-default-small diff --git a/docs/frank-default-large-benchmark.md b/docs/frank-default-large-benchmark.md index 0d1e146..be994cf 100644 --- a/docs/frank-default-large-benchmark.md +++ b/docs/frank-default-large-benchmark.md @@ -7,10 +7,19 @@ llama.cpp build on that machine (upstream, b10133+). ## Summary - **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline. -- **Spec (2026-08-14)**: MTP sidecar drafter (`draft-mtp`, `a4lg/Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`, `--spec-draft-n-max 3`). Replaced DFlash, which was retired because it releases a generating slot prematurely under parallel load (see below). -- **Context**: 3 parallel slots x 200700 (pool 602100). +- **Spec (2026-08-15)**: DSpark sidecar draft (`draft-dspark`, + `magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max + 7`). Replaced bundled MTP. **Single-slot (PARALLEL=1)** because DSpark + crashes the ROCm backend under two concurrent 16k prompts (see + [[qwen3.8-spec-matrix]]). +- **Context**: 1 slot x 262144. - **KV**: `q8_0` K / `q8_0` V on GPU. -- **Batch**: 8192 / ubatch 8192 (2026-08-14, uses the VRAM headroom; peaks ~65.5 GiB of 96 GiB). +- **Batch**: 8192 / ubatch 256. + +> The A/B tables below are retained for history but predate the 2026-08-14 +> Qwen3.8-27B `UD-Q8_K_XL` switch to bundled MTP and the 2026-08-15 DSpark +> switch. The current production spec decision and the DSpark parallel crash +> are recorded on [[qwen3.8-spec-matrix]]. ## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt) diff --git a/docs/index.md b/docs/index.md index 4ea2cc1..8d55444 100644 --- a/docs/index.md +++ b/docs/index.md @@ -17,10 +17,10 @@ MoE experts on CPU, port 7770] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] D[gfx1151 APU 30GiB phys / 96GiB GPU-visible] - E[llama.cpp MTP sidecar + E[llama.cpp DSpark sidecar draft frank-default-large] - F[Qwen3.6-27B uncensored Q8_K_P -MTP-ONLY draft, port 7770] + F[Qwen3.8-27B UD-Q8_K_XL +DSpark Q8_0 draft, port 7770, PARALLEL=1] D --> E --> F end @@ -52,7 +52,7 @@ MTP-ONLY draft, port 7770] - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU - [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug) -- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL bundled-MTP spec matrix; winner draft-mtp n_max=5 +- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = DSpark sidecar draft, PARALLEL=1 (2026-08-15 DSpark ROCm parallel crash) - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small - [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings diff --git a/docs/qwen3.8-spec-matrix.md b/docs/qwen3.8-spec-matrix.md index e2f1802..bbbf571 100644 --- a/docs/qwen3.8-spec-matrix.md +++ b/docs/qwen3.8-spec-matrix.md @@ -4,7 +4,18 @@ This page records the speculative-decoding benchmark and the production tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw (`frank-default-large`). -## Result (applied) +## Result (applied, 2026-08-15: DSpark) + +- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP). +- Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF` + (`Qwen3.8-27B-DSpark-Q8_0.gguf`, 1.36B, block_size 7), `--spec-type + draft-dspark`, `--spec-draft-n-max 7` (clamps to block_size). +- **PARALLEL=1.** Two concurrent 16k prose prompts crashed the ROCm backend + with `ROCm error: an illegal memory access` during per-sequence block + sampling. See the DSpark parallel incident below. +- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot). + +## Result (superseded 2026-08-14: bundled MTP) - Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP). - Spec: `draft-mtp`, `--spec-draft-n-max 5`, no `--spec-draft-p-min`. @@ -12,6 +23,51 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw - Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under 2 concurrent streams. No premature stop. +## Incident: DSpark ROCm Crash Under Parallel Load (2026-08-15) + +A two-stream parallel acceptance test crashed `frank-default-large`. + +```mermaid +graph TD + A[Two concurrent 16k prose prompts
high prompt-cache reuse] --> B[ROCm illegal memory access
in common_speculative_impl_draft_dflash::draft] + B --> C[status 6/ABRT core dump] + C --> D[systemd auto-restart, 503 during reload] + D --> E[pinned PARALLEL=1] +``` + +Trigger: two distinct-but-overlapping ~16k-token realistic prose prompts +started simultaneously on PARALLEL=2. The ROCm backend aborted at +`ggml-cuda.cu:106` with `ROCm error: an illegal memory access was +encountered`, surfacing in `hipStreamSynchronize` inside +`common_speculative_impl_draft_dflash::draft` (the class used by +`draft-dspark`). `systemd` caught it (status 6/ABRT) and restarted the +unit; in-flight streams saw `Remote end closed connection without response` +until the ~reload finished. + +Mechanism: DSpark reuses the DFlash block-diffusion draft machinery. The +DFlash draft model uses one shared draft context (`ctx_dft`) across all +parallel slots; under concurrent block drafting on ROCm it performs an +illegal memory access (HIP stream sync). This is the same multi-slot +class of bug that retired `draft-dflash` for premature EOS; here it is a +hard ROCm crash instead of a bad sample. + +Fix: pin **PARALLEL=1** (single slot). With one slot, concurrent requests +queue safely and each completes to `max_tokens`. Revisit after an upstream +multi-slot DSpark fix (DSpark is known-good single-slot). Single-slot 16k +prose generation was ~steady (measuring only the correct-size signal, not +this page). + +## Parallel results by stream count (2026-08-15, correct-size tests) + +| Parallel | Two 16k prose prompts | Two 67k prose prompts | +| -------- | -------------------- | --------------------- | +| 2 | crashes (illegal mem access) | passes | +| 1 | queues, both complete | n/a | + +Note: the crash manifests on the smaller (16k) concurrent streams, not the +larger 67k pair. Size is not the trigger; concurrent block drafting on the +shared draft context is. + ## How MTP works on Qwen3.8 The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside @@ -67,17 +123,19 @@ Two failure patterns appeared during early production tests: `scripts/frank-default-large.env`: ```ini -SPEC_TYPE=draft-mtp -SPEC_DRAFT_N_MAX=5 +SPEC_TYPE=draft-dspark +DRAFT_FILE=Qwen3.8-27B-DSpark-Q8_0.gguf +SPEC_DRAFT_N_MAX=7 SPEC_DRAFT_P_MIN= BATCH_SIZE=8192 UBATCH_SIZE=256 CTX_SIZE=524288 -PARALLEL=2 +PARALLEL=1 # DSpark multi-slot crashes on this box (2026-08-15) ``` -Window: 524288 total / 2 = 262144 per slot, matching the pi -`models.json` contextWindow for `frank-default-large`. +Window: computed at 262144 per slot (the pi `models.json` contextWindow for +`frank-default-large`). The unit reports `n_ctx_slot = 262144`; CTX_SIZE is +a pool ceiling and per-slot stays 262144 regardless of the single slot. ## See Also