diff --git a/AGENTS.md b/AGENTS.md index 55c730b..6008176 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ ## Overview -This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with a DSpark sidecar draft (`draft-dspark`, single parallel slot, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (DSpark single-slot, 2026-08-15; DFlash retired earlier for premature EOS). +This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with llama.cpp's `--spec-default` (ngram-mod, no draft model, Vulkan0 backend, 2 parallel slots, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (--spec-default + Vulkan0, 2026-08-15; DSpark retired same day for a ROCm parallel crash; DFlash retired earlier for premature EOS). ## Benchmarking Policy (MANDATORY) @@ -61,7 +61,7 @@ Target map: | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| -| Dankclaw | dank@dankclaw | llama.cpp (UD-Q8_K_XL + DSpark sidecar, `draft-dspark`, 1 slot) | frank-default-large | 7770 | +| Dankclaw | dank@dankclaw | llama.cpp (UD-Q8_K_XL, `--spec-default` ngram-mod, Vulkan0, 2 slots) | frank-default-large | 7770 | | Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 | ## Runtime & Troubleshooting @@ -108,7 +108,7 @@ graph TD - Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first. - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). - Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding. -- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also **single-slot only** on this box: two concurrent 16k prose prompts crash the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). +- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also unsafe here: two concurrent 16k prose prompts crashed the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). Production uses `--spec-default` (ngram-mod, draftless) at PARALLEL=2 on the Vulkan0 backend. See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). ### Dankclaw memory model diff --git a/docs/DFlash DSpark Speculative Decoding.md b/docs/DFlash DSpark Speculative Decoding.md index 225aa6c..d19ce30 100644 --- a/docs/DFlash DSpark Speculative Decoding.md +++ b/docs/DFlash DSpark Speculative Decoding.md @@ -197,18 +197,17 @@ docker run --rm --name vllm-qwen36-dflash \ ## Deployment on Dankclaw (AMD Strix Halo gfx1151) -Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with a **DSpark sidecar draft** on the ROCm (HIP) backend. +Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with **no dedicated draft** on the **Vulkan0** backend. Current facts (2026-08-15): - Main model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (bundled MTP, unused). -- Draft model: `magnitudedev/Qwen3.8-27B-DSpark-GGUF` `Qwen3.8-27B-DSpark-Q8_0.gguf` (DSpark, 1.36B, block_size 7, loaded as a sidecar via `--spec-draft-model`). -- Spec command: `--spec-type draft-dspark` with `--spec-draft-n-max 7`. -- **PARALLEL=1.** DSpark multi-slot crashes the ROCm backend (`ROCm error: an illegal memory access`) under two concurrent 16k prose prompts. See [[qwen3.8-spec-matrix]] for the incident. -- Backend `--device ROCm0`; `--gpu-layers all --fit off`; KV `q8_0` K / `q8_0` V. -- Context: 1 slot x 262144; batch 8192 / ubatch 256. +- Spec: llama.cpp **`--spec-default`** preset = ngram-mod (n_match 24 / n_min 48 / n_max 64). No draft model. +- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes 5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm wins prefill 119 vs 106 t/s. See [[qwen3.8-spec-matrix]] for the bench table. +- **PARALLEL=2** — passes the 2x32k parallel stream test. +- KV `q8_0` K / `q8_0` V; context 2 x 262144; batch 8192 / ubatch 256. -History: before DSpark, the box ran bundled MTP (`draft-mtp`, auto-detected b10411+, retired 2026-08-15). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`) and the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`) remain in `/home/dank/models/` as reference only. +History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only. ## Notes diff --git a/docs/deployment.md b/docs/deployment.md index ca7c9ac..20733ee 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -139,22 +139,21 @@ curl -sf http://127.0.0.1:7770/health The active service on Dankclaw runs llama.cpp serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the alias `frank-default-large` on port -7770. It uses a **DSpark sidecar draft model** (`draft-dspark`, -`magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max 7`) -for speculative decoding on the **ROCm (HIP)** backend, with **1 parallel -slot** at 262144 context, KV cache `q8_0` K / `q8_0` V on GPU, full layer -offload (`--gpu-layers all`, `--fit off`), and batch 8192 / ubatch 256. -The unit file is `scripts/frank-default-large.service`. It launches through -`scripts/serve-llama-frank-large.sh`, which reads tunables from -`scripts/frank-default-large.env`. +7770. It uses llama.cpp's **`--spec-default`** preset (ngram-mod, no draft +model) for speculative decoding on the **Vulkan0** backend, with **2 +parallel slots** at 262144 context each, KV cache `q8_0` K / `q8_0` V on +GPU, full layer offload (`--gpu-layers all`, `--fit off`), and batch 8192 / +ubatch 256. The unit file is `scripts/frank-default-large.service`. It +launches through `scripts/serve-llama-frank-large.sh`, which reads tunables +from `scripts/frank-default-large.env`. -> Note: DSpark replaced **bundled MTP** (`draft-mtp`, in-model, auto-detected -> b10411+) on 2026-08-15 at the user's request. **DSpark is single-slot -> only**: two concurrent 16k prose prompts crash the ROCm backend with -> `ROCm error: an illegal memory access` (status 6/ABRT) during per-sequence -> block sampling, so `PARALLEL=1`. DSpark is the semi-autoregressive -> extension of DFlash; DFlash was previously retired here for a -> shared-draft-context premature-EOS bug. See [[qwen3.8-spec-matrix]]. +> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k +> prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) → +> `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went +> ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster +> (5.68 vs 3.83 t/s). See [[qwen3.8-spec-matrix]] for both decisions and +> the bench table. Also retired earlier: bundled MTP, the vLLM-ROCm CIRU +> INT4 Ling-3.0-flash runtime, and the llama-serve/vLLM units on Dankclaw. ### Dankserver — frank-default-small diff --git a/docs/frank-default-large-benchmark.md b/docs/frank-default-large-benchmark.md index be994cf..97c75a0 100644 --- a/docs/frank-default-large-benchmark.md +++ b/docs/frank-default-large-benchmark.md @@ -6,20 +6,22 @@ llama.cpp build on that machine (upstream, b10133+). ## Summary -- **Backend**: ROCm (HIP). Vulkan is not viable for the full pipeline. -- **Spec (2026-08-15)**: DSpark sidecar draft (`draft-dspark`, - `magnitudedev/Qwen3.8-27B-DSpark-GGUF` Q8_0, 1.36B, `--spec-draft-n-max - 7`). Replaced bundled MTP. **Single-slot (PARALLEL=1)** because DSpark - crashes the ROCm backend under two concurrent 16k prompts (see +- **Backend**: **Vulkan0** (2026-08-15). At 262k ctx Vulkan decodes 48% + faster than ROCm (5.68 vs 3.83 t/s); ROCm only wins prefill 119 vs 106 + t/s. Generation is the interactive cost, so Vulkan0. +- **Spec (2026-08-15)**: llama.cpp `--spec-default` preset (ngram-mod, + n_match 24 / n_min 48 / n_max 64, draftless). Replaced DSpark, which + crashed the ROCm backend under two concurrent 16k prompts (see [[qwen3.8-spec-matrix]]). -- **Context**: 1 slot x 262144. +- **Parallel**: 2 slots — passes the 2x32k parallel stream test. +- **Context**: 2 slots x 262144. - **KV**: `q8_0` K / `q8_0` V on GPU. - **Batch**: 8192 / ubatch 256. > The A/B tables below are retained for history but predate the 2026-08-14 -> Qwen3.8-27B `UD-Q8_K_XL` switch to bundled MTP and the 2026-08-15 DSpark -> switch. The current production spec decision and the DSpark parallel crash -> are recorded on [[qwen3.8-spec-matrix]]. +> Qwen3.8-27B `UD-Q8_K_XL` switch. The current production spec decision, +> the DSpark parallel crash, and the high-context ROCm-vs-Vulkan bench are +> recorded on [[qwen3.8-spec-matrix]]. ## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt) diff --git a/docs/index.md b/docs/index.md index 8d55444..113cbbf 100644 --- a/docs/index.md +++ b/docs/index.md @@ -17,10 +17,10 @@ MoE experts on CPU, port 7770] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] D[gfx1151 APU 30GiB phys / 96GiB GPU-visible] - E[llama.cpp DSpark sidecar draft + E[llama.cpp --spec-default ngram-mod frank-default-large] F[Qwen3.8-27B UD-Q8_K_XL -DSpark Q8_0 draft, port 7770, PARALLEL=1] +Vulkan0 backend, port 7770, PARALLEL=2] D --> E --> F end @@ -52,7 +52,7 @@ DSpark Q8_0 draft, port 7770, PARALLEL=1] - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU - [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug) -- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = DSpark sidecar draft, PARALLEL=1 (2026-08-15 DSpark ROCm parallel crash) +- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = --spec-default (ngram-mod), PARALLEL=2, Vulkan0 backend (2026-08-15) - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small - [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings diff --git a/docs/qwen3.8-spec-matrix.md b/docs/qwen3.8-spec-matrix.md index bbbf571..c2dfee4 100644 --- a/docs/qwen3.8-spec-matrix.md +++ b/docs/qwen3.8-spec-matrix.md @@ -4,7 +4,21 @@ This page records the speculative-decoding benchmark and the production tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw (`frank-default-large`). -## Result (applied, 2026-08-15: DSpark) +## Result (applied, 2026-08-15: --spec-default) + +- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP). +- Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24, + n_min 48, n_max 64). No draft model. Chosen 2026-08-15 after DSpark + crashed under parallel load (see below). +- **PARALLEL=2** — passes the 2x32k parallel stream test (both streams + complete 256/256 concurrently, no crash). ngram-mod is draftless and + parallel-safe. +- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes + 5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm only wins prefill (119 vs + 106 t/s). See the high-context bench table below. +- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot). + +## Result (superseded 2026-08-15: DSpark) - Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP). - Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF` @@ -13,6 +27,7 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw - **PARALLEL=1.** Two concurrent 16k prose prompts crashed the ROCm backend with `ROCm error: an illegal memory access` during per-sequence block sampling. See the DSpark parallel incident below. +- Retired the same day in favor of `--spec-default`. - Batch 8192 / ubatch 256, ctx 524288 (262144 per slot). ## Result (superseded 2026-08-14: bundled MTP) @@ -118,24 +133,42 @@ Two failure patterns appeared during early production tests: EOS on a wall of repeated text. This is sampling variance, not a corruption bug. Realistic prompts never stop early. +## High-Context Backend Comparison (2026-08-15) + +Tool: `scripts/bench-high-ctx.sh `. Both runs at ctx 262144 +with a ~200k-token realistic-prose prompt, `--spec-default`, batch 8192 / +ubatch 256, single slot, on scratch port 7892. + +| Device | Prompt tokens | Prefill (t/s) | Generation (t/s) | Wall (single req) | +| ------ | ------------- | ------------- | ---------------- | ----------------- | +| ROCm0 | 202822 | **119.1** | 3.83 | **1770 s** | +| Vulkan0 | 200235 | 106.3 | **5.68** | 1929 s | + +Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs +3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost +(users stream tokens), so **Vulkan0** is the production backend. Both +`tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s). +High-context decode collapses on both backends versus ~25 t/s at 16k ctx; +that is the long-context attention penalty, not a spec artifact. + ## Production Configuration `scripts/frank-default-large.env`: ```ini -SPEC_TYPE=draft-dspark -DRAFT_FILE=Qwen3.8-27B-DSpark-Q8_0.gguf -SPEC_DRAFT_N_MAX=7 -SPEC_DRAFT_P_MIN= +SPEC_DEFAULT=1 # llama.cpp preset = ngram-mod (48/64/24) +SPEC_TYPE= +DRAFT_FILE= BATCH_SIZE=8192 UBATCH_SIZE=256 CTX_SIZE=524288 -PARALLEL=1 # DSpark multi-slot crashes on this box (2026-08-15) +PARALLEL=2 # passes 2x32k parallel stream test (2026-08-15) +DEVICE=Vulkan0 # 48% faster decode at 262k ctx ``` Window: computed at 262144 per slot (the pi `models.json` contextWindow for `frank-default-large`). The unit reports `n_ctx_slot = 262144`; CTX_SIZE is -a pool ceiling and per-slot stays 262144 regardless of the single slot. +a pool ceiling and per-slot stays 262144 regardless of the slot count. ## See Also