docs: record --spec-default + Vulkan0 + PARALLEL=2 final state; 262k-ctx ROCm-vs-Vulkan bench
This commit is contained in:
1 parent
5296cf11d7
commit
b6f2ee2945
6 files changed
+77
-44
No files matched your search
@@ -2,7 +2,7 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with a DSpark sidecar draft (`draft-dspark`, single parallel slot, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (DSpark single-slot, 2026-08-15; DFlash retired earlier for premature EOS).
|
||||
This project manages deployment scripts for two LLM inference servers. Dankclaw runs a llama.cpp service for Qwen3.8-27B (unsloth UD-Q8_K_XL) with llama.cpp's `--spec-default` (ngram-mod, no draft model, Vulkan0 backend, 2 parallel slots, port 7770). Dankserver runs a llama.cpp service for Qwen3.6-35B-A3B (HauhauCS uncensored Q6_K_P) with ngram-mod speculative decoding and CPU-offloaded MoE experts. See [docs/deployment.md](docs/deployment.md) for the full deployment and update process, and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the spec decision (--spec-default + Vulkan0, 2026-08-15; DSpark retired same day for a ROCm parallel crash; DFlash retired earlier for premature EOS).
|
||||
|
||||
## Benchmarking Policy (MANDATORY)
|
||||
|
||||
@@ -61,7 +61,7 @@ Target map:
|
||||
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankclaw | dank@dankclaw | llama.cpp (UD-Q8_K_XL + DSpark sidecar, `draft-dspark`, 1 slot) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | llama.cpp (UD-Q8_K_XL, `--spec-default` ngram-mod, Vulkan0, 2 slots) | frank-default-large | 7770 |
|
||||
| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 |
|
||||
|
||||
## Runtime & Troubleshooting
|
||||
@@ -108,7 +108,7 @@ graph TD
|
||||
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
|
||||
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
|
||||
- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding.
|
||||
- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also **single-slot only** on this box: two concurrent 16k prose prompts crash the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md).
|
||||
- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also unsafe here: two concurrent 16k prose prompts crashed the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). Production uses `--spec-default` (ngram-mod, draftless) at PARALLEL=2 on the Vulkan0 backend. See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md).
|
||||
|
||||
### Dankclaw memory model
|
||||
|
||||
|
||||
Reference in new issue
Block a user